I don't have a great deal of info on this one yet, but twice within a few minutes I tried to delete an unused fieldset field and it resulted in an exception page saying that Unable to delete field 'my_field_name' because it is in use by these fieldgroups: 'message'
I tried this on a dev and testing server for the same site and both times I tried to delete an unused fieldset field and it randomly tried to delete a field that is being used. The report field that couldn't be deleted was different each time.
Makes me think that maybe at some point these fields were in the fieldset that I deleted (which was successfully deleted), but at this stage I don't really know, but it feels a bit dangerous.