You're right to be concerned about asbestos exposure. However, the likelihood of asbestos on the surface of the wall is minimal. The "cotton-like fibers" are most likely the paper fibers from the surface of the sheetrock that were roughed up when you removed the panels. There is a possibility that the joint compound could have asbestos in it and you wouldn't want to sand that if it does. If you want to proceed cautiously, contact a local testing lab and collect a couple of samples that can be tested by a qualified, local analytical lab. This would not be very expensive. If there isn't any asbestos, you can proceed with your project. If there is, you wouldn't want to keep working on the wall in a way that could raise dust. Instead, you could patch, but not sand, and then seal the surface with paint. If the surface didn't look as nice as you wanted, you could wallpaper over it with something that had a little texture.

Here is a good summary about asbestos in the home and how to involve professionals if you need to. Your state and local environmental health organizations can give you guidance on this, and should be able provide lists of qualified labs and removal specialists. (However, I don't think it will come to that.) Good luck.

Here is a link that might be useful: Asbestos - EPA

Thank you very much for your reply! It was very helpful. I think I like the idea of patching and covering it up. We've also purchased an asbestos home kit and will send in some samples.

Thank you again for your insight and for the information!

I had a similar issue two weeks ago. Buyers of my home noted possible asbestos panels in a wood stove chimney chase. I never got it tested, but assumed that it was and disposed of it myself. My state allows for non friable panels up to 160 square feet to be removed by the homeowner. I pulled a permit for it and everything. Our county has a hazardous waste collection program (6 times a year you can bring it to get disposed of for free). State says for quantities less than 160 square feet, contact the local authorities for disposal info. Called the town, and he said either call your local garbage company to see if they're licensed haulers, or use the county haz waste day. Called haz waste folks up, they told me how to bag it properly, and where to bring it. It passed final inspection this past Monday. It cost me nothing but my own time. But I couldn't have done it myself if it was friable.

And note what 'friable' means to the EPA.
If you can crush the material with your bare hands it is friable.
Everything else is non-friable.