You might try wiping them down gently with a rag with warm water and vinegar. Once it is thoroughly dry, then prime with gardz and repair all the spots, and once sanded, then see if you need to skim coat. You just might now have to do that. It will need to be primed again after the sheetrock has been repaired, and yes, its ok, to curse the person who did this in the first place. :-)

I have successfully used an electric hand sander with fine sandpaper and a light touch to get the wall to smooth. Since you are going to prime and skim coat and then prime again, any damage you do in getting the glue off will be repaired in the process.