There's no way to easily get the panel adhesive off the walls without some damage. However, if you're careful, you can get this job done without having to put in new sheetrock. You need to go to the hardware store and buy yourself a big box of single edge razor blades, and a blade holder, like those used for stripping wallpaper. Then remove the adhesive by slipping the blade between the paper surface of the sheetrock and the dried adhesive. Sometimes it will be easy and sometimes you'll cut into the paper...persevere! Sometimes it will be easier to use the blade without the holder and run it nearly flat along the wall surface. Once you've got the adhesive removed, you'll need to lightly sand any roughness with some fine sandpaper, and repair the wall where needed with some joint compound and more sanding. Then put a ooat of primer on the sheetrock when you've restored the wall. If you have any roughness remaning caused by fibres getting raised by the primer, you can knock those down with a light swipe of fine sandpaper. This will not be a lot of fun, but it will be much easier than new sheetrock. Good luck.

When we pulled off our paneling, we really didn't want the extra cost of all new drywall either. We were pleasantly surprised when our sub said that if we just sand down the glue, they could add a thin layer of mud and then the skip trowel. Our glue was 30 years old, so it was very dried out and easy to sand. There were a few large holes to fill, but now it looks great.