If a room is "unconditioned" it should be insulated from any "conditioned" space that is adjacent to it or above (as required by the building code for new construction). Adjacent conditioned spaces do not need insulation in the walls or the floor-ceiling assemblies that separate them. If the adjacent rooms are not conditioned to the same temperature then insulation might help but wouldn't be critical or required by code.

"Do I have to insulate the stud walls I am about to put up that are on the cinder block walls that divide the room from the other rooms?"

No.

"Or just on the "exterior"/foundation walls?"

Yes.

Just an idea..we insulated all walls...the "shared room" walls were insulated for soundproofing reason.

Another thought - if you are not using the other rooms, close the vents. We insulated the ceiling and all the walls - made it very cozy compared to the other rooms, and as Lido said, helps with soundproofing.

In terms of comfort, you will want to insulate any wall where there are different temperatures on each side of the wall.

Also, if you are in a basement, you likely have some noisy things in the unfinished space - washer, dryer, heating, waste pipes etc. If that is the case, I would insulate interior walls just to keep the noise down.