Check that the vent fan is actually connected to something and goes out through the roof. Make sure the flap is working too. How often is the fan run? Maybe get a timer or humidistat. If there is no HVAC, it is impossible to keep the outside weather from getting into the bathroom.

We have restrooms on the golf course in FL and even with AC they still smell damp and musty. I'm sure no special steps were taken when they built them to make them more insulated or weather proof!

A desk top fan on a timer would do it. Just have it run a little a couple of times a day, anything to move the air a bit.

Or -

We always kept an appliance like this one on the boat when locked up and tied to the dock. They don't get hot, just gently warm (you could lay your hand on it and hold it there) and cost about the same as running a light bulb. It was enough to keep the bathroom, kitchenette, bunks nice and dry. No condensing windows.

Thank you both for these great ideas!!