Maybe it is from the fan, rather than the door. Can you get into the attic and check out the area above the fan? Is fan vented out through the roof? or into attic, where it could have caused mold and that is the smell.

Thanks I considered this as well, but only the door seems to have a smell when the fan is off. If it was the fan wouldn't the whole room have the smell?

Can't currently go outside due to the weather, but will try soon.

A previous house I owned had a wet bar and it reeked of a mildew/mold odor. We removed the sink and drain and it only helped a little. I cleaned and put all manner of odor absorbers inside the cabinet. We lived there 21 years and it never dissipated completely. Wood absorbs odors. Perhaps a paint job with a special primer might help. Good luck.

If you believe the door is the cause;, remove and replace the door.

Since smell is worse with the fan, it could be that the door is what has absorbed the smell from the fan. I would check out the fan first.