Mushrooms growing in potting soil are very common. They are almost always saprophytic fungi or those that survive on the decomposition of any organic matter in the potting mix - peat, bark, actual soil, etc. They are not harmful to the potted plants directly but can be an indication of excessive watering or high humidity. Rosemary is not fond of these conditions so any decline in the plant itself can be attributed to excessively moisture retentive potting soil, not the mushrooms it may generate.

Remove the mushrooms as you wish but I'd strongly suggest repotting in a faster draining, less moisture retentive soil mix....something like one would use for cacti and succulents. Rosemary likes it lean and dry.

also keep in mind.. that many things slow down in winter.. as the sun declines ... even in FL though not as much as up north ....

and as they slow down.. they dont need as much water as it may have needed in the high heat of summer ...

either reduce water.. or change the media a gal notes... but even new media may not help.. if you continue to over water the plant...

media is a water management system.. and as it wears out it loses its ability to retain the water the plant needs while shedding excess ... medium is manipulated based on the needs of the plant...

i think of rosemary more as a shrub.. which would need a sip of water.. and then near drying before the next watering... it is not what we would call a water loving plant that wants to sit in a sponge of water ... taken to the extreme.. the woody roots will rot ...

its apparent your media is rotting.. as that is what fungi does in nature ... they are natural biodegraders .... and again .... that is normal as the media ages and breaks down ...

ken