Is this a pole that they installed just for the panel or was there already a pole there? It's odd that they would do something like that without asking first. Mounting to a wood pole sounds strange to me, but perhaps that's a regional thing. Here if they need to install equipment above ground they typically put in galvanized unistrut rails to mount to.

we installed a pool last summer and had the exact same problem, not only was it unsightly it was not a location that was easily accessible when having to do maintenance on the pumps/equipment. I insisted the contractor relocate it (at their costs). The electrician installed it when neither the pool contractor or myself were onsite. A NEW electrician was provided by the contractor, the panel was moved to behind/on the concret wall we built to hide all of the equipment.