Do you have an attic without flooring? If so, you can go up there and get at it easily. Thats also most likely what the electrician that installed it did.

Nope, we don't have an attic and no access from above. We renovated the house before we moved in and had the entire house rewired. So everything was open for the electrician.

In that case, you will have to cut out a section of drywall, and then remove the bracket, and install your new one for the fan, and install the new box, then put in a new piece of drywall, and get it taped and mudded and sanded. It will need to be primed and painted and probably that entire ceiling will need to be repainted.

You can buy extensions -- bit goes in one end, attach to a handle made to accept a hex bit, or right angle drill.

Biggest problem will be seeing what you are doing.

@cat_ky All of that mess is exactly what I'm trying to avoid.

@PPF. Yup, tried that. And you're right, not being able to see makes it extremely tricky.

One of these would help, and do not cost much. I'm linking the first one in a search and not recommending this specific one.

https://www.amazon.com/Inspection-Fantronics-Waterproof-Borescope-Adjustable/dp/B071HYRPND?source=ps-sl-shoppingads-lpcontext&ref_=fplfs&psc=1&smid=A2UPMZGQGXZTQ0

Maybe? Cut a larger round hole where the J box to give you better access? Patch with drywall and cover with a ceiling medallion?

Or cut a small access hole directly in front of where the bracket is attached and use a right angle drill or drill attachment to remove the screws?

Look like drywall screws holding it in the photo. Were the ones that you removed drywall screws? If so, they are brittle and snap easily. Manhandling it...grasp, twist, pull...lock onto the bar with vise grips...might get lucking and be able to snap the screws.