I think tearing out the brick wall or the hearth would be a very messy job, and risky too, because you don't know what's behind it -- maybe some bad surprises.

Think about sort of "getting around" the problem. You could paint the brick or cover it with sheetrock. Or you could build shelves over the brick to store books or to display s collection of things that please you.

I think you can greatly transform the appearance without tearing it out. The challenge is to turn what you do not like to your advantage. The hearth is solid space that could be put to good use.

Good point about now knowing what's behind everything. I just hate that the hearth spans across the entire wall and I can't put furniture all the way against that wall like I want too and I'm losing space because of it (and the rest of the room isn't an option, I have one long sofa against the only actual wall, the other one goes into the kitchen and the other has a sliding patio door to go outside.

Ah, the joys of home ownership!

You didn't mention your budget. If you don't have the budget to make some changes, just paint the brick. If you want to make changes, consider a direct vent gas fireplace. That could solve your heating problem. DV fireplaces are "an investment", so you might find it cheaper to add the ducting for your heating system. The brick might only be facing (most likely) so the removal isn't as difficult as you think. A contractor or fireplace company can probably tell you.