I agree. Those "walls" should be removed!

The stone above the mantel looks strange, also. Almost like it's not flush against the wall

Those walls need to go and with that heavy firplace the shelves need to be quite thick. Take the art down on top of the window and lower your other art. Unfortunately I think you will have to remove all those builtins to remove those weird walls the cool thing is if you did that the TV might fit in one side on an articulating arm so that you would have a better viewing height.

black square represents larger object eg. tv or picture, orange depicts mantal, stone brick should be painted out and those sidewalls should go but if structurally it can't be done consider removing shelves and extending them out to edge of frame, add addition to extend to ceiling....and if that is also not feasible then consider a flush frame to ceiling and adding doors to enclose that space

pic

drywall the top of the stone fireplace, paint everything out and add lush hanging plants/basket on top shelf either side... you'll have to add/contruct the top shelves but if simplicity is your goal then this could very well work for you

sorry my diagrams are far from perfect, I'm using my phone which doesn't have the proper design app
anyway this may not be a concern for you but ideally I would hang 96" or longer curtains as that would balance out with the fireplace wall.
As for this diagram the white represents an area completely drywalled flush across the upper partiion...I would still consider some kind of cabinetry doors for the lower, side sections

this pic is more realistic and you could get a look that resembles this...I hope I've been of some help to you

What did you end up doing? Please post after pictures :)