I wouldn’t. I would try another color behind the tv. Navy or even wallpaper, then bring color and pattern into the room with a throw and pillows. Consider artwork on the walls and new fabric on the door. I think the stone adds some warmth and texture that you might miss.

I personally love the stone, it is very warm and comfy.

Playing off of theresen's idea, here's your accent color behind the TV along with a throw pillow. Even more, I would cover up that prefab fireplace with a beautiful Custom Fireplace Screen and throw in some interesting andirons and fireplace tools. Plant on the left for balance.

The fireplace accessories can also be done in a bronze color to tie in with the chandelier, but they just show up better in this mockup in pewter or satin nickel, which also blends well with navy.