Depends on your budget. I find the stone too busy and wood paneling awkward. If budget were no concern, I'd get rid of both and create a floor to ceiling fireplace surround either from tile, ledgestone, bricks, or wood planks. Inspo:

House on the Hill · More Info

Plan #011S-0003 · More Info

Calistoga Farm House · More Info

Swedish Farmhouse NE 38th Ave · More Info

Model Home, Starr Homes LLC · More Info

Oh - and if you keep the stone, I would probably drywall over the paneling and paint the mantle to contrast with the white (perhaps brown or gray).

That stone is wild. so not traditional as you say you are more of.

Do you have the budget to get rid of the stone and wood above and redo a traditional FP surround.?

That is what you need to do.

As said, try working with the stones' colors. If not the whole thing, why not alter just the "worst stones," using expert paint help on the darkest, or on the most tan.... if a mess of contrasts.... not sure if that's the worst part. From here, it seems too nature.

A transitional by a pro... using a material to cover over the upper wood only.

Houzz quote by Merzbau Design Collective: "The fireplace surround was built in place: wood framing with cement plaster on lath."

Wild Basin living room · More Info

That fireplace is a hot mess, indeed. It has to go.

Depends on how expensive your new FP surround/mantle turns out to be.

Start by browsing Houzz and fall in love with a FP ! Then call in 2-3 contractors who have experience doing fireplaces and get estimates from them re: demolition and new surround like your inspiration pic.

Keep us posted.

Good luck,