A picture of the entire wall is needed to make suggestions.

If covering the wall with stone, I would remove everything and would also cover over the granite around the fireplace with the stone and change the hearth as well.

I agree with leaving off the mantel and surround if you're using stone. I'd keep it clean and monochromatic like Maureen's picture shows. Otherwise, it will be very overwhelming on that large chimney. I'm curious, how will you deal with the side of the chimney. If you look at the picture Maureen posted, you'll see that the built ins and shelves come out past the stone, which I presume wraps all the way around the chimney. They were probably built after the stone was done. In your case, your built ins are recessed. It appears that there is no good place to end the stone on the sides. Just a thought.