Not typically difficult but we cannot see how it is attached. If there are anchors in the field stone any patching will definitely always be visible. If you damage the stone during removal it will be incredibly difficult to repair. The finish will more than likely need to be reapplied as the area where the mantel is not will be a different color.

Another idea is a tv wall mount that pulls down - its made for over fireplaces (as it will still be too high, even without a mantle, for some of us.) Google MantleMount - amazon has it. HTH! (FWIW I was going to say I wouldn't move the mantle - because the filler will always show (where the current mantle anchors are) but: the tv will require more holes in the stone anyway. The tv will cover up the 'old' mantle holes-repairs.)

Lowering the already lower mantle, will not look all that great, if you decide to do that. I would leave it, and find another place for your tv. TV should be placed at eye level when you are sitting down, and you shouldnt have to look up at it.

You need to check on the required clearance between the fireplace opening & the mantel. Varies, depending, for one thing, on how far the mantel projects.

Lowering the mantel may not be an option.

Thank you all for your thoughts. I'm looking into the mantel mount. That was a great suggestion I didn't know about. Appreciate all of you weighing in!