would a backer rod stuffed in there and then caulked over work?

That's probably what I'd do, only using a sealant instead of a water cleanup caulk.

Definitely stay away from expandable foam for that task. Bear in mind that if you use a sealant and you don't have good caulking skills, you could end up with a smeared mess on the stone that could be hard to clean up and make neat. You could tape everything, but you may not end up with the kind of neat caulk line you were hoping for when you pull away the tape. Also be aware that almost all the good sealants are harder to paint over if that is your plan: they need to be fully cured first and then typically require the use of something like acrylic paint to get the coating to adhere. So read labels carefully before you start filling that gap.

How about removing the stone that is causing the issue to begin with?

That's sort of like using a cannon to kill a mosquito...;-)

the stone will not be removed ;)