Have you ever thought to make the door and side lights work more with the stone? just curious. That being said, i'm not sure but I think with the right professional lime wash could work.

Back up and take more photos.

There are some videos you can see for "removing stone veneer." It is a big labor intensive job judging the one video I watched.

If your budget can swing replacing it, no harm in trying to paint or lime wash or whitewash it first.