I would go with warm gray/taupe toward brown more

Some kind of

Thank you for that rendering, that's very helpful! That darker gray is exactly what I've been picturing, can you tell me which actual colors you would suggest? Would Knitting Needles by SW be a good choice? I should also mention that we're adding a black metal roof.

Well, I am happy to hear you are changing the roof, since the brown roof you have doesnt go with black windows either. Knitting Needles has a purple undertone, so you may want to rethink that paint color. I would pick a paint that matches the limestone in color, so the house looks all one color.

I actually did not have any particular color, just painted with what I like.

But my picker tells me that is Garret Gray SW or Bear Creek BM

https://www.benjaminmoore.com/en-us/paint-colors/color/1470/bear-creek.

You might want to check shades of Bear Creek. Matching colors are all in your stones.