Very attractive home with lots of curb appeal! You're correct--the stone is dictating the color of the siding and trim. At present the warm color of the stone doesn't complement the cool green/gray of the siding. I'd see if I could find a warmer neutral in a beige/taupe color range, and pick a lighter version of that for the trim. The bright white trim gives too much contrast.
You might try these combinations from Benjamin Moore's Historical Collection:
Siding--Huntington Beige, Quincy Tan, Yorkshire Tan, Lenox Tan, Brookline Beige, Tyler Taupe
Trim--Monroe Bisque, Dunmore Cream, Pittsfield Buff, Putnam Ivory, Bradstreet Beige
Q
Similar stone to ours though a little more orange. I like the lighter color with the darker trim.
Q