As a builder, I would say this is purely a matter of personal preference. It is your house so the facade should be stone where it looks good to you.

In my opinion, the elevation looks nice as drawn, but adding stone on the left side would not hurt either.

Ask about the cost implications too...that often plays a major factor in veneer choice.

If this is a brick house IMO you already have too much stone

Yes, it is a brick house. The current builds he is doing involve brick and stone. As far as cost, the builder offered to add stone there if we would like, no extra cost.