Primarily, it's your money, so you should get what you want. Having said that, ostensibly your architect knows what he is doing. Perhaps you are trying to cram too many things into too small of a footprint. You can shrink rooms, but they may end up functioning poorly. A bathroom where you have to dance around the door to close it, a too-small-to-actually-hang-stuff-properly-sized closet or a kitchen where two people is too many can be a constant source of irritation. Be sure you are alloting the proper amount of space for all you want.

The finishes on the exterior are generally open to interpretation. Changing those shouldn't be difficult. Just be sure to listen to your professional if he posits a reason or suggestion. Chances are he laid out the exterior for a specific look and possibly cost-point.

Your architect may be absolutely inept or he may be trying to give you what you want in a functional format. You'll have to decide. If you do decide to walk, it's cheaper to do it in the design stage than at any other time. Be sure and ask if you will be able to take whatever drawings have been done to this point if you even want them. Different architects have different policies as to who actually owns the plans. You may wish to check your contract/agreement.

How is the architect's fee calculated? Providing multiple elevations with a lot of client input is desirable but also time consuming. It is also difficult for a designer to know where a client will tolerate cuts in floor area. This usually requires a face to face meeting with tracing paper and a calculator but that does not always result in a space reduction. The bane of any design meeting is that owners (and designers) fall in love with ideas and can't let go of them. These days most people want many different kinds of spaces even though in normal use they are to some degree redundant.

We don't have a written contract. He basically charged a fee at the beginning which we paid him once we had a basic floor plan. We were then told to be as picky as possible when going over the plan and point out any changes we would like to make. We could make as many changes as we liked. There were a few right off the bat. After we pointed them out to them, he made the comment that he hoped those were our only and last changes. We started dealing with the designer about six months ago. We gave him a detailed folder with what we wanted in each room. This was the actual room size (give or take a little), built in desk in the three children's rooms, walk-in closets, large walk-in pantry and any extra storage space. For example, we had a great room that we wanted approxmately to be 20' X 20'. They changed it to 24' X 24'. They also put 60" widths for all the hallways. Every room in the house was made "huge" with lots of extra space. When we started working with this architect, we were referred to him from the developer of the subdivision that was being started. They said they only had a "working relationship." We recently found out that they are very good friends even though they live a couple of hours from each other. Now, we don't feel our architect is looking out for our best interest. The bigger the house we build, the more his friend will make. We don't really trust him at this point.

I doubt your suspicions about the architect wanting to help his friend are correct. It is more likely the architect is trying to finish the project with a profit. Reducing the size of a house is a lot of work not only in terms of finding ways to accomplish it in plan but in revising all of the other drawings. If the fee is fixed the architect could lose money of the project.