You are absolutely right that it doesn't make sense to rip up the hardwood flooring in the family room. Instead I suggest you remove the inset carpeting and fill the space completely with hardwood flooring. I know you thought it would be too formal, but if you place a casual rug there, it will be even less formal than it looks now.

One thing that would really help reduce the formality of the area is to change the glass coffee table and the end table to more casual ones. To make people feel relaxed in the space, they need to feel that there is nothing that is easily breakable or easily damaged. I also think it better if they not match each other. It gives a much more current and relaxed feel to the room. The finishes need not match either.
Carolyn Albert-Kincl, ASID

How about just laying a larger area rug on top of of the inset carpet for your new dining space? Then, put a large complementary area rug for the new family room? Caroyln's option is best, however. I agree to create a casual space with warm furniture. I also think floor to ceiling curtains in the dining area would warm up the space and lend to the family room feel.