The hill concept is to prevent water loging and roting the roots, stem.
With limited space, you can plant bush types that will not run for ever.

Yeah, but the ones I am interested in growing seem to all be vining types only...

Yes, you plant the squash seeds in a group, 4 or 5 in a hill about 18" across with individual seeds 6-8 inches from each other. Then you thin to the best 3 (I have to admit that I don't always).

The roots don't mind growing together as long as they have the room between the hills to spread out in.

You'll find that the plants in the hills vine out from the center (even the "bush" varieties spread their stubby vines this way), in an arrangement that shades the roots to keep them cool -- very necessary for squash, which do NOT like direct sun on their roots.

It goes back to the way that the wild ancestors of the squash would have grown from the clump of seeds in the rotted fruit.

The minimum space required for a very compact, bush variety squash would be the center of a 3 foot square. But only the most carefully-bred compact varieties would stay that small.

For ordinary summer squash of the bush type you'll want to think of each hill as the center of a 4-5 foot square. For long-vined winter squash you'll want to put your hills at least 5-6 feet apart, but you can fudge that on one side by gently turning the vines back on themselves and directing them into an area where they are free to run.

If we're strictly talking about the amount of space needed to keep the plants health in terms of nutrition, water and the like, do you think a 3x3ft square is good enough for a hill of 2-3 plants? I can run the vines into an area where there is no tilled soil, or onto a trellis or something to give the vines space. I guess I was more curious about how much space the roots need.

Thanks for the info!

IMO, that would be the absolute minimum space.

I pulled up my first planting of summer squash this week and even though they were not the most vigorous squash on the face of the earth their roots were 2-3 feet long.