Maybe because they get so large.

Not really. For a tree they are medium sized. Oaks,Ash,Pines..on and on get much larger. Plus here in the bay area there is a Redwood in every yard. Like a Blue Whale as a pet.

There are some very large front yard ones in the 1950s neighborhoods in Fremont. The trunks always seem too big for the canopies. Even the city govt. has pointed some out: https://fremont.gov/2599/Centerville etc.