Could have been where a gravel pile was located during the original construction of the house(s). Builders are not always fastidious about cleaning up. The only downside I see is the hassle of removing all the rocks. :-] You might consider making a sifter with a wood frame and some chicken wire. If the soil is fine enough to sift. Too many clods and it may not work too well.

Well, I got my rocks out and added some compost to the clay, went down only about 3 inches. Was going to plant perennials tomorrow. As I lay sleepless in bed tonight it occurs to me to worry that this rocky area might be related to my septic system. It is between the tank and the house. Would there be 2" white gravel so close to the surface related to the sewer line? I got down to the end of the gravel and did not encounter a pipe.

Whoops, it could be your leach field! Depending on where you live, there can be regulations determining things like distance from house, depth, etc... but where I grew up, part of our leach field was exposed - wait for it - white rocks.

Do you have the plans, or drawings for your house? If not, check with the City, Town, County...

Rocks naturally sink into the soil if there is no weed guard or something.