I am sure that there is something to your theory. I bought a one gallon sized petunia last year because my wife liked the color and the 4 count pony pack I bought out grew the other. I am betting that in addition to what you are thinkng about that there are additional set backs caused by transplant shock. Some plants just hate having the roots messed with. Squash, watermelon, and other members of that family are affected very easily whereas tomatos, hardly seem phased at all by transplanting. I personally tend to buy things as small and cheap as possible because I think it is better to spend a small budget on a more diverse selection.

Your theory has some validity, I think. It could also be that Lavender is a fairly drought tolerant herb and prefers drier soils to wetter ones- meaning you actually gave it more optimal condidtions than the neighbors did.

Plants have a predetrmined growth pattern that can be fairly predictable.
In one season a plant should grow X amount.
\The reason theirs was in a 5 gallon container is it was started before yours was in the season, so it had a headstart.
But apples to apples... at the end of the season it grows the same amount no matter how big (or small it was when planted)
Also the root system in the 5 gallon container is fairly established, so transplanting it into the ground, verses the smaller pony packs, will cause greater trauma when transplanting, and the plant is in shock longer witht he bigger roots.
Also as mentioned above, the Lavender is drought tolerant, meaning, your neighbors were adding more stress to their 5 gallon plants.
Over all, it is soooo much better to buy small and save time, and stress to the plants.
Have a wonderful day! :-)