I have an olive grown from a pit taken from a salad I had on a trip to the French Riviera in 1991 - it's planted in a protected area near the house, and has never produced an olive. That's probably typical of seed-grown varieties, but it has great sentimental significance and makes a nice foliage plant. Based on this experience I would say that olives are a possibility in this part of southeastern VA, until we get a really exceptional freeze, which tends to happen every 15 years or so (we had one in 2004, but my olive was still indoors in a pot at that time).

I have a potted 'Arbequenia' olive. It grows fast and blooms in late winter. It has never gone through the process and made any olives but one of these years the timing will be better and pollinators will be around and it will happen. I plan to always have it in a large patio pot and never plant it in the ground.

Glad to hear that others have had some success at growing the trees. I would say that my tree is probably 10 years old and has been in the ground the last four. It is surrounded on two sides by high brick walls and has survived a low of 17 degrees with about 12" of tip dieback on some of the most exposed branch tips. I keep my fingers crossed for the mild winters to continue but will take protective countermeasures to keep it alive if necessary.