Hi motownnj,
I have also olive trees in pots.
Olive trees are stong at winter, but not at freezing for a long time, the best way to protect them is to cover all the pot (down-around and up) with an old blanket or put the pot in a larger pot adding between them perlite or pieces of newspapaer, so that to protect the roots from freezing, if the roots are proteceted you will solve the tree.
You can see bellow my pots with olive trees covered with snow, only for a few days eithout problem, of course that is n't the rule.
John
from Athens-Greece

{{gwi:500518}}

an eating olive.. or a weed plant known as an olive????

the weed plant belongs outside.. or.. it will take whatever your sun room will give ...

no experience in warm zone eating olives

ken

The olive tree is an eating olive...sold as a tuscan olive tree, so its far too cold here for outside full year. It has already been well below freezing, with lows in the 20s - teens outside.