I don't know how experienced a gardener you are, so forgive me if this is info you already know. Olive tree sare relatively difficult to grow in New England since they aren't hardy in this area, but they need a lot of light. So they would need to go outside in warm weather and inside in cold weather, either in a greenhouse or in a sunny window with grow lights adding additional light both in intensity and day length.

Here is a discussion started by someone in CT on issues he is having with an olive tree.

Edited to add:

Logee's, which is somewhere in CT I believe is always a reasonable place in New England to check for plants that like it warmer than our usual weather. They do mail order, but I have never bought anything from them either in person or via mail order.
Logee's olive trees

Logee's is in Danielson, CT. They have several varieties of Olea available for mail order in their online catalog. If you want to take the ride, it's a very interesting place to visit. I've gone there for over 50 years now, although not as often lately. They have some things that you simply will not see anywhere else.

Not sure if you are in Boston area but there is a guy in roslindale who sells olives on american legion. City farm or something. Google figs.net roslindale joe. It'll come up.