I'm no expert on growing olives in containers but it sounds as though "wet feet" might be the cause of the leaf loss. Olives like very good drainage and are drought resistant. Perhaps the potting medium retains a little too much moisture. Try giving it less water and perhaps a little more often and don't have the container standing in a saucer.

Did find this US site which gives information on growing olive trees in a container and hope you find it useful.

Here is a link that might be useful: http://www.whiteflowerfarm.com/growing-olive-tree.html