The most common problem is root rot. When you water it, does the water drain out the bottom right away? If not, the roots are keeping it from flowing, and sitting in water. That can be fixed by repotting, and trimming the roots. If any are black, cut back to the white. Sometimes just the change from cold outdoors inside will cause problems like leaf drop, but most get over it in time. Also, keep it away from any gas heater vents.

The water flows right out the bottom. I was afraid of root rot so I let it dry out and pulled it out of the container and it all came out no problem so it is root bound. Since doing that I'm afraid that the the water has been flowing around the sides therefore it's actually not getting the water. I'm not sure though. Do you think it would be of to repot in the winter ? Thank you

I would repot when it is showing vigorous growth. Have you checked for pests via magnifying glass or jeweler's loop? I had an olive tree from a workshop, and in its first winter with me I started having the same problem. I don't remember the pest now, but I remember it was a real infestation- enough that I just pitched the plant I'm now sorry to say. I could have sprayed it with isopropyl alcohol or maybe an insecticidal soap.

ez