How often do you water it? Once a week?
Maybe the soil is cooling off too much since these really are tropical.

It will do better in following years if you can bulk it up larger and thicker next summer before it has to suffer in the colder weather. They basically can simply almost melt away in cooler temps until there is nothing left. If you can give it some bottom heat when you first bring it indoors to keep it growing, that will help. I know if it is difficult to keep going even here in our zone 9b northern California winters unless it gets planted in spot with full warm winter sun, reflected heat from masonry walls and surrounding concrete paving, etc. I had a pair I struggled with left outdoors year round here in Berkeley, and after the third year, out they went. I like using Cotinus coggygria 'Royal Smoke' to similar effect, but it isn't quite as tropical looking...