Recently, my ficus shed most of its leaves on the side of the plant away from the window (I was gone for 2 weeks and my house sitter closed the blinds against my explicit instructions)

It's now outside for the summer (I live in NC) and the half of the plant with leaves is doing well - leaves are large and very dark, and there's a lot of new growth. The branches on the other side, however, remain bare.

I've scraped a few of the bare banches and do still see some green, but now the foliage on a few limbs close (on the same trunk) to the bare/dry ones are starting to look wilted. Should I just prune off the bare limbs, even if they make up half of the tree?