You didn't say, but I'm assuming, after noting your USDA zone, that your plant is in a container. The symptoms you describe, especially the wholesale shedding of individual branches, is usually the sign of an extremely pot-bound/root-bound plant, which could also compromise your ability to irrigate properly. How long has it been since you potted up/repotted/did root work?

Al

I noticed the last year or two in my orchard (SE Maryland) in June, early July that I have something similar happening where an otherwise healthy individual branch or trunk on a multitrunk fig will suddenly whither. I've decided that an insect of some sort (Asian Ambrosia Beetle?) has bored into the wood killing that section. I was concerned at first thinking it would wreck havok through the orchard, but damage has been very limited -- so I'm guessing something in the local eco system is keeping the pest in check.