That's almost certainly a paper wasp (or hornet) nest getting started on the branch. A few things from experience:

1. Don't knock it down during the day. Treat it in the evening after dark, when the wasps are all inside and least active.
2. Use a wasp/hornet spray that shoots from a distance (15+ feet) and saturate the nest entrance first, then the whole nest.
3. Wait 24-48 hours before removing the dead nest, and wear long sleeves, gloves, and eye protection while you work.

To keep them from coming back next season: prune back the dense branch growth in early spring so there's less sheltered nesting space, and knock down tiny starter nests (quarter-size or smaller) in spring before a colony gets established. Also clear away fallen fruit and keep rubbish bins sealed nearby, since sweet smells attract foragers in late summer.