The caterpillars are eating your tree, you need to decide if you are in the all you can eat caterpillar business or the growing of landscape trees. If you spray your tree weekly for three weeks with BT you can control the caterpillars. Bacillus thuringiensis a bacteria that will only be harmful when ingested by a caterpillar. Al

But if they're sawflies, Bt won't work.

If you handpick, they'll be gone today.

If the yellow butterflies are 'cloudless sulfurs' perhaps your tree is a cassia / senna -- both a host and nectar plant for sulfurs.

Phoebis sennae

Yes, those are the caterpillars of sulfur butterflies. I planted my senna to attract them. My bush's flowers only last a few weeks anyway, and the cats don't eat enough flowers or foliage to ruin the show. For camouflage, the caterpillars are usually greenish in color if they are eating the foliage; yellow like the one in your photo if they are eating the flowers. Although photos can be deceiving, yours appear to be about the size that they will wander and get ready to make a chrysalis. Below are some photos. The chrysalis can also range in color, usually from green or yellowish-green to red.

I actually grow plants to encourage butterflies, and am delighted when I see caterpillars. But perspectives vary.