I don't do broccoli, but it seems like BT (Bacillus thuringiensis, Dipel, Thuricide) is the obvious way to go. Some broccolists need to weigh in here.

never had that problem. Only fly maggots

BT or row cover

Agree with yolos and Dan. BT works if you feel like dusting or spraying your plants regularly. Or use row covers to keep the adults from laying eggs in the first place.

Rodney

Row cover as soon as you plant, or seed.

Loosely blouse over the plants but secure it tightly at the edges.

I hand pick for the most part, but I only have a few broccoli plants each year since I am the only one who eats it in my house. To get rid of the current worms, soak the heads briefly after harvest in hot tap water or salted water.

I only have a few plants this year so I am soaking them in salted water. Next year I will use row covers. I have discovered that my Eggplants need to be covered also.

Do you know what kind of caterpillars?

If you see those pretty white "butterflies" flitting around your garden, they're likely at least part of the problem. They're cabbage moths, and their larvae will devour brassica plants in relatively short order, if you let them. Row covers for certain, next time. :)

I've sprayed weekly with a tank mix of BT and Neem oil and haven't seen a single caterpillar despite seeing cabbage moths every day.