Sometimes tomato hornworms (some of which are tobacco hornworms) will attack peppers, also. Did yours look like this? http://www.tomatodirt.com/tomato-hornworm.html

When they are on my tomatoes I yank them off, usually with my fingers, but sometimes with tongs if I'm feeling squeamish. Then step on them and try not to get grossed out if the guts shoot out from underfoot. After a few tries, you can learn just how to step and twist your foot to keep things from getting ugly. If you have a large garden, BT works if applied when the caterpillars are small.

I haven't seen any of this type this year, but there are some white, fuzzy caterpillars that seem to love my Phoenix Palm. I'll go out early in the mornings to water all of my plants, and there are anywhere from 4-15 caterpillars, just on my poor Palm. There's 27 plants out there, but they single out the lonely tree. I don't understand it, lol. I just pick them off with my bare fingers & toss them into the dogwood tree closest to my 2nd floor balcony. Most of the time they land on the tree, but sometimes they fly on past it, and hit the ground. Oops!!

I get the big hornworms, they're determined things and latch onto my plants so tightly I can't pull them off and put them somewhere else. I wish I could--the moths are cacti pollinators. So, I wind up decapitating them and letting the birds figure out how to get it off my plant. There's no need to spray poison when a few minutes and some rocks will work fine.

Parts of the Southwest have a rainbow of native hornworms, some are very neat looking. The moths are important pollinators.