I don't know ID. but if a reasonable number, I'd start flicking them into soapy water.

I am not recognizing the plant. I suspect that it might be a pest that is species specific to that plant, since no eggs appeared to be laid on others in the garden areas. Let us know what the plant is and I'll bet that an ID of the pest will follow. Meantime, do as Jean suggests...remove all that you see.

Perhaps sawflies? Very similar to caterpillars.

And yes, please tell us the plant they're on.

Spraying something, until you know what your have, is not what Integrated Pest Management is about, it starts with identifying what you have so the proper, and least toxic menas of control, can be used.
Take a couple of these, sealed in plastic, to your counties office of your state universties Cooperative Extension Service where they should have the resources to properly identify what you have.

Thanks for the replies. :)
I do not know what the plant is called, nor does the owner, my neighbor. It was a gift to her years ago.
Kimmsr, I didn't mention spraying and I don't see where anyone else did. I asked for ID and for an organic solution to the problem.
I went out the next day to collect the worms, but most were gone. There were bits of white here and there. I suspect the birds found them. I asked my neighbor if she would mind if I cut the plant down (with the intent of burning the remains) and she said she would prefer to dig it up.
So thanks again. :)

The birds were probably watching you take pictures of the pests and decided that they must be pretty special.

rhizo - LOL! Makes perfect sense. :D