Big worms on plants are always eating the plant. Smush every one you see, Small worms may be good predators (syrphid fly larvae, ladybug larvae) or pests (rose slug on bottom of leaf). Spiders and little black pirate bugs are good. The brown balls may be feces.

Generally pests are close to the damage and don't move around much. Predators are active and often you can observe them hunting or eating bugs.

For rose slugs & picture, see the thread by redsox titled something about insecticides.

The big green worm is the culprit and the brown balls are his droppings. They can do a huge amount of damage because they're such voracious eaters. The eaten leaves won't grow back but others will take their place. The little black bug might be the one eating the blooms also. Hopefully in time insect predators will come along and take care of that one. I don't spray but I would try hard to pick off the worm/caterpillar.