If I found that in my own garden, I would know it was a deer. Is that a possibility for you?

Rudbeckia hirta is a larval host and/or nectar source for:
Bordered Patch and Gorgone Checkerspot butterfly caterpillars. Looks like caterpillar damage to me, though no way to tell from this pic if they are/were desirable butterflies or some kind of moth or armyworm.

http://www.wildflower.org/plants/result.php?id_plant=RUHI2

We certainly have deer and rabbits here. Are the butterflies mentioned likely to live in NY? I looked them up - they seem to be other places but not here. I'm going to try deer stopper and see if conditions improve.

I don't see caterpillars on them. I'll go out at night and see if I can spot anything. Thanks

We certainly have deer and rabbits here. Are the butterflies mentioned likely to live in NY? I looked them up - they seem to be other places but not here. I'm going to try deer stopper and see if conditions improve.

I don't see caterpillars on them. I'll go out at night and see if I can spot anything. Thanks

I've always been fortunate, in a garden sense, and never dealt with deer in the yard. It may look like deer damage too, IDK. Caterpillars are usually very good at camouflage, and visible during the day if they are on a plant. They are usually on the bottom of a leaf, to stay out of sight. Slugs and pill bugs are nocturnal, but rarely so thorough that far above the ground.

I didn't research those butterflies to that point, what range they inhabit, just looked to see if your plant is known to host butterfly caterpillars generally. Good job on the follow-up. If you are out of range for them, that eliminates those from consideration, though indiscriminate caterpillars like cabbage whites, various moths, and armyworms are a possibility on too many plants to count. IDK their exact range either, but both can really munch up plants here.