Aesthetics and practicality sometimes conflict with that decision. A large infestation looks terrible. It is not aesthetically pleasing but it does not hurt the plant (much at least). However, do consider that fungal spores are all over the bad looking leaf and these spores may spread further so, cutting off the leaf would help prevent the spread. But then aesthetic reasons crop again... if you cut too many leaves in an area, the shrub may look weird or ugly so then you end up thinking should I? or should I not? Also to consider is that the leaves will shortly dry out anyways....

Oooooooohhhhhhhhhhhhhhhh, well.....

Thanks Luis,

I don't mind the aesthetic issue at this point. My concern is whether is will harm the plant if I cut off all the leaves. If I do, sounds like I cut them off where the petiole attaches to the stem, correct?

No harm? Well, these leaves continue serving their primary function of providing food and sugars to the shrub. Their days are counted this time of the year though. So try to remove the worst of the worst and let Mother Nature do the rest. But, if the infestation is large (and looks very ugly) then maybe it is best to cut off all/most the leaves. Tough question, hu?

Yes, cut the little string that connects the leaf to the stem so you do not prune off the flower buds now growing at the ends of the stems.

Thanks Luis!

As a general thing, removing the fallen leaves soon after season's end may help reduce problems next year as many diseases overwinter in plant debris.