My advice in cases like this ....is always to take a few leaves to a nearby high quality nursery ...they will know for sure what causes this in your area

In our area overwatering/poor drainage? seems to encourage bugs to do this...

Thank you. I will try that. I do think I have poor quality soil- it's so incredibly dry and parts of it are like sand. However, I haven't had a problem with this in the past which is why I was thinking it's a pest of some sort. I even saw these small whitish-grey spiders on one of them.

It definitely looks like pests...and a garden pro can tell what kind of pest and how to treat it...as well as what to add to your soil to make it a better host to plantings...a sandy soil does drain better so that is not all bad...if it was not so bad I would spray the entire plant with Neem..especially under the leaves...this pretty much stops the attack...in this case you need to re-set the conditions for a happier growth environment...and maybe prune quite a bit...