Look on the underside for spider mites. They're super tiny and look like dirt or dust. You may need a magnifying glass.

Have you seen anything that you assumed was spider webs on the plant?

Yes, there are spider webs but there are a few small spiders and a couple big spiders that hang around on my plant. I thought it was just them

Here is some information you may find helpful.

Could also be russet mites, which I believe don't make webs. But that speckled damage is pretty characteristic of mites.

Not russet mites. And I doubt it's spider mites either.

Compare OP's leaves with image on this page: http://www.austintexas.gov/blog/mite-be-spiders-0

The damage looks different to me.

Speckles traveling down the leaves from the center out, starting on the plant from the bottom up... looks like classic early stages of mite infestation to me. To me, it looks just like the photo in your link, just that it's less advanced than what is shown in that link.

I'd love to know what else does that kind of damage for future reference.