That isn't unusual and you aren't doing anything wrong. The lower leaves deal with all kinds of issues from pests to pathogens to being shaded due to their proximity to the soil. Often splashback from the soil during watering and rain causes many of these issues.

It is usually recommended to remove the lower leaves below the fruit to reduce disease and the potential to transfer to the rest of the plant. It will also allow air to flow under the plant to reduce humidity there as well, which also helps. It's also good to put mulch under the plant, especially in sandy soil to cut down on splashback and retain moisture.

Thanks Capoman. I'll keep trimming them off and see what happens. Appreciate the tips.

It looks like a nutrient problem to me. It's hard to see the curled-up leaves, though. The curl is stress from something. It could be from the Nitrogen in the worm castings.

What fertilizer do you have in the soil? I've never been a fan of foliar feeding.

Here is a link that might be useful: foliar feeding link

Looks like you need mulch - that will help with watering frequency and splash carrying spores.

Oops