Are the touching the soil at all? If so, try to keep them off the soil. As far as watering, if you have to ask, I am guessing that over watering is a definite possibility. They can be picked green but the preferred stage for picking Cayennes is Red (Ripe). Also, is your weather supporting good growth at this point in the season? You don't state which growing zone you live in, but things are about to end up here in Wisconsin and if you recently had a night or two of freezing or near freezing weather, it could be that the peppers were damaged at that time.
Bruce

Hi Bruce,
I'm sorry, I should have said I'm in Alabama. It's still pretty warm here - in the 80's to 90's during the day and 60's to 70's at night.

No, the peppers aren't touching the soil at all. I did pick some of them while they were still green and made pepper jelly with them. Of course, the jelly had almost no heat. That is why I wanted to leave the current peppers on until they turned red. I want to make more jelly with some heat this time!

I will cut back on the watering a bit and see if that helps. The peppers that are still green are beautiful and don't have any blemishes at all. It is only when they begin to turn color that this is happening.

Kind of looks like bacterial soft rot to me, link below.

Here is a link that might be useful: Soft rot

Or sunburn?