I think it's probably all the wet weather. The native white ones (P. digitalis) survive in the woods crowded by other plants. Now if it was a western species I'd say they would be prone to mildew. I've tried them. After visiting the west I wanted some of those deep reds and purples for myself, but it was no go in Iilinois. I think the two most colorful native species you can grow here in the east are cobaea and smallii. Anyway, I wouldn't worry. Make sure your Penstemons have good air circulation and aren't crowded by weeds or other plants. Good luck!