I believe it is the heat. My tomato plants are loving it but my pepper plants were doing better on the days when the highs are in the 80's. Lately we have been having 90's and reached 102 yesterday-yikes! My pepper plants had a couple of the peppers that were just starting, just before they flower, get yellow stems and fall off. A couple of the leaves got a yellow color and were justing hanging on the plant. This happened a couple of weeks ago in our last 90 degree week. Then we had 70's and 80's for about a week and they started to form more peppers. Of course after they started we got back up into the 90's. I have found that with my tomato plants they are really growing better in the heat however I have to water them 3 times a day instead of once a day. My plants are on my deck though not in the ground. My pepper plants I have been needing to water everyday in this heat where when it is in the 70's I was watering them every couple days.
Tomatoes love all the sun they can get and the pepper plants need the sun but the peppers themselves need a little shade from the leaves on the plant. I am no expert. This is my first year also but have been doing a lot of research to keep my plants happy.

Could be heat. The temerature just dropped from 104 degrees to 101 degrees outside. I think i may go outside and water them now. I was out there earlier and the tomatoes looked great, and most of my other stuff. But i planted a cayenne and 4 jalapenos and one bell to close to a couple of zucchini's and i think the zukes are zapping alot of water and nutrients from those plants, although the cayenne and jalapenos are producing peppers still. This heat is insane for NY, and to think i actually worked an 8 hour shift today in this heat on heavy highway construction, i must be outta my mind! oK, YEP I THINK I AM.