Are you fertilizing them, too much may stop them from fruiting. They like it warm when flowering but cool once the flowers are pollinated. Frost when flowering can also have an effect, It took almost 10 years before my Carmine Jewel set fruit on a regular basis, but Juliet has produced at about 3 feet tall but is affected by frost more easily I think. The deer won't leave Juliet alone though, it must taste really good.

My guess is either cold damage to the flowers after they have opened (and possibly been fertilized) OR as is the case with any number of so-called self fertile fruits, self pollination is weak and they really need a second, different cultivar for best pollination (which is always a good idea regardless).