The first thing that comes to mind is lack of pollination. You may need to hand pollinate the females flowers. There are a number of threads on hand pollinating you might want to do a search here or look in the FAQ.

Another thing that may be causing it is if the plant is stressed (too much rain, too little rain, too little nutrients.) The plant will abort the fruit if it can't support it.

I'd say lack of pollination as well.

Thanks! Well, we have not had much rain here in SW Tn as of late. Will try watering and pollinating