If you've ever listened to ralph snodsmith of "the garden hotline" you know what to do . go outside clip of the fruit and the flowers. I couldn't do it no way I would be to excited for the little fruit and come later in the season when the plant didn't produce as well I would hate myself and say I should of cut off those flowers. However next spring if it happened again I know I would leave those flowers. So if you can snip if not well your not alone.

I agree with daylily. When I do, unfortunately (as I don't like store bought plants) buy plants, I always try to find one without bugs, diseases, plant damage, etc., but most importantly one w/o buds, flowers & fruit. I like to let the plant mature in my environment, not the store's or nursery's. Also, at that height, it's still almost a seedling and will not produce well if you let it fruit now.