The majority of my hibiscus buds are dropping. When I inspect the buds I see insect poop and a hole where something has eaten its way out of the bud. Occassionally, I'll find a little green worm hanging out of the hole. I'm in Texas, so it could be gall midges, but the hole and the green worm don't match with what I'm reading about that pest.

Does anyone have a clue what this is? I want to make sure I'm treating the specific problem, not carpet bombing with pesticides.