Its probably those darn caterpillars that seem to loooove brugmansia. I always have a battle with them on some of my brugs. I hand pick and I spray water on them to remove some baby ones and eggs. I look at the underside of the leafs and I can see tiny whitish eggs.i used to brush them off but I seem to do it every single day so I know just spray them with water when I water my brugs and it helps a lot to keep bugs away. Caper pillars seem to mostly love the brugs with velvety texture leafs. I seen plant food with insecticide in it and it suppose to keep bugs away I want to try it and see if it would work but its so darn expensive. I will try and ask and see if anyone has used those kind if things.

He's right, the caterpillars/moths are attracted to the furry leaved brugs the most. Like Jesus said, hand-picking is the best, but they are camoflaged so well that sometimes you don't see them until they are munching on the center whorl of leaves, which is very disheartening.
Try Safer brand Caterpillar Killer. It's organic, a bacterial fix that only attacks the worm, and does not harm the environment.