While I think that wearing gloves is an appropriate precaution when handling this plant, I'm not sure how your entire garden will become poisonous by its presence. Many folks including myself grow various species of Datura and Brugmansia intentionally with no ill effect...

Do you know what species you have? If it's one of the annuals, you may have a seed base built up in the soil that will eventaully be exhausted. If it D. metelioides or wrightii there may be portions of the roots that you are not getting when pulling.

The plants are pretty easy to ID when young; I would probably do the pulling/weeding/spraying when they are small and vulnerable.

Unless you are eating the plant parts or getting significant amounts of plant juices on your skin you'll probably be fine, (I've heard of a few people that are highly susceptible and brushing by the leaves might be enough to cause them problems).

Mike