Yes, they are a mushroom, probably one of the scores of puffball types. You can't get rid of them permanently........they are just an occurrence of nature and the spores exist in the soil. But they do tend to be very seasonal and you can just rake them away when they appear if you dislike them. They are also harmless to anything growing in the garden.

Like with a lot of fungus the bulk of it is underground. What you see on top is just the fruiting body, the reproductive part of it where the spores develop. The spores are so fine they blow in the wind and can travel long distances, so it's true you have no chance of effectively getting rid of them.

Definitely a puffball, most likely Bovista pila. None of the true puffballs are poisonous; some are edible and choice, though not this one.