Not dangerous a bit to handle them, just don't eat them! :)
I've read that milkweed "milk" is poisonous, so I imagine if you should injest some of that, either from the plant or a caterpillar that's been eating the leaves, you'd have some significant heart problems, I think it is. I know if you get some in your eye, it can do damage, though probably not permanent, to your cornea.
I've handled plenty of them with no harm.
Sherry