It's not that the leaves are poisonous to your worms, but rather the rhubarb leaves make your wormery ph acidic. When that happens the pile will begin to sour, encourage mites and fungi, and your worms will not live or breed well acidic conditions.

The 3 plants to never feed your worms are citrus,onions, and rhubarb.

Here is a link that might be useful: worm composting faq's and tips blog

My worms do OK with onions, and they get a lot of them. I have also fed oranges, They seem to be OK wit the oranges, though it took a few weeks before they were willing to eat them.

I have more trouble with grains that anything else.

Thanks for the info on the rhubarb leaves. I do feed mine onion ends and oranges. They chow these down with no problems. They really like grapfruit too.