As long as they're normal earthworms or redworms they're fine, they eat waste not roots. If you're leaving it in the pot for any amount of time make sure it's more shady than you'd normally put peonies, the pots will heat up in full sun and can kill the plant and the poor little worms!

Thank you for your answer, appreciate it. I'm new to gardening. No green thumb here. ;P