Over the years, peonies form BIG roots. Unless it's a really BIG container, I wouldn't suggest it. Maybe a nice planting of Iris, daylilies, hydrangea, clematis, etc, would work better? Try to stick with plants with a finer root system.

Thanks for the suggestions. After the canna removal, I planted Glads. They had a very fine root system and it was matted to the bottom of the container.I also removed them. The containers are about 16"x16" and I wouldn't call them BIG. I already have hydrangea and the deck isn't a suitable location for the clematis. I have three elsewhere in the yard.This has been such a crazy year for my garden, the one Day Lily, Blue Eyes, didn't even bloom. Back to the drawing board, I guess.
Hmmm, I might plant cannas again and divide BEFORE they get so crowded. We truly enjoy the tropical look and I can wheel them into the garage over winter.