If the stems are pale in color, dry looking and with no leaf growth, they can be removed at any time. And should be removed, as any dead wood provides a safe haven for disease and insect organisms to breed and spread.

And I would recommend cutting back at the base with sharp, clean pruners. Snapping or pulling just risks damage to the root crown.

General rule is cut NOW if no growth from a stem. Or you can leave a few to support new stems that make massive blooms like on Endless Summer Original. Old stems typically won't snap with your fingers until they've been dead for 2+ seasons. That said, it's a good idea to snap the really old stem stubs off the crown each year to allow new shoots room to form and grow.