Wooly thyme develops a pretty dense root system, doesn't it? That would be the threat rather than the above ground growth, it'll push right through that. Peonies don't like root competition, though, so you might want to remove it unless it's very shallow and won't interfere with the peony root.

Thanks for the reply. The roots aren't really deep but as you said peonies don't like root competition. As the woolly thyme spreads the mat gets denser too. I will remove the growth and perhaps put a rock to slow it down, that way I can remove it before it reaches the peony again.