I really like this plant but I noticed it's really starting to take over a few of my gardens. I've started pulling them out where they're growing close to or on top of other plants but I'm wondering if I need to do that.

I know they're extremely invasive but will they harm other perennials?