From my little knowledge about bananas yes they will affect the growth of the mother plant in some way, and also the fruit if its to have any. There is a video on youtube by a guy who grows and sells banana plants that talks just about this. He removes all but one sucker from mother plant each year and sells the rest. You can also replant them elsewhere and not let them go to waste.