Just a couple of points of clarification. Agapanthus are neither lilies nor bulbs - they are much more like daylilies in that they are a fleshy rooted herbaceous perennial (actually in the Amarylis family, Amarylidaceae). And they prefer full sun, tending not bloom much or at all in very shady locations.

Your best two options for removal are the ones you have considered - digging out or killing off with a herbicide. Since you are not inclined towards the herbicide (and I totally understand your reluctance to use these types of products), then that leaves digging up the plants. If this is more labor that you are able to provide, consider hiring someone to come in and do it for you. Even cutting back or killing off the top growth will leave a dense mass of fibrous roots that will be a lot of work to get rid of.

Do you know of something I could use on them that would kill them but still leave the ground healthy enough to replant in it afterwords? Sorry, I wasn't sure where to ask thi question. Thanks Gardengal45

Most herbicides - like RoundUp (glyphosate) - will break down in the soil very rapidly and leave it perfectly safe to plant in. Or you could try something a little less disruptive, like vinegar. But plants need to be in active growth for these sorts of products to be properly effective, so winter time in North America is not the best season to address this :-)

And I would also recommend applying whatever herbicide you choose to cut stems and foliage. This tends to speed up the translocation of the active ingredient to the root system, where it will then kill the plant. You may need to make more than one application to get complete control and household/table vinegar (5-7% concentration) also requires sunlight to be most efficient. If you can find horticultural vinegar (20% concentration), it will work much better.

And you will still need to dig up the roots before you can replant or allow then to properly decompose, which could take a number of months - probably at least one full growing season, maybe more.