It's the browning and dying back stems and leaves that are feeding the bulbs for next season's blooms. You can pare them down a bit now by cutting off the "candelabra" where the blooms were attached - that stops seed production which saps some of the energy from the bulb.

As opposed to ripping them out, plant something in front to hide them when they start going through their less than attractive stage. I use nepetas mainly, but there are any number of plants.

Plant something all around the lilies. New bloomers will draw attention away from the faded lilies. You can plant thousands more since you have so much empty space.

I read an article by an English gardening author who mentioned how unsightly many American gardens are because they are filled with mulch instead of plants. Why fill your garden with mulch when you can plant something more interesting?

I plant thickly and closely in order to reduce evaporation, weeds, and irrigation. Plus, I am greedy. I want every inch in my garden filled with something more beautiful to look at than mulch.

I aim for a succession of blooms and interest year round. That way newcomers will draw attention away from faded plants/flowers.

I agree with pitimpinai. Plant something that will bloom after the lilies. Or move them in the fall to an area that won't be so noticeable for when they finish blooming.