You want to wait until the leaves all brown and dry up and fall off and the stalk turns brown as well. As with all bulbs, the lilies need that remaining foliage to remain in place and photosynthesize to recharge the bulb for proper flowering next year. You can take 1/3 off at most before this happens - otherwise, leave the rest in place to mature and ripen naturally.

Once the stalks turn brown and cut back, I leave about a 3" length in place so I know where the bulbs are located to avoid damaging them when cleaning up the bed or adding other plants.