If I recall, leaf spot is a fungal disease. By removing the infected leaves, you are limiting its spread; if you leave them on, the fungus can spread more easily to further leaves. Also, I believe it can infect other related plants in the area, so check them, as well, as they may be helping it continue to spread. Removing the infected leaves doesn't necessarily completely stop the disease progression, especially since spores that have already hit healthy leaves will still infect those leaves over the next few days, but leaving infected leaves will certainly let the fungus spread more if conditions are correct (which it sounds like they are).

Also, I'd recommend planting your tomatoes in a different location next year if you can, or sterilize the container if you're growing in one.

They need to be removed no matter where they are located on the plant. They serve no purpose for the plant except to stress it and help spread the disease.

Thanks!