If the leaves still have spider mites on them, bag them and remove them or they'll keep spreading. If you have sprayed something to remove the mites, it's not necessary. For fungus, remove the leaves because there will be spores on them.

I have started spraying with neem oil. First application last Thursday and again Sunday morning. Does the spider mite damaged leaves harm the plant growth vs pruning them this late in the season to grow new ones?

The brown and yellow are fungus infected leaves?