Have you tried just hosing down the plants with water? Water dislodges and smashes the aphids and is typically suggested as the first go-to method for aphid control. Next on the list would be insecticidal soap (the real stuff....not the homemade substitute). My last option would be the neem oil.

When/if you do spray with something other than water (and that's all I ever use), do so very early in the day. Once the product dries, the chance of phytotoxicity decreases substantially.

Yes I have used water to wash them off but they come right back - aphid populations here are apparently out of control! - which is why I began using the neem spray, which seems to at least help for awhile to deter them. When I spray the neem solution I do it in the evening so the plants are not in the sun but the leaves still get damaged after being exposed to sun the next day. My assumption is the neem residue by itself is enough to lead to phytotoxicity. I've also tried a spray with just water and hydrogen peroxide, but once again the aphids come right back. Would love to find an effective aphid deterrent that also did not damage my plants!

Do you grow your plants in containers? You are talking about the "need to leave the plant in the shade for a few days, before putting it back in the sun"?

Yes the majority of my plants right now are in containers.

I would try a lower concentration. The labels of neem oil usually have instructions on the ratio of oil to water. If the concentration is too high, it will burn the leaves. I have also heard of people trying to use neem oil as a systemic with a soil drench. Not sure if that would work, but a systemic could work on aphids in theory. I'm not sure there's any definitive research on whether neem oil is effective as a systemic. I'd start with lowering the concentration. Not the greatest answer but that's the way it goes. Have you tried importing some lady bugs? You may not be able to completely eradicate the aphids, but maybe get the numbers down to more easily controlled amounts.

Your plants are stressed. Stressed plants easily succumb to insect damage and diseases. One of the reasons for stress could be too warm (hot?) soil in the container due to the strong sun operation. My advice is: shade permanently the containers. This may not help immediately but for sure will make the plants more happy.