I cut down my milkweeds last week because they were smothered in aphids. Once they are larger (mine were 4ft tall) they are able to handle some damage from aphids. I haven't found anything that will keep them away. I've seen occasional ladybugs but they aren't enough to manage the population.

Melle, do you get aphids anywhere else?

We have a few spots around the yard with aphids, but they are either green or black and there are other predatory insects intetspersed (ladybugs, syrphid fly larvae and lacewing larvae). The oleander aphids build up into much denser, larger clusters. i think i saw a syrphid pupae on the milkweed so maybe they just need to get found by the predators.

I'm afraid to grow Milkweed, it almost seems like they bring aphids to the garden.

I have never seen the yellow oleander aphids that are commonly found on milkweed plants on any of my other garden plants, but it depends upon what you grow. They much prefer host plants from only a few families: Asclepiadaceae and Apocynaceae. They are primarily seen on milkweeds, vinca and oleander.