My butterfly weed (asclepias) did pretty good in a compact clay spot I hadn't amended. So did my pincushion flower (scabiosa), though I've never seen that attract that many butterflies. Most everything else I've ever had does so much better with amendments.

If you mulch it every year, and turn the mulch under as deep as is easy with a shovel in the spots where you don't have plants in the fall, it will take 3 years to notice a difference but it will help your soil a lot without too much effort. (I say in fall because then you won't disturb your perennials because you'll still know where they are.) So if you plan to stay where you are that long it will be worth it eventually.