There is an entire (and very active forum) dedicated to the discussion of soil issues, so this may not be something easily addressed in a simple exchange :-)

Building a good soil base is not something that is accomplished overnight. It tends to be a longterm, ongoing process unless one is constructing a new garden from the ground up (pun intended!) and importing a large quantity of quality soil mix. Typically the recommendations to lighten a heavy soil and improve water penetration and drainage is to add quantities of organic matter - compost, aged manure, leaves, grass clippings, etc. As all of this continues to decompose and breakdown once in contact with the soil, what seems like a huge amount of the stuff initially really doesn't make much of an impact unless repeated regularly, season after season. Eventually you will achieve a very friable, loose, organically rich and biologically active soil that encourages deep water penetration and excellent drainage.

Drainage issues from sloped conditions are a bit different but certainly can be complicated by heavy soils. Without being able to see your specific situation firsthand, it's difficult to suggest how to best address it. Terracing or stepping down the slope may help but one needs also to be conscious of how you are affecting natural drainage and the impact it may have on surrounding properties. Altering drainage so that it has an adverse impact on neighboring properties is illegal in most areas. If drainage is a serious issue, consulting with a soils engineer may be appropriate and the most efficient route.

In general, additives like pumice, perlite, vermiculite, etc. are not recommended for landscape (inground) use. They add nothing to the soil and are needed in such large quantities to affect drainage that they are impractical and expensive. These types of soil amendments/conditioners are most often recommended for use in container or sometimes, raised bed plantings. And again, container plant culture is vastly different from ingound planting with its own very specific requirements. There is an entire forum dedicated to this complex topic as well.

You might find it helpful to visit both of these forums and read through a few of the postings or access the search feature to look for past threads that have addressed similar concerns.