Lavender needs really good drainage. I usually mix small gravel (chicken grit, to be precise) into the soil where it is going to be planted. I also rarely water my lavender, except during the first season after planting. No guarantees though..... it's just not the right climate here. Hope this helps!

There is a lavender farm in central Monmouth County, NJ where they have grown about 20 varieties of lavender. If it can be grown here in central NJ, I don't see why it would not also succeed in the Philly area. They did sustain damage in last summer's heat and then Sandy, so are renovating their fields, but you might want to consider a visit - very nice people who might be able to recommend some varieties more tolerant of our climate. And, yes, they do require excellent drainage.

Here is a link that might be useful: Pleasant Valley Lavender

As stated above, Lavender needs good drainage, and benefits from shearing during/after bloom. My best specimen grows out of a stone wall and the stem looks tortured.