Vegetables won't take up asbestos, so that isn't an issue. You yourself will take up vastly more asbestos just living in, and breathing in such house with asbestos components. Certainly lead should be avoided. Asphalt is pretty inert when cold. I don't think it can dissolve in water, which is how it could be absorbed by plants. If it could dissolve in water, asphalt roads would wash away pretty quickly.

Agree with Dan that the only things you might have to worry about are things that will dissolve in water. That's the only way into fruiting plants - leafy greens are another matter. And rain water just falling from the sky, especially if you live in a city or suburb, isn't nearly as clean as we'd like to think it is anyway yet we consume the food it falls on. Plus mulch and soil are pretty good filters.

I do get lots of sand off my asphalt shingles in my barrels but it settles out. Same with the lime. My biggest issue with my rain barrels is mosquitoes breeding in them so I keep some Dunks floating in them all the time.

Dave