You don't need to remove any. Neither honeysuckle or butterfly bush are the slightest bit fussy about soil conditions and will grow just about anywhere. If you were growing an organic veggie garden, my response may be different but with these two plants, no worries about any contaminates.

I would make sure the soil is loosened properly as it is probably suffering from compaction and adding organic matter (like compost) will help to lighten and aerate the soil as well as refresh any missing nutrients.