What are your eventual plans for the area? If you are going to try to get the rubble removed, you don't want to plant anything that will have to also be removed first. Or will you plan to add more dirt to bury the rubble? Or leave it as is.

If you don't plan to change it much, you could get dense trees and other vegetation started (they will grow surprisingly well if they can just get a little dirt to start in). This will shade the ground and out-compete the weeds over time and mean that you won't be able to mow or weed eat. Spraying with an herbicide also just maintains the barren nature and provides fertile ground for weeds to invade anytime you stop spraying since they will have no competition.

Mainly, doing nothing to establish the growth you want will guarantee that you will continue to have problems with weeds. Just what to plant to meet your goals depends on where you are and your climate. Check out your state ag school web sites for ideas on vegetation adapted for your area.