It depend's on what kind of clay you have. We have a red clay and a black clay within 20 miles of one another. The black shrink's and cracks when dry then swell's when it become's wet. The red stuff is stable from wet to dry. If you were on our red clay I'd say refill with what was removed. On our black I'd suggest importing enough red to fill the hole.