I am gonna take a stab at it and say Olive tree?

What did it smell like? Could be Osmanthus Fragrans, but I would look at a general picture of the plant itself as well as a properly focused picture of the flowers to tell...

Hard to make out the flowers in the pics but I'm guessing it's an elaeagnus.

https://floridata.com/Plants/Elaeagnaceae/Elaeagnus%20angustifolia/1153

Thanks very much for all the help. It does look an Osmanthus but we are a bit far north for fragrans and I took a look at the leaf with a magnifying glass at it is likely Elaeagnus. I appreciate the Osmanthus suggestion because I need to smell one of those things sometime. It could have been angustiflolia but my flowers are almost perfect white and it's just my pic that makes it look a bit yellowish. This I think is an Elaeagnus umbellata, although I am not totally sure. It smells super sweet and it's very perfuming a large area in front of a building. Really a good fragrance for a weedy looking tree in a weedy area. Thanks very much for everyone trying to help and getting me on the right track because I had no idea at the beginning.

Cool. I just found this site and the tree was blooming in Delaware in May in 2003.

http://delawarewildflowers.org/plant.php?id=0739

Perfect, White Clay Creek park is very close to this location, only a few miles. Yes I also saw those brown spots on the leave with my magnifying glass. Thank you for the link, great close up pics.