I have a couple of ideas.

It might be Carolina buckthorn.

Carolina Buckthorn

It might be wild black cherry (Prunus serotina).

Wild Black Cherry

It might be something else entirely.

Both Carolina Buckthorn and Black Wild Cherry can have somewhat variable leaf shape. They have markedly more sparse foliage when grown in heavy to dappled shade, but fuller foliage in more sunlight. One way to tell the two of them apart is by their flowers. The flowers of Wild Black Cherry are in a longer raceme, and the flowers of Carolina Buckthorn are more of a cluster. Both produce berries that ripen from red to a purplish black. If you want to compare your tree to online images of the Wild Black Cherry, which is what we have growing as an understory plant in our woodland, keep in mind that the young trees have a very smooth bark and the older ones have a shaggier bark.

In the woodland, I just leave them alone and let them do their thing. They grow underneath much older and larger pecans, hickories, walnuts and oaks. The birds eat the berries, though sometimes not until winter. I understand the berries are very astringent, though I've never tried one myself to see for sure. I don't know anything about the Carolina Buckthorn fruit, but in photos I've seen, it appears larger than the fruit of the wild black cherries (which are very small). That size difference, though, could just be because the fruit in the photos I saw grew in areas with a lot more moisture.

Only you can decide whether to leave it or not. I leave all the native plants I can because wildlife needs them for food. If yours is under a solitary tree and ruins the look of the landscape, I think no one could fault you for removing it. Based on how little our black cherries have spread over the years, they do not seem invasive. It is my impression that Carolina Buckthorn can be more invasive in conditions it likes, but I have no first-hand experience with it.