Can you take a photo of the plant growing in place? Where on the leaves are the thorns located? Any flowers you have seen?

Thorns on the leaves .... Above or underside? To me it looks like either a Blackberry or Raspberry.

Maybe an Aralia.

No flowers yet, the thorns are both above and under (technically they are prickles, non). I took another photo in case this reveals more

"thorns are both above and under"

That pretty much rules out Black/Raspberries..... Your new picture also shows the leaves aren't right for those berries either. Hopefully someone will come along and positively ID 'em for you =)

Might it be stinging nettle?

No, it is definitely not stinging nettle.

There seem to be a lot of new seedlings (you can still see their seed leaves). Is there a more mature specimen nearby that we could see a picture of?

These are the largest specimens so far, but I'll keep an eye on them and take a photo if flowers emerge. I am still not into pulling them out, in case they turn out to be a valuable addition, such as a native perennial that can revisit me every year. I hope I won't regret this decision :)