These are likely soil dwelling worms. They come up to eat and then return to the soil and poop in it leaving organic material in the earth. I'd just use the soil, any worms that tag along would either crawl away or help disburse organic material to your garden. Some others will likely make their way there anyway.

You could try luring some away but since they are there on their own already I doubt you'd get them all. In a worm bin there will be worms squirming around in finished compost that is predominantly worm poop so not adding new material to the compost pile may not do what you are expecting. If they like it there then there's little you can do to convince them otherwise, other than by physically removing them.