The answer depends on the type of worms you have in your barrel. E. foetida tend to stay in the upper 6"-12" or so, E. hortensis can be found in high numbers throughout my 30" deep flow-through bin, and E. terrestris make deep more or less vertical burrows (and don't like to be disturbed). Since you have a barrel sitting on the ground and it sounds like you think they crawled in themselves, I could see any of these or others living in your compost. So take a look at what you have. The three species I mentioned are pretty easy to distinguish using on-line resources.

Personally I like to use some compost to cover freshly added food waste. I find this helps keep any odors and flys down (okay, I do through in small amounts of dairy with everything else). You could add your extra compost a bit at a time to your barrel this way.