Worms or caterpillars? I doubt worms could climb on a house can they? I do know that when the soil gets saturated worms have to come out or they will drown, but I didn't knwo they can climb. After the rain stops my daughter and I like to go around and rescue the ones stranded on the sidewalks and driveways before they bake in the sun.

Here is a link that might be useful: caterpillar id

Worms and millipedes do that during periods of heavy rain fall. If you've got flowered beds around the house that you keep heavliy mulched you'll see a lot of this.

Worms are less of a problem than millipededes. Worms will leave when it drys up. Like a bad relative, Millipedes may very well decide they like it on or in your house and stay.

When I had a basement, I had a bad millipede problem there. I'm starting to see a lot of them on the porches here.