Yes you can, and you should, if you can move them to a protected location. The longer they stay out, the more birds and bugs can get to them. In particular parasitic wasps. Move them to an unheated garden shed or garage location. Avoid locations that overheat in the summer sun.
The important thing is to move them slowly, gently, and without bumping them. If the larvae get jarred loose from their food supply, they are unable to reattach and will starve. So I've read anyway, so I play it safe and handle them with care. Once Fall rolls around, you can handle them with wild abandon. The are fully cocooned and dormant by then.
I try to have my bees into the garden shed by the first of June. That's good enough to avoid most of the parasitic wasps. In fact, the sooner the better.

Hee hee. I just went out side to move them (June 11th) and apparently one females was still at work. I gently pulled out a nest block and she came out to the edge of her tube and looked at me, probably wondering what the heck was going on.

Still, I noticed two wasps nearby, so I'm taking it inside. Better safe than sorry.