Let it dry, then you can often just brush it off with a soft brush. Pressing scotch tape onto the pollen will also often lift it off (again let it dry first). If those attempts don't get it all out, try laying it in the sun for a few hours.

This time of year, I've almost always got a shirt or shorts lying out in the sun to get rid of lily pollen. Takes a while, but it works.

We should start a fashion for wearing orange shirts in the garden.