Shaking it is something that a raccoon might try. They are quite smart. You can never stop every predator. Even a snake can slither into the hole to the nest.

But at least from my point of view, I usually support the house in such a way that it could not be shaken. Not because I thought something will shake it, but just that I've usually seen bird houses on something stable, such as the pole you mentioned.

I would find a way to put a fence backboard next to the house, and the fasten the bird house to that in some way. If there are no babies yet, you can also try to move the bird house next to one of the larger main branches and fasten it there. At least it won't be able to be shaken.

I can't see the hole but also do not have any branches near the opening, otherwise other birds and predators may attack the parent sitting in the nest or the babies.

Poor Raccoon. You're taking away his food. Raccoons need to eat, too! They're probably more endangered than birds. Think of that fuzzy, fluffy, little teddy bear of a Raccoon crawling off somewhere to die of hunger, cold and alone. Just because you moved a bird's nest somewhere he couldn't reach.