I don't think it will be a problem. The summer sun will help increase flower bud production. It will also help new growth to mature or "harden" as they say. This lessons the odds of winter damage. Winter sun hitting camellia plants when temps are below freezing can do a lot of damage. So plants that receive early morning sun during winter are much more likely to have this problem. Remember to water during mild periods. Winter winds sometimes damage plants with northern exposures. This is especially a problem with plants that need water. Don't fertilize until spring. I have six hardy camellias and I am very happy with them. Good luck.

I grow several Japonicas on the North side of my house. They are in full hot summer sun for 6 months, and in full shade from my house for 6 months. They are extremely healthy and bloom lots. I just make sure they get some water from drip irrigation every day in the summer.