Part of the struggle is that your contractor has two flat materials that he's trying to make curved. Should this opening be perfectly round (or at least not wonky to the eye)? Yes. Is it reasonable to say it's difficult? Also yes.

Your drywall guy should be working from a template of the window, drawing it onto a completely covered space, and cutting out the circle from there. To me, it looks like he eyeballed it and didn't get it right.

This is not a sanding issue. This is how the opening is shaped. They need to reshape the opening to make it correctly round.

If your contractor's drywall guy cannot do this, he should find someone who can and not charge you extra for doing the job right. Good luck!

If you have it redone, you may want to the porthole around the window larger so it slopes away from the window. This would let in more light and may be easier to hide the not perfectly round edges.

I honestly like that it's imperfect, especially with the light coming through. It looks organic, kinda like the moon. Not sure how your wall is curved, as the whole staircase isn't visible, but maybe if you live with it awhile, it could be fine there.

much easier said than done. You can cut a perfectly round hole in the face sheetrock, but now add the return and a corner bead, and it will never be perfect. If anything, maybe look into a trim kit of some sort pre-made to the finish of your liking.

As long as the possibility is there to get this right, I think it's not too much to ask that it gets done properly.