What's wrong with just using the piece of plywood? Instead of duct taping it there, you could add a hinge or just a latch to hold it in place. Or magnets. Magnets are fun.

I would create a frame and fill it with a plywood backer and drywall. There are ready-made panels

I have the exact same configuration in my house, and the previous owners had framed the drywall area with door framing material, and then cut a piece of wood to fit into the cutout and up against the frame. The wood panel is tight enough that I can push it into place and it stays there.

Its not a great looking installation, but in my case it ends up behind a bureau where you can't really see it. I have considered adding Velcro strips to the back of the panel and to the studs behind it to make it hold very firmly in place.

Bruce