If you don't use the insert, then get rid of it. It'll make it easier to block off the flue.

We always plan on using it but probably only turn it on 5 times a year….

You need to uninstall it. The reason for the draftiness will probably be obvious. Take pictures and come back.

If you are rarely using it, remove it and put a Glass Door over the opening to stop the draft.

If you need ideas for fireplace doors, I can help you with that.

If you need it for emergency heat and want to leave it, you could figure out a way to put an insulating cover on it, like a thick quilt with insulation under it, or similar. If you know anyone who sews, they could tailor some kind of cover so it looks more intentional and custom. The cold air is coming down the chimney into all that metal and causing a draft, I suspect.

I cover is unconventional, but I don't think it would be any less attractive than how it looks now. Maybe something like this surrounding the stove, with a cover over it.

https://bangmarts.com/product/_1644199.html 

You could use your google skills to locate a ready made cover the correct size to go over the insulation. This is for an outdoor heat pump and looks to be about the right size. Probably comes in different colors. It would be great if you could find something the color of your brick.

https://www.coversandall.com/outdoor-air-conditioner-covers.html?gclid=Cj0KCQjwyt-ZBhCNARIsAKH1175o2AT-iC3OoWaHG48vWpEkow13YdbelfhHKihNiiNHoC5hTr6b1CoaAl2VEALw_wcB

For some reason the cover pic I posted above isn't showing. Trying here

The stove itself should be close to air tight so I doubt that wrapping it up in a quilt will help.

The mouth of the FP appears to have been left open which calls into question other aspects of the installation. How the fresh air intake for the pellet stove was handled, how the stove chimney was routed, how the FP damper was or wasn't blocked off, how the top of the FP chimney was or wasn't blocked, how the FP chimney was or wasn't insulated. Don't know how the stove flue was routed...horizontally out the side of the house or vertically up thru the FP chimney...what kind of chimney...clay, prefab metal...

We don't have XRay vision so the photo in the OP is next to useless.

Oh, you’re right cilantro. I thought the dark on the surround was a metal plate used to seal off the opening, but I can see now that it’s open. A covering on the stove wouldn’t do much if the back area is open. Better to stuff insulating material into that void!