My down comforter is in a duvet cover (pillowcase). I remove it regularly to wash it. I have the comforter drycleaned twice a year.

Unfortunately sounds like you need to restructure the comforter to minimize feather migration and loss and still keep the fleece on the outside.

There's a fabric called ticking that has a very tight weave. Use the ticking to encase the down/feathers. Use the microfiber and fleece to make a removable outside cover.

I did just this many years ago using directions from Threads magazine. My comforter is still in excellent shape with no signs of wear or down/feather shifting inside.

Warning--this is not a project for the faint of heart. Transfer the down outside on a windless day or you'll be vacuuming it up forever inside your house.

Good luck, it's a lot of work but well worth it in the end.