One suggestion is to resaw a piece of the trim to make it thinner and just add the trim(same as the left side) on the right wall, cutting the base, and using caulk to fill the top. The right door would actually close just as it goes now, but be behind the trim piece. That would allow clearance for the right door handle.

you mean, so I would attach it to the right wall.. "sideways" ?

Nothing you can do with what you have.

Back to the store and buy a swinging door or bifold doors. (I promise that you will detest those by-pass doors so having to replace them with something else is a blessing in disguise.).

The right hand side has to be framed out sufficiently to accept trim. That is the only way it is ever going to look right. (Swinging door common widths are 30", 32", 36"; more size latitude with bi-folds. Do some research on what size will best work with the available space; get the best fit and then frame the opening to fit the door.)

One other thing. The door casing (trim) goes down to meet the floor. Not supposed to stop and sit atop the baseboard. It looks like the baseboard is thicker material than the door casing that you have. You'll need either thicker casing material or build out the trim (flat stock ripped to the width of the trim & nailed onto the wall and then apply trim to the build out.). That way, the baseboard will die nicely into the casing.

Yes, attach it sideways.

Upon further consideration, it might be easier to simply remove the trim and make the opening plain with sheet rock, like an inset window is done.

I just realized that our old house has two closets like the one pictured in the OP and there is no trim around the doors.

Me too. Five of those closets here and no trim molding on any. In fact, inlaws had a new home built recently and no trim on theirs either.

Agree, lose the trim.

Or you could just leave it!

"talley_sue_nyc

Or you could just leave it!"

No.!