I would use the largest door that fits the wall. You don't have the width for U-shaped shelving, so i would just keep it simple with deep shelving.

Are you planning on having shelves along the 3' wall? If so, that cuts into the door size you can fit on that shorter wall (i.e. if you're having 12" shelves along the long wall, that only leaves you with 18"-door jam for a door).

It might flow better to have the door on the short side, but it could ultimately be a better storage solution if you leave the door on the long side. Can you post the layout?

Also, measuring your various items to get a sense of the min/max depth of shelves you'd need will also help with designing the space.

3’ is way too deep for a pantry. It needs to be no deeper than 18” if you want to find things. Even then, that’s still going to be jumbled.