what is the wall color? might you paint the closet door the wall color?

The wall color is SW Collonade Gray. That might work. I know it won't look right if it is white. But I'm really concerned about the pantry door, even though it is in the kitchen area, it is still visible because of the open concept.

if the pantry door is closer to the white cabinets than it is to the wall colored closet door, i'd not worry.

Right now the closet door is white. Do you think I should paint the closet the color of the wall or should I paint it black as well? (leaving pantry door white).

no, i think a single black door is enough. whether you want to conceal/de-emphasize your closet door a bit by making it wall color is up to you. i'm just making suggestions without a photo.