It depends. Is the cabinet door full overlay or partial overlay?

2 inches should be fine. Does your cabinet company offer extended stiles? If so, get them instead of a separate filler. Extended stiles give you 3 inches. Then you can scribe them and cut them down as much as necessary.

If the adjacent door extends beyond the depth of the pantry cabinet, you could go as low as 3/4" and be fine. However, if it does not, the cabinet hardware will need to be taken into account. For instance, if your hardware is 1.5" deep, then you should fill at least 2.25" between the wall cabinet and the pantry. (3/4" + 1 1/5" = 2 1/4")