Here's an article that will tell you how to calculate the fill space requirements for your box. Basically, 2" per 14 ga conductor, 2-1/4" per 12 ga, with some subtleties about how you count. The new outlet will list how many cubic inches it requires. The box should have its volume stamped on it somewhere.

However, the manufacturer's installation instructions take precedence over the code requirement. I have to admit I've never seen an instruction based on the number of wires in a box, there are certainly GFCI receptacles that can be installed in gang boxes with many wires if yours is not one such..

Just at a guess I'd say you needed a bigger box.

It seems the "no more than 4 wires (not incl grounds)" limit in the Leviton sheet is more about how-to install simplification for non-electricians (and of course liability) than conductor/space limitation in the box.

I appreciate you taking the time to write up your reply. But I need to clarify my question:

I put the incoming black+white coming from the switch on the Line terminals. Where do I connect the remaining two whites (whose corresponding blacks are tied elsewhere)?

I assume Line, right? (Since I am not actually feeding another box with a hot that can be protected.)

You need to get an electrician involved:

1. the box is too small to meet code with that many wires, and

2. though I am not trying to be insulting, your level of electrical knowledge does not sound sufficient to safely accomplish what you want to do (totally apart from the fact that this outlet needs to be completely replaced).

This installation could have been done by a previous owner without proper knowledge/skills, and there may be other connections we can't see that also compromise things. I know you want to be given advice on how to fix this, but there are enough warning signs here that I don't want to try to do that.

Agreed. Once you've traced all the wires and know exactly what they are, where they go, and what they are doing how to wire them should be obvious. If you can't do that an electrician is called for. We can't tell what the wires are doing from a picture.

Ditto rwiegland & kudzu.

This is certainly not a homeowner modified electrical box. The cabling is all original 1967 NMC cloth covered like the rest of the house (hard to see from the pics). Also, the box looked professional: the grounds were twisted and pigtailed soo tightly and the wire nuts were done so perfectly and bent and packed in the back of the box it all looked like an educational diagram. It was difficult for me to pull it all out.

As for the box fill, after much hassle I was able to measure the box. Again, hard to see from the pic, it is huge at approx. 5"x2" metallic with a 1" extension single gang cover. The closest on the table are 4 11/16"x2 1/4" which allows 18 #12 conductors.

I think you were all being mislead by the pictures. It is original work by an electrician home builder.

Through trial and error I let the Leviton GFCI answer my question. There is only one way the device even operated: put the other two neutrals on the Line terminal side.

If I wasnt going to protect by running the dishwasher hot through the Load terminal, then the two neutrals from the direct feed panel->dishwasher need to be on Line not Load. That was what I was asking confirmation for.

It is all working now. If you all still have concerns, I am listening. My only practical choice is to put back the outlet (edit: returning it has been for the past 15 years I have owned this home) as it was and call an electrician.

For those finding this topic in the future: (Not sure if it's against forum rules to post links to other forums) but I did find a very healthy discussion on Installing a GFCI with more than 4 wires not including ground. Be sure you read the fine print comments, also. Very good notes in there.