Well, you'll have your four 18-2 switch legs, plus the load side leads of your transformer wire nutted together. That being said, wire nut packaging will (*should) show you the wire range and number of wires they are rated for. Off the top of my head, I'd probably use a red Buchanan Bcap (maybe a yellow) which are readily available at the big box stores.

The nut that fits and holds them tight...? If it doesn't tighten down far enough and cover the exposed ends then it is too small. If it doesn't seem to grab well then it is too big. I am of the pretwist school of thought, you pretwist it tight the same direction as tightening a wirenut and they shouldnt come apart by themselves before the nut is applied. Then tighten on the nut which should keep them from ever coming apart. I have had to redo plenty of loose junctions which were not pretwisted. Have never had to redo my own pretwisted junctions.