I would get them off there somehow, if nothing else hit them with a strong blast from the garden hose, since you said naturally if you have to. Perhaps someone else can identify them and that will give you a clue on how to deal with them.
Personally, I think that I would be inclined to spray them with some insecticide or sprinkle some sevin dust or something like that on them. That really looks like an infestation that could easily get out of control.

I do not see any damage to the leaves at this point, but with the number gathering on them, that could be a temporary thing. Another thing to keep in mind is that insects seem to know when a plant is sick or stressed and go for it first for some reason. You said that it was yellowing and the otheres were not. They seem to be some of the first in my garden to do that too, but you probably need to check and see if you can determine any reason for it.
Just my thoughts on it.
Bill P.

OMG, be careful with that sevins dust advice. It can come back and bite you on the A#@

That's interesting. They look like some kind of midget aphid. The aphids I got last year were much larger, black and brown colored, and they formed around the base of the vines. I'll be curious to see what those turn out to be!

Did you try using soapy water on them? A few drops of dish soap mixed with water in a little pump sprayer will work great.

Thanks for the replies.. I shot it this morning with soapy water. We'll see what happens. It's so funny out of 15 plants how just one could get attacked like that. I'll try to get a more detailed picture out later today. I grew this guy from seed so I'm kind of attached to it but I'm tempted to yank it out if these bugs don't go home.

Yeah, hit it with Sevin Dust...no, just kidding, sorry Lakeshore

I would hit it with Safer Insect Soap or similar insect soap. They look like aphids to me, and those are easily controlled with soap.

Thanks Landarc,

I'm on my way to the store right now. The plant still looks pretty healthy however, It is definitely infested. I'll take some pictures of some of my plants shortly