Those are aphids. They're hard to spray in those curled up leaves. I'd either spray or just ignore. Cutting them off makes the tree look better but that's about all it does.

You can use a systemic with imidacloprid labeled for fruit trees. It will take awhile to work its way through the system as you water it into the soil but once it's in, the aphids start to chew on the leaves and die. I've been very happy using it on my trees so far. It effects any sucking insects so it is great on citrus also.

Do you have ants all over the tree? Applying tanglefoot around the trunk can help solve the problem if that is the case.