Yes, any growth that results from your graft will ALWAYS be kumquat, which is simply using the root system of the other plant. You'll need to be conscientious about removing any sprouts that may arise from that root system, though. Those WILL be bitter orange and they will be very happy to take over your grafted plant.

It will be important for you to promptly remove any sprouts that emerge from below your graft.

Thanks rhizo, I was hoping to hear that!

There used to be a nursery in Harlingen, TX that sold kumquats on sour orange. The grafts would fail after a few years. Use Cleopatra mandarin, or trifoliate or one of the trifoliate hybrids for rootstock. I think Thomasville citrangequat should give a good cold hardy tree. Some have used Swingle citrumelo to get huge trees fast and lots of very sweet fruit. Swingle is less cold hardy than Thomasville but a lot more so than Citrange.

I have a friend who grafts several dozen trees a year on feral trifoliates. He has given up on grafting kumquats as being too difficult. He only does a wedge-cleft graft.
Commercial grafters use T-budding on trifoliate.

I'm glad Tantanman responded. I was going to give you some vague warning as well (not based on experience, just on anecdotal info I've read) about the rootstock combination with kumquats.

Chip budding works as well as cleft grafting on meiwa or nagami kumquat. IMHO, easier also. Changshou kumquat however, probably a hybrid of kumquat and mandarin, t-buds easily.

Here is a link that might be useful: mrtexas