You need to get rid of it or it will sap the strength from your Kumquat. Tease away the soil gently until the whole attachment point is bared; then either cut it off or pull it gently to take it away with a heel. Keep the wound dry until it heals over. Maybe even use a big of fungicide to help protect it.

Find out about your Kumquat. What was it grafted to? It's just possisble the rootstock is worth keeping, either for its own sake or to use for future grafts.

But it sure does have ugly spines! Could make a great barrier to keep unwanted visitors out of the yard!

Rain2Fall

It's the trifoliate orange(Poncirus trifoliata) rootstock(notice the three leaflet configuration). Cut it off, and keep any new shoots that might arise pinched/rubbed off.

You probably could root the trifoliate cuttings - it is a good dwarfing, cold-hardy rootstock for most citrus, but the fruit is just this side of inedible - very seedy, very sour, and there's a really nasty gummy oil in the rind. But, a dense hedge of P.trifoliata would certainly be a barrier!