Imagine it bigger. No. Bigger than that. At least 20m high, and it has width, too. How much yard is it taking up? Where are the leaves falling? What is the acidity doing to the concrete? How many little oaks are popping up in seriously inconvenient places? How much shade is your neighbour enduring?

Some people plant parkland trees in the darnedest places, unfortunately. And, once they reach a certain size at least one of several things happens: the local authority decides it is a 'heritage tree' and you can't do anything to it without approval; it costs you zillions to get the arborist in to remove it; branches fall on your house or worse; the roots heave up your foundations.

If you fancy having trees in the area then perhaps an espalier fence with cordon apples or pears on dwarfing rootstock so the roots don't forage in your richly composted veggie garden. (They can reach a long way. A friend has a glasshouse more than twenty feet uphill of a pine tree - and the pine roots snuck in over one growing season to take snacks with the tomatoes.)

The sooner out, the better. And, if you know any wood turners, they'd probably be glad to take the timber if you don't want to use it yourself for woodcraft or firing.

The veggies you will grow will balance the karma from the loss of the tree. As said above, an oak tree of any type is huge. If it is bothering you now, that will only increase. I also think the "lost" veggies you would have to buy at the store would be much more environmentally impactful than removing a baby tree from where it should not be. However, I don't know how it relates to your house, in terms of offering shade so your a/c doesn't run as much as it would without the tree.

If it would make you feel better to go plant a tree somewhere else, where a tree should be, please do so. Also, a more suitable replacement for your own yard could be good, like a cute little japanese maple, or hydrangea shrub... or more veggies!