Do not use neem oil in the soil , use neem seed meal , just sprinkle around the plant and work in the first few inches , or use Mosquito dunks broken up to a powder the same way. Diatomatious earth spread on top helps too ....

Most important is not to keep your soil so wet, actually letting the top few inches dry out will help

Ok, thanks. But would they be fungus gnats as the soil is very dry.

Probably not , maybe fruit flies ..... Are you growing in a pot ? it can be dry on top and wet on the bottom Fungus gnat looks a little like a tiny wasp , if you have fruit more likely fruit flies heres a link to a picture fruit fly vs fungus gnat