bird feeder question

i feed finches � year round � four finch feeders in the garden area � do the hulls from the nyjer seeds do harm to the garden soil

i have heard that sunflower seeds hull harm the soil � but not sure that is true

any advice would be appreciated

thanks, mr toad