If you are in Manhattan or Brooklyn, try Lower East Side Ecology Center. They have a stall in Union Square Green Market (where they collect food waste for composting) and also a location in Gowanus. They do deliveries within Manhattan.

I use them for potting soil and compost.

https://www.lesecologycenter.org/products/

If you need mulch, Green Wood Cemetery maintains a free mulch pile all year round. You just need to bring your own bag (and shovel or spade if needed) during the hours the cemetery is open. Ask the guard at the front for the location.

Town of Islip in Suffolk has good stuff for sale, cheap.