You have a negative pressure issue in your home. When an exhaust fan is turned on air leaves the home. This makes the pressure inside the home lower than outside the home. Replacement air will leak in through any opening in the house to equalize the pressure.

If your house is sealed well the ”make up” air may be pulled in through the dryer vent when it is not running. Since this will be direct outside air any odors outside will be pulled inside.

Turning on additinal venting will only make the problem worse. New build homes routinly include an energy recovedy ventilator that is designed to help worh this problem. As long as proper filters are installed that can capture odors this problem ahould not be present.

One option would be to seal the dyer vent and install a ventless heat pump dryer, like the Miele T1.

The problem will NOT go away however. Outside air will leak in somewhere when an exhaust vent is running. Around doors, windows, through unsealed areas in the walls/attic, through a chimney (if you have one), the vent hood above the stove/range, etc.

Low hanging fruit check all your exterior exhaust covers. Most builders use the cheapest ones available which don’t seal from the get go and after a year the fins fall off and they are wide open. I opt for the stainless steel type with magnets or the canister type with the whiffle ball inside.