Are you sealing it from the outside (I don't know if that is or isn't advisable because of the need to vent moisture) or sealing it from the interior space above?

Sealing from the basement

The planes between floors require fireblocking. So the small gaps around pipes, ducts, cables, etc are called annular spaces, and need filled. These can be filled with any product that does NOT meet ASTM E136 (a combustibility test), so spray foam, painters caulk, joint compound, probably some gum, etc. are acceptable. You can choose a product that is easy to install and is compatible with the adjacent wood, like canned spray foam. Air sealing is a side benefit of fireblocking. The return duct spaces can use the same product you choose for the floor planes.

Many residential inspectors (and DIYers unknowingly) think it must be "red-orange" foam or caulk, which are fire rated products. They are beyond code but certainly a good choice too.

Open chases between floors, however, require other solid materials like 2x wood or 1/2" drywall (and their annular gaps can then use non-ASTM E136 products).

Source: IRC R302.11

" Sealing from the basement "

Sorry, I thought we were referring to the basement. Is your concern to seal the basement from the interior area above, or from the outside?

@3onthetree @Seabornman So all those penetrations in the subfloor need to be filled correct? I assumed the penetrations in the return ducts needed to be filled because its just common sense (you want ductwork air tight so no basement nastiness getting into supply air). But I was unsure of all the other holes in the ceiling/subfloor. What are pros/cons of sealing those holes?

When we built an addition, the codes inspector required all floor penetrations to be sealed. I used the fire rated foam. I personally would rather not have plumbing lines sealed, as if there's a leak, maybe I'd see it in the basement instead of rotting the subfloor out. I think its a good idea to plug the holes in the panned ducts, although they'll still be somewhat leaky due to their construction.

Yes all floor plane holes that allow pass through of smoke/fire/air. If there is a plumbing pipe or something that needs accessed, foam can just be pulled off or cut with a knife.

Yes on ducts, too small to affect performance of your return, but doesn't hurt to fill so do it. If any of the penetrations are through the sheetmetal, tape would be better than foam though.

You're using the term "air sealing," which is usually used in reference for thermal comfort, thus why Elmer is asking about exterior walls. That is typically where you'd focus air sealing at the rim joist area for comfort upstairs, then insulating that area sufficiently. Air sealing for the general basement ceiling is not considered to help thermal comfort, as your basement looks modern (>1950s) and in the conditioned envelope. If you have other issues with stack effect pulling smells or something upstairs, fix the root of those problems.

For wires and smaller penetrations, you can use fire-rated foam or fire-rated caulk, as @Seabornman suggested. For larger penetrations, such as ducts, etc., you can use mineral wool insulation, which is also fire-rated and code-compliant. Be sure to wear gloves and eye protection.