It is probably some sort of glazing compound that was used to "putty" in the windows.

Might be just some old mortar from when the window was set.

I would be looking at some that is a bit better thermal brake.

Are you retrofitting them double pane?

Thanks for your comment, windowsonwashington. We're keeping the windows original with the single pane wavy glass.
What would you recommend to use to fill in the space?

How were they affixed to the block? Are you trimming them out or are they just caulked?

Some backer rod or spray foam to seal the gap would work. Spray foam, given the narrow profile of the window, is probably a messier idea.

The windows were screwed into a wood frame from within where the window is in the open position. (This is 1925 building technology!)

Now that I think about your question, the putty probably helped the metal channel to be more weather proof, since there are 6 oblong screw holes to accept #10 brass wood screws in the channel.

+1

Probably some sort of sealant.

I high quality sealant will suffice now.

Do not try to fill the entire gap with sealant and some foam backer rod will probably be wise here to give yourself a solid and consistent caulking joint that can accommodate movement.