What is above the windows...sub-floor? How does this window open, Is this a tilt-down or tilt-up?

In any case, just have to find the best way to frame everything out so you can apply drywall, etc... Every case is different and there is always a way, in your case you can do something like this:

As the I-Beam goes and the pipes, the concept is the same, just box everything in so it can be finished:

yes subfloor is above the windows. The windows are glass block windows and a couple of them have the window vent tilting up.

If the windows are stationary that is even better, you can put blocking right up to the windows, use glue and small screws so they don't penetrate the sub-floor and you can wrap everything with sheetrock and have a nice finish.

With all due respect, those windows are tiny and depressing. If you're going to go to the trouble of finishing the room, why not do some checking into how much it would cost to enlarge the windows and get new windows put in? That would make a huge difference in how it feels to be in the room.

That would be a heck of a job cutting through that concrete wall with rebar in it trying to make a bigger opening.