Take both portions of the wall out. Live with it for a bit. If you find you want to define the spaces, put a section of tile down by the door.

Since the door opens right into the dining room, you could close it up with dry wall, shutters or beaded glass (). You could put shallow shelves. A version of one of these would be interesting: and

Not sure what your style is, but I think you need to obscure the view from the front door. I am all for an open floor plan, but think the entry should have some definition. If you have a contemporary style there are some beautiful architectural resin panels available that could add a buffer between the door and the dining room. Lots of styles available, but I have attached the seaweed and maze examples. There are also more subdued options. Would need to frame the panel, or create multiple frames and insert in the opening. It just looks unfinished at the moment, either do a wall, add a panel but not sure about removing the barrier.

Knock it down

Since it's the entrance I vote for closing off (some nice ideas above) as opposed to removing

Thanks, everyone! The windows may work really well, especially as we'll be taking out some of the old single-panes in the house anyway to replace... Thinking of taking the closet door off altogether and putting in a small bench with storage underneath for shoes, shelves above, etc. - do you think that will look silly/too cluttered since it will still be recessed as a closet?

What about dressing it out with a column or pillar?

Not sure of your dimensions, but Acurio makes a vinyl panel that comes in various styles and are less than $50. Acurio Latticeworks 1/4 in. x 32 in. x 4 ft. Black Ginger Dove Vinyl Decor Panel

How about a freestanding, free form curvilinear wood screen wall, about 5-6 feet high??? See attached image.

Whatever you choose, think about dusting!

Great ideas here. I think the closet is too small for a seating area. You could replace it with recessed shelving, but that might be a bit contemporary with the built in and wall of old sashes (which by the way will look awesome!)

Has the wall been there from the beginning? If your remove it, you could find that the flooring has a gap under the sill of that wall.

You could also continue the knee wall up so it's like a bar counter height and case it out and add columns to make it seem symmetrical so it's like a little window so it still seems open. I know this picture doesn't resemble your layout or anything but something like these columns would look good with that style door

I"d go for the decorative screen ... make it as open or solid as you like, but fill the open space between ceiling and knee wall. You could do it with rotatable photos -- travel on one side (to entry), personal on other side (so strangers don't get too personal a look at your family..use these as conceptual visions -- then have what you need custom made: http://www.ebay.com/bhp/frame-room-divider; or twrap a 1x3 board with some fabric you love (in any degree of sheerness) screw it into the ceiling, then another at the knee wall to stretch the fabric taught -- instant screen.