By now, you know that something fishy is going on. Lets start with the obvious. There is 120 VAC supply and a 12 V filament bulb. For this to work, there has to be other circuit elements. Applying 120 VAC to a 12 volt bulb would vaporize the filament in a blinding flash. That does not happen, so something else is going on.

I suspect there is a 120vac to 12 vac transformer in the base of the fixture, and if not in the fixture, it is elsewhere. That would be the simple setup.

There are other possible arrangements, but would be more costly and likely not used. One arrangement could be a transformer followed by a full bridge rectifier to give full wave rectified DC. This DC would be very rough without smoothing elements, rising and falling 120 times per second, but a filament light bulb would work on this. Many cheap volt meters may not give correct value readings. An ordinary voltmeter will show a reading on either its DC or AC setting when full wave rectified voltage is applied.

The puzzle is that you report a zero reading on either the AC or DC scale with the light fixture disconnected. I assume that you did turn the breaker off in the distribution panel before you messed with the fixture wires. Did you turn that breaker back on before you made the meter measurement?

Do you have the complete wiring diagram for this lighting setup? You might have what is called low voltage lighting. The feed to the fixture is low voltage, not 120vac. This system has a voltage reducing transformer and relays. The wall light switches do not send power directly to the light fixtures, Instead, the wall switch operates a relay and the relay sends low voltage to the lamp.

I think you are right. My cheap multimeter can't keep up. I found a similar problem posted on another site, with a slightly more detailed solution posted... the fixture is probably fed 12 V by a high frequency transformer that is adding enough ripple that the meter is confused.

It all depends on the type of meter. If its a digital, it could respond in a variety of ways depending on its circuit design.

The needle movement in an "old fashioned" multimeter has a moving coil in a permanent magnet field. This is inherently a DC meter with auxiliary circuitry to read AC. The AC input is rectified and applied to the meter movement. The AC scales are adjusted to provide proper readings. An internal cell supplies voltage for resistance measurement.

If you are curious, look at the supply with an oscilloscope to see what is there. Be aware that many oscilloscopes have fairly high input impedance and can display a voltage on a wire that is not connected to anything provided that wire is in an alternating field.

My meter wasn't an old fashioned needle type, but it was a $3 Ebay special. For fun, I'm going to put a Fluke multimeter on it, to see if it's also confused. And perhaps a scope. But I'd have to haul the scope from work.