First, the OEM module output voltage is 1.5V, so you would want one alkaline battery, or a bank of them in parallel, not 3 in series achieving 4.5V as in the unit shown in your third image.

Second, LEDs need current limiting, and we don't know if that is happening in the OEM module, or at the individual bulbs, or maybe in the wiring. Some experimentation with a battery and some resistors might help determine whether a series resistor is needed. Start with a high resistance and work down to similar brightness as the OEM module produces. High in this case is probably tens of ohms. You may have to sacrifice a light string to home in on the ideal value. Keep in mind the power requirement for the resistor. P = I^2R.

Why do I get the spidey sense that with that transformer plug the UL label is fake?

Battery operated lights (from the factory) are pretty cheap, and there are lots of sales right now.

Kaseki is on the right track. IMO, the 4.5V power supply looks to be just that, strictly a power supply with no current limiting. That means the current limiter is likely already in the wiring after the power supply, which is typical in my LED string light experience. If so, you could use the 4.5V battery supply and add 2 silicon diodes in series with it. The silicon diodes add in a ~0.65 voltage drop each into the circuit. The final voltage would be 4.5 - (2 x 0.65) =3.2V nominal, which should be AOK. Any silicon diode will probably do, the 1N400x series diodes are common and available at Amazon..

I think I misread the label on the original supply to be 1.5V instead of 3.5V. In the 3.5V case the battery module with diodes should work until the batteries are drained, which might not take too long. The diodes may get quite warm if the actual load is represented by the power supply label indicating 0.45A. Have fun experimenting.

For longer effect, if one has two strings of lights, using a 6V UPS lead-acid rechargeable battery or a non-rechargeable 6V lantern battery may work.

Transforming your decorative LED Christmas lights into a battery-operated setup sounds like a fun project. By replacing the plug with a battery pack, you'll gain more flexibility in where you can use your lights, without being limited to outlets.