There is no compatibility issues with the products.
Usually , on a transformer there is a common terminal, with the other terminals being the different voltages available.check foe the common terminal

The 300 watt transformer should carry your 50 watt lamp load with ease. Check for voltage match. A 12 volt lamp should be connected to a 12 to 14 volt output from the transformer. If you've connected 24 V lamps to a 12 V output, those are going to be very dim or have no light.

If you have 5 of 12 V lamps, these should be parallel connected to a 12 V source. If the lamps are wired in series, it would require 5 x 12v or 60 V to drive the lamps and this is greater than "low voltage".

Did you measure the output voltage with the lamps connected? What was it? Maybe the transformer is no good. When measuring the output voltage of a transformer, it is advisable for it to have some load. The open circuit voltage of a faulty transformer may appear to be ok, only to fall to almost nothing when a load is applied.