I'm wondering what your seed, or plant source was. Also do you mean the original(pink) Brandywine, or one of the many others with that name. The pink Brandywines I have grown have always had very large, malformed blossoms that don't pollinate well. It's possible you don't have the real BW, but one of the many imposters that are on the market. Sorry I can't help more, but since nobody else was responding I decided to.

Do small blossoms mean small tomatoes?

As a 'general" rule, no. For most part the size of the fruit is determined by its genetics and its growing conditions. The true Brandywine is often noted for very large blooms and with a different appearance from other varieties.

You don't indicate where you are located and you are using unusual growing conditions (containers filled with organic compost, banana peels, etc.) so it is difficult to ID exactly what the problem may be but the most likely culprit is your container size and the mix you are using in it.

Basically it will be the size of your container and the mix you are using will determine how the plant will do when it comes to blossom size, fruit size, plant health, production, etc.

Dave