If you got red berries, you got some female plants there.

Nope!! Berries would indicate there is a girl or two involved in your asparagus patch.

The Jersey varieties are often advertised as 'All male' but often there are 10-20% female plants. You can pick off the berries or let them fall and then pluck the new then wispy seedlings next spring and transplant them.

So, is it only the male plants that will produce enough or big enough spears to harvest?

IMO, nope- not just the boys. Girls are harvestable too. Most wild patches that folks harvest from have plenty of females and males. So you don't to worry about having to remove females because of their performance. Like LoneJack said, the girls are probably a fairly low percentage of your plants.

But to my understanding, all boys can/do produce more/bigger more consistently than girls do. Mark your girls this year, and if you find they are underperforming compared to your males next spring, then cull them out and replace. If you don't notice enough difference to make it worth a replacement, it's all good :)