My first thought was about your weather. Spring for me is supposed to be rainy, but we got lots more rain than normal with many more wet days. All of that wet caused fungal infections that damaged plants, flowers and fruits. Anthracnose is a fungus that attacks a lot of plants. On blackberry it isn't noticeable until the fruit begins to ripen. Then the fruit wilts or turns brown. Plants need to be treated with fungicide, especially after extended rainy periods. There are several viruses that are much worse because they have no cure. Here is a LINK that talks about them.

Should I spray with Neam Oil now or is that not safe with the fruit already set on?

Once sprayed with anything but water (or rain), blackberries are very difficult to clean because of all the little crevices, and because of the softness of the fruit when ripe.

The first topic on the webpage below covers your partialy-formed blackberries, and it confirms what others have typed above.

https://www.uaex.edu/farm-ranch/crops-commercial-horticulture/horticulture/ar-fruit-veg-nut-update-blog/posts/fruitdisorders.aspx

I researched this some more, and it appears my blackberries have dry cell syndrome, which is caused by various fungus, including Anthracnose, which is what others already said. Will this be a problem every year or is it something that is mostly determined by weather patterns from year to year? Does anyone know? If my blackberries are going to look like this every year, then I might as well replace them with something that is more tolerant to Anthracnose and other fungus'.

Once a plant has anthracnose it will return. The disease winters over in the canes and spreads via spores the following spring. Infection is not hopeless. Fungal and bacterial diseases can be treated. Here is an ARTICLE with information on treating anthracnose on blackberries. Most if not all state universities will have similar articles on this.