Looks like it might be anthracnose or bacterial black spot. It can get bad under wet conditions at lower temperatures. It will damage flowers and any fruit that manages to develop. Copper based fungicides can be effective against it but spores are everywhere in nature so you can't get rid of it completely. In commercial operations mangos are sprayed regularly from onset of flowering right up to harvest to keep it under control. The wetter the climate during flowering/fruiting the more of a problem it becomes.

How often to spray them from onset of flowering to fruit?

Every two weeks. The spores are everywhere so reinfection can occur when the fungicide is washed away or dissipated over time. You can get 'latent damage' in developing fruit which isn't visible until the fruit gets close to maturing.