What setup do you have? Is there a filtration system and how often do you test the water chemistry?

It sounds like a fungal infection but most likely it's ammonia burns from wastes that have built up in the water. Change the water and add any needed additives....especially if you have chlorinated water.
I'd take him in to a fish store and ask for advice and any meds they think may help.

Black on a goldfish is normal only if it's a young black fish who is beginning to change to orange or some other color. Young fish are brown or black so they won't get eaten and their color usually develops later.

If a fish is turning black, it usually indicates some sort of bacterial infection. The inability to swim correctly could be that he's dying from the infection or can be caused by a condition called swim bladder disease. Healthy goldfish are pretty active and easily come to the surface to eat.

I would raise the temperature of the tank with an aquarium heater and add some kosher or aquarium salt (don't use table salt because it will kill him) to the tank and see if that helps, or you can treat him with one of the medications for bacterial infection you can get in the pet store. I don't think there's much you can do for swim bladder disease, but I suspect your fish's problem is bacterial. Goldfish don't usually need heated tanks, but higher temps help them recover from infection.

You should also test the quality of the tank water with a kit you can get in the pet store. If ammonia is high, it will make your fish sick, so you'll need to do partial water changes to lower the ammonia.

Hopefully, your fish is in a tank with aeration and a filter. Goldfish produce a lot of waste and don't live long in bowls or without filters. Fish diseases are very opportunistic and tend to show up whenever a fish is stressed, by a move, by dirty water, or from season changes for pond fish, so you didn't necessarily do anything wrong in caring for him.

You could try the aquarium forum for further help. There aren't many fish keepers here and this forum is mostly owners of cats and dogs.