Hello W, you certainly have a long to-do list with this new house. Those "caps" look to be homemade with a 2 1/4" door casing trim (similar to homedepot.com/Woodgrain-Millwork but profiles are slightly different depending on supplier, where, and when purchased). If you have a bunch rotting, and are DIYing this, I would think it might be best to just remove them all, scrape the paint, and touchup the paint.

Whoever built the fence made them. Copy them if you like the look, remove them, scrape and paint if you want a low labor approach, or saw them off flat and use the pre-made post tops.

Thanks @3onthetree and @rwiegand for your help!

Just curious, is there any PVC fence post cap that would fit with such pyramid shape of the existing fence post (without saw them off flat)? Thanks again!

Possibly but an oscillating tool will make short work of sawing the tops off flat. Will take less time than buying an assortment of caps with the long shot chance of one fitting & then having to return the ones that don't work.

Be sure to measure the post, too, to make sure that the post is the same dimension as standard modern 4x4's.

Not an expert on PVC caps, but I assume most simple cheap ones are "hollow." So then it might sit over some of a pyramid-shaped post, but the angle of pyramid on the PVC will probably not match the wood cut, so you have to either fit one to see how far down it will extend or cut off a small portion of the wood.