I believe the ants are probably after some leftover cocoons or freshly hatched worms that your eyes may not be able to see.The aphids are looking for fresh green plant growth and you usually see that with chemical fertilizer.The lush green growth attracts hungry insects.
When you remove your castings set them aside for about another month with a moist cover of newspaper in a cool dark spot and see if more worms hatch.Then, harvest the tiny worms and add back to your bin.
Then use the castings.