Well, I don't know anything about koi, but I would assume that they are similar to gold fish in response. My fish come when they see me, or at least something like me, and if I stand where I normally stand. If I stand somewhere else around the pond, they may hide. The first year, I believe it took a few weeks before the original three fish felt confident about me.

They certainly don't like odd shadows and we actually have a hawk that lives nearby and who has probably been the source of occasional loss.

Yes, I think you should be concerned. Are you feeding them pellets or flaked food? I use flake and even if not nearby, I could see fish eating. I suppose non floating food might be harder to view.

Try a sinking food and toss it down where they are hiding- they'll eventually come to associate it with your presence. Make sure its a fresh good quality food and you should pre soak it in pond water prior to feeding to soften it. Its good to get them taking food from you - otherwise how would you know if they're not eating.... which is one of the main indicators that something's wrong.

If they are coming up and eating when you're not around, I'd wonder if some predator is bothering them?

If there is a predator that comes and visits your pond regularly, they won't come up to the surface. I could always tell when a heron had come to visit because my fish would hide for days. I wouldn't even be able to see them.

Oops didn't see the above post, but yes, I agree it might be a predator. Do they have a place to hide? Every spring I used to have to put chicken wire over the pond.