I used epoxy coated gravel, and I think it's actually safer. You won't get any unknown minerals breaking down and leaching into the water from the gravel. You should still rinse it before adding it to the tank, but 'raw' gravel would be MUCH dirtier.

Yes, the experience that I've had with raw gravel is that it seems to never get clean no matter how much I rinse it. I wasn't sure if the epoxy coating would be somehow detrimental to the plants if their roots were to come into contact with the gravel. I guess if that's what they sell for fish aquariums, it probably doesn't harm the fish so it should be fine for plants. I guess I should use deductive reasoning more often. :)