Hi Bub,
Are they the ones that you can lower? Since you're not sure that purple martins were in there, I would clean them all out and watch for other birds, like house sparrows that might try to set up house in them, and then evict them.
I had some tree sparrows hanging around mine one year, and the bird people told me that they were a good thing to have there too. Unfortunately, nobody stayed.
You can get covers for those holes. And If I'm remembering right, you keep them covered until you start seeing Purple Martins, then uncover them. That will keep the house sparrows out. Good luck.