Why not? Sounds lovely! I was just this morning leafing through Justina Blakeney's New Bohemians handbook and spotted a wall of hats as decoration.

ETA: One drawback would be dust.

You sure can. I like the look in the right setting. Always liked that photo Saypoint just shared above.

What's more, if you have a few you know you aren't going to wear again, you could hang them from the ceiling -- I can see three to five hats in a hanging group looking really good.

Of course you can! I don't have an examples, but you could always organize hooks on the wall and hang the hats on them. Or layer them on a shelf (or shelves). All I have hat-wise in my house are baseball caps my husband leaves all over. Lucky you!

It would look nice.

I hang a sun hat on my front door every spring for years. The trick is to stuff it so it stays in shape. I use a smaller kids kick ball for this - works great.

Here's the hat collection picture from The New Bohemians Workbook.