Use a soft brush for your vacuum. It will loosen surface dirt so you could brush it away. I use a regular one - but you need to keep in mind that there are different kinds of velvet, and what works for me might damage your sofa. So test your brush on a small spot first. If everything is fine - then move forward.

In addition to what Heather said, I would like to suggest using a hand vacuum. They have somewhat lesser power and are made for such things only. They will not damage your sofa fabric.