Formal is kind of a vague term. And paperback is kind of a vague term. Some paperbacks are the size of the average book and would blend very well if the spine isn't all crunched up. If you are speaking of the original size paperback, I would not add them to a library in a formal room.

Books are books. I consider them to be 'precious' friends. I like to display my books with regard to size and colour. If I have a cover (paper back or other wise) that could pass as art, I face the book to the room = colourful and provocative. Display your books in the same way you would precious nick-nacks and brick-a-brack. With a little skill and a good eye for symmetry and colour, you should be able to use the hard covers and soft covers effectively. Throw in some nick-nacks as "book ends" and you will end up with a beautiful display that invites friends to "shop" your library.

A true library should have hardcover and paperback books.