5% of the height. 120'' x 20%=6''.

In my opinion there is no formula. Consideration should be given to the molding treatment in the entire space... amount and size of ceiling molding, width and height of door and window casing, clearances, if there is paneling on the walls, interior transoms, use of plinth blocks, etc. etc. so that the baseboard assembly is proportional in size and bulk.

To begin with refer to the Windsor One molding website. They have several molding series...Colonial, Craftsmen, Greek Revival. You can see how the sizes run which should help you in configuring your entire space and not just the baseboard.