Actually, a rectangular duct of the same sectional area as a round duct will have higher friction (pressure loss) due to a higher surface to area ratio. However, in most cases, the dominant pressure losses in kitchen ventilation are from the hood baffles and make-up air restrictions (such as a filter and a heater), so a little more duct restriction will likely not have a significant effect on the CFM-vs-pressure operating point on the fan curve.. Aim for a rectangular duct that is nearly as large as the joist construction allows.

• Broan 4-1/2 in. x 18-1/2 in. to 10 in. Round Duct Transition (at Home Depot)

This is done all the time. Run it up and a 90 across the top of the cabinets and conceal it with crown molding. Or run it up and through a joist bay and out. The disadvantage of the joist bay is drywall work and cutting a top plate. Neither is a big deal if done correctly.

To @kaseki's point, rectangular generally needs about 20% greater face area. How the transition is handled is important as well and differs depending on which side of the blower it's on (e.g., are being pulled vs air being pushed).

As @John Liu mentioned, a good HVAC shop should be able to make the right thing for you.

It's also important to not go too large. You need to keep a minimum airflow or else you'll get too much buildup (grease, dust, etc.) on the duct walls.

Can you not take the venting through the roof ?