I am not a bee keeper, but honey bees decided to start a very large colony under my castor been plant this summer. They were thriving there until winter hit.
I would be more concerned about human consumption of the honey IF castors are pollinated by bees.

Bees know not to pollinate toxic plants, not to worry about honey. Besides I think its the oils in the seed that are toxic, not the flower pollen.

(Yes, pesticides do show up in honey-but thats a different story)

I planted 12 castor bean plants this past summer and the blooms always had several kinds of bees on them, apparently without incident.
Kris

Actually, Bees will pollinate and gather nectar from poisonous plants like rhododendron. There are cases of people becoming very sick or even dying. Below is a link to one instance

Here is a link that might be useful: http://www.stuff.co.nz/4449422a11.html

see this study: http://www.academia.edu/6416344/Toxicity_of_castor_bean_Ricinus_communis_pollen_to_honeybees

Bees make Mad Honey from Rhododendron plants. Some places in the world collect and sell this Mad Honey to people who consume it to get high from it.