You can remove old leaves with holes, fungal spots. I would venture to say it needs a lot more light. It will be okay in a lower light area - it will eke by through the dreary winter months but if you could provide an overhead light 12 hours a day, it will make a world of difference. A lot of my canes have to sit away from the lights and they limp along but the ones under lights often put out new growth and flower.

The first 3 photos are of canes under lights (cheap shop lights)

This is a mixed pan (one of those lasagna pans that are cheap and very effective) of canes that is close to lights but not under them.

Thank you, your plants are beautiful! I had it in a room full of windows that sees the sun rise (on the right) and set (on the left) not too much direct sunlight, just in the afternoon, this is the spot...

How long are your days? And how many sunny days compared to overcast? Is this a southern exposure?

I have a few in the kitchen on a baker's rack and this is on the southern side of the house so on sunny days it may be too intense at times (at least when I move them there to begin with). The bulk of mine are in the basement under lights but I do have a few close to a window in the basement.

The days right now are about 9 hours and more overcast lately but the plant has been there all summer and fall. I guess I was just worried because it was so vibrant and perky when I first got it.

If it was wilting or drying up, then I'd say you might have a problem which is usually a problem with over or under watering. Old leaves need to be removed when they become unsightly to you and/or if they have a fungal problem (more than likely it would be mildew). A few small holes or even tears in the leaves is nothing to be alarmed about. Older leaves will typically start yellowing and fall off by themselves but you can remove them at any time.

You can read some culture tips on Brad's Begonia World.

http://www.bradsbegoniaworld.com/