It might be against your jurisdiction's building codes. If you are outside city limits, I would still be extremely careful with this plan. A single spark can set dry building materials aflame, not to mention the air pollution you get from smoke that close to your home on a consistent basis. You'd need to keep that chimney regularly cleaned out to ensure that debris inside doesn't light and then float up and onto the roof. Pizza ovens are really hot. I've usually only seen them built in locations somewhat distant from the main house. I would think it might be hard to get a really good pizza oven effect making it 1/3 of an existing oven. Most of these are dedicated units. There are kits if you want to build your own and put it out on the grill patio if possible based on your building codes.