Same place you do not without going with bigger anchors.

Clean up all the loose plaster, and raised spots. Fill in the holes with spackle. Use drywall tape on it all, and drywall mud, and make sure it is sanded smooth. Since you have textured walls, you can use the end of a dry paint brush, to see, if you can match in the texture. Then make new holes for your curtain rods. Primer that drywall patch well, when you finally have it just how you want, and then repaint. I see a couple marks further on down the wall, that you may as well patch too, while you are at it.

I patched with wood patch and spackled, then tried to put toggle in and what you see is what happened

Wood patch isnt what you use on a plaster, or sheetrock wall. You are also better to drill new holes, than try to reuse those. I can see, that someone has patched at one time. A proper patching job, should never been seen, it should blend in and look like the rest of the wall. Might be time to call a drywall person to come and patch it.

I would treat all of the raw edges of the drywall with Gardz, let dry and then use hot mud (Dura Bond...) to fill the holes.

I am experimenting with using non-fusible fabric interfacing....glue on with Super 77...and then top coating with drywall mud. A top coat of Dura Bond would be best. Sand and then top coat with premixed drywall mud.

I would use toggle bolts and not those plastic drywall anchors.

Better way to repair using a backer board:

https://www.youtube.com/watch?v=ZPn8dvQcTTM

After repair is completed, attach the rod bracket with screws long enough to get a solid purchase into the backer board.