Where are you from, what zone, state, etc... would be very helpful. I am from Minnesota and we have not seen any butterfly or catepillars yet, a little too soon.

Oh silly me, I meant to include that. I am in central Texas.

Are you planting enough in the way of host plants? I can almost promise a bunch of pretty caterpillars if you plant some dill, parsley, or Queen Anne's Lace, for example.

Caterpillars show up shortly after you begin seeing butterfly and moth activity, usually in the spring. Is this school in session in spring through fall? That's prime caterpillar time.

Identify your particular host plants, as you've only mentioned milkweed, the host of Monarchs. In your location, caterpillars would be munching away right now, if they are present at all.

Flowering plants (annuals and perennials) attract these insects into the garden and reward them with nectar. But you must have a variety of host plants for specific species to lay eggs.