Hi and welcome!
Do you think it's a plant choice thing or do you think it's the soil? Afternoon sun is tough especially if it's a hot afternoon sun and then shady most of the day.

Are they in the ground or potted?

Yes, start with the soil. Mix in some handfuls of peat moss and/or potting soil to add water-retaining materials + nutrients. If high in chunky clay, remove some and replace with a mix of compost/top soil/and a little play sand.

Petunias should cooperate in the sun but can become stressed if there is too much dryness or heat.

Some personal recommendations if you haven't looked at these already:

1. Dianthus (the "annual" kind). These come in many colors, and with the winters PA we've been having, they will actually start growing back in February! I got all mine at Walmart 3 years ago and haven't lost a single one yet. They can grow to 1' tall and will bloom all spring/summer! For me, they actually do better than the smoky-foliaged "perennial" dianthus, which you see at stores growing up to 1' tall but mine never grow past a few inches each year.

2. New Guinea Impatiens. Highly sun tolerant, profuse flower producers and strong growers with very attractive foliage. You can find these at Kmarts, who always carry a ton that are healthy.

I have had good luck with annual salvia (usually blues though there is a red I like), Profusion zinnia, snapdragons (come in all different heights), angelonia, lantana. I love shade impatiens but the conditions have to be right. Do add some nice compost as you plant. I use Osmocote for once a season feeding.