I think pics are tough to tell.

The fence may certainly be a problem but so is the silt fence. You really have to get in there to see what the issue is.

If the county is that involved, can you bring them over to ask?

In my last town and current city, we would be able to call someone over to look at the issue and they would enforce whatever neighbor action (if any) was responsible. It is generally illegal to stop the flow of water (in my area) that impacted a neighbor. And that would count a fence or grading.

we can't tell what we're looking at

New construction projects generally require a site development plan which includes a grading strategy to deal with storm water. The plan is typically reviewed and approved as part of the building permit application process. If you have an approved site development plan, that will show what you--or your builder--need to do.