Might be a hickory (or some relative) instead.
Paw Paw leaves are supposed to be toothless I believe.

http://www.chesapeakebay.net/bfg_pawpaw.aspx?menuitem=45494

http://www.wegrowit.com/adams_images/AsiminaTriloba.html

also seems to indicate the leaves are alternate for paw paw.

You may want to take a look at some of the basic algorithms for tree identification that would be available on the internet. One of the first points is to distinguish compound leaves (like that of your hickory) composed of a number of leaflets (seven in your photo) from simple leaves. The pawpaw has simple leaves of a characteristic shape that are simple to identify - I can't think of another tree in the forest that you're likely to confuse it with. But they don't appear randomly in the forest like an oak or a maple might. Instead they appear in groves, often near rivers or creeks. They are out there but it may be some time before you stumble upon them.

You were exactly right guys. That was a hickory tree. I did find some pawpaws though! They aren't as close as I would like but I found a grove :)

Thanks!