If you want it to look natural why where you thinking of building next to the fence? It seems the area slopes away from the fence so incorporate your berm and boulders into it, kind of like a gaul on a tree or a cancerous growth. Then it won't look out of place. Don't need loam over the top. Just use compost or manure when you plant whatever you are going to plant. Spend that money on mulch instead.

Here is a link that might be useful: Propagating Perennials

Plant trees and shrubs behind and it doesn't matter if a mound comes out of nowhere, if an incongruous background can't be seen it is not startling. If you think you need different soil for the plants you want to grow (maybe you really don't) then do add a layer of that at the last - I wouldn't have different, cheap dirt in there like the middle of a sandwich unless you really need that much more height - instead of compost as this will decompose and return you to the original condition. And you don't want to plant in small hole or pockets of amended backfill for multiple reasons. You do want to mulch after planting, perhaps in this case with some kind of locally derived (and therefore convincing) rock product.

I think it will be okay looking by the fence area because that is a natural slope. I am more concerned about it leaning up against the fence. I still want to add more soil because I do want it higher and I have lots of dirt I have to do something with (although its very rocky so its not good for much else).

I want to add a layer of dirt on top of it to smooth it out. The soil I have is mostly rock and I am concerned about my kids playing on it with that many rocks exposed. Most of them are small so I think they will be easily covered by dirt.

Thanks for the advice!

Yes: you don't want to rot the fence.

I'm not sure I understand what is intended for the kids' play surface at the top--loamy dirt? a planted garden? Will it be more like for exploring than intensive use?

I am currently in the process of making a couple of raised beds in our yard by building up low hills of excavated sand, surrounding them with rock, and then stacking soil on top. The first bed is going to have a base of sand about a foot deep, and will hopefully have another foot of good soil on top of that. I have no idea how it is going to turn out, though. :)

Just don't push your little hill directly up against the fence, and the fence will be fine.