The "stuff" you are seeing is lichen. Lichen is harmless to the tree, but, at least in this case, is a sign of other problems. The tree in the picture looks like it's in extremely bad health (at least the part in the picture).

There's really nothing to go on, in your post, to help determine what happened to the tree (it was NOT the lichen). Weeping willows and flowering peaches (especially flowering peaches) are not the most robust trees to start with. So, don't give up on trees in general, although you might want to remove these!