Are the binary formats of *.lyr and *.mxd files published?
They're not published, and you're probably not supposed to reverse engineer them and still expect free cookies at the ESRI UC, but they're basically OLE2 Compound Document format files, like MS Word & Excel documents. With a nice tool for cracking them open (e.g. http://www.gemboxsoftware.com/CompoundFile.htm), you can get to the hacking part pretty quickly. With a keen eye and a little knowledge of the ArcObjects object model, you'll be reading interesting data in no time. I've even heard rumors of people creating LYR files from scratch using non-ESRI code.
It is not and I guess by nature it cannot be, since an .mxd is essentialy composed by ArcObjects (toolbars, maps, layers, renderers, symbols etc.) serialized via standard IPersistStream mechanism.
With the open ArcObjects ecosystem, where a map can contain lots of custom components each with their own serialization, not to mention changes in serializated data between component versions, I cannot imagine how this would be possible, unfortunately.