Filling in Hobbit Gaps

As we work our way through The Hobbit book - a really distinctive work, charming and less ponderous than the epic to follow - the narrow perspective of the fairy tale structure can be maddening for the adult reader.  Many plot points are left open, most of which relate to Gandalf:

• The dwarves want treasure, but why is Gandalf involved?
• Why does he think it so important to have a hobbit on the team?
• If you're Thorin running a dwarf commando team, why are you agreeing to have a strange hobbit involved?
• If you're going to pick a hobbit, why a 1st-level rookie like Bilbo for such an important mission? 
• Where does Gandalf go as the party crosses Myrkwood?
• Why does Gandalf, despite experience and good local knowledge, steer the party first into a mob of goblins in the Misty Mountains, and then (in absentia) onto an incredibly dangerous forest path, when there are apparently alternative ways (e.g., boats) to get to Long Lake. 
• Why the hell can't he brief the dwarves or Bilbo on the potential dangers, rather than just yell at them to "stay on the path!"  It just seems "stay on the path or giant spiders will eat you" would have better prepared the party for their journey.
• It's apparent at the end that this has all happened just in time to break up a major offensive by the baddies.  So, um, Gandalf, what are you not telling us?

In the mid 1950s Tolkien addressed these and many other questions in a retcon piece entitled The Quest of Erebor.  Formerly available only from unscrupulous sweaty fat men in the back rooms of college book stores, it finally appeared in Unfinished Tales in 1980, and is also available in full in The Annotated Hobbit, which I obviously have to now go out and buy.

But why do I get the feeling that it won't answer all of my questions?

Oh, my younger son now refers to Beorn as "Chekhov's Beorn".  And so he is.