But we'll get to that. First we need to temper expectations.
If you are interested in being a DM because you are hoping to be the next Tolkien or something, man you are barking up the wrong tree. If you want to direct film (or plays) or write for television...go do that. If you want to write a gigantic fantasy novel series like Sam Donaldson or George Martin or (God help you) Robert Jordan...then go do that. Write your books. Write your screenplays. Film them...whatever it takes to scratch your artistic itch...DO IT. Draw your magnum opus comic book with Frazetta-like characters or Aspirin-like situations or WHATEVER. Go do that! Hopefully, you'll make some money on it, but REGARDLESS, you will find satisfaction pursuing your creative endeavor of choice.
D&D doesn't look like that. Oh, I know...there are folks on the internet that play D&D that seem to be creating some sort of entertaining "story." That's fine...that's another form of scripted entertainment, created to entertain the subscribers. That is NOT what D&D play is, nor what it looks like in practice, nor (I daresay) what makes it an incredibly enjoyable pastime.
D&D, in its best form, is a fantasy adventure game, allowing players to experience fantasy adventure. But what does that really mean?
It means that it is a game. A game that creates high emotion in its players...fear, joy, excitement, exhilaration, despair, anger, sadness, triumph, etc...emotions that are EXPERIENCED in play, much like a competitive sport or game of high stakes poker. That's what it's like to PLAY the game...that is the "fun" of the game...for the players.
For the Dungeon Master, the satisfaction comes from creating a world and running a game that creates these emotions in players.
But if you expect players to "perform" in the manner of a trained actor or "behave" in a manner that "befits the story;" no, in most cases that's NOT what you're going to get at the table, even with trained actors for participants. That's just not what the game is built to do...not in ANY edition of the game. You might have a group of very creative individuals sitting at the game table with you...screenwriters and novelists with a modicum (or more) of acting talent, and your game will still fall far, far short of the satisfaction of writing your own novel (in which you can control the content and dialogue) or directing your own show (same).
Trying to administrate these things is missing the point of play. And, over time, you will end up very, VERY disappointed if this is your expectation of play.
In running the game of D&D, the DM presents a world, with various things for players to do. And by "do" I mean Choices that players can make about Actions their characters take in hopes of having an Impact on the imaginary (DM created) World. THAT, my dear friends is the game of Dungeons & Dragons. That is the essence of what the Dungeon Master does...at least, as far as the players are concerned. What happens behind the scenes (and mostly away from the table) is the world building done by the DM to create the dynamic campaign with which the players interact.
Done right, this creates a campaign the players want to return to and engage with on a regular basis...it is the well-run campaign that creates the drive in players to play. Which, in turn, provides feedback to the DM that is used for additional dynamic world construction, giving the DM (as Bath wrote) "full reign to his creative genius."
It is this interplay between players and DM, facilitated and moderated by the rules, that govern and express the play of the Dungeons & Dragons game.
So what rule set to use?
Despite all the internet "ink" spilled on the subject, the edition/version of D&D used is of secondary importance (at best) to the play of the game as just described. Every DM will, with time, make adjustments to the ruleset to better suit their needs in play, and EVERY edition requires additional work by the DM to ensure that long-term, satisfying campaign play is achieved.
The various Basic editions (Moldvay, Mentzer, Holmes) provide an adequate introduction to the rules, and are the most accessible way to learn the game system. Coupled with addition of an "Expert" set (Cook/Marsh, Mentzer) they provide instructions for the basic procedures of the D&D game: character creation, combat, advancement, exploration, etc. They are all well written for their purpose and excellent in obtaining their objective: even players as young as nine and ten can grasp the basic systems and processes of the game after a reading of (one of) these Basic books.
The Advanced game (Gygax) in its first iteration provides a more robust system for play that anticipates and addresses many of the issues that might hamper long-term campaign play. This includes a more robust selection of player choices (no less than 35 combinations of race and class, more spells and equipment, and some inherent/integrated world building considerations), expanded rosters of monsters and treasure, and a variety of options and decent advice sections for the Dungeon Master. Gygax's AD&D (sometimes called 1E or "first edition") codifies many of the "best practices" developed over years of actual play and experimentation. The core volumes of play include the Players Handbook, Dungeon Masters Guide, and various Monster Manuals. Later volumes in the 1E series provide additional rules and ideas, but most are fairly superfluous to running the game.
The original D&D game (Gygax & Arneson) provides the basis for both the 1E and Basic editions of the game and for the historic-minded or unapologetic game tinkers, it provides the foundational bones on which the game was built. "OD&D" (as it is sometimes called) along with its supplementary volumes can provide a better understanding of the game's evolution and rules development, but to make the game function requires many choices and additions from the DM/referee. While this holds a great appeal for DMs who enjoy performing their own customization, it is a distraction from the actual creative imperative of the DM (world building and campaign creation). Also, many DMs find that the additions and modification they end up using are already anticipated by the AD&D (1E) version of the game.
The 2nd Edition (Cook) version of AD&D is similar to 1E in most ways, simply streamlining and restructuring many of the disjointed and scattered rules of the system. However, 2E's restructuring of the advancement system (the method in which players earn "experience points" or x.p.) is problematic in that it A) removes an easily understood, objective measure of success, and B) removes an incentive for cooperative and creative play. This has consequences in both the short-term AND long-term campaign play, but can be rectified by reinstating the "gold for x.p." standard (and several published adventure modules for 2E strongly advocate taking this specific action).
Both the 3rd Edition of D&D (sometimes called D20 and currently published as the Pathfinder game system) and the 4th Edition of D&D have multiple issues that make them rather unsuitable for long-term campaign play of the type I aim to describe in this blueprint; they represent their own, very different...and divergent...types of game play. They fall outside the purview of this series.
The 5th edition is the current and most widely played version of D&D at this moment in time. Its publishing longevity surpasses any edition except 1E and 2E. It has had no less than three introductory box sets published for it, helping to make the rule system accessible for the new player, and is well-supported on-line by both the publishing company (WotC/Hasbro) and a large community. It is largely streamlined and simplified compared to prior "Advanced" editions (1E through 4E) yet enjoys robust player options far surpassing "Basic" editions of the game. And because of its variant rulebooks options, 5E provides a wide-spectrum of ways for the game to be played: it is customizable (like OD&D) and provides the DM with vast means of rewarding players (via fiat or "story awards"). It models a very different game play experience from earlier editions of the game...an experience based on expectations set by standards of video game design...which may have a broader appeal to a younger demographic of participant. Unfortunately, as such it also falls outside the purview of this series; while it's certainly possible the 5E mechanics could be adjusted to make it conducive to satisfying, long-term campaign play, the effort and analysis in doing so is more than I care to perform, given that there are already alternative systems available that provide a more-than-adequate head start on the task at hand.
Aficionados of Dungeons & Dragons generally have a favorite edition, based on the point at which they entered the hobby and/or the point at which they "mastered" game play. Preferences of style and design vary from player to player and (as stated above) almost all Dungeon Masters, with time, will modify the rules somewhat to meet their own preferences of play. As such, I think it's important not to become too distracted from the subject at hand by quibbling over which edition of the game is "better." Each edition can be made to work, and even amongst the older (pre-2000) versions, each will give a slightly different experience or "flavor" of play.
In the end, we are unconcerned about flavor. We are concerned about long-term play. Choose a rule system and stick with it. As this blueprint continues, adjustments to system will be inevitable.