Lessons from the Live Team #6:

Documenting Live Updates

by Sandra Powers
2006-07-11

I've been thinking a lot about documentation lately. Now, documentation is not a popular topic for game teams in general, and in my experience it's even less popular with teams working on MMO games post-launch. In fact, documentation is one of the first things that a live team throws overboard when the big post-launch storm hits. But documentation can be very useful – and not, perhaps, for the obvious reasons. So it is definitely worth doing some thinking about how you can best use documentation in your particular situation.

The Usual Scenario

The kinds of documentation we write go through a number of changes during the development of a game. In my experience, you usually come out of pre-production with a mishmash of documents that includes very high-level blue-sky system stuff plus a lot of incredibly detailed back story. During production you may solidify your system specs into a reasonable organization, but the closer you get to launch the more last-minute system changes you make that never get documented anywhere. During production you probably also start documenting content – and unless you've put some thought into a set of templates, this probably means that your content docs are each unique to the designer and involve varying levels of detailed story and mechanical vagueness.

Of course, from beta through a few months after launch you probably don't have the resources to document anything at all – not fixes for the exploits you missed before launch, not the modifications to the combat or death systems because you found out that what you had was just no fun, and most certainly not the new content that you are slinging out as fast as you can to try to hold your audience long enough to fix some of the underlying issues. You just have to get through this mess and then maybe you can go back and straighten things out.

After you've been live for 4 to 6 months, things probably do start calming down a bit – at least as much as they are ever going to. At this point you may start thinking about documentation again. There’s no way you're going to start with the system or gameplay docs – the old docs are too outdated and you don't have time to rewrite them yet. Plus just to make things interesting much of your team's tribal lore (the unwritten knowledge of what's been done and why) is gone – many of your team members are taking their comp time or have rolled to another project or left the company (and possibly the industry) entirely. But you could start by adding docs for new content as you go, you suppose ...

But considering how much work this is and how well you've done in the past months with no real documentation, is it really worth it? Sure, the docs you wrote back during production served a purpose, but that purpose is past. A live game is very different from a game that hasn't shipped, and investing time in documentation at this late date is just pointless, right?

Well, not quite. Documentation actually serves some rather important purposes in the live development cycle; it's just that those purposes are a good bit different than the ones it served during pre-launch development. Perhaps the most important thing to understand is what post-launch documentation is not good for: it is useless – worse than useless, a waste of time and effort – to attempt to comprehensively describe every aspect of your vast and ever-changing game. You don't need to describe the game – the game exists; it describes itself. And you'll never be able to fully describe what the players do with the world and with each other anyway.

But documentation is perfect for a number of other purposes, including:

• Providing a basis for intelligent QA.
• Structuring the approval process.
• Recording intentions, motivations, and alternative solutions.

Let's look at these functions one at a time.

Intelligent QA

Just as your development team needs to be as efficient as possible to keep up with the pace of live development, so too does your QA team. Unlike many single-player games, you can't afford to merely toss a new copy of your game over the fence to QA every time you've made a change and expect them to regress every aspect of it in a vacuum – not if you want to ship anything on a reasonable schedule

So your QA team needs a basis from which to test new features and content. You could use some sort of automated report about what's been added or changed, but ideally you want them to test more than just simple functionality – they will also need to be on the lookout for potential exploits, for unintuitive behavior, for the unfortunate lack of fun. In other words, your QA team needs to be assuring the quality of the total gameplay experience. And in an MMO game, that's a pretty broad mandate.

In order to accomplish this with limited resources on a tight time limit, month after month, the QA team needs to go about their task as intelligently as possible. In addition to knowing how you think the new feature or content is supposed to work, they also need to have some basic background on your goals for this addition – who is it aimed at, what need does it fill. Furthermore, while they can't regress the entire game every time, they do need to be alert to unintended interactions with existing content. You can help by providing them with pointers towards other systems or features that might tie into or be affected by this change.

In many ways, QA is the primary immediate consumer of your documentation. It helps to have their needs as an audience directly in mind when you write your docs.

Approval Process

If QA is the primary immediate consumer of your documentation, then other members of the team are the secondary immediate customer. Good documentation makes an excellent basis for a formal approval process. A doc is a concrete item; it helps you generate useful deadlines – the doc must be written by this date; you have this long to give feedback; any issues or questions must be resolved by that date; etc.

If you have a team of one then you probably don't need an approval process – but any more than that and it's an absolute necessity. This isn't because you don’t trust the other team members – not at all! But an MMO is a monstrously large game with a devilish lot of systems and details to keep in mind, and any addition or change is probably going to benefit from the feedback and constructive criticism of other people.

So why don't I refer to this as a feedback process instead of an approval process? Because feedback sounds optional: if you have the time, please take a look at this at your leisure and let me know if you have any thoughts. And we all know that we won't have time for optional feedback – everybody is working as hard as they can every moment just to keep things running. But approval is not an option, it's a requirement: before we release this change to the world, it will be entirely thought-out, it will be reviewed by your peers, and we will discuss any potential issues. And we will get it done even though we don’t have the time for optional things like feedback because this is a set part of our process.

An approval process based on written documentation can help you catch a multitude of problems before they become disasters. Potential exploits, unbalancing items, useless quest rewards, lore inconsistencies, inappropriate Easter eggs ... The earlier you correct these issues, the less resources you waste on them and the fewer players you lose because of them. And that’s really all there is to it.

Motivations, Intentions, Alternative Solutions

And finally, there is another audience for your documents – a more distant audience that you nevertheless should not neglect: yourself (or other developers) in the future.

A live game is not a static entity; it grows and changes over time. A good MMO is expected to live for many years, and in fact it may well outlast many of the current team members. (No matter how much you love the game, you may want to try something else after five or six years.)

But many of the design decisions that you make are very much based in the now. These decisions grow out of what your current players want, the current market environment, the current pressures on your team and the resources available to you right now.

So given that the next time you consider this decision may be several years in the future, it is important to record this context. That way your future self won't have to spend time fumbling around trying to remember why you made this decision, or even worse trying to find someone – anyone – who may have been around at the time. It’s especially important to record any alternative solutions that you may have investigated and discarded so that you don’t waste time years down the road implementing a solution that won't work. In fact, you probably want to document any large investigation you undertake even if you end up not making any changes because of it. That information can save you a lot of time in the future.

Practical Considerations

Unfortunately, knowing what documentation can do for you in live development doesn't necessarily make it easy to manage. And the harder it is to manage, the less useful all those benefits start looking. So there are a number of questions you should consider when you are figuring out how best to use documentation in your process.

For example: how and where will you store the documents? The two most popular methods I have seen are MS Word docs in Perforce (or some other source control system, but Perforce is my favorite) or an internal Wiki. There are benefits to each method.

A Wiki is generally more flexible – it's arguably much easier to make quick and painless changes, and that can help you encourage your team to keep things up to date. Plus, you get the ability to cross-link docs in a very natural and useful way. On the other hand, I’ve not been impressed with the revision control or collaborative feedback features of a Wiki – and there's no reason I should be, since it's not build for that purpose. Using Word gives you built-in change tracking and commenting, and of course Perforce is built for revision control. I personally prefer this latter route, especially for any document that is going through an approval process. In my time on Asheron's Call 1, we actually used both methods: Word docs in Perforce for design specifications that needed approval and a Wiki for internal process and guideline docs that didn't.

You will also need to think seriously about how to organize your documents so that you can find things both now and in the future. On AC1 we organized docs primarily by month, which was quite convenient for QA ("Test everything in the July 2006 folder") but made finding older documents quite challenging ("What year did we do that thing with the scarecrows?"). Some good indexing can help with that, as can Google Desktop search, but there may be better options for your particular game.

For example, a heavily zone-based game would probably benefit from a zone-based organization: perhaps with major folder for zones and subfolders for areas within the zone. Or perhaps a level-based organization would suit you better. It really depends on what content and features you plan on adding. If you don’t organize things into monthly folders, however, you will probably want a monthly overview document to help you keep track of everything that is going live at a given time. In any case, you will probably end up with a clear division between features and content, and probably some divisions between types of content as well. And don't forget to consider how the documents for expansion packs might fit in!

Incidentally, be careful about using your bug/task tracking tool for documentation. It is often useful to give a good solid explanation of how you fixed a bug when you resolve it, but if that explanation contains important changes that other people on the team will need to know then you probably need to edit the appropriate doc and reference that doc from the bug. Likewise, we found it useful to reference the bug number in the doc itself any time we made a change based on a bug. This is a quick and simple way to give some context to changes without duplicating information or losing it in your bug database.

One last word on organization: be very wary of any scheme that duplicates data. Duplicating your index is fine so long as you don’t move docs very often – for example, you may want one index that tells you where to find docs based on the name of the quest and another that helps you find docs based on name of the big sword you get at the end of the quest. (Since players tend to identify quests by the reward, the second one will likely be very useful for Customer Service as well!) But the indices should point to the same place. Duplicating information is an invitation to serious confusion whenever you neglect to update one of the locations. And you will.

Anyway, there you have it – my thoughts on documentation. Like much of our business, it's usually boring, often difficult, and yet essential for keeping things running smoothly. Just keep your goals in mind as you plan your documentation process and organization – and keep in mind that those goals are quite a bit different than they would have been pre-launch – and you'll be fine.

[ <— #5: Efficient Additions | #7: Social Engineering —> ]