Lessons from the Live Team #9:

The Bug Process

by Sandra Powers
2006-10-10

The basics are easy: Bugs happen; we fix them. But in a post-ship, live MMO game, handling bugs can be a bit of a headache. Bugs can come in from multiple sources on multiple builds on multiple timelines: from the current live build; from a build in testing for the next live update (internal or external); possibly from a parallel expansion build (also internal or external); even from the dev team itself. (Actually, keeping all those different builds straight isn't so easy either, but that's a story for another day.)

For example, you can imagine a scenario like this:

• On the live build, there is an NPC in town with a typo in their dialog. Whoops! Bug!
• On the test build, scheduled to go live in two weeks, this dialog has been replaced with a seasonal version of the dialog in which the NPC waxes lyrical about the winter festival. The typo doesn't exist in this version – no bug!
• But on the same test build, this NPC does use the wrong script call to address the player and therefore addresses the player by the NPC's name. Whoops! Bug!
• But on the main development build, scheduled to go live in one month, the seasonal dialog has been replaced with the old generic dialog – except that the original typo has been fixed (no bug!) and a new typo has been added (new bug!).

So we have a lot of bugs coming in from a lot of sources. And we have to be able to route and prioritize these issues correctly, or things – sometimes quite important things, like customers – get dropped. It's possible to flail about and keep more or less on top of bugs by virtue of vigorous action. But if you'd rather be smarter and more efficient, then it pays to put some thought into how you plan to handle bugs, preferably before you ship.

Everything is a Bug!

First things first: don't get too hung up on the word "bug". I use "bug" interchangeably with "issue" and sometimes "task" to indicate something that needs work. It may be a defect in the code or in the data where something doesn't work as it's designed. It may be a defect in the design – maybe this skill is overpowered or that quest is just no fun. It may be a suggestion for a minor new feature (say, an auto-AFK option) or even a work item that doesn't immediately affect the game (like standardizing the grammar used on crafting ingredients so you can better automate their generation at some point).

Even fairly major projects – say, the bullet point features of your next expansion – can be handled as bugs (or tasks, as you prefer) and tracked through the normal bug process. In fact, this can be a nice way to keep a quick high-level eye on the status of the major features. Of course, you'll need to have a lot of supporting documentation outside of the bug process for this, but you can easily reference the Wiki page or location of the docs from your bug.

Path Planning

Bugs come in from many sources. Someone needs to be responsible for taking the bugs from those sources and getting them into your database of bugs. (This doesn't need to be a real database – at the simplest end it could just be a list of issues, maybe in Excel or even, possibly, written on a napkin. I recommend using an off-the-shelf tool with a real database, though. There are some decent free ones; see below for random recommendations.)

At any rate, you'll want to define a path for each source and make sure that the responsibilities for each path are clear. Your QA department will probably be heavily involved with this: for instance, they may be responsible for entering bugs from an in-game bug submission form or handling bugs sent over from Customer Service. QA can also verify the validity of bugs coming in from outside sources. And of course they'll be entering any bugs they find from their own testing.

One of the interesting questions I've run into several times now is whether the developers themselves should enter bugs or if they should route all new bugs through QA instead. On the one hand, QA tends to have more experience with defining the steps to reproduce a problem (repro steps) and will likely enter a better quality bug description. On the other hand, sometimes it takes a lot of extraneous communication to explain the details of a technical task – at this point QA will just be cutting and pasting your e-mail anyway, so you might as well do it yourself.

A simple rule of thumb might be that if the problem has repro steps, then let QA handle it. But if it's a task or work item that is more easily described in terms of what needs to be done, then the developers can enter it. Of course, even with this guideline you may run into timing issues. If a designer is testing their own brand new quest in progress and hits a snag, they'll of course fix it. But if you are past the point where you have frozen your code prior to a release, you'll want to track this change, which means entering a bug on it, which means that the fix is in before the bug is entered and what's the point of that? The answer: tracking. I'll talk about this in more detail below, but suffice it to say that for a live MMO game, it's just as important to have a record of the bugs fixed as it is to fix them.

Once bugs are in the database, someone also needs to be following up on them: making sure they are prioritized and assigned and scheduled and handled. This someone might be the various leads, although I prefer production to be in charge of this – but then, I'm a bit of a control freak. The important thing is that someone is looking at the bug database frequently. How frequently? On Asheron's Call 1 I reviewed the incoming bugs every morning; anything I couldn't immediately prioritize and assign out was held for a weekly discussion meeting with the leads. At my current gig I also review new bugs daily, but because of the larger scale of the project I will follow up with the leads each day on questionable bugs or tasks.

When the work is done on each bug, it generally goes back to QA for verification and closing. But you will also want an acknowledged process for closing bugs without fixing them. Who has the authority to say that a certain bug won't be fixed? What kind of review process do "Do Not Fix" or "Not a Bug" issues go through? Make sure the entire team knows about this stuff!

Incidentally, I don’t expect that any of this is terribly new or interesting. You don't need or even want to innovate with your bug process. No, the important thing is that you have a bug process, and that everyone on the team knows what it is.

Tracking

For fire-and-forget single-player games, the bug process is all about – well, fixing bugs. But for a live MMO game, the bug process is just as much about tracking changes as it is about fixing defects. There's almost no change so small that you won't want to track it. Well, no, that's not true – there are plenty of changes so small that you won't want to track them, but months from now when you are trying to figure out what happened, you will wish that you did track it back when you made the change.

Likewise, it's important when you are resolving a bug to tell what you did to fix the problem, not just that it's been fixed. This is helpful for the rest of your team when your fix blows up in three weeks while you're on vacation. Of course, you probably also recorded this information in your source control tool when you checked the change in, but that just means it was easy to cut and paste the explanation into the bug database also. The bug database is way easier to search by keyword than the revision history on your source control.

A bug database is a also record of intentions and goals – just as much as your Wiki or your design documents. When you need to revise the crafting system six months from now and you are trying to figure out why you made that weird tweak – what was that supposed to solve again? – this is when a record of the change and the issue it was intended to fix is useful. Even for bugs you decide not to fix, it's important to record why – if only so the community folks can figure out what to tell the players when this becomes a cause celebre on the forums.

The Tools of the Trade

There is some quite decent software out there that can make the bug process a lot less painful. I have personal experience with DevTrack and RAID (Microsoft's internal bug tool), and second-hand experience with Bugzilla. My favorite is RAID, but unless you're close with Microsoft it's not likely to be a feasible choice for you. DevTrack isn't bad. I’ve also used Excel as a back-up in certain situations. Excel is the all-purpose workhorse of production – but I strongly recommend a full-featured application for serious projects.

Which software you choose will depend on your situation and your needs, of course, but I can point out a few features that I’ve found incredible important:

• Make sure searching is easy and powerful, both by title and by description. I end up searching for bugs by keyword several times a day – namely before I enter a new bug, to make sure it's not a duplicate, and when I'm trying to track down the status on something that came up in a meeting.
• Similarly, make sure you can easily sort and filter in the ways you'll likely need to: by person, by release date, by department (i.e. show me all the bugs assigned to coders), by priority or severity, by zone, by whatever. If you can dump your searches or sorts into Word or Excel easily, all the better.
• The history of the bug is important. Who found it or entered it, who has done what work on it, who changed the priority, who assigned it to the dead queue ... Of course, you probably don't want to have to dig through all the history every time just to get a summary of the problem, but you will want that information available when you need it.
• Flexibility is important in the long run. If everything works out, you'll be using this database for years – and during that time, your process and the way you organize your tasks will likely change several times. If you can stick with one piece of software that can be adjusted as your process changes, you’ll be ahead of the game.
• On the other hand, in the short term it is more important that daily operations are quick and easy. Otherwise, you'll end up bypassing your own process to avoid the tedium of entering and tracking bugs.

Some bug tracking software can integrate with popular source control packages. I've never used this feature; I have no idea how useful it is. Some tools also come with helpful e-mail notifications: when a bug is assigned to you, you get an e-mail. This is a nice feature to have, but it's not essential – nor would I rely on it overmuch. It’s just as easy for a busy developer to ignore an e-mail as it is for them to forget to look at the bug software. Much more difficult to ignore is the producer standing in your doorway with a bug list and a pencil, waiting for an update.

Conclusion

A good bug process is one in which the bugs get handled and don't get dropped. Generally speaking this means having some idea of how issues come in and what happens to them and where they go – and who is responsible for them at each stage. The details will vary based on your team and your project and your situation, and it's tempting to think that anything that mostly works is good enough. But handling bugs for a live MMO game is complicated by the multiple streams of development all going on at once. So if you want to use your resources efficiently and cut down on headaches, it pays to give it a little bit of thought at the start.

[ <— #8: The Yearly Rant ]