Series Info...Engines of Creation #13:

Plan Your Fight

by Dave Rickey

Ever since I first got exposed to this industry, there's been an ongoing struggle over how to define what is and is not a "Beta" for a game. Theoretically, there is a definition already, this version taken from The Free On-line Dictionary of Computing:

/bay't*/, /be't*/ or (Commonwealth) /bee't*/ n. 1. Mostly
working, but still under test; usu. used with `in': `in beta'. In
the Real World, systems (hardware or software) software often go
through two stages of release testing: Alpha (in-house) and Beta
(out-house?). Beta releases are generally made to a group of lucky
(or unlucky) trusted customers.

However, there are variants on that definition, one of the most common (and for our purpose, most troublesome) being the substitution of "feature complete" for "mostly working". To my knowledge, no MMO has ever reached Beta truly "feature complete"; most have launched with major features not yet implemented. In some cases, the lack of these features has gone beyond being a source of complaints to becoming a running joke ("paid Beta", "fun to be patched in later"). In the worst cases, expansion packs have shipped as "empty boxes", none of the features listed on the boxes actually working until weeks or months later. In addition, Betas have become inextricably part of the marketing plans of the games: a good beta can provide an irreplacable boost to the word of mouth for a game (and a bad one undo millions of dollars worth of traditional marketing through bad buzz).

The problems arise because of the differing agendas of the parties involved in bringing the games to and through the beta process, and their differing expectations of what they are supposed to accomplish. The fact is, our traditional Alpha/Beta division does not serve the needs of these games well. Well before the full featureset is in place, the complexity surpasses the point where internal QA processes are adequate to cope. You simply can't even make a serious attempt at full-regression organized QA; the manpower requirements are far too high. Even by the softer, "mostly working" definition, what works at a peak load of 30 players may fail disastrously at 300. More subtle problems, like database corruption, may only appear after weeks or months of heavy abuse, or only under extreme conditions of variable network latency combined with unlikely (but statistically inevitable) events, like two people logging into the game in the exact same picosecond.

Beyond that, there are the marketing implications. Ideally, Beta isn't about marketing, it's about making the games work. In the real world, only a fool ignores the impact of "beta buzz". Beta testers telling their friends "You've got to try this game" is the most powerful marketing force in the industry. A few thousand evangelists working their personal social networks is better than millions of dollars of paid advertising. By the same token, beta testers saying "don't bother, it sucks" can kill a game, no matter how much money is spent on marketing. In fact, it's quite possible that the only function of traditional marketing for MMO's is in "validating" the buzz; if you have full-page spreads in magazines it makes it easier for those hearing positive buzz (especially those working for the magazines) to give it credence. NDA's are of only limited usefulness in containing this. In some cases they may be counterproductive (when they are widely ignored by those that dislike the game, but respected by those who like it).

On the other hand, for the purposes of getting high-quality feedback that tells you what is wrong and where to look in order to fix it, Beta sucks. Comparatively few of the testers submit feedback of any kind, and those that do tend to submit the same reports in slightly different forms (and with wide variations in detail). The most common return from any in-game bug reporting tool is the word "bug", with nothing else. Most of the testers are there to get a sneak peek and/or play for free; most of the remainder doesn't know *how* to be a good tester. Also, the Beta process is a continual contest over the design of the game, both between the company and the tester community, and usually within the company itself; bug reports often become weapons in that conflict.

So Betas become exercises in community management, usually long before the team is ready to make the transition from developing a game to operating one. Meanwhile, an increasingly jaded marketplace is judging the Beta against the same standards they judge games at launch, or even years past their launch. All games, from the venerable established title with nearly a decade of operation, to the newest and rawest creation just leaving "technology demo" status, are competing for the same "mind share" in the marketplace. This is a long-term trend that bodes ill for the industry, but is beyond the scope of this week's column.

What is within the scope is to try and better define the *types* of beta, to establish that there are distinct stages with distinct needs that are currently being lumped into the generic label of "Beta". To the best of my ability to ascertain, these are the categories (names assigned-semi-arbitrarily):

  • Development Beta
  • Playtesting Beta
  • Scaling Beta
  • Load Beta
Development Beta:
This phase starts very early, and could also be termed a "Public Alpha". Gameplay features are under constant development, fundamental architectures are still being established, severe and critical bugs are commonplace, and content is nearly non-existent. Because so much is not yet in place, the game can still undergo major changes. Because so much is not in place, the game is not very fun, and tester dropout rates are high and average hours played is low. Number of testers is in the sub-500 range, with maybe 100-150 truly active in the testing (and a high proportion of those engaged in a dialog with the developers). In many ways, this is the "honeymoon" period of game development; the community tends to be cohesive and coherent, and the active participants highly engaged and hopeful for the game's prospects. Until all major gameplay systems are 80% in place, it is inadvisable to move to the next stage.

Playtesting Beta:
This phase appears slowly out of the Development Beta. Systems and features are mostly in place, but tuning and playability factors are woefully inadequate. Content is sparse, generic, and mostly uninspired. The community of active testers is only slightly larger (150-250), but considerable attrition of the original tester base has occurred, and there is some friction between the remaining "old hands", who have personal relationships with the development team and a high degree of investment in the game, and the newer testers. There is often considerable acrimony between various factions of the tester community, especially over fundamental game issues. The newer testers are less engaged and motivated than the "old hands", and there is usually a faction of "sore losers", who do not like the direction some game features have taken, but are too heavily invested to walk away. At this stage, the basic gameplay becomes "carved in stone" or a "dead horse", and most arguments are over details of implementation within those bounds. Some testers are "gaming the team", falsifying and fabricating reports to either get the changes they desire, or just stir things up.

Scaling Beta:
Here the project is entering its big push, gameplay and major systems are 95% done, but endless tuning, tweaking, boundary conditions repair, and bug-fixing is underway (and in fact, will never stop). Content is being developed at a rapid rate, and the "feel" of the world is being established. The tester community grows throughout this phase, from around 350-500 up to an amount that "fully loads" one world instance. At this stage game balance and featureset arguments reach a fever pitch, as the community becomes aware that what is in place is very much what will be in place for launch, and final decisions are made on what to cut from the design. Overall feedback volumes are so high, and quality and reliability so low, that the development team becomes almost entirely dependant on a pool of trusted testers (usually survivors from the earliest months) for actionable feedback. If measures are not taken to replenish this pool, they will be almost entirely depleted through attrition by the launch of the game. Most of the testing community is increasingly feeling that they are shouting into a void, an increasingly over-tasked team lacks time to interact with any of them at all. If some development team members have not been added specifically for the purpose of community management, and steps are not taken to maintain a pool of reliable contacts, community relations may break down completely at this or the next stage.

Load Beta:
The broad strokes of the world have all been laid out, and fine detail is being added. At least on occasion Gameplay is almost completely as it will be for launch; although major changes may be made, it is unlikely they will actually fix the problems they are aimed at, and highly probable that they will create other issues that will haunt the team for months or years. The tester community is now enormous, numbering in the thousands, almost all of them being "tourists" checking out the game to see if they want to play it. Feedback volumes are incredible, average quality is nearly nil, and only careful filtering and management can maintain communications. The vast majority, if not all, of the development team is in "bunker mode", focusing on their assigned tasks and trying very hard to ignore the increasingly hostile and hysterical forums (by this stage, any forums open to the general tester community have become solely the domain of "board warriors"). At this point, the die is cast and either the game will do well, or it won't, but no reliable indicators have been found to detect which based on the activity or discussions of the testers.

This is probably not the only way to define these phases, and I'm sure some would take exception to my advice for dealing with them. But we have to start this debate somewhere, and the key point is this: we are doing ourselves a great deal of harm by not recognizing that the process we call "Beta" is actually several different things that have been lumped into one label, that must be recognized and handled distinctly. What Beta means for non-MMO's applies only to the final stage of these beasts.

[ <— #12: Nuts and Bolts | #14: What's So Great About Realism? —> ]

Recent Discussions on Engines of Creation: