|
Plan Your Fightby Dave Rickey
Beta However, there are variants on that definition, one of the most common (and for our purpose, most troublesome) being the substitution of "feature complete" for "mostly working". To my knowledge, no MMO has ever reached Beta truly "feature complete"; most have launched with major features not yet implemented. In some cases, the lack of these features has gone beyond being a source of complaints to becoming a running joke ("paid Beta", "fun to be patched in later"). In the worst cases, expansion packs have shipped as "empty boxes", none of the features listed on the boxes actually working until weeks or months later. In addition, Betas have become inextricably part of the marketing plans of the games: a good beta can provide an irreplacable boost to the word of mouth for a game (and a bad one undo millions of dollars worth of traditional marketing through bad buzz). The problems arise because of the differing agendas of the parties involved in bringing the games to and through the beta process, and their differing expectations of what they are supposed to accomplish. The fact is, our traditional Alpha/Beta division does not serve the needs of these games well. Well before the full featureset is in place, the complexity surpasses the point where internal QA processes are adequate to cope. You simply can't even make a serious attempt at full-regression organized QA; the manpower requirements are far too high. Even by the softer, "mostly working" definition, what works at a peak load of 30 players may fail disastrously at 300. More subtle problems, like database corruption, may only appear after weeks or months of heavy abuse, or only under extreme conditions of variable network latency combined with unlikely (but statistically inevitable) events, like two people logging into the game in the exact same picosecond. Beyond that, there are the marketing implications. Ideally, Beta isn't about marketing, it's about making the games work. In the real world, only a fool ignores the impact of "beta buzz". Beta testers telling their friends "You've got to try this game" is the most powerful marketing force in the industry. A few thousand evangelists working their personal social networks is better than millions of dollars of paid advertising. By the same token, beta testers saying "don't bother, it sucks" can kill a game, no matter how much money is spent on marketing. In fact, it's quite possible that the only function of traditional marketing for MMO's is in "validating" the buzz; if you have full-page spreads in magazines it makes it easier for those hearing positive buzz (especially those working for the magazines) to give it credence. NDA's are of only limited usefulness in containing this. In some cases they may be counterproductive (when they are widely ignored by those that dislike the game, but respected by those who like it). On the other hand, for the purposes of getting high-quality feedback that tells you what is wrong and where to look in order to fix it, Beta sucks. Comparatively few of the testers submit feedback of any kind, and those that do tend to submit the same reports in slightly different forms (and with wide variations in detail). The most common return from any in-game bug reporting tool is the word "bug", with nothing else. Most of the testers are there to get a sneak peek and/or play for free; most of the remainder doesn't know *how* to be a good tester. Also, the Beta process is a continual contest over the design of the game, both between the company and the tester community, and usually within the company itself; bug reports often become weapons in that conflict. So Betas become exercises in community management, usually long before the team is ready to make the transition from developing a game to operating one. Meanwhile, an increasingly jaded marketplace is judging the Beta against the same standards they judge games at launch, or even years past their launch. All games, from the venerable established title with nearly a decade of operation, to the newest and rawest creation just leaving "technology demo" status, are competing for the same "mind share" in the marketplace. This is a long-term trend that bodes ill for the industry, but is beyond the scope of this week's column. What is within the scope is to try and better define the *types* of beta, to establish that there are distinct stages with distinct needs that are currently being lumped into the generic label of "Beta". To the best of my ability to ascertain, these are the categories (names assigned-semi-arbitrarily):
This phase starts very early, and could also be termed a "Public Alpha". Gameplay features are under constant development, fundamental architectures are still being established, severe and critical bugs are commonplace, and content is nearly non-existent. Because so much is not yet in place, the game can still undergo major changes. Because so much is not in place, the game is not very fun, and tester dropout rates are high and average hours played is low. Number of testers is in the sub-500 range, with maybe 100-150 truly active in the testing (and a high proportion of those engaged in a dialog with the developers). In many ways, this is the "honeymoon" period of game development; the community tends to be cohesive and coherent, and the active participants highly engaged and hopeful for the game's prospects. Until all major gameplay systems are 80% in place, it is inadvisable to move to the next stage.
Playtesting Beta:
Scaling Beta:
Load Beta: This is probably not the only way to define these phases, and I'm sure some would take exception to my advice for dealing with them. But we have to start this debate somewhere, and the key point is this: we are doing ourselves a great deal of harm by not recognizing that the process we call "Beta" is actually several different things that have been lumped into one label, that must be recognized and handled distinctly. What Beta means for non-MMO's applies only to the final stage of these beasts.
[ <— #12: Nuts and Bolts | #14: What's So Great About Realism? —> ]
|
