The Launch Pad is Ready... For Testing

When it is time to start charging a fee to play a massively multiplayer online game, the industry talks about ‘the launch.’ The intent is defined thusly (from Dictionary.com):

launch¹ (lônch, länch)
v. launched, launch·ing, launch·es
v. tr.

1. 
   
   1. To throw or propel with force; hurl: launch a spear.
   2. To set or thrust (a self-propelled craft or projectile) in motion: launch a rocket; launch a torpedo.
2. Nautical. To put (a boat) into the water in readiness for use.

So, how well do we match up to that standard? We don’t even come close. It is a scene that is becoming all too familiar in this industry; launch before the game is stable and hope like Hades you can patch the worst problems quickly enough not to lose your core market. If Detroit launched new car models the way we launch new online games, you’d take delivery at the sales lot only to find that the engine wouldn’t start, two wheels were missing, the driver had only twelve inches of head space and unshielded EM radiation from the dashboard instruments sterilized everyone within thirty feet in ten seconds flat. The only thing that gets propelled with force is the box the game came in.

The implications for the genre are enormous. At one extreme, we’re at the least getting a bad reputation. At the other, this practice could conceivably kill the genre or limit it to the hard core niche we’re serving now. What makes this particularly poignant now is one recently – and badly – launched MMOG and another much-anticipated MMOG scheduled to launch later this month.

Case in point: the recently launched World War II Online. In this case, the game ‘launched’ in the same way the first few space program rocket boosters launched in the late 1950s, rising a few feet before giving up in despair and coming back down to a spectacular crash and burn. WW2O’s launch may go down as the textbook case launch failure, much the way Robot Monster became the textbook bad film. Here are some of the choicer tidbits:

• After buying the WW2O retail package, players found themselves in the position of having to download a mandatory 67.3 megabyte patch. That is not a misprint; players were basically forced to download the game they had just purchased. If you happen to be stuck with 56k or lower dial-up Internet access, that means several hours of download time. I don’t know about you, but that would cause me to become mildly annoyed, in much the same way Jehovah became mildly annoyed with Sodom and Gomorrah;

• The patch and registration servers for the game didn’t exactly work right. According to Cornered Rat/Playnet, the developer/publisher, the servers were swamped from unanticipated high sales on the first day. Those sales numbers? Supposedly 15,000. How many registrants could the servers accommodate? Less than 800, partially because…

• The servers were crashing regularly or otherwise were unavailable. This was originally blamed on new hardware installed shortly before the game launched. As of the time of this writing, the problem doesn’t seem to have been solved completely, according to complaints on various message boards;

• Even after the ‘patch,’ the game client doesn’t seem to work for a surprising number of players. If you read the various ‘reviews’ and message forums associated with WW2O, it is apparent that at least half the purchasers couldn’t get the game to run on their machines;

• When the above problems were fixed well enough to allow access to the game, even players with maxed-out game machines were getting frame rates below 5 fps. Some were getting frame rates below one. Some of this was solvable for some players by switching to or from the Z buffer, but others seem to still be experiencing problems;

• Billed as a land-sea-air simulator of World War II, the game launched minus a bunch of features promised by the developers. OK, this is not unusual; feature whacking to get a game out the door is fairly normal and doesn’t necessarily have to be a bad thing. However, the features whacked here include the "sea" part, meaning naval operations. For that matter, machineguns didn’t make the release, which seems wacky, to say the least. This would be like releasing Cowboys and Indians Online minus the horses.

All in all, it was a horridly bollixed product launch. It is not as if developers and publishers can claim they were caught unaware, not with so many examples from which to chose – and learn. For that matter, some of the WW2O developers have developed and launched combat simulator MMOGs before, most notably Warbirds. It isn’t like they have the excuse of being newbies to the field, like Funcom.

And Funcom is the developer of the previously mentioned and much-anticipated game that is set to launch later this month, Anarchy Online. Even before launch, AO is starting to exhibit some stress fractures. The standard complaints about client lag in art-intensive areas and trouble zoning from one area to another abound. There are also more serious complaints. For example, the front end client for the game went gold a few weeks ago, to get the retail package on the shelf in time for the launch. Last week, it was discovered that using the AO client with Windows 2000 could corrupt the Win2k registry so badly that one couldn’t even boot the machine. Some AO Win2k users had to boot from the Windows CD and recover the pre-AO installation version of the registry to get their computers working; even then, some had to reinstall sound card and other drivers.

With problems of that nature so close to the premier date, you have to wonder how well AO’s launch is going to flow. First impressions – reviews – can be critical to the long-term success of an MMOG. Players can be pretty tolerant, but even they have their limits. And one has to wonder how many players get frustrated early and leave, never to return. Consider: in the days of hourly fees to play an online game, the player retention rate was much higher than the approximate 45% we see today with monthly fees. Of course, games used to launch with a lot fewer problems…

Now, can you imagine trying to launch a mass-market MMOG with those kind of problems? I sure can’t. What causes this kind of thing? What are the root problems here? No one expects an online game (or any other online product, for that matter) to launch problem-free, but what we see today borders on the absurd. Why have MMOGs been so prone to the Crappy Launch Syndrome since Ultima Online launched in 1997?

I’ve talked about some of the problems before, such as a general lack of experience in the industry in dealing with the requirements of client/server code development, and underestimating cost and time of development until the need to get a game out the door for a return on investment is paramount. That is quite a bit of why we’re having the current problems and we’ll talk more about it again, I’m sure. Something we haven’t discussed much is the need for more and better testing.

At the risk of sounding obvious, testing an MMOG is hard. Unlike the mere thousands of man-hours which go into testing home PC games, which generally consists of testing hundreds of machine/peripheral configurations and performing bug hunts, even a simple MMOG requires literally millions of man-hours to test. You not only have to perform the testing you’d do for any other front end client, there is the additional complication of all that backend network and game systems code to deal with.

This is can be far more complicated than it sounds. For example, consider a massively multiplayer fantasy RPG. There are a bunch o’ game systems that have work to together perfectly to perform even simple acts reliably. If a player wanders up to an NPC monster and attacks, the game systems have to perform a number of checks on both combatants’ weapons, armor, weight, physical speed, range of attack, inventory encumbrance, magically-induced modifiers on weapons and persons, et al, then come up with a number, resolve the attack results and return those results to the player.

OK, this is what computers are for, right? Pretty much a simple matter of multiplying, dividing, adding and subtracting as needed, making an invisible random dice roll and applying the results, right? Right… as far as that goes. But now consider this: when you have hundreds of modifiable weapons and armor pieces combined with thousands of individual player characters and NPCs, the attributes and skills for which make up tens of thousands of combinations that can change on the fly, the number of possible game systems combinations that a QA department must test can reach into the millions pretty quickly.

And if you’ve ever worked QA on a MMORPG, you know the following dirty little secret: some game systems bugs are so deeply buried, they may occur only once every few million or tens of million commands processed by the server. Or only when exactly 1,531 simultaneous players are logged in when an orc spawns in exactly the right place, or only if six +8 Blood-drinker axes initiate combat on the same NPC at the same time tick. Or only when a monster has been surrounded with flour bags to immobilize it and 50,000 gold pieces are then dropped on it. You get the idea.

The upshot is, there is just no way to test all the possible conditions inhouse. That’s why we open our games to external testing and try to encourage a thundering horde to participate. Not only do you get load testing to see where the cracks in your software and hardware are, you get to pick off some of the more hidden, elusive bugs, too.

But not in six months, when over half that period includes less than 2,000 testers, which seems to be the norm right now. With 30,000 to 50,000 testers pounding on the game for six months, you’ll catch the most readily apparent bugs and anomalies and still have time to fix them and test them (as well as the new bugs you created with the fixes) in time for launch.

Bear in mind that some of these bugs require such complex conditions for activation, they just won’t pop up until sometime after launch. These are the bugs that give QA directors gray hair; they are reported once every month or two, but just can’t seem to be duplicated. This can go on for months or years, with players swearing publicly they’ve seen the bug and castigating the publisher for not being able to fix it. Finally, a programmer will make an intuitive leap and the problem will be found and fixed, while the players sarcastically note it should have been fixed long before. If there are even a couple dozen of these elusive bugs at any one time, which is likely, then your QA department is running in crunch mode pretty much from Launch Day until the heat death of the Universe.

If MMOG game code remained frozen from launch, eventually all but the most elusive game system bugs would be found and fixed and the QA department could eventually go on an extended vacation and act like drunken sailors. As we all know, however, MMOG code does not remain frozen; we tinker with it constantly. The game is never quite balanced correctly for our tastes, new skills, objects, quests and other content are added over time. Not only will standard, run-of-the-mill bugs be added by all this, it is virtually certain that it will add one or more hidden, elusive bugs.

Most maddening of all, when we find and fix bugs, there is a tendency for new bugs to be created by the fix, which must then also be found and squashed. It is a never-ending process. In a perfect world, even one small code change would result in the game being tested from scratch, top to bottom. Obviously, that just isn’t possible or realistic.

So what can we do different? First, developers and publishers need to understand that longer test periods with high volumes of simultaneous players are required exercises, not luxuries. You can’t go along for five months with a couple thousand testers, then assume that hitting the last month before launch with 20,000 to 50,000 ‘load’ testers is going to cut it. You need to stage a rising number of simultaneous users gracefully, giving you time to detect and correct problems before rising to the next level of intensity.

Next, such games need to have a minimum of three months testing at max tester loads before launching. Most of the really weird bugs aren’t going to show up until you max out player counts on the game servers for an extended period.

Executives that can demand a game ship before it is ready need to back off a bit. These games always take longer to develop than expected; cutting the testing process short is not exactly a good way to compensate. Part of the problem here is that Marketing and Sales have probably locked in ship dates with retailers to guarantee that shelf space will be available and advertising will hit at the proper time, which is, after all, their jobs. A change in the process is necessary here; publishers probably shouldn’t be scheduling a launch date and unleashing Marketing and Sales until the first true maxed-out server tests.

There also needs to be a comprehensive code change control process in place. Cowboy programming – making and implementing code changes on the fly, bypassing the testing process - has caused more problems in MMOGs than any other factor I can think of. What looks like a simple fix usually has other ramifications; changes need to be controlled and tested well before being released to production servers.

For the client side, which probably has to go to the duplicator about a month before the launch date: Pound on it early, freeze the code as early as possible and for god’s sake, make as few changes as possible after you send the gold master to the duplicator, or you’ll end up with 67 meg download like WW2O.

None of the above will deliver a perfect, bug-free game; that just isn’t possible. It would deliver games with far fewer launch bugs than we see today.

That’s a start, I think.