But the question posed was not “how can I find a quick fix that may or may not improve product quality?” but “how can you solve the problem of adequately playtesting end game content?”. That second question is equivalent to a larger “how do I improve my test process?” question. Any company asking that second question should be looking for a realistic, long-term solution. Any company asking the first question (which would be similar to any company who ask the second question, gets a right answer and then says “oh, that would be too much work”) obviously will not want to completely overhaul their testing methodology. However, these are the same kind of companies who feel that what matters is ‘quickly’ churning out the next game, not improving the overall process of making games so that each successive one becomes easier and less risky.

So, anyway, I don’t accept that investing in a real test department is the wrong answer simply because many game companies, as they are now, have antiquated and backwards software engineering practices.

Further, the philosophy of QA in the industry as a whole is messed up. For most of the publishers, QA peons are hired and fired on a regular basis as projects demand. It’s not unusual for a QA tester’s job to end once the product ships. (In some traditional publishers, this can be true for regular development jobs, too.) There’s no career in being a QA tester, only in working up to a lead position that just herds the cats. On top of this, QA is seen as a stepping stone into “real” game development, so your most competent QA people aren’t going to want to stay in QA; they’er eyeing those junior designer and assistant producer positions. All these factors means that QA is seen as a temporary job, and there’s little reason for a company to invest in a QA team.

Yeah, the obvious solution is, “don’t do that.” But, we’re talking about changing how most of these companies operate on a fairly fundamental level. So, it’s much, much easier said than done.

I would like to highlight one point SirBruce brings up. Response to the feedback you receive has a lovely impact on the players who participate. It makes them feel more important and is seen as a reward for their time spent testing something that could be the ‘dumb first iteration’. Not to mention it sometimes sheds light on a system mechanic that the players then turn around to help them in their normal play - which is more reward and adds to the warm fuzzy feeling those people get for helping you make your game/content more fun.

I really like the idea of rewarding the end game core groups. I just need to work out how to apply it to my project.

It’s a multi-level process. The developers first should be good enough to playtest. Then there’s the other teammembers who should “eat their own dog food”, but as you say, they’re also busy with their own thing. Internal playtesting is the next level, then lower level QA, then push it to the test server. And you still have to be willing to *respond* to what you learn on the test server, modify, and send it through the whole process again.

And this also needs to continue *after* release of the content.

The second-choice does seem to be relying on player input, but that can only go so far. Players (mostly) won’t understand what it takes to make a good bug because they will be looking only for things that seem to hamper their play experience. So while they may stumble on the real issues with a particular piece of content, what it much more likely is that your dev team will have to doggedly filter through mountainous stacks of player complaints to actually figure out what’s wrong and what to do about it. Good testers, on the other hand, can explore the root cause of issues and clearly communicate to you waht the problem is and can collaborate on a way to fix it.

The other advantage of a professional testing staff is that they will, contrary to David’s assertion, likely reduce your test head count; and hopefully costs. Depending on how you scope their work (and who you hire) you can end up with a professional team capable of reducing their own work load and yours. Testers who are hired for a computer science/engineering background can create automation and other tools: freeing up dev time to work on the actual code and freeing up test time to really look at the fun, interesting and difficult stuff. Like thoroughly examining raid content.

That said, just up and hiring a bunch of professional testers and saying “go at it guys” is not going to magically solve anything overnight. Like a proper Dev team, you can’t just make a test team appear out of thin air and expect them to suddenly turn your test process into a thing of beauty. However, as long as you’re relying on a bunch of minimum wage temps to properly validate a hugely complex system like a game, you’re doomed to failure; or at least a huge number of bugs. Trying to shore up dependence on a poor testing infrastructure by relying on players, who are not only outsiders to the project and the code but who do not have your best interests in mind, only their own sef-serving goals (which often will conflict with good game design) is an even more broken strategy.

Guilds I’ve always been in that have been top notch have always had a good mix of people that would be great both in an alpha as well as a beta environment. That’s usually what makes them successful (besides the insane amount of time they have available).

You have the stats nerds, the balance freaks and the people just looking for a fun experience. A well balanced and motivated guild like this to me should be ideal for testing both alpha and beta. It’s just a matter of motivating them.

Speaking from a gamer perspective here it’s not too easy to get a guild like that interested in testing a game when they can actually play it and make progress.

BugHunter: While it’d be great to have a QA department that has the experience and bandwidth to thoroughly playtest and analyze end-game content, it’s unfortunately out of the reach of some companies. It’s especially hard when you want to test content that requires a large number of people, and when you have to convince the financial backing to spend gobs of money on more staff. That’s one reason why I like alphas. Blizzard handles this issue well, with their gigantic QA/Test staff, but financial concerns aren’t an issue for them. So it’s a poor situation, but I’d wager it’s a common reality.

From personal experience the players you want in on the testing will play along during beta. File their bug reports and help the game develop in the right direction. Once the game is released they are playing the game to win and won’t be interested in getting access to a test server where there are no records to be broken or glory to be had.

If you want the right people to test your game properly you need to give them something in return besides access to yet another “beta”.