Subscribe
Playtesting
March 20, 2008 9:30 am
Let’s stay on this handcrafted content thing. Let’s say you have awesome tools and a great pipeline, you have an enormous and productive team which never complains about having to make stuff they don’t like, and you’re just full of great ideas with what to do with it all.
Your bottleneck is playtesting. Not QA — your awesome tools help ensure that stuff is mostly not broken on the first iteration! You need playtesting to make sure that experiencing that content is everything that it can be — your awesome tools cannot help you.
Let’s say you’re, oh, working on a mature game and making end-game content to shut whiny guild leaders up.
Your QA team is a bunch of minimum wage temps who don’t actually play the game. They can’t help you test raid content.
You can round up the whole office and ask them to play through it a few times a week, but that’s time they could have spent coding or modeling or whatever it is they do. And they’re not top-notch players either.
You can recruit some player volunteers and ask them to please, please not tell anybody about your dumb first iterations. Maybe they will, maybe they won’t! At least they play the game. (And woe if anyone outside of the program finds out. It’s so unfair!)
Alternatively, you can throw it on test and move on. Then everybody knows about your dumb first iterations! It’s so embarrassing.
How can you solve this bottleneck?
Vargen wrote:
How do you solve it? Why by being perfect designers who get it right the first time. I mean how hard can it be? Doesn’t my $15 per month entitle me to perfection?
*ahem*
I’m sure this argument will crop up. Best just to lay it out in all its naked ignorance now.
Posted on 20-Mar-08 at 10:13 am | Permalink
Matthew Weigel wrote:
Why is QA a bunch of minimum wage temps who don’t actually play the game?
How is this related just to hand-crafted content, and not a problem of generated content as well?
Posted on 20-Mar-08 at 10:32 am | Permalink
Darius K. wrote:
Have a trusted community play the game, and interview them about it via private forums and record and analyze metrics on their experience.
Posted on 20-Mar-08 at 11:52 am | Permalink
Makaze wrote:
Why is QA a bunch of minimum wage temps who don’t actually play the game?
Excellent question! Perhaps at least part of the solution is to make QA (for MMOs) a bunch of minimum wage slackers who sit around and play the game all day.
Want to join our guild? Minimum 8 hours of raiding per day, 1 days off per week. Must relocate to . Pays $30K, no benefits.
You’d have people moving out of their parents basements and lining up around the block for that.
Posted on 20-Mar-08 at 12:16 pm | Permalink
Makaze wrote:
Oops, supposed to say “Must relocate to *insert dev city here*”
Accidentally used pointies around it.
Posted on 20-Mar-08 at 12:18 pm | Permalink
Patrick McKenzie wrote:
Make access to the test server part of the elder game and your players will tear themselves apart trying to get into it
Imagine if the first 3 guilds on some randomly selected servers who downed Grand Lord Foozle got their accounts marked for 1 month of access to the test server for playtesting the epic Revenge of the Undead Reanimating Foozle event. Reward guilds who are particularly useful on the test server with public praise and in-jokes, perhaps specific to their server. (”Pathetic mortals, who do you think you are, $GUILDNAME? Taste my blade!”)
Similarly, if you’re testing casual stuff, then let access to the test server be determined by casual accomplishments. (Beating a Heroic 5-man, etc)
Interesting wrinkle: do you tell players about the exact conditions in advance, or do you let them sweat it out?
(I also think you could do some REALLY interesting feedback loops from the test server to the real world. Prestige items, titles, public recognition, a leaderboard, be creative. It doesn’t have to be something with an in-game effect.)
Posted on 20-Mar-08 at 9:04 pm | Permalink
David Hunt wrote:
I agree with Darius on this one. I think a solid alpha community can go a long ways towards addressing the issue.
Posted on 23-Mar-08 at 12:24 pm | Permalink
jonte wrote:
I doubt giving players access to the test server will work for the most part.
From personal experience the players you want in on the testing will play along during beta. File their bug reports and help the game develop in the right direction. Once the game is released they are playing the game to win and won’t be interested in getting access to a test server where there are no records to be broken or glory to be had.
If you want the right people to test your game properly you need to give them something in return besides access to yet another “beta”.
Posted on 24-Mar-08 at 7:29 am | Permalink
BugHunter wrote:
Hire a real QA department like successful business software companies. If those employees are only costing you 30k you still haven’t done it right.
Posted on 24-Mar-08 at 2:12 pm | Permalink
David Hunt wrote:
jonte: The ideal composition of an alpha group is quite a bit different than a beta group. You don’t want typical players in alpha. The players who are interested in that sort of thing derive enjoyment from it on a different level than access to beta. There isn’t a large supply of those kinds of players, and there are plenty of things you can offer as perks.
BugHunter: While it’d be great to have a QA department that has the experience and bandwidth to thoroughly playtest and analyze end-game content, it’s unfortunately out of the reach of some companies. It’s especially hard when you want to test content that requires a large number of people, and when you have to convince the financial backing to spend gobs of money on more staff. That’s one reason why I like alphas. Blizzard handles this issue well, with their gigantic QA/Test staff, but financial concerns aren’t an issue for them. So it’s a poor situation, but I’d wager it’s a common reality.
Posted on 24-Mar-08 at 10:33 pm | Permalink
jonte wrote:
David Hunt: I guess I was a little unclear in what I was saying. I also have the feeling that I’m missing some underlying message in her post after your response.
Guilds I’ve always been in that have been top notch have always had a good mix of people that would be great both in an alpha as well as a beta environment. That’s usually what makes them successful (besides the insane amount of time they have available).
You have the stats nerds, the balance freaks and the people just looking for a fun experience. A well balanced and motivated guild like this to me should be ideal for testing both alpha and beta. It’s just a matter of motivating them.
Speaking from a gamer perspective here it’s not too easy to get a guild like that interested in testing a game when they can actually play it and make progress.
Posted on 25-Mar-08 at 4:02 am | Permalink
Reed wrote:
Ultimately, I think that BugHunter is right. You can’t magically solve the playtesting problem without people. Dedicated, passionate, and most importantly professional, testers are key. Not high school dropouts who think it’s awesome to work in the game industry.
The second-choice does seem to be relying on player input, but that can only go so far. Players (mostly) won’t understand what it takes to make a good bug because they will be looking only for things that seem to hamper their play experience. So while they may stumble on the real issues with a particular piece of content, what it much more likely is that your dev team will have to doggedly filter through mountainous stacks of player complaints to actually figure out what’s wrong and what to do about it. Good testers, on the other hand, can explore the root cause of issues and clearly communicate to you waht the problem is and can collaborate on a way to fix it.
The other advantage of a professional testing staff is that they will, contrary to David’s assertion, likely reduce your test head count; and hopefully costs. Depending on how you scope their work (and who you hire) you can end up with a professional team capable of reducing their own work load and yours. Testers who are hired for a computer science/engineering background can create automation and other tools: freeing up dev time to work on the actual code and freeing up test time to really look at the fun, interesting and difficult stuff. Like thoroughly examining raid content.
That said, just up and hiring a bunch of professional testers and saying “go at it guys” is not going to magically solve anything overnight. Like a proper Dev team, you can’t just make a test team appear out of thin air and expect them to suddenly turn your test process into a thing of beauty. However, as long as you’re relying on a bunch of minimum wage temps to properly validate a hugely complex system like a game, you’re doomed to failure; or at least a huge number of bugs. Trying to shore up dependence on a poor testing infrastructure by relying on players, who are not only outsiders to the project and the code but who do not have your best interests in mind, only their own sef-serving goals (which often will conflict with good game design) is an even more broken strategy.
Posted on 25-Mar-08 at 12:31 pm | Permalink
SirBruce wrote:
You need a real playtesting group. A lot of people say this should be QA, but you can’t expect all your QA people to be that experienced with every particular aspect of content in your game. So you at least need a playtesting team, either part of or adjunct to QA, to catch these sorts of things.
It’s a multi-level process. The developers first should be good enough to playtest. Then there’s the other teammembers who should “eat their own dog food”, but as you say, they’re also busy with their own thing. Internal playtesting is the next level, then lower level QA, then push it to the test server. And you still have to be willing to *respond* to what you learn on the test server, modify, and send it through the whole process again.
And this also needs to continue *after* release of the content.
Posted on 31-Mar-08 at 9:29 am | Permalink
Hate wrote:
In my studio daily play testing is almost a mandatory requirement. However the budget argument often wins when the crunch is on. On the design side I love feedback from the texture artist who has no clue what he is doing.
I would like to highlight one point SirBruce brings up. Response to the feedback you receive has a lovely impact on the players who participate. It makes them feel more important and is seen as a reward for their time spent testing something that could be the ‘dumb first iteration’. Not to mention it sometimes sheds light on a system mechanic that the players then turn around to help them in their normal play - which is more reward and adds to the warm fuzzy feeling those people get for helping you make your game/content more fun.
I really like the idea of rewarding the end game core groups. I just need to work out how to apply it to my project.
Posted on 03-Apr-08 at 4:51 am | Permalink
Will Jennings wrote:
I’m with Matt Weigel, BugHunter and Reed - if the only people you’re employing to test the game are minimum wage temps then you don’t, in fact, have a great pipeline or a maximally productive team. Experienced, professional testing and support, fully integrated with the rest of development, multiplies the effectiveness of the team even when you mostly aren’t breaking things on the first try. A quality testing department that can handle playtesting, among other responsibilities, is more readily attainable than the rest of the premise.
Posted on 04-Apr-08 at 12:07 pm | Permalink
Psychochild wrote:
The problem is that a “real” testing group costs real money; in finance terms, this is a “fixed cost”. If you’re Blizzard, this may not be a big issue, but for the smaller developers, this can be a significant drain on resources. A few lean months and bad things happen.
Further, the philosophy of QA in the industry as a whole is messed up. For most of the publishers, QA peons are hired and fired on a regular basis as projects demand. It’s not unusual for a QA tester’s job to end once the product ships. (In some traditional publishers, this can be true for regular development jobs, too.) There’s no career in being a QA tester, only in working up to a lead position that just herds the cats. On top of this, QA is seen as a stepping stone into “real” game development, so your most competent QA people aren’t going to want to stay in QA; they’er eyeing those junior designer and assistant producer positions. All these factors means that QA is seen as a temporary job, and there’s little reason for a company to invest in a QA team.
Yeah, the obvious solution is, “don’t do that.” But, we’re talking about changing how most of these companies operate on a fairly fundamental level. So, it’s much, much easier said than done.
Posted on 10-Apr-08 at 5:27 pm | Permalink
Reed wrote:
Psychochild, you’re quite right. For a lot of companies, it would be a huge paradigm shift to actually pay for professional Developers In Test instead of high school graduates who think the game industry is cool.
But the question posed was not “how can I find a quick fix that may or may not improve product quality?” but “how can you solve the problem of adequately playtesting end game content?”. That second question is equivalent to a larger “how do I improve my test process?” question. Any company asking that second question should be looking for a realistic, long-term solution. Any company asking the first question (which would be similar to any company who ask the second question, gets a right answer and then says “oh, that would be too much work”) obviously will not want to completely overhaul their testing methodology. However, these are the same kind of companies who feel that what matters is ‘quickly’ churning out the next game, not improving the overall process of making games so that each successive one becomes easier and less risky.
So, anyway, I don’t accept that investing in a real test department is the wrong answer simply because many game companies, as they are now, have antiquated and backwards software engineering practices.
Posted on 16-Apr-08 at 10:01 am | Permalink