We Can Fix That with Data

Our latest dev blog talks about our iterative design process here at Spacetime, including an application of a simple data mining system.

Gamasutra has a brief article on production-phase metrics for single-player games at Bioware. I don’t know how true this is of the rest of the industry, but it sounds like theirs is driven by QA, rather than design.

Ideally, QA should feel comfortable giving subjective design feedback to the team — walking across the office to tell the designer that the new minigame sucks. Unfortunately, designers aren’t always willing to listen. Walking across the office with a printout of this chart could be more successful.

That kind of stupid designer pride leads them to be unwilling to back metrics initiatives or to listen to community feedback. Grow up and take advantage of the tools available to you — they make your job easier.

Via a colleague at NCsoft, Nielsen is yet again trying to get into game demographics to sell to marketers. They’ve tried this before, but failed. The old plan, in addition to the antiquated diaries they already use for television :

Nielsen intends to roll out two products this year, the more formidable one being a “tag” that PC, console, and online game developers can build into their software to be used by Nielsen to measure all sorts of in-game activity, especially response to advertising. This includes how people navigate through games, what levels they reach, and how long they spend on each level.

But, because tag placement requires the participation of game developers, Nielsen doesn’t expect the first tagged game to be released until the second half of 2005.

I suspect this also failed because the developers didn’t give a shit about “[measuring] all sorts of in-game activity.”

The new plan is a People Meter-style “audio scanning device.” I didn’t know how People Meters work, the devices they use to track television habits outside of the diaries, and Wikipedia’s article sucks, so I Googled it. Here’s a description of a portable one.

The Portable People Meter, developed by Arbitron Inc., is a pager-sized device that is carried by a representative panel of television viewers. It automatically detects inaudible codes that broadcasters embed in the audio portion of their programming using encoders provided by BBM and Arbitron. At the end of each day, the survey participants place the meters into base stations that recharge the devices and send the collected codes to BBM for tabulation. The Portable People Meter can measure exposure to any electronic media, which has audio that can be encoded – television, cable, and radio, even cinema advertising and in-store media.

I am not making this up, and Nielsen is planning to do the same thing with games: “the equipment detects each game’s unique ‘audio signature,’ compares it to the reference library of audio signatures compiled by Nielsen, and determines what games are being played when and where.”

Alrighty then!

They’re also getting data from Sony, which is much less bizarre: “which Sony games are being played, by whom, and for how long — ‘to enhance the data we are collecting and to give us the opportunity to take a hybrid-based approach to measurement.’”

They’re not specific, but I imagine they’re talking PS3 data. I’ve curious about how seriously Sony and Microsoft are taking their mountains of free data from online consoles — really, the Xbox 360 knows everything I play, for how long, what I do in game, when I log out, what TV and movies I watch, which DVDs I purchase. One would like to think that they’re using it to make the games better, instead of selling it to marketers so we might one day enjoy the wonders of dynamic advertising in everything we play, but hey.

Most World of Warcraft server forums maintain a list of which guild has gotten how far in each noteworthy raid instance. Here’s one, for example. They require a representative from the guild to post every time the guild manages to down another boss.

Via the Elitist Jerks forums comes an automated guild progression site, WowJutsu. According to the FAQ, it runs a script on the Armory and looks at guild members’ gear. If it sees my character wearing a drop from a specific raid boss, it assumes my guild has killed it.

This isn’t entirely accurate — at the low end of the raiding scale, players sometimes participate in multi-guild raids. Sometimes players don’t log out in their raid gear. The layout is cool for top-tier guilds with more than one raid icon to show, but for most crappy guilds on most crappy servers, most people only care how many bosses they’ve killed in Karazhan, and that requires a mouseover. The individual guild reports are pretty cool, though.

I was going to rant here about WoW guild recruitment tools and the lack thereof, but I’m just bitter that I ended up in a terrible guild when I was shopping a few months ago. Clearly, I didn’t read the progression thread. And while some kind of raid epeen rating in an EQ2-style in-game guild recruitment tool would make things even harder on these crappy little guilds, I sure wish that I would have had some more obvious warning.

– Edit, July 5
Brandon has some discussion on the percentage completion stats on the main page’s sidebar.

Google has patented the idea of analyzing in-game behavior to better target ads.

The patent says: “User dialogue (eg from role playing games, simulation games, etc) may be used to characterise the user (eg literate, profane, blunt or polite, quiet etc). Also, user play may be used to characterise the user (eg cautious, risk-taker, aggressive, non-confrontational, stealthy, honest, cooperative, uncooperative, etc).”

The information could be used to make adverts that appear inside the game more “relevant to the user”, Google says.

Players who spend a lot of time exploring “may be interested in vacations, so the system may show ads for vacations”. And those who spend more time talking to other characters will see adverts for mobile phones.

I, for one, look forward to the cosmetic surgery ads in character creation.

• WoW talent distribution metrics. Somewhat flawed methodology (he only examines a set of characters with Allakhazam profiles), but interesting nonetheless. 12% balance druids, represent!
• An RMT solution to the Chinese government-mandated anti-addiction system.
• Pictograms and culture. Worth thinking about if you’re planning to localize your game. You can’t bake the word “Blacksmith” into your sign textures. Well, you can if you want to redo all your signage for every version of the game. Good luck with that, and I’ll spend a lot less time a) thinking about universally understood icons, and b) asking the artists to do the textures once and only once.
• Information design is graphic design: improving the usability of information software (via Juice Analytics). MMOs are information-heavy games; we should be working harder to present the players with the information they need in a format they can use. WoW has a decent default interface, but still, how many of the most popular WoW UI mods are ones that improve information display?

Thanks for all the kind comments and emails. My wrist’s recovery isn’t quite on schedule — I was supposed to be able to type by now, but the doctor’s telling me to wait for another freaking month. This has significantly hampered my ability to work, but after a few days learning to target with the keyboard, I’m playing WoW at about 90% efficiency. It helps that I’m a caster druid, so my job is to either “tab to target, hit Moonfire hotkey, hit Insect Swarm hotkey, hit Wrath hotkey a bunch of times, tab to next mob” or “hit f key to target groupmate, hit Lifebloom hotkey, wait … hit Lifebloom hotkey, wait … oh shit, hit Healing Touch hotkey, wait …” About all I ever have to type is the initial “am I healing?” when I join the group.

This may say something about group gameplay and dynamics in WoW — not that I’m complaining. I’m happy I can do something other than watch Star Trek: The Next Generation on DVD.

In “stuff I wish I could comment on further that’s come up in the last few weeks:”

• The discussion on guilds at Will’s is timely, because another one of my recent WoW activities has been shopping for a raiding guild. It’s been an interesting experience, and I plan to rant about it later. See also Kevin’s followup commentary.
• Terra Nova talks PvP balance. I’ve done PvP balance for a long time. I eventually discovered a secret to making changes that feel responsive, rather than change-for-the-sake-of-change or an attempt to jerk players around: wait until the last minute to commit changes. Producers hate it, QA complains, but if you fix a perceived problem and sit on it for two months, nine times out of ten that’s enough time for player ingenuity to solve the problem on their own. Then you post the patch notes and they think you’re an out-of-date idiot, and you are! (Well, regardless, they always think you’re an idiot.) (And assuming that the game mechanics allow for sufficient emergent behavior.) I’ll have lots more to say on this in the future, heh.
• Darius Kazemi’s new metrics middleware company gets off the ground. They have an extremely awesome blog so far. I expect they’ll be an extremely strong competitor to Emergent.
• Holy shit, Rush Limbaugh intelligently defends games. Well, he defends games by way of comparing them to guns, which doesn’t really work on the target anti-game audience, but hey.