Evidence-Based Design

What happens when baseball commentators love a player, while his statistical performance is average at best? Snark.

It may be that the presence of a fun guy improves the performance of his teammates, but I can’t put that in a spreadsheet. Thus, I’ll laugh with the guys who are laughing at him, and we’ll chortle over the idea of the “grit over replacement player” stat. You have to admit it’s hilarious.

By way of Juice Analytics, one of my favorite blogs, comes news of a new social networking/sports picking site.

PicksPal is a FREE, national, online sports pick competition that allows Members to challenge, communicate, make friends regarding sporting events, and compete for prizes.

The Washington Post has more.

Predicting winning teams is an arcane art, and it seems like a logical next step to data mine large numbers of amateur picks. Funny illustrative links of experts failing:

• “19 baseball experts from the nation’s leading sports network try to predict which team will win the World Series, and not one of them even picks a team that makes it to the World Series, let alone wins it.”
• Some football expert is getting out-picked by a swimsuit model. (Link to video in here.) (I thought it was kind of stupid — hardy har har, he’s getting beaten by a woman! — but hey.)

There are other ways to do it — mathematical analysis rather than pulling predictions straight out of your ass. Win expectancy is fascinating stuff, but I don’t really see anybody doing it outside of baseball (after all, it has over a hundred years of data to mine).

To bring it back to MMOs, I wonder if there are ways to use sports picking tools to look at PvP. I can sort of mathematically predict who’s going to win a one-on-one PvP encounter, but once you have 20 classes and 200 players of varying personal skill levels in a single event, my spreadsheet is worthless. I just hope that the one-on-one or small group-on-small group balance works out on a larger scale.

This strikes me as something worth a lot more thought when I have the time.

Ryan Shwayder says it’s fun. Brian Green says it’s a waste of time. I say the data management makes it harder than you think.

We’re all talking about whether or not mobs should drop their equipment. I agree that it’s a neat little feature, and I think neat little realism features can add a lot to a game world, even if nobody consciously notices them. (My pet peeve: worlds with no female mobs. Little thing, nobody’s going to really notice, would make the world feel more organic.) This isn’t a game design blog, though. This is a game data blog.

I worked on a game where humanoid mobs dropped their equipment. When the worldbuilder assigned equipment to a mob, he could specify a drop rate (e.g. “Generic Sword, 25%”). There were a couple of problems with this system:

• When combat formula are extremely complex and mob stat values are different than characters’, it’s already hard enough to predict mob performance without adding equipment on top of it. (We eventually redid the way mob stats were calculated, making all mob equipment cosmetic. This is probably the way that most MMOs should handle mob performance.)
• The treasure system wasn’t smart enough to compensate for the humanoid mobs’ extra loot. The treasure system was balanced to drop stuff worth a certain amount, and mobs with equipment would drop that stuff plus their equipment. Score!

Solution A: Extra Tables for Everyone. Most RPGs have dumb treasure systems. The worldbuilder assigns a treasure table to a mob, and every time that mob spawns, it rolls some random numbers and drops the resulting crap.

You could build a second set of treasure tables and assign them to all the humanoid mobs, but that’s a pain in the ass. (Especially when the treasure system is stupid in addition to dumb, and looking at the tables makes your eyes bleed.) This is what we did. It really, really sucked.

Solution B: We Love Overengineering.You could build a smart treasure system, where you just let the mob know the actual value of the crap it’s supposed to drop. It’d have to take into account the equipment they drop, then add enough extra random crap to meet that desired amount. If a mob without equipment died, it’d get extra crap.

Loot would be more predictable — all mobs of the same level would be equally good choices to kill, reducing the explorers’ joy in finding new camps to exploit for cash. You could engineer the system around that — the worldbuilder could say “these guys drop 25% more stuff than other mobs of their level” or whatever — but then the system is even more complicated. (Also, then the worldbuilders would able to fuck up your careful economic balance. Worldbuilders!)

Solution C: Waste Your Time. Ryan says that you can avoid all these problems by making mob equipment worthless. “Waah!” Right, let’s overburden the item database with totally worthless shit that nobody wants. They might as well just drop melons.

I’m considerate. I don’t want to waste your time. I also don’t want to overengineer the hell out of a system unless there’s a damn good reason, and letting you kill a guy to take his stuff isn’t good enough. And finally, I don’t want to duplicate data and make new treasure tables unless there’s an even better reason, because that work just sucks.

Sorry.

I got married a week and a half ago, and I spent the next week honeymooning in the Caribbean. I’m back now.

Check out wowecon.com, an extremely cool web-based auction house tracking system. I love the mouseovers, where they show you the vendor price and current auction house price for the item in question. And their visualizations are great.

One of the benefits of running the most popular MMO in the world is that you get a lot of people doing work for you for free. When it comes to metrics, PlayOn’s analysis is fantastic, and Blizzard doesn’t have to do anything for it. Thanks to wowecon.com and the auction house system, Blizzard doesn’t have to do any internal price tracking. (Those of us without auction houses still have to jump through a lot of hoops.)

I hope talented people do free work for me someday.

Danah Boyd uses the power of actually looking at data to rip Comscore a new asshole. Excellent!

Danah has done very interesting studies on social networking phenomena. I especially love this analysis of why all the kids these days are on MySpace. We shall talk about that another time!

Earnest Adams has written a piece for Next Generation on “50 Books for Everyone in the Games Industry.” I have a few additions.

See, when I was in high school, I wanted to go into politics. I wanted to be a public policy advisor type, one who pontificated about how things should be. It turned out that college-level political science and economics, at least at my school, were all about math and charts, and at the time, I wasn’t big on math and charts. I switched to an English major and merrily read some Derrida.

Several years later, I ended up pontificating about how things should be, only it was even cooler — I was dictating game rules, where I really define how things are. And it turns out that all that math and all those charts are really interesting and useful. I think I learned a lot about problem analysis with all that literary theory, but I now wish I’d moved forward with the econ department.

Now that I’m trying to catch up, I’m reading economics books and finding them tremendously instructive when it comes to online game design. My job, as an MMO designer, is to get groups of people to do what I want them to do. Economics is the study of human behavior in response to incentives. Bingo!

The new breed of pop economics books have been the best game design books I’ve read. A few you should consider:

• Freakonomics, by Steven D. Levitt. This is the one that everybody talks about, and it’s for a reason — it’s accessible, easy to read, and looks at interesting problems. I’d like it better if every chapter weren’t prefaced with a quote about how awesome the author is, but hey.
• The Armchair Economist, by Steven Landsburg. This one’s a little older, and has a little more hard econ work. If Freakonomics is a discussion of issues through an economic lens, Armchair Economist is a discussion of economics with issue examples.
• Hidden Order, by David D. Friedman. This one has charts and formula, so it’s quite a bit more hardcore than the others. I must admit that I haven’t finished it, but it appears to be in much the same vein as the prior two books, and is thus probably pretty useful for design.
• Naked Economics, by Charles Wheelan. Naked Economics is a straight-up econ book, unlike the others I’ve recommended. Nonetheless, it’s still very accessible and fun to read. I enjoyed it quite a bit.

Later edit: for an example of applying economic analysis to something that doesn’t have to do with, like, the GDP, check out Marginal Revolution’s take on housecleaning. As usual for that blog, don’t miss the comments.

Signal vs. Noise talks about how “Netflix nails the customer experience. I’m interested in two of their points: that Netflix does great “interactive emails” (when they request a survey response, it’s a one-click deal), and that rating movies on the site itself is quick and easy.

Netflix is sitting on a mountain of data. I mean, they’re giving a million dollars away to whoever can process that ratings data better. (I thought about downloading the data, but then I realized that I don’t have a machine that could handle a database with a hundred million records in it, and people a lot smarter than me are working on the problem. In fact, one of them is already winning. So much for my BMW and swimming pool.)

MMOs sit on mountains of data too, but players generate it merely by playing. I’d love to augment that hard data with more subjective evaluations — survey responses, actual textual feedback — but that’s hard. In that respect, a good community manager is the best tool we have.

I won’t say that we can do better than good community managers and their skills at reading and interpreting player posts, but we can further augment that data with well-run surveys. Netflix gets it right online.

Current game survey systems are clunky. Some games have surveys at login. Why would I fill out a survey at login? I’m trying to log in. I have actually playing the game on my mind.

Some games have surveys in game, which is a step better. City of Heroes’ famous beta quest reports are the best example I’ve seen so far. During beta, when you finished a mission, a window would pop up and ask you to rate it on a couple of scales. Like Netflix’s, it only took a few seconds to complete. I don’t know if Cryptic got good results from it — maybe players tended to hit all the 5s or all the 1s — but I suspect it was pretty damn useful.

What if we want to know more than that? Knowing how a player feels about a particular quest is fantastic, but what if we want to know how they feel about the game in general? At AGC this year, Rich Vogel said that one of the most important things to ask beta players is “would you recommend this game to your friends yet?” Well, how do you find that out? Run a little survey on the boards? Fuck that. It just tells you whether or not the guys who post on the boards like the game. In any survey, the sample is overwhelmingly important.

The answer, I believe, is in passive messaging. The most prominent current example of passive messaging is World of Warcraft’s tutorial system, but it can be used for so many other things.

I’d have a passive message pop up during downtime and ask if the player has a second to fill it out. Make sure it’s small, quick survey with a light UI, so they can see if they get attacked and can bail out if necessary. When the system selects players to ask, it can be sure that it’s asking a player from the appropriate sample (make sure you get representatives of every class, for example, and if possible — though I know it’s tricky — it’d be great to sample by playtime, too). When you look at the results, filter by those categories. “Hey, priest players really hate this game!” Fix that class. “Huh, people don’t start to recommend the game until they’ve played for 10 hours.” Fix your newbie experience.

You could run class-specific surveys and compare the in-game responses to the reports on the boards. You could compare survey results from different times of the day, to see how in-game population affects responses.

All in all, you could run your game with a whole lot more insight into how your players — the ones who are actually logged in right now — really feel.