We Can Fix That with Data

Signal vs. Noise talks about how “Netflix nails the customer experience. I’m interested in two of their points: that Netflix does great “interactive emails” (when they request a survey response, it’s a one-click deal), and that rating movies on the site itself is quick and easy.

Netflix is sitting on a mountain of data. I mean, they’re giving a million dollars away to whoever can process that ratings data better. (I thought about downloading the data, but then I realized that I don’t have a machine that could handle a database with a hundred million records in it, and people a lot smarter than me are working on the problem. In fact, one of them is already winning. So much for my BMW and swimming pool.)

MMOs sit on mountains of data too, but players generate it merely by playing. I’d love to augment that hard data with more subjective evaluations — survey responses, actual textual feedback — but that’s hard. In that respect, a good community manager is the best tool we have.

I won’t say that we can do better than good community managers and their skills at reading and interpreting player posts, but we can further augment that data with well-run surveys. Netflix gets it right online.

Current game survey systems are clunky. Some games have surveys at login. Why would I fill out a survey at login? I’m trying to log in. I have actually playing the game on my mind.

Some games have surveys in game, which is a step better. City of Heroes’ famous beta quest reports are the best example I’ve seen so far. During beta, when you finished a mission, a window would pop up and ask you to rate it on a couple of scales. Like Netflix’s, it only took a few seconds to complete. I don’t know if Cryptic got good results from it — maybe players tended to hit all the 5s or all the 1s — but I suspect it was pretty damn useful.

What if we want to know more than that? Knowing how a player feels about a particular quest is fantastic, but what if we want to know how they feel about the game in general? At AGC this year, Rich Vogel said that one of the most important things to ask beta players is “would you recommend this game to your friends yet?” Well, how do you find that out? Run a little survey on the boards? Fuck that. It just tells you whether or not the guys who post on the boards like the game. In any survey, the sample is overwhelmingly important.

The answer, I believe, is in passive messaging. The most prominent current example of passive messaging is World of Warcraft’s tutorial system, but it can be used for so many other things.

I’d have a passive message pop up during downtime and ask if the player has a second to fill it out. Make sure it’s small, quick survey with a light UI, so they can see if they get attacked and can bail out if necessary. When the system selects players to ask, it can be sure that it’s asking a player from the appropriate sample (make sure you get representatives of every class, for example, and if possible — though I know it’s tricky — it’d be great to sample by playtime, too). When you look at the results, filter by those categories. “Hey, priest players really hate this game!” Fix that class. “Huh, people don’t start to recommend the game until they’ve played for 10 hours.” Fix your newbie experience.

You could run class-specific surveys and compare the in-game responses to the reports on the boards. You could compare survey results from different times of the day, to see how in-game population affects responses.

All in all, you could run your game with a whole lot more insight into how your players — the ones who are actually logged in right now — really feel.