Evidence-Based Design

When people look at web pages, what are they really looking at? Give Eyetools some money to answer the question for you. They’ll do a little study and send you some cool visualizations.

Check out the heatmap, which reports the behavior of many users. The “individual session images” are equally fascinating.

They do all the work at their office. I guess that’s so you don’t get too scared of the tools. Unfortunately, it reduces their outreach into fields outside of web design — like, say, game design.

UI usability is normally totally ignored. If Microsoft is your publisher, they’ll send your game through the Usability Lab (and if you live in Seattle and are suitably multicultural, you can be a tester, too!). If you’re anybody else, you’re lucky if a single designer on the team gives enough of a shit to say something about it.

Eye tracking is important. Minimal eye movement should give the player access to all the information they need for the task at hand (during combat, for example, maybe your power cooldowns and the opponent’s health shouldn’t be on opposite sides of the screen). Unfortunately, without the crazy Eyetools lab, you’re not going to have any hard data on that front.

Intuitiveness is something that you might have more luck with. Is it intuitive to find your character’s inventory the first time you log in? You’d need some way of tracking clicks (log each open-HUD event, for example). Alternatively, send the peon junior designer into the newbie chat channels during the first few days after launch, and find out what people are asking about. They probably won’t go to the boards to say that they can’t find the inventory. They’ll ask in chat and wait for a beta player to answer.

To validate the default UI layout, you need to know how players have rearranged theirs. This requires you save UI options on the server — probably not a bad thing anyway, as it means they don’t have to arrange their UI just right every time they log into a new machine (just be sure you account for potentially different resolutions). You get a mountain of fun data to play with, too.

Option C is doable by just about every average MMO. Why don’t you?

Over two years ago, Raph Koster published some Star Wars Galaxies economic metrics on the game’s official website. The post disappeared some time ago, and I was pleased to find it online today.

I’m always interested in learning about other MMO developers’ use of metrics, considering that most everybody I talk to just looks at me funny when I ask what they’re tracking. Back then, SOE was at the forefront of innovation. (A year later, I developed a system for Shadowbane, and while it was pretty cool, it was just a designer hack. It’s an unfortunate commentary on our industry that it was probably the best thing going at its time.)

Even though SWG’s metrics system was the best thing going at its own time, Jeff Freeman revealed that it wasn’t perfect in a comment at Terra Nova. When Ed Castronova noted that “300 accounts (out of 5,000 or so, I am guessing) have 95 percent of the wealth,” Jeff told us that “we continue to track money in cancelled and inactive accounts.” That’s a really basic, easily avoidable mistake.

Their metrics don’t seem to account for the value of goods to players, either. I don’t know that anybody’s tracking that right now, outside of the World of Warcraft Auctioneer mod. (Go read that Auctioneer link; it’s a Wiki article written by somebody who actually knows what he’s talking about. Very cool.)

If you have basic trade logs (and you should, because they’re really important for verifying customer service claims), it’s not hard to track the value of goods. Let’s assume that you have a basic item game, where items don’t really have unique stats. A Vorpal Sword of Bunny-Smashing is just like every other Vorpal Sword of Bunny-Smashing and so on.

Do a query on your logs database (if your logs aren’t in a database, do whatever it takes to get them into one — they’re only about a million times easier to use that way). Select trade transactions and group by item received, and print whatever was traded for them. Stick it into Excel and have some fun.

Of course, players often trade items for items, rather than items for cash. Items for cash is easy. Items for other items means that you need to know what the relative cash value for that item is, and you can’t rely on the item’s “value” stat. In some cases, you can — Shadowbane players would use a high-cash-value item to stand in for cash during trades to avoid a gold limit, for example. In other cases, I might trade my Vorpal Sword of Bunny-Smashing for an Uber Mace of Crab Destruction, and we need to determine the player value for the Uber Mace of Crab Destruction first. We can’t programmatically distinguish between the two.

To make our first iteration easier, let’s filter our query for cases where items were exchanged solely for cash. Graph it. It’s probably a bell curve. Save your median as the average value of that item, and do the query again without that where clause. Substitute the items-traded-for with the cash values we just determined.

Now you know how much items are worth to players. Hooray! Save that data and track it over time. There are probably some surprises — hey, that item has a typo in its stats, so it’s worth a lot more than it should be. Fix your bugs. Have fun. Enjoy being more informed than any of your contemporaries.

No matter how popular a game may be, someday, nobody will play it. Someday, World of Warcraft will need to consolidate its servers. In other cases, the game is never popular in the first place. More than one MMO company has been disappointed in a new title’s sales and subscriber counts. Long before launch, design needs to account for both ends of the spectrum.

Lord of the Rings Online just announced a system where players can play monsters, one of those ideas that’s been kicking around MMOs forever. It’s neat to see somebody finally try it. One part of the announcement gives me pause, though:

Players, on the other hand, will have optional series of quests that will pit them against monster players, giving them reasons to go out and mix it up. PvMP is optional and consensual – you won’t “stumble across” a monster player. PvMP will be allowed in specially delineated areas of the game.

What happens if nobody shows up? What happens if the game is massively popular? What happens five or ten years from now, when the game’s population naturally and inevitably declines? Will there be enough players who want to be monsters to populate those quests?

I don’t mean to pick on Turbine all day, but a great example of a cool system that failed because it didn’t scale well was the crafting stations in Asheron’s Call 2. To craft most items, you needed to stand next to a crafting station — if you wanted to blacksmith, you had to stand next to a forge, etc. By virtue of standing next to a forge, your crafting skill would get a little buff. You could put resources into the station (ingots into the forge, etc.) to increase the buff, but the resource cost was so high, it ensured that a number of players would have to band together and donate. It encourages crafters to work together, to make friends, and to congregate.

I thought it was a very cool system, but it relied on a certain density of crafters. If you’re the only one at the station, you’re not going to waste your resources on it, because you can’t possibly afford the cost. If I remember correctly from beta, it took five or ten players to fill up a station.

In live, sadly, I never saw that many players around any single crafting station, and the system went unused. AC2 was shut down in December of last year.

Signal vs. Noise links to a Forbes article about why Project Runway is about more than fashion design.

Project Runway celebrates creativity and innovation. It also conveys tough-minded lessons about work in a competitive, market economy.

Forbes goes on to talk about how the show works as a business and marketing example: “merit and subjective judgment coexist. […] the marketplace doesn’t measure value on a single dimension.” The Signal vs. Noise guys go on to explain how Tim Gunn should really be a Unix developer: make it work first, then make it work well.

Like the fashion competitions in every episode of Project Runway, game design — especially live MMO design — is about taking what you have and making it work as best you can. During development, they just made it work. Now, you have to make it work well. What problems do you need to solve right now, and how can you solve them with the most efficient use of limited resources?

Design docs in the development phase often come with a “Goals” section at the beginning. What is this system setting out to do? One of your quest system goals, for example, would be “give players something to do.”

During live development, I like to replace that section with “Problems Solved.” How is this system going to solve an existing issue? If you added a quest system after launch, one of those problems solved would be “players don’t have enough to do.”

It’s still the same bullet point, but it puts your job into perspective. In live, you don’t do things just because they’re cool; you don’t have time. There’s shit to fix. Get it done, and carry on.

How do you decide whether emergent gameplay is an exploit or a “creative use of magic?”

In baseball, it’s against the rules to intentionally hit a player with the ball. It still happens frequently — so frequently, in fact, that a recent California Supreme Court 6-1 ruling overturned a college batter’s lawsuit against a pitcher who hit him in the head, causing injury.

[Justice Kathryn Mickle Werdegar] … said it didn’t matter that intentionally throwing at a batter is forbidden by baseball’s own rules. She cited an impressive array of baseball legends — including St. Louis Cardinals manager Tony La Russa, Los Angeles Dodgers Hall of Fame pitcher Don Drysdale and New York Giants pitcher Sal “The Barber” Maglie — who have claimed brushbacks are an integral part of pitching tactics.

The sole dissenting judge noted that “intentionally hitting another person in the head with a hard object thrown at a high speed is highly dangerous and is potentially tortious, no matter whether the object is a ball thrown on a baseball field or is a rock thrown on a city street.” Astute, eh?

Jeffery Standen, a law professor at my hometown’s own Willamette University, writes an interesting blog on sports law. In his articles about this incident, he laments the legal system’s reliance on “grand empirical pronouncements based on anecdote, if based on anything at all.”

The usual approach to questions of this type is for a court to wonder whether or not people in this situation (batting in a baseball game) reasonably “expect” beanball pitches. This is a factual, empirical question, and the court’s answer to this question was derived from anecdote: the court’s opinion recites various instances of beanballs and statements concerning beanballs and concludes that beanballs are within the common expectation of batters. […] what degree of a shared opinion [amongst players] would suffice for a court to decide that beanballs are in fact “normally expected”?

There are two interesting parallels here for online game management:

• Anecdotal evidence — posts on the boards, the developers’ perceived experience — is almost always taken as fact, because the industry, as a whole, doesn’t use metrics nearly as much as it should.
• Rules infractions, if not disciplined, tend to become “part of the game.”

Baseball has metrics. Fans have tracked the “Hit By Pitch” stat for over a hundred years, and a quick glance at Baseball Prospectus shows that the current top 30 pitchers by innings pitched range from one to eighteen this season alone. I’d love to see incidents per game, per team, but I wonder if the California Supreme Court would have cared. There’s no way to objectively tell if a pitcher has deliberately chosen to hit the batter, or if he got unlucky trying to throw an inside pitch.

Let’s say we’re running a live PvP MMO. Players have discovered they can … drop a house on top of a character to move it out of the way. (And really, if you visualize it, it’s about as funny as watching somebody get nailed in the head with a ball.) We can track that! Let’s search our logs for instances of new house placement near locations where lots of characters are dying. Boom — you know who’s doing it and when. What do you with that data?

You can forward the list of offenders to customer service. You can write code that disallows players from placing houses where there are characters. Or you can see that it happens frequently enough, and think it has enough humor value, and really, it doesn’t really hurt anybody, and you can let it go.

How does your opinion change if everybody on the boards is raging about how they hate having houses dropped on top of them? What if everybody thinks it’s fun and creative?

What if you have that player feedback, but you haven’t bothered to check your logs? You have no earthly idea how often it’s happening, even though the guys on the boards say it happens all the time. You don’t know who’s doing it, so even if you want to enforce rules against dropping houses on guys, you have to rely on players to report it in real-time and hope that a CSR’s at work. You’re working blind, just like the California Supreme Court, and when you go back to the players and tell them “well, everybody tells us that dropping houses on people is cool,” you’ll sound just as ignorant.

Sadly, that’s the way it works in just about everywhere.

For fun, here’s a blog about Craig Biggio getting hit by balls. He holds the modern-era record, and has been hit 281 times.