Metacritic, Part 1

There has been quite a bit of sturm und drang in the video game world over Metacritic and the role it plays in the industry. For those unfamiliar, Metacritic is a site that aggregates the review scores of many different review sites and aggregates them into a "metascore" (through some non-public method of weighting) that is meant to represent a consensus opinion on each game. In theory, it is using a wisdom-of-the-crowd method to determine how "good" a game is in a way that eliminates the biases and quirks on individual reviewers.

One of the criticisms Metacritic gets caught up in is the value of review scores themselves. How can you possibly represent the value of a videogame in a single number, given all of the things which affect that number? I won't be addressing that here, as it applies far more to individual review sites' decisions to use scores in the first place and philosophical questions about how to judge art. Also, people are free to ignore those numbers if they feel that such scores are useless.

I'm far more interested in Metacritic's place in the business side of the videogame industry.

The videogame industry shares many structural similarities to the tabletop game industry. There are developers (the equivalent of tabletop designers) and publishers. Developers create the game and publishers pay the bills (though the timing of payments differs between the two industries). Publishers pay developers some combination of fixed amounts for meeting design milestones (in tabletop this would be getting the game contract-ready) plus royalties based on sales.

This is important because these developer-publisher contracts have more and more frequently been using a game's Metacritic score as a basis for determining some amount of the developer's final payments. Specifically, some bonus payments will be withheld unless the game achieves a specified minimum metascore.

This has caused a subset of the community to be angry at Metacritic. They feel that it unfair for developer's livelihood to be controlled by something so opaque and arbitrary. They point out that review sites don't all use a 0-100 system and that Metacritic's conversion of, say, 3/5 stars to a score of 60% doesn't represent the true opinion of the reviewer. The inability to know which reviewer's scores will be weighted how is attacked as making it impossible for a developer to ensure that their hard work will be rewarded. The fact that a single reviewer's giving a game a low score for what Metacritic-critics call incorrect criteria (such as messages conveyed by the game's story or art) can push a game's metascore below the contractual threshold is a common nightmare scenario raised.

I've simplified this a lot. If you want a fuller example of the criticisms of Metacritic, I would point you to the below video by Totalbiscuit (one of the most popular YouTube videogame reviewers), or to this post of his.

In the next post, I will share my thoughts on the subject, leveraging my zero years of experience as a video game developer coupled with my zero years of video game reviewing.