The Problem With Numbers

People are impressionable. That is one statement that can’t be understated. There can be a four-page review, highlighting every single emotion felt during a forty-hour campaign, but most people will skip to the little number at the end of the page. Why they do that is simple; there is this notion that you can understand the ins-and-outs of a game by simply seeing its final score, and comparing it to the scores of other games. This is among the biggest problems in the industry, and this is why:

Misrepresentation of the Scale

Metacritic exists so people can get a sense of the overall reception of the product (in this case, a game). But this already creates a problem. Different review sites have different scales. Some sites use a 5-star scale, which is then translated in a manner that it was not meant to. Other sites (that share the same number system), treat the numbers differently. Metacritic is pushing all these variables together into one pool of reviews. The outcome is questionable at best.

But that isn’t the biggest problem with the grading system. The out-of-10 system has a bigger problem than Metacritic’s relevance in the industry. Instead, the problem is with how critics abuse the scale. As a result, readers are growing into the same trend. Let’s take a look at one of my favourite publication sites; IGN. God of War: Ascension was given a 7.8; .2 away from what IGN considers “great”. That means that the review should be positive, since it has been awarded with such a high number. It implies that the game is really good, right? Reading the review, editor Alex Simmons thought otherwise. I’ll let you read the closing paragraph yourselves.

“Unfortunately it’s a criticism that can be aimed at Ascension as a whole – it’s an enjoyable game, but one that doesn’t quite live up to the pedigree of its predecessors. The story, which promised so much by revealing a more emotional side to Kratos, stumbles along and is so flimsy it’s almost incidental. The combat has been refined and is therefore as satisfying as ever, but the lack of any really memorable set-pieces – combined with its tiresome reliance on carbon copy battles against waves of exactly the same enemy – result in a game that is at times spectacular but all too often forgettable. Kratos fanatics will no doubt revel in the challenge it presents, especially the gruelling Trials of Archimedes, but there’s no escaping the fact Ascension is the weakest in the series.”

Does this sound positive to you? To me, the game comes off as a disappointment, and not worth my $60. In the most recent Podcast Beyond, a fan writes in complaining about the score. He implies that the score was, essentially, terrible. If that’s the impression given from an almost-8, what should we think of 6s? The same response was given with many games that received an 8 or lower; such as Gravity Rush, Resident Evil 6, and Assassin’s Creed Brotherhood.

While this may just be my outspoken opinion, I believe that the scale should be far more strict. An enjoyable, memorable game shouldn’t be a 9; it should be a 7. But that’s the problem; it is my opinion. Clearly, we can’t all agree on a universal scale. And this takes us back to the original point. We use numbers to compare, but the numbers are hardly comparable.

angry

I’ve waited months for this game, and It only got an 8.5?

The best review site that I’ve come across is Kotaku. Instead of numbering their scores, they have an entirely new system for people who want a quick look at the page. They score it with a “Yes” or a “No”. If the game is worth playing, it is a “Yes”. If not, a “No”. Isn’t that why we should be reading reviews anyway? To find out if its worth our money. They even have a “Not Yet” score, which addresses another huge concern with numbered reviews. And that is…

The Ever-Changing Industry

It is no secret that the next generation is going to have an online focus. Taking a look at one of the most documented launches of the generation, SimCity is a good example of what is to come. “Always-on DRM” is annoying, but it is also here to stay. If you have lived under a rock for the past few weeks, then you wouldn’t know that SimCity experienced significant server issues upon launch. A week later, when reviews started to surface, the numbers were fairly low. Destructoid gave the game a “4” in their written review. But that is the problem. Let’s take a look at the final paragraph, once again.

“I wanted to like this game, I really did. At first I started to enjoy it, but soon all I found was frustration. I can’t recommend this game to anyone, and I don’t want to play it anymore myself because I am afraid of seeing all my efforts lost due to server issues. It’s a decent game if it worked right, but the online dependency, forced multiplayer, and DRM ruin it.”

If you take a look at the entire review, you will conclude that the heart of the issue is in the server-dependancy. A “decent game” does not receive a 4. But what do we take out of this review now, since the servers are working better than ever? I play SimCity with absolutely no problems, and I have been for about a week. The reviewer was obviously frustrated with the server issues, but that problem is in the past. This review is officially outdated, and has been for the past week and a half.

revyes_01

Because sometimes its as simple as a Yes or a No

We are in an industry with DLC and constant patch work. The game at launch can be an entirely different game in two weeks (as it was with SimCity). The review scores, or its Metacritic, however, don’t reflect this. Placing a number at the bottom of the review makes it contemporary. If the game doesn’t work right at launch, Kotaku’s “Not Yet” makes a lot more sense, doesn’t it? Especially when “It’s a decent game if it worked right”.

Paid Reviews

As a regular reader of many press sites, it hurts to know that they are constantly betraying our trust. We get to know these editors; who they are, what they like, and why they like it. We trust them with our money, so it hurts when we hear reports of paid reviews. That’s exactly what we heard about a week ago, though. In this article, Play4Real reports on an Ex-IGN employee’s confessions. He remains anonymous, but considering the number of layoffs over at IGN recently, it is somewhat believable. This employee has a lot to say, and it is worth reading the original article. To sum it up, he or she says that games have secured numbers that have been paid for. The Last of Us, which is due out in June, has secured a perfect score. This is the full list leaked:

Grand Theft Auto V: 10/10
Metal Gear Solid V: 10/10
The Last of Us: 10/10
BioShock Infinite: 9/10
Assassin’s Creed IV: 8.5/10
Watch Dogs: 8.5/10
Beyond: Two Souls: 9/10
Lost Planet 3: 8/10
The Wonderful 101: 6/10
Pikmin 3: 8.5/10

Looking back, how many games have been “adjusted” with some extra cash? Grand Theft Auto IV is almost a dead give-away, with its hardly deserving perfect score. IGN isn’t the only site that does this. Reports are scattering that the majority of popular press sites make these agreements. And this is why so many people are angry about IGN’s exclusive review of Bioshock, which will be posted before the embargo date is lifted. People will flock over to IGN to read the first and only review of Bioshock Infinite this week. Either 2K Games is very confident in Bioshock’s critical reception, or the above list is somewhat accurate; it can’t get below a 9.

1252866429IQ5RZf

What score does this buy me?

I am not saying that Ryan McCaffrey, IGN editor who is in charge of the infinite review, is not creditable. I am not saying that it will be a bad game, despite its inevitable high score. In fact, I generally trust his reviews and rely on them before purchases. I also generally trust Ken Levine, director of Bioshock Infinite, with making great games. Even so, there is reason to be skeptical of some reviews. It is not the editor, but the parent and publishing companies.

So What’s Next?

Even I am guilty of scoring my reviews, but that is about to change. From now on, inspired by the excellent review system used over at Kotaku, I will keep my reviews entirely written. If one wants to simply glance at the review, they can read the closing paragraph, which will nicely sum up my thoughts on the game. Why? Because numbers are misleading. They are being used to manipulate us, and they don’t properly illustrate the evolving state of the industry. I think its time to think twice. Its time to take this more seriously. I’ll leave you with this to think about, from the aforementioned Ex-IGN employee;

“Unfortunately, a typical reader of IGN only cares for the number at the end of the review,” the employee laments, “They only write the review because it’s necessary. If they could, they would rather just put up a score number instead of a review and call it a day. This is how stupid video game culture is. This is how stupid IGN thinks you are. And sadly, no one thinks twice.”

 

EDIT: I’ve been informed that the source in the “Paid Reviews” section is not legitimate. Regardless, I still believe it is a relevant argument. Thank you “V” for the correction.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s