Reviews: What Level Am I?

I just updated my Twitter feed with a snarky comment about MMO reviews. Then I realized I had more to say than the Twitter window allows.

The source of this morning’s grumbling was a reporter friend of mine. I love him to bits, he’s a smart guy, knows the genre, really enthusiastic, and above all, experienced. He was telling me about one title avoiding the usual MMO pitfall by having the endgame in place. And then he told me he hasn’t personally gotten past 18. Basically, he is taking the studio’s word for there being an endgame at all.

ARGH! No! Even if the studio is telling the truth (and the studio would have to be pretty stupid to lie, considering it’s the kind of lie that will be revealed within thirty days of launch), you can’t take the studio’s word for it and still call yourself a reporter. Printing anything without analysis and personal experience (or at minimum in-depth research) makes you a member of the White House press corps, circa 2002.

Worse than absorbing and regurgitating the party line is reprinting without comment the kind of fluff handed out at press tours and trade shows. Just because you got a tote bag, a sticker, and an early beta key does not mean you are obligated to print the things on the handout someone stuffed in the tote bag.  Also, screenshots on the disc the studio gave you for your convenience? That’s nice, but if you weren’t allowed to take your own screenshots, congratulations, you’re doing the marketing team’s work for them, not reporting.  It’s fine to use the official shots in the early days of a product’s cycle, when there is no playable version for you to use. But if the studio wants to control what is being posted in the final days before launch, you should be asking questions, not glumly following orders.

Reviewing an MMO, especially if the open beta period is short and controlled, is very difficult. The studio cannot allow you enough time to play the endgame, not and have a prayer at finishing the endgame. The usual dodge is to boost your level, outfit you in gear, and throw you at the final scenario. That’s fine. But you are obligated to tell your readers how you accessed top level gameplay. And you owe it to your readers to ask your handlers what you are wearing and how the normal player will acquire each specific item.

Finally, I think MMO reviews would be better if the headline contained the level that the reviewer had achieved without bonuses, benefits, or boosting. Don’t make me wade through all the euphemisms for “well, I didn’t see it myself, but I heard you can do X.” Tell me how far you got, tell me how you got there, and tell me how you enjoyed the journey. Otherwise, take “reporter” off your resume and replace it with “press officer.”



  1. TPRJones said,

    September 5, 2008 at 4:10 pm

    “you can’t take the studio’s word for it and still call yourself a reporter”

    That’s SO last century. Modern journalistic standards do not include such outdated concepts as fact-checking and investigation. When’s the last time you watched MSNBC or FOX News?

  2. September 5, 2008 at 4:52 pm

    […] Posted by Kendricke in The Gaming Industry. Tags: game reviews trackback Sanya Weathers put up a great commentary over at Eating Bees today regarding MMO reporters who review game content they haven’t personally experienced.  […]

  3. Elovia said,

    September 5, 2008 at 5:31 pm

    Irresistible force meets immovable object. Hype vs Marketing. Hysteria vs Persuasion. Reporting vs Journalism. First time poster, long time reader.

  4. Joho said,

    September 5, 2008 at 7:05 pm

    To be fair to the poor reporters, MMOs can routinely take 10 days played to reach the endgame. Assuming this is work for the reporter rather than something done in their free time, that’s a month of 8 hour days. No reporter has that kind of time available for something as basic as a review. Even if they did that, they’d have played only one class and faction, a small part of the total game.

    Book or movie reviewers have the luxury of experiencing the complete product before writing their piece, but it’s simply not practical for game reviewers.

  5. sanyaweathers said,

    September 5, 2008 at 7:48 pm

    Sure. But that’s rather my point – tell me how far you got on your own steam, and clearly delineate which experiences you had in an assisted hothouse environment. At this point we all know the reporter didn’t get to 50 on his own in the lousy week he was granted, but it seems to me that transparency demands that we as readers know what exactly the reporter is reviewing. Let me judge the review in the context of how much is being reviewed.

  6. Apache said,

    September 6, 2008 at 2:13 am

    Pretty sad

  7. kfsone said,

    September 6, 2008 at 4:37 am

    A while ago, Battleground Europe got a showing on G4. They did not like it, their review consisted of showing lots of scenes of people doing not a lot. Unfortunately, it’s an all too common experience, but it appears that they decided to judge it by FPS genre standards and it is a boring single-player FPS. It’d be like reviewing DAoC RVR by running a level 5 over to the battlegrounds and filming people heading in and saying “this is ‘entertaining'”.

    But it just validated by ongoing opinion of “computer journalism” which is that they should all just be honest and rename themselves “blag-oriented marketing reworders”. I can’t remember the last time I saw real, honest to god, investigative computer reporting that qualifies for the term “journalism”. Take “Toms Hardware” which is “Tom’s Sale Pitches and Marketing Confusion about Hardware” these days. The most investigating I’ve seen in a computer magazine or forum is heavy use of to find the least recognizable words to replace the advertising brochure the journo is basing his write up on.

  8. September 6, 2008 at 6:39 pm

    “well, I didn’t see it myself, but I heard you can do X.”

    Wow, you just summed up every press release ever. And this doesn’t apply to gaming really, but nearly all world news. kfsone summed it up nicely as well.

  9. Merlyn said,

    September 6, 2008 at 7:01 pm

    When’s the last time a review of a game was really accurate? Most reviews come out when a game is released and are based on a hurried playthrough of a pre-release beta. I take anything said by most game reviewers with a grain of salt anyway…especially about end-game.

  10. Apache said,

    September 6, 2008 at 7:30 pm

    I’m guessing the person in question doesn’t have a very tough editor.

  11. Frank said,

    September 7, 2008 at 1:52 am

    This is honestly why you can’t take reviews at face value, ever. I think the point applies to players who read the reviews and not just the reviewers themselves. Too many times these days, you see reviewers laying 10’s at the feet of many a game, and those that read the reviews also giving it more premature praise than an awkward high school kid on senior prom night.

    Direct, actual experience trumps reviews every time. People should stop overestimating the credibility of reviews and what they bring to the table,a nd that includes companies. Reviews are just opinions, is all, but you’re the one making the choice about whether or not to believe it or see it for yourself, and I tend to take up the latter.

  12. Zaphod said,

    September 7, 2008 at 12:51 pm

    I don’t mind all that much. Every MMO review I read is taking fairly lightly, and I’m really only looking to discover one thing… will this game pass my 10 minute rule.

    Since my days in Everquest, I have an extremely low tolerance for bad game design and wasted time. To make certain I never suffer through another game clearly designed by people that actively hate their players, I created the following rule:

    If I walk/run/do nothing for 10 minutes in a row in a game, I uninstall it and throw it away. Period. No second chances.

    Naturally, I don’t want to buy games that are going to head straight for the bin, so I tend to wait before buying, and take in a number of reviews. If perusing regurgitated press releases is the only way to get the information, I’ll take it.

  13. Elovia said,

    September 7, 2008 at 5:29 pm

    So is the cop out to poor or inadequate reviews (think of the reviewers as cheerleaders at a lemming stampede) … is the cop out the all too familiar disclaimer “game experience may change during online play” ?

    It is far too easy to review a game while isolated from the hundreds of kiddies, d00ds, and gank-squads with whom you will be required to play (these, took, may influence the overall fun factor). How many reviewers actually review the game playing community along with the game itself? Or submit a more thorough follow up review six months after release? Screenshots of graphics and user interfaces, and explanations of combat mechanics can only provide reference to a small portion of the game. At pre-release, I’m sure we can all agree that, reviewing the community is near impossible unless the game caters to an extreme niche, and even then assumptions must be made.

    On the other hand, I believe what Sanya is trying to unsuccessfully argue for is the need for some standard of playability by which the reader may judge a game without actually having to experience it firsthand. But therein lies the rub. How do you generically describe a learning curve applicable for a broad swath of game playing population? If it took Player A only a few hours to reach level 20, how long will it take Player B? That, of course, depends on the differences between the two players: their time commitment, their attention to clues/mechanics, and their goals.

  14. Elovia said,

    September 7, 2008 at 5:31 pm

    *blush* … “took” = “too”

  15. Apache said,

    September 7, 2008 at 8:10 pm

    You can in theory write a good MMO review on launch day. Provided you’ve been playing the beta for a few months, have your characters all maxed out, and the test server you were on was lively enough to be able to thoroughly sample the end game and pvp aspects of the game. Plus, screenshots and movies or it never happened. 🙂

  16. Staleek said,

    September 8, 2008 at 2:08 pm

    Long Time Reader (All the way back to IGN DAOC Forums) First Time Poster:

    The most honest review of any game in any genre I believe is Zero Punctuation.

    If you get over his language and crudeness, he actually takes games to their final level and describes the end game play. This is why he rarely reviews games pre-release. He likes the full copies of games so that he can play them to the end and comment on them correctly. To me he is a true gamer because he actually takes the time to play the games.

    I still agree with Sonya, in that it’s next to impossible to review end-game play without actually playing it. 🙂

  17. neofit said,

    September 10, 2008 at 12:22 pm

    Actually, a “well, I didn’t see it myself, but I heard you can do X.” concerning the end game is good enough for me. Because I know that I will never reach said end game before the end of the free month (and if somehow I do then it must a relatively content-free game like AoC, so I’ll quit anyway).I want to hear how this game plays in the levels that an addict with a day job that starts at 7:00 AM can achieve in the free month :). Who cares how the game will play like in, say, 3 or 6 months? I’ve played all the western games since EQ1, some bored me before I reached the end game, yet I consider that I had enough fun to be worth the price of the box.

  18. September 10, 2008 at 11:32 pm

    […] More recently I pre-ordered Warhammer Online, on reccomendation from members of my Guild. Let’s face it, I’m going to value the opinion of people I have played with for years more than any post-release review. […]

  19. Jason Ballew said,

    October 1, 2008 at 7:51 pm

    I’ll be honest, I agree with a lot of what Sanya said, and I’m primarily a MMO reviewer.

    We’re doing our Warhammer review a bit differently (and it’ll be out next week). We’ve got two people reviewing it, myself and another person. We’re going to write our review based on what we’ve played so far….and then do monthly followups. MMOs are almost impossible to fully review simply because of the amount of content available, and the fact that they NEVER END.

    Thankfully, Mythic was nice enough to give us press accounts, so that we don’t have to put money into it (which can limit what we’re able to do, honestly, being freelance and not paid journalists), so we’re willing to keep the writing up. But I think this may be the best way to do MMO reviews: On a rolling basis. Obviously, the scores don’t change after the initial review, but if you’re doing monthly articles, people can see if the game starts to tank.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: