My Dream Patch Day, And The Junk.

This exercise is taking place entirely inside my head, since I don’t know any software company that actually does it this way. Also, since this is MY dream, everything has been set up for the convenience and peace of mind of a community weenie. But I admit… I’m being a bit disingenuous when I say that. See the question at the very bottom of this post.

Up to several weeks before the patch:

I have been getting patch notes as work is completed.

– I am getting ALL of the patch notes, meaning a note for each completed task. The programmers and designers have indicated which notes they do not believe should be made public, and their opinions carry considerable weight, but the actual decision to post or not post is up to me and the producer. If we are not certain why the programmer or designer has tagged a note as Do Not Release, all we have to do is ask, and the answer is provided in complete sentences without the arrogant son of a bitch smirking and saying “because I said so.” However, I have learned over time that the response to such obnoxiousness is not “try again or I’ll punch you right in the junk.” If the person with the information is behaving badly, the problem is one of management – a manager is encouraging this attitude, and without fixing the manager, the problem will merely continue for the next thousand years.

I have been translating the notes into plain English as I go along, and checking with the programmers to be certain I have not changed the meaning of the notes.

When I see a note certain to cause drama, I raise the issue with the producer immediately. The item may require additional research, volunteer input, rephrasing, or outright removal. Honestly, sometimes people are working in a vacuum, and may not have considered the repercussions of their decision. This is rarely because the programmer or designer is stupid, but because he lacks the data he needs to make the right call. My job includes providing that data, if my company does not have many internal metrics. If I am lucky and my company DOES have those metrics, my job is communicating that information with the context of how the users behave. A community person serves the community – and a development team is as much a part of the community as the customers are.

Back to the potential drama. It might be best for the game if we just go ahead and kick the hornet’s nest. But even if I think the producer is forty kinds of wrong, my job is to advocate for change right up until patch day, and then to shut up and post it as though it were my idea. Unlike patch note phrasing (my job), deciding what things go in or out of the game is HIS job. My job may be eating bees, but I’d rather have it than his job, which looks to me like a combination of spreadsheets and pure political ass. Gah. Technically, you probably could pay me enough to be a producer, but I wouldn’t enjoy it.

One week before the patch day:

I receive notice from the producer or his designated minion that there will be a patch at a certain time and date. I immediately post this information to the community in as many ways as I can, including but not limited to the patcher itself, my community web page, the product’s blog, and the marketing website. (The marketing website and the community website are not the same webpage, having different goals, different language, different purposes, and different people controlling them.)

One day before the patch day:

I receive notice from the producer that everything is going swimmingly, and we will indeed be patching on the morrow. I immediately post this information, along with my best guess as to how long the servers will be down.

– I have learned that the programmer’s estimate means “actual server downtime” and he is NOT including the amount of time the servers will be up but not available to players due to testing. I have given up attempting to explain that all I care about is whether or not my players can PLAY.

– I have drawn my own conclusions as to how intelligent the producer is and added or subtracted time accordingly.

– I have cross checked the estimate I was given with customer service and QA.

– I may add an hour or two to the official estimate based on the patch notes, because certain types of changes have not once failed to break the servers in the decade I have been watching MMOs run patch days.

Patch day:

The person in charge of taking down the servers was told in advance what time patch day festivities would kick off, and took down the servers on schedule. He made an in-game announcement thirty minutes prior to doing so, to take care of the players who do not read announcements on the patcher or the web. He did NOT learn about the patch from reading the company website, and if he did, the producer got punched in the junk.

I hang out on forums, and I have forum moderators at my fingertips via IM. Forum traffic is insane, because the players who normally post during the day while they are at work are now joined by the players who normally play during the day. I have prepared a statement about why we do not patch at 4 AM on Tuesdays, why we always “screw over” the same group of players by patching at the same time every month, etc. I have already discussed this topic with the producer, and totally lost the argument, so the topic is closed. I made him help write the explanation, though. I cut and paste from this statement repeatedly.

I address the drama-prone note to the best of my ability. A significant number of players mainly want to hear the explanation and be reassured that we didn’t just randomly nerf the most popular weapon in the game by 50%, but rather, we did it for valid reasons… such as the massive improvement we made to the statistic that had been so pathetic that everyone felt compelled to get that one particular weapon. This explanation is given a test drive first internally, then on the boards, and the final refinement goes on the website and added to the patch notes. Since the population on the boards is a minority of the player base, what is essentially a draft explanation is fine… but they’re so vociferous that they can hammer a piece of coal into a diamond.

I do not collect any feedback whatsoever on patch day while the servers are down, though I might finish getting information from those who played on the test server. All other feedback being sent in on patch day is being based on what people think of my patch notes, not the actual patch. The development team does not need data based on anything but results.

Two hours before the estimated time of return, I start asking questions. Hint: When the programmer in charge stares at me wild-eyed and cannot articulate a response, I add three hours to the estimate and post the new time immediately without pestering him for more information. I also post the reason for the delay. The truth is always better than reposting “we are working as hard as we can” fourteen times. If I have a producer who is not smart enough to recognize the value of truth, it is my job to attempt to compensate, and help the players to see that the developers are in fact working. This can be done by describing the process, the number of people working, or any other details that can be scraped up. I am the only window to the development team that the players have, and parroting boilerplate is not good customer service.

If I am not able to get the information I need, I take it up with the producer after patch day if he is frantic, and on the spot if he’s just sitting there playing a competitor’s MMO.

If the downtime has wildly exceeded even my worst case estimate, I stop posting times and retreat into “as soon (TM) as possible.” I also stay on the job. It is vitally important to the sort of user who is camping the website to see changing time stamps, and proof that I will in fact post at 1:30 AM just to let them know the score. That user is disproportionately influential among his less-anal retentive friends. Also, once downtime has gone six hours or more overdue, the most senior community person is the one who needs to be posting, to prove to the customers that the company is serious. Staying on task during a disaster is why senior personnel get paid the big bucks. Well, buck, but compared to the rest of the team, it looks like a big one. Forcing a low level person to bear the load after bedtime is crappy, intolerable behavior, and that sort of manager should be punched in the junk.

I announce the return of the servers, but I hang around for another hour doing shots with the exhausted and hilariously insane team. I’m still hanging out in case either a) the servers need to come right back down, or b) the servers are going to come back down the following day. Admittedly, if it’s 1:30 AM, I may go straight to bed, but the customer service team has my phone number, and they know they can wake me up to make a post.

After the patch:

I gather followup feedback according to specific, scheduled benchmarks and present it to a producer who has not been drinking recently, and he takes action according to established protocols. Message board traffic is only one data point, and used as supporting, NEVER primary, evidence.

***

That is my selfish, selfish dream. Well, in a DREAM some of those steps would be skipped and we’d go straight to doing celebration shots because the servers are up on time with no trouble, but even on a bad patch day, communication and teamwork can go so well that I think I’m dreaming.

Here’s why I dare to dream: Which of those steps in my dream would not be in the best interests of the producer, the development team, and the customers?

Advertisements

26 Comments

  1. Don Neufeld said,

    December 14, 2007 at 11:25 pm

    I can do you one better: your ideal game servers should be handling the downtime broadcasts themselves at regular intervals, including a popup right when a player logs in if the server is coming down within an hour of their login.

  2. Ibn said,

    December 14, 2007 at 11:54 pm

    You know, what you’re describing sounds pretty much like the average monthly prop for Asheron’s Call during most of the time that I was community rep there, at least once we’d bought the game from MS and were handling everything in-house.

    Of course we had the luxury of a fairly small tightly-knit team as well as the experience of having done this once a month for several years in a row. Some of the notes you reference were things that I started doing out of sheer self-defense (translating check-ins into conversational English patch notes as they come in), others were things that got done because we had experienced Ops people (knowing what our prop schedule was as well as we did).

    I’ve always had a lot of empathy for other MMO community reps — especially the ones who lasted longer than I did. Thinking about the fact that most CR’s DON’T get all the stuff you listed above… damn, my empathy just turned to sympathy.

    (My one disagreement is on the “I don’t want to be a producer” thing. When I found myself having at least a little presence on every aspect of the game — again out of sheer self-defense — I realized I was essentially shadowing our producer. That’s when I realized I could probably go into production full-time when I burned out on OCR.)

  3. sanyaweathers said,

    December 15, 2007 at 12:31 am

    Don: You mean automate simple tasks to remove the possibility of human error!? 😉

    Ibn: Well, there’s no accounting for taste 😉

  4. Makaze said,

    December 15, 2007 at 1:49 am

    “do not believe should be made public”

    Uggg that kills me. If you’re going to punch me in the junk at least tell me in the patch notes, preferably with an explanation. The fact that stealth changes exist and actually exist on purpose is why people insist they’re there every patch whether or not they really are. It also makes the “Oh, that programmer must have forgotten to tell me he made that very large and controversial change” line completely unbelievable even though I’m sure that in reality it does happen.

    With the way players dissect MMOs these days they’re going to find out eventually anyways. By trying to hide things it only leads to a loss of trust and credibility.

  5. Drew Mayo said,

    December 15, 2007 at 1:59 am

    “do not believe should be made public”

    There are always things that shouldn’t be made public. like: “Found a case where a hacker with a packet sniffer and generator can cause 10x attack speed followed by immediate world server crash. This does not prevent the hack but instead flags the account for the fraud team to immediately shadow and ban.” should be delicately reworded by experts in community. Certain details of a patch fix are, er, better left unpublished.

  6. Makaze said,

    December 15, 2007 at 3:08 am

    Of course situations like that should not be disclosed but then I don’t think there’d really be any dissent among the team on that type of issue.

    On the other hand I’ve been both involved in discussions about not including changes because “they’ll never notice anyways” and been that programmer that really did forgot to mention some changes I made. I’ve also been the programmer that “forgot” to mention a change according to the same people that decided not to inform the public.

    Just never on an MMO.

  7. sanyaweathers said,

    December 15, 2007 at 3:29 am

    Makaze, take my word for it – on an MMO (in my experience and in that of a number of my close friends) there are only three options. One, and this is the bulk of the situations: Drew described it for you. Two, it’s something you don’t need to know because it doesn’t affect your character, but you probably want to know. Spawn changes, drop rates, etc. There’s always a note, because this is an easy one for the community weenie to win, but it’s “the spawn rate of X has been increased.” Three, it does affect your character, but there are SO MANY VARIABLES that the developer cannot possibly give you a formula.

    In the case of option three, the community weenie is fighting for as much detail as possible (but sympathizes with the dev), and the developer would rather drink a frosty mug of his own pee than even try (although he’s also a player and wants to be open), because any note he provides is incomplete without you knowing the underlying code of the game, and the one factor he doesn’t mention (because it only happens on Tuesdays in May with a full moon) is going to be seized upon as evidence of stealth nerfing. A compromise is usually reached.

    It’s not done with malice or any intent of hiding anything. I sometimes wonder if the loss of trust is a paranoia issue left over from when devs DID hide things because they didn’t realize how… skilled at reverse engineering the players were.

  8. lxsli said,

    December 15, 2007 at 3:30 am

    I wish I was a dev at your game. I’d be quite happy to follow that process.

    The question becomes more motivating in the positive. How does this process benefit every member of the team? It’s much easier to sell someone a process which directly helps them, rather than just doesn’t cost them much.

    As a (non game) coder I spend about 80% of my time trying to figure out exactly what people want. If you could help me gather an accurate and precise understanding of current perceived issues, I’d be all over you. Ofc I’d be taking input from the Test team and architects as well, but in WoW at least it often seems the top-end players have a much better grasp of the synergies and scaling at play than the devs.

    Ideally, and admittedly this is something of a Holy Grail, it would be possible to draw a connection from complaint to requirement to design to patch to verification.

    If you’re hiring, y’know, feel free to ask… 😉

  9. lxsli said,

    December 15, 2007 at 3:31 am

    Ahaaaaaa! I guessed those little boxes correctly!
    In case anyone else is wondering, check out gravatar.

  10. Dartwick said,

    December 15, 2007 at 2:22 pm

    I think things like the old evade formula in DAoC demonstrate one of the biggest problems devs can have with changing mehcanics in a game.

    -No one ever released the formula officially
    -the formula was complicated
    -the formula had a huge impact on game play

    It was a recipe for disaster. Although I have always suspected that if you did take the time to release this complicated forumulas on some forums at least the word would get around to all us uber geeks and it would prevent a lot of gnashing of teeth in the long run.

    But it probably is a good idea to keep something like x=[(b/(500-a)+a/10](charlevelA/charlvlB) off of the official forums where it would just frustrate normal people.

  11. srand said,

    December 15, 2007 at 6:37 pm

    I can confirm what Ibn said: Especially towards the end (of the period when Ibn and I worked on AC1 together) we had a beautiful smooth process that at the time I figured was pretty standard. I have since learned that it’s not. (No offense to Don — the automated notifications on EQ2 were pretty slick, once we sorta got them working, but the entire patch note process was a nightmare.)

  12. Jobrill said,

    December 16, 2007 at 2:30 am

    I would probably give my eternal devotion and respect to any company that ran patches according to this formula.

    well, assuming, you know, their game didn’t suck.

    And even then I might grin and bear it.

  13. December 16, 2007 at 7:33 pm

    Y’know, I think for MMOs a list like this goes right up there with The Joel Test for “good places to work.” It’s a good way to evaluate where we are and what we need to do to improve our community relations.

  14. Blackblade said,

    December 17, 2007 at 3:41 am

    First to register punchthemanagerinthejunk.com wins a very strong slushy… Then they must link it to this post.

    Excellent post. If only such a reality was possible, eh?

  15. Bishop said,

    December 17, 2007 at 11:46 am

    One thing is for sure. If I ever got the priviledge of working with you, I would wear a jockstrap with a hard cup all the time, just to be on the safe side.

  16. Krinsath said,

    December 17, 2007 at 2:35 pm

    Excellent thoughts Tweety. Of course, being excellent thoughts they’ll be ignored by those in charge. 🙂

    I have noticed though that there are some games that come close to what you describe from at least the player’s perspective. I’ve always been impressed with CCP’s openness about any screw-ups they make (“boot.ini? Who needs THAT?”), but that’s about the closest I’ve seen a company come to this ideal.

    One day, the companies that make MMOs will realize the biggest boost to the longevity of any MMO is the community of that MMO, and that the community weenies can be a major part of that game’s success if they can keep the players involved in it.

  17. =j said,

    December 17, 2007 at 5:40 pm

    Any producer found playing a competitor’s MMO during patch day (especially if^H^Hwhen it goes south) should be punched in the junk on the spot. Prefably with an instrument similar to the Vociferious Hammer weilded by the forum kiddies.

    srsly

  18. Matt D-K said,

    December 17, 2007 at 10:24 pm

    From a coder’s perspective, I agree strongly that the Joel Test mentioned above is great. I would add”

    1. ALL code submissions to the mainline must be reviewed by at least one other engineer. and that all code submissions to version control have clear notes. This should be enforced both through peer pressure, but also through software controls ( integrate version control with review software )

    2. ALL code submissions must have clear change notes. These should include:
    (a) a one phrase or sentence overview of the change ( e.g. Spell Database Update) ,
    (b) a list of all changes made from the non-coder perspective ( e.g. expanded the range of spells ),
    (c) an elaboration of the changes suitable for coders ( e.g. updated the spell index field from an unsigned short to a uint32_t ),
    (d) Bug numbers as appropriate ( e.g. FogBugz Ref 421 )
    (e) Testing notes as appropriate ( e.g. Please create several spells in the range of 70,000 to 200,000 of various types and make sure they work )

    3. On a daily ( M-F ) basis, the code should be rebuilt ( from complete scratch if at all possible ) and emails sent with the aggregated ( and nicely formatted if possible ) change notes from #2. Backups made, emails sent on failures.

    4. Automated tests should be run immediately following #3. Emails sent on success or failure.

    5. Automated scripts should be used to aggregate changes for notes to Test and Live

    That’s all off the top of my head for now.

  19. Alex said,

    December 18, 2007 at 9:31 pm

    From a users point of view it would be nice to see more games which don’t take hours on a patch day, especially ones which seem to do it more often than the main patch day!

    It’s quite disappointing for MMO providers to take up most of a working day to install a patch. I know that occasionally there is going to be a patch which goes wrong and takes extra time to frantically fix a bug, but routine patches seem to take longer than ever. My hope is that with increased knowledge of the game and the patching process you’d be able to make it faster after many years of doing it.

    Am I expecting too much to see servers returning before the afternoon?

  20. Andrew said,

    December 18, 2007 at 10:05 pm

    Alex – A short downtime for patches is prioritized below backups, system maintenance and testing. The patch day is usually doubled/tripled up as a time where a full system backup can be made in a consistent state, and any outstanding maintenance can be performed. In the cases where a short downtime has been prioritized highly, testing suffers and more problems occur.

    If enough budget was diverted towards QAing – including paying developers to sit idle until bugs are discovered – then the patch duration could be minimized. MMO studios always have more work ahead, and so every usable asset is scheduled 100% (and then some) which results in a lot of work only being done when (or if) it’s required (i.e. at the last minute).

  21. Solvoug said,

    December 19, 2007 at 5:00 am

    Good stuff! It isn’t just MMOs where this dream should become reality! I work for a large company with an e-commerce arm, several “mini sites” and other online properties. Patches in any software that integrates real-time user interaction needs to be managed like this.

    The issue is as Andrew described — the dev resources are always allocated 100% plus to the work ahead; the end result is that a bunch of work that *should* be done, won’t be. The irony, of course is that if it *were* to be done, it would avoid a lot of work that *must* be done all of a sudden (and usually with out the controls of “normal” enhancements and bugfixes) because something that went horribly wrong on patch day and it cannot be ignored.

  22. December 20, 2007 at 8:20 am

    […] Weathers hat in ihrem Blog einen tollen Artikel darüber, wie ihr optimaler Patchablauf aussieht: My Dream Patch Day, And The Junk. Patches (oder, wo wir schon einmal dabei sind, auch Wartungsarbeiten) sind nämlich nicht nur ein […]

  23. January 8, 2008 at 3:18 am

    […] About the perfect patch day. […]

  24. January 11, 2008 at 6:58 pm

    […] Weathers, who was the community director for Mythic Entertainment has written an excellent blog post about what a perfect patch day for an MMO should look like, from the point of view of a community […]

  25. Kerri Knight said,

    January 28, 2008 at 4:21 am

    Hour 3 of a “15 minute downtime rolling realm restarts”

    2 deleted posts asking for an update

    no sticky

    no announcement

    nothing on the client launcher

    nothing at the login screen

    clearly this is community management at its finest…..

    How do people keep their jobs with this kind of performance?

  26. Gareth Eckley said,

    February 12, 2008 at 3:38 pm

    This is exactly how CCP handle EVE Online upgrades.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: