Lessons learned by working at Funcom for five years

I have an upcoming jubilee of sorts in a few months, at which point I can celebrate having endured life in the game development industry for five (measly) years. This cause for celebration is somewhat diminished by the fact that Funcom announced earlier this month that they are restructuring and consolidating offices – which means that I, along with the majority of the other developers at Funcom’s Montreal office, are being let go. For my part this means that I have at most two and a half month left before my official last day at FC, and having started working for FC in late March 2008, this means I might just about pass the five-year mark (yay!) before I’m officially out of a job (nay!).

Throughout these last soon-to-be five years I’ve had the pleasure and privilege of working with a diverse bunch of awesome people (and I hope that I’ll be able to work with some of them again in the future!) on two different MMORPG projects, and I would not change that for the world, but what exactly have I learned after this time spent working in the game development industry? Which of the preconceived assumptions and expectations I brought with me have held up, and which have been thoroughly shattered? What knowledge have I acquired that I can bring with me where-ever I go next?

I’ve played with the idea of writing a post along these lines in the past year or so, but what I’ve found out is that it’s not easy to summarize several years worth of experience in a simple blog post. Instead, I will try to focus on some of the more obvious lessons I have learned, the ones I can point at and say “that might have been useful to know/realize the value of when I first started”. Some – or maybe all – of them are perhaps obvious enough that they’re hardly worth mentioning, but then again – everything is obvious is hindsight.

Lesson learned: Design documents don’t survive first contact

No matter how elaborate or detailed the design document, it will not survive first contact with the implementation process. Unforeseen complications will arise, new information will come to light and changes to the original design will be necessary for implementation/playability/fun/balance-reasons.

Since I started working as an AI Designer I’ve found, through my (albeit) limited exposure to design document-writing, the following process to be a realistic and efficient compromise to overly detailed design documents: Concept the design in limited detail, prototype the initial implementation and then iterate on that until it’s where it needs to be.

Lesson learned: Don’t tie combat damage to animation lengths

A technical design decision that plagued Age of Conan for years after the game launched was to tie combat damage directly to the duration of and “impact” timings of specific character animations. Since the motion-captures of these animations were unique for each gender and the animations were then created directly from these sessions with no attempts to eliminate differences between the two genders, a situation arose (and remained unchanged for several years) where one gender had a combat advantage over the other. Because of different animation durations and impact-timings between male and female animations, a character of class A and gender X would be more efficient in combat/do more damage than a character of the same class A , but gender Y.

It took less than a month after the game launched before this became public knowledge amongst the players, but there was no easy fixes for the problem. There was zero chance of re-doing all the motion-capture sessions, and even cleaning up all the existing material with an eye for balance between the genders would have taken an insane amount of time and resources. In the end (after lengthy deliberation) what we ended up doing was manually adjusting the worst-case scenarios directly in the animations, while everything else was done through speeding up or slowing down animations until the impact-timings, keyframes (often nudged in one direction or the other) and animation lengths matched up as best they could.

This approach required a lot less time and resources, but was still a huge undertaking because of the sheer amount of changes necessary and the amount of data that had to be generated and then continually revised as changes were made. Also, because of the nature of the tools required for this work, this was a task that could not easily be spread out across several developers. Ultimately it ended up in my lap. The work itself consisted of manually tweaking animation-speeds (both for visual look and balance), impact/done-keyframes (when does this attack do damage? when is it over so I can start my next attack?), transition-keyframes (transition points between different combat animations) and continually updating and recalculating numbers on various spreadsheets all the while testing every change in the client to make sure no new bugs were created in the process – a process that in the end took well over a hundred (maybe as much as 150) hours of work with no other distractions. In the end this allowed us to reduce the combat-differences between the genders to within a very small and insignificant percentage (but not eliminated completely).

That is not even the whole story, as the differences mentioned were also partly responsible for a bug that allowed some players even greater advantages in combat. Some of them saw this as a feature because it went unfixed for too long, and much forum drama took place after the above mentioned balance tweaks were made – as they also directly affected the effectiveness of said bug/feature.

All of this extra work (and blood, and tears, and suffering, and forum drama) could have been avoided had different choices been made when the combat-system was first designed.

Lesson learned: Testing is Alpha and Omega

One trap I have seen many developers (and my own conscience is not clean either) fall into, is that of not testing their own work extensively enough. While usually there is (or should be, if not!) a QA department dedicated to thoroughly testing everything the developers produce before it is “shipped”, even a minimum amount of testing on the part of the developer can potentially save both them and the QA department a lot of headaches.

Example of such a situation: The developer (me, in this example!) performs a last-second minor tweak to an NPC AI script intended as a tiny bit of polish, does not bother testing after said small and seemingly insignificant tweak, only to find out the next day – after it has passed through the QA loop – that the final minor tweak caused the NPC (and thus the feature that depended on said NPC) to stop working altogether. Doh!

In that situation, one simple test-run before shipping it off to QA would have revealed the problem and it should have taken only a couple of minutes (or seconds, even) to fix once discovered. Instead it looped through the QA processes, then back to the developer (still me), who then had to figure out what broke, why it broke and how to fix it – before then sending it back to QA for another round of testing.

The ideal process for testing one’s own work might go something like this:

1. Test your stuff as often and in as many different ways as humanly possible, while always looking for ever more devious ways of breaking it
2. Get someone else to test your stuff while you passively observe and make notes
3. Fix all the bugs and issues that arose during steps 1 and 2
4. Rinse and repeat steps 1-3 indefinitely until no more problems can be found, THEN ship it off to QA

Of course, back in reality, there are such things as budgets and deadlines and leads/project managers breathing down your neck, which means that step 4 would more likely turn into “Rinse and repeat steps 1-3 indefinitely until you run out of time” – which would usually just before or shortly after you reach step 1.

Lesson learned: The Law of Unintended Consequences kicks ass!

The law of unintended consequences is definitely real, alive and kicking when it comes to game development. Small, seemingly insignificant tweaks/changes can have widespread consequences far beyond whatever one is specifically working on.

Semi-made up example 1: A specific feature (say, for instance, the ability to perform animated emotes) stops working for NPCs. Developer finds the problem, fixes it, verifies that yep – the feature now works for NPCs again. They’re performing their animated emotes exactly as intended. Problem solved, right? Except…it turns out this is also a feature that is used by players, and while the fix solved the problem for the NPCs, it also created a brand new one for those players – their animated emotes now cut off one second after they start playing, or the players are now frozen in a permanent state of emote, unable to move or abort the emote!

Example 2: By fixing (in data instead of code where it should have been fixed – but could not because of limited resources) an issue that could cause players to become “stuck” in combat mode, resulting in them always having their weapon drawn no matter how many times they tried to sheathe it, an unintended consequence was that the auto-sheathing feature in the client that kicks in when players leave combat would now trigger as soon as their opponent in combat was dead – even if the attacking player was still in mid-strike. Result: Character sheathing their weapon while in the middle of an advanced combo-finisher, and the character then performing the rest of the finisher-animation as if the weapon was still drawn.

Example 3: Modifications are made to a specific data-file dealing with character creation, and patched to the Live servers. Unintended result of those modifications? Breast-sizes for all female characters in the game are now reduced. Much amusement (and lack thereof among certain players) takes place.

Lesson learned: Documentation matters

Documenting changes, fixes and additions in enough detail that other designers/developers (or even yourself) can look at it months/years later and understand what was done – and why – is more important than a lot of people think. It can in some cases mean the difference between hours of unnecessary debugging and testing, and a couple of minutes. Unless a bug-report specifies exactly what the problem is and how it should be fixed, commenting on that bug-report with anything as generic as “Fixed” or “Problem solved” with no further details ought to be a crime punishable by having to write the following a thousand times on a blackboard every day for two weeks straight: “I will always document my fixes from now on.”

Imagine that you’re trying to solve a particularly tricky problem (a bug that needs fixing, a feature that needs implementing or some content additions that need adding), and after hours of concepting/prototyping/debugging/testing/scratching your head you discover that a similar problem has existed before, and a solution was found! You throw your hat in the air and shout, with joy: “Hurrah!” …Then reality sinks in as you realize that there is no documentation in the existing task/bug-report for what exactly the problem was and what the solution was. Also, the developer who implemented/fixed it the first time around has now died, or – worse! – left the company!

Document your shit, people.

Lesson learned: Details matter, but there is such a thing as too much detail

Many developers (myself included) get hung up in the details of our work. We are never fully satisfied with what we’ve created, and if it was possible we would likely keep on polishing and perfecting our work indefinitely. However, the law of diminishing returns eventually kicks in, and lines have to be drawn.

Sometimes those lines can be hard to see, though, and we end up spending relatively absurd amounts of time on details that don’t really matter. A particle-effect that lasts half a second and only plays once every full moon does not need as much polish and attention as one that will play regularly and last ten seconds, or loop forever. The texture for a small critter/rock does not need to be as crisp and clear as one for a large animal/statue. A 20m tall support-beam in a dark room does not necessarily need a carefully crafted, beautiful and complex pattern running along the top of the beam – the players will never see it.

Lesson learned: Revision control matters

I did not grok the true value of revision control until I had actually worked with it actively for some time. Now, the idea of not being able to access the version history and actual different versions of a particular file seems alien to me. With a system of revision control in place, away goes any worry about things like file-corruption from 3ds Max-crashes, overwriting the work of others (or having your own work overwritten), not knowing who the last person to work on a specific file was (and when), etc.

Manual file-backups and renamed copies can only take you so far – switch to revision control before it’s too late! My experience from Funcom is with Perforce (proprietary), others worth checking into: SVN or Git/GitHub.

Lesson learned: Optimization matters

Whether it’s an unnecessarily heavy particle-effect, a texture that’s too large, a playfield with too many different unique assets/NPCs in it or AI scripts that execute too many commands in too short a time – optimization matters. Reducing the amount of particles emitted, bringing those textures down to a reasonable size, re-using more of the same assets across a particular playfield rather than insisting on using unique assets for every single corner of the map, making sure those AI scripts don’t bring the server to its knees – these things can mean the difference between a game with an unplayable framerate on laggy servers and one that runs stable at 60fps with no hick-ups!

Lesson learned: Developers suck as players

Regular players are more often than not better at playing an MMORPG than the developers of said MMORPG. This is due to the sad fact that developers do not have enough hours in a day to both play the game as much as regular players do AND actively develop said game. This becomes extra apparent when considering that specific developers by necessity only work on and possess knowledge about specific parts of these games on a need-to-know basis. Server programmers don’t need to know where item X drops in the game to do their job. Audio guys don’t need to know the details of character progression to do their job. A 3D artist don’t need to know the optimal ability rotation for class Y in combat. A gameplay designer isn’t necessarily an AI designer isn’t necessarily a systems designer. There’s just too much information for any one developer to possibly obtain.

This specialized knowledge problem can be offset somewhat by something like mandatory playtime, where the developers are either forced to or given the option to play the game during regular work hours, though this is a double-edged sword. While it allows the developers to become more experienced and knowledgeable at playing their own game, it also eats into the time they have to actually develop new features, more content, fixing bugs, etc.

Lesson learned: Marketing decisions trump game design decisions

This might differ from one company to another, but the impression I’m left with is that marketing decisions trump game design decisions.

Semi-made up example: You implement some content in your MMORPG where players will, after completing a series of long quest-chains and faction grinds (a topic for discussion on its own), be rewarded with a specific and cool-looking mount. Shortly after releasing this content, the marketing department decide that they need something to give new players incentive to sign up for X month subscriptions as they buy the game. One of them notice this cool-looking mount and decide to bundle it as a “reward” in the subscription package.

Result: The effort that the existing players put in to get the cool-looking mount is now completely wasted as brand new players can get the same mount (perhaps in a different color) by choosing to subscribe for X months when they buy the game. To add insult to injury, the subscription-based version of the mount might only available to players of new accounts. No sane designer would sanction this, but unfortunately they’re not always given a choice in the matter.

The same can hold true for things like item-shops, where business decisions might be made to put up items that are likely to be detrimental to the long-term health of the game, but which will also sell like hotcakes right here and now.

Another example of this would be the decision on whether to launch a game as subscription based or as a F2P/B2P title. This choice can have a major impact on the design of the game and for which features will be prioritized and implemented, yet it’s a decision that largely rests with the marketing/business people.

Lesson learned: Knowledge Obscurity is Job Security

Whether intentional or not, knowledge obscurity can mean job security. For instance, as a BCC designer at Funcom, I was one of very few people who were willing (and required) to learn and use a very specific and not very user-friendly tool that is (still to this day) crucial to both Age of Conan and The Secret World.

This was the tool where animation systems and the character state-machine were hooked up to one another, and if setup incorrectly or not maintained properly it could literally break all players and NPCs in the game. Knowledge and experience with this tool is partially (I like to think that my winning personality and good fashion sense also played a part) what made me survive three rounds of layoffs at Funcom – though not the fourth, which took out an entire office full of people!

Obscuring this knowledge was not intentional, as there were both extensive documentation and tutorials for using this tool available, but the effect was the same – some measure of job security because hardly anyone else really knew how to use it, wanted to know how to use it, or had known how to use it and then later deliberately brain-washed themselves to get rid of that knowledge.

Lesson learned: There is no pleasing every player

As it turns out, you cannot please all players. No matter what you add, fix or change there will always be players who didn’t like it, didn’t consider it a bug to begin with and who liked the old version better. This is also one of The Laws of Online World Design, and as such is probably one of the more obvious lessons I could have learned, but there’s no replacement for first-hand experience, like after having modified a broken/non-optimal particle-effect or animation only to receive a back-lash from players who are not pleased with the new version.

Lesson learned: Don’t make promises that can’t be kept

Whether it’s marketing people promising features that have not yet left the conceptual/prototype stage, game directors giving information on things to come in the next patch or community managers giving a very specific ETA for when servers will be back up after patching – players will hold them to it. If those promised features are canceled/cut, if those things to come are postponed to the next patch, if the servers are down a minute longer than expected – the players will immediately pull out pitchforks and torches and demand one head or another served on a platter.

This is unfortunate, as it means that information given to the players will often have to be sanitized for anything that can be interpreted as a promise or an ETA, unless the company is large and/or faceless enough to be able to handle the heat generated by the players upon the failure to keep those perceived promises (whether they were intended as such or not is irrelevant).

In conclusion…

To conclude this post I’ll leave you all with an ancient Chinese pro-verb that seems appropriate to use in this context:

How can you put out a fire set on a cart-load of firewood with only a cup of water?