Welcome to CodeCraft: Under the Hood with the Blizzard Engineering (part 2).
Derek: Good morning, BlizzCon. Welcome to Day 2, hope you had a good time yesterday. It has been a great show so far. So, today we actually managed to find five of our engineers who are still awake. Maybe they just woke up. Don’t know. Anyway, so we’re going to go through some introductions, then have a few questions for them and then we will get to your questions.
So, to kick things off, I am Derek Dupras, the engineering manager on Hearthstone, and…
Nick: I’m Nick Rivera, senior software engineer on the online group on the Diablo team, and I’ve been at Blizzard for 9 years.
Andrew: Hi, I am Andrew Murphy, I’m one of the lead engineers on Battle.ne., I’ve been at Blizzard for about three and a half years.
Seyil: Good morning, BlizzCon. My name is Seyil Yoon. I’m the server engineering lead on the Hearthstone team, and I’ve been with Blizzard and Hearthstone for about three and a half years.
Ryan: Good morning, my name is Ryan Newsome. I’m a mobile software engineer in the web and mobile group. I have been at Blizzard for just over a year and I’m a part of the team that just released the Battle.net mobile app.
Andy: Good morning, my name is Andy Tran, and I am a software engineer at Blizzard for web and mobile; and I’ve been here about nine and a half years, worked on a lot of web projects including the forums, our new systems, the WoW Armory and StarCraft II profiles.
Derek: So, let’s get things started with a couple of neutral questions here. So, Nick, what was one of your biggest blunders, and what did you learn from it?
Nick: Okay, well the key part of this story is that I’m still here, not fired yet, so this was around the time that Necromancer was about to come out, and we were really excited about it and not just because Necromancer was turning out to be a really cool class, but for us on the online group, it was important because we were going to release it worldwide on the in-game store.
At that point, the in-game store was only available in China because the game is free-to-play over there. The in-game store is a cool system, because it’s completely data driven, which means you just add something to the catalog in a data file somewhere, and it just appears; and that mostly works without having to touch the game or the servers.
I’d say “mostly” because this was the exception. For the server in the game to know about the necromancer, they have to know what a necromancer is, so as soon as we added it to the store catalog this meant that all of our currently live servers threw up a little warning in our monitoring software saying: “I don’t know what this is.”
It’s important to know that it’s just a warning. We expected this warning to come about, nothing bad was happening, but we don’t like having strangeous warnings in our monitoring software because it creates noise that makes it hard to track real problems.
So we evaluated, I took a look at what it would take to get rid of that warning, and came up with what I thought was a very simple solution which is: “Well, we’ll just patch the currently live servers so that they know what a necromancer is, then don’t complain about it.”
They won’t send it down but, they won’t sent it down to the game but it will call the warning, and we will get on with our day. It was like a two line fix, so I basically fast tracked to the live deployment and I was like: “What’s the harm?”
So, a couple of hours later, after checking that in I got a call because apparently after it had been rolled out to China many players were reporting that they could no longer make any purchases. And I was, in the back of my mind I’m like: “Well, this couldn’t be that little fix, it was only two lines;” and we worked to track it down, and guess what it was… that trivial trivial two-lines fix.
So I had egg on my face for sure, but we worked through, we basically just reverted it because it was simple to do that, and everything went back to normal just fine. Albeit I was a bit more embarrassed.
So, I learned two things that day, which was always test everything no matter how small it is, but more importantly it’s okay to make a mistake every once in a while — even if it seems like it’s the end of the world, there’s a solution to every problem. So if you make that mistake, you just have to own up to it, you solve the problem as quickly as possible, and you’ll be able to get through it.
Derek: Thank you, Nick for sharing that story, and I’m glad you’re still with us.
Nick: Me too.
Derek: So, Andrew, four engineers were not on the game dev teams. Can you tell us what some of the challenges are for the games that are releasing new patches, new releases, new expansions?
Andrew: So, I work on the Battle.net Purchase Team, so if you’ve ever bought something in-game, if you’ve been on the web shop, if you’ve put a key, used a key in a box or anything like that you’ve gone through our platform; and purchasing is one of the things that no one thinks about. In fact, when it works no one even really realizes we’re there, but we sometimes get some really interesting engineering challenges with the fact that we support every single game.
We built a really highly-scalable system, it’s distributed, you can login anywhere in the world and play with the stuff that are in sync game, but sometimes we get challenges like Heroes 2.0 where they say: “Hey, we want to give players a bunch of loot chests on day one.”
We go: “Okay. Well, how many is a bunch?”… “Somewhere between 12 and 70.”… “Wow, okay, so that’s a lot, when do they get it?”… “Right as soon as they login.”… Okay, we do the math and we look and realize our system is going to be hit a hundred times harder than it’s ever been hit.
So we then have to go and start doing load testing, we have to come up with faster ways of writing things to the database, we have to sometimes come up with new ways of communicating with the game; and then there is Customer Service teams that have to figure out how do you make sure the players get their stuff, and if something goes wrong… how do we correct it, and all those other things.
So there is a ton of other things that are on Blizzard outside of just the engineering for games that every patch, new things have to be figured out, made sure that it is solid so that way you guys don’t even have to realize those teams are there.
Derek: Well, thank you Andrew. I know there’s a lot of work that goes under the hood, so we really appreciate that. So, Seyil. We’re on the same team, I’m very curious, what was the first memorable thing that you ever worked on on Hearthstone?
Seyil: So, let’s see, I started on the Hearthstone team about a month and a half after launch with which if you recall was in March of 2014.So when I first started on the team, I kind of did what you normally do when you first start on a team. I was fixing bugs, and at the time we were getting ready for releasing the Curse of Naxxramas adventure.
So I was just kind of doing bug fixes there, but the first really memorable thing I worked on was for our first expansion Goblins vs Gnomes; and in that set (if you recall) there is a card called Steamwheedle Sniper— and Steamwheedle Sniper was really interesting, because it did something that you previously couldn’t do in our game before, which was change the way targeting works for the hero power; and our targeting system was probably a little more complicated than it needed to be at the time.
So it was really interesting, because I was digging into this new system that I had never previously worked with before, and engineers like solving problems; and so it’s kind of interesting to dig into that stuff to see how it really works. And the other really interesting thing about it was I came from a more technology-focused industry, and so it was the first time I got to work with a technical artist — so that’s a shout out to Becca Abel if you are out there.
So that was really cool. I had never worked with an artist before, and it’s really interesting because there’s a certain visceral feel good moment that you get sometimes when you’re working on gameplay systems, because you write this code, and you work with an artist, but then to actually see your code come alive on the screen when you play Steamwheedle Sniper and all of a sudden the hero icon changes and you can actually start targeting things, it was really cool; and it was kind of a really cool feel good moment. So, yeah, that’s the first really memorable thing I worked on for Hearthstone.
Derek: Thank you, Seyil. So, Ryan… you’ve been working on the Battle.net app. I hope everyone here has been downloading it, using it. It’s awesome. I really appreciate it. Could you share with us what some of the challenges were working on that app?
Ryan: Yeah, absolutely. Well, when we started the Battle.net mobile app, we knew that we wanted to try and make iPhone and Android users equally happy — which is a challenge in itself. One of the things we needed to do to do this… is to make a user interface for each of the individual apps that respects the given platform. What I mean by that is: if you are an iPhone user using the iPhone application, everything that you’re going to see in the app and experience is going to be familiar to you; and the same would be true for an end user in the Android application.
To accomplish this, we had the engineers and the designers work very closely together, so the designers could leverage the experience and the expertise of the engineers for their familiar platform and analyze every UI component that we put in the application to make sure that it was appropriate for the platform.
We were trying to keep everything as consistent as possible. An example of this divergence is actually the tabs on the main screen. If you look at the Android and iOS application they are in completely opposite positions, and even access to the profile is in a different way for both Android and iOS. And when we actually started making the app and engineering things out, it was another challenge which is actually one of the hardest things I thought I’d have to do as an engineer… and that is to throw away something that I’ve worked hard on.
A lot of times when we made a component and we actually physically use it, or like have users test it, we determine that things are wrong. Either that, or the design changes, specs change, more requirements are made in there.
Sometimes we can just get away with making some significant changes to what already exists, but a lot of times we actually have to make a hard decision and that is to throw away what we already have and start from scratch. This just produces better code, a better component, and in the end a much more Blizzard quality application, and it’s something that is a much better user experience for our players.
Derek: Thank you, Ryan. We really appreciate it. Alright, so before I ask my last question here, we’re going to give you guys an opportunity to ask questions, we just ask that you queue up over there to your right… there’s a nice light over there, and there will be someone there to help you organize your questions.
So, Andy, there’s a lot of great eSports events going on, and I know you’ve been working on the eSports platform. Could you tell us what some of the goals were for the platform, and what sort of challenges; and how did you help the team, you and your team, achieve those goals?
Andy: Yeah, Blizzard has always played a big role in eSports. So we wanted to provide and validate one of the best worldwide experience for all of our fans to follow all of the eSports games. What that specifically meant for my team in particular is we wanted to build a shared platform of tools and services to do really common things like bracket logic, paired team management, caching, and just other general good consistent API features, so that all the various eSports programs like the Heroes Global Championship and Overwatch World Cup can focus on doing their own website integration to tell their own unique eSports experiences.
So, right from the get go my team decided we really wanted to focus on having a good documentation that’s accurate and up to date, and had samples for everything, which is a lot easier said than done. Our experience with documentation has thought us it is really easy to be rushing out a feature and forget documenting one new field you added, or just forgetting to document the entire feature at all; and I thought that it’s really easy to forget updating all the samples you’ve added to your documentation like months ago.
So, the way we combat this feature was we actually added a step into our test automated testing that actually will record the request to responses, so that we could then use them in samples and our documentation, and that pretty much ensures that our samples and our documentations are always up-to-date, and on top of that our tests will actually detect if there are missing fields that were introduced that don’t have corresponding documentation and it will actually fail a test.
It actually forces the developer to actually document everything. So, as a result our platform pretty much has 100% documentation coverage and it’s been instrumental in rolling out all the various eSports web experiences out there all the way to production for us.
NEXT: COMMUNITY Q&A
| CodeCraft Panel Transcript | ||||
| 1. Engineering Team | 2. Community Q&A | 3. Community Q&A (Part 2) | ||
Hope you enjoyed this article. Please, support Blizzplanet via PayPal or Patreon, and follow us on Twitter, Facebook, YouTube, and Twitch for daily Blizzard games news updates. |
 |
