BlizzCon 2017 CodeCraft: Under the Hood Panel Transcript

Andrew: I’ll take this one. Hi Jim, so first Blizzard game I played was Warcraft II. First time I played it was against my dad. It was pretty cool. I got absolutely destroyed by it, but the thing that I really thought was interesting was like this is something completely different.

Yeah, it was fun and just got me super excited about playing games, and like wanting to build them… like just building small projects, and I ended up going down more of the web tech side, but that passion for being involved in this community and Blizzard was something I always wanted to do.

Andy: Interesting experience. My first game I played that was Blizzard was Warcraft II. It was the first RTS game for me, and it was just blew my mind that you could actually control an army and build stuff like that; and my experience getting into Blizzard was also really interesting.

The way I got in was I was pretty much competing with another warlock on our server to level 60 — once the game launched, and he didn’t work at the time, so he would beat me to it. I was like number two, so I like to think that I was number one because I actually had work; but we ended up being really good friends, and he actually had a cousin that worked at Blizzard, and he just sent over my resume, and then the rest is history.

Andy: I think it’s unique per team, but one of the key things is once you have APIs out there, it’s really important to preserve backwards compatibility — as you never want to make anyone break. So it’s really difficult to migrate off a backward compatibility stuff, but you always want to just make sure it works for your existing clients and increase communication to help them migrate off of it.

Andrew: One other thing is with version control, more and more, some of our teams are starting to really use centralized version control like we can see like web mobile projects and Battle.net projects were more or less in the same area.

As far as like making sure the change control and stuff like that works, every commit you end up doing, why did you work on this, what is the story. We have software that goes and can correlate those stories with the code review, and when we actually go to do a release this is what’s in the release, so we can actually start checking; and more and more we’re striving to get to the point where we can identify this change went out, and if there is a problem, how do we deal with just this change. We want to avoid getting more and more surprises, and we aren’t there everywhere yet, but we’re trying to continuously improve that.

Nick: To add to that, I think it has been touched upon, but typically each game team has their own solution for this problem because the only people working on the game are the people on the game team, but it’s sort of a different challenge for people on social platform teams such as Battle.net or Web teams who have to share their code across the entire company for multiple usages, so there’s sort of different considerations to be taken there, which is why they tend to use different solutions.

Seyil: One of the really exciting things about the opening ceremony video yesterday, there was a section if you guys saw it where we were talking to people with a white backdrop, and it wasn’t explicit, but you could kind of sense the feeling of diversity in that montage; and it kind of really made me feel good about the inclusiveness that we’re trying to have as part of Blizzard and our community.

In terms of kind of concrete things, well first of all, everyone should go to the diversity meet up that is on the second or third floor of the north hall. So that’s kind of a safe place for us to talk about these sorts of issues, but in terms of kind of concrete things there’s a couple things Blizzard is doing, and it kind of starts from the hiring process up to how we keep people; and so kind of on this end, one of the things we’re doing is with our interns program we’re trying to really focus on increasing how many female interns we get for our various positions; and so last year, we actually doubled the percentage of women that we’re getting for these positions, and then kind of on the retention portion, I know that- I believe next year in 2018, there’s going to be a “Women at Blizzard” Summit; and so this is a place for people to talk about these sorts of things, and obviously this is a really important issue for us; but we can always do better. There’s always more to be done, but we’re trying.

Seyil: I can talk a little to that. I mean… maybe this doesn’t directly answer your question, but we have a lot of DDOSs, and so we approach it on many different levels. All the way from our services being able to handle that sort of thing, up to our ISPs, and so that’s a problem that continually plagues us a bit, and we’re always playing this game with them in terms of– we figure out new techniques, they figure out new techniques, etc etc. So that’s a big problem for us.

Ryan: Yeah, I would like to add on to that. In mobile-specifically DDOS and network screening is an issue that we have to handle up front, because we are kind of at the mercy of app reviews, so if there is a problem that we can’t actually fix it sometimes it can be a couple of days, sometimes even up to a week, and so we have to kind of be really diligent about what we’re going to need up front.

Andrew: I’d love to take that one, because that was what I was mentioning in Heroes 2.0. One of the big things we had to do was… we try to do best thing first, try to design the data… typically, at least on my team, we are a SQL based, we try to normalize the data, we try to not repeat things and so on; but sometimes you reach a point where performance-wise the right thing is flatten your data, denormalize it.

There are teams that do end up using no SQL solutions for my platform, the one that I work on. It’s not appropriate for many of the cases we’re using, but there are actually teams that are using no SQL solutions like Couchbase and some of the other things like that, throughout the company. So it’s kind of a what fits what you want to do.

In the case of e-commerce, where we are linking accounts and we are linking like what licenses you have, and a lot of the things like that, that sort of structured data fits really well, but games teams I know have a lot of different solutions.

Seyil: I mean… one of the great things about Blizzard is that each of the teams is given the freedom to implement whatever solution works best for them, and so one of the things the game teams are really fortunate about is that we have this amazing platform that can support all these things, and we have experts in the field who can work with us to help us make these decisions in terms of what technology works best. So you’ll find pretty much everything.

Ryan: Before I actually was at Blizzard, I had a lot of side projects, and a lot of that actually led to me being a mobile developer which was really fun; but I don’t think I managed it too well in the beginning, and I really burnt myself out spending too many late nights, and too many weekends. I think that the true balance comes when you just set aside some time for it, but you don’t let it interfere with your daily job; and I think it’s very good to try and have your main job be someplace that you’re passionate about.

That way your side projects can help your learning, but you don’t want to have something that’s more interesting on the side that is going to take your focus. And so, it’s good to have balance. It’s really good to have that extra side project learning, it helps you grow outside of what you’re already doing at your work, but just not heavily diving into things too deep, and really being avoiding of your burnout is a big thing.

Seyil: One other thing, aside from just side projects, it’s also not a bad idea to get involved in some open source initiatives, and these are not your own projects, obviously; but there are ways to contribute to these things as well that maybe you can manage your time a little bit better.

Andy: Yeah, I wanted to echo that. It is really helpful just to check out the open source communit. You are not there to contribute, but it’s available for you to look at. It is really interesting to study what they’re doing, why they are doing it, and why exactly they are doing it, and incorporate that into your own skill set.

Andrew: My first job actually I got was contributing to an open source project. So like the fact that I was like… put that on my resume, really helped there. We also at Blizzard have Hack-a-ton for summer teams where we dedicate time to, if you’ve got a project you’ve been working on, maybe this is something that can turn into something. I think Ryan’s Battle.net app sort of came out of that.

Ryan: Yeah, there is some influence in the Hack-a-ton, definitely.

Derek: So, I’ll just add a little bit to that. On the Hearthstone team, we actually do a Free-Your-Mind at the very end of the year. So last couple weeks of December, everybody on the team actually has a chance to work on projects that they are passionate about.

Most of these projects are central to Hearthstone, but they don’t have to be. What we found is that because of Hearthstone’s relentless schedule of expansions, which we love working on, it is very important that we don’t crunch. Otherwise, we wouldn’t really be able to maintain that practice, and at the end of the year giving people a chance to express their own inner creativity I think allows us to just explore and kind of use that time at the end of the year just to come back and feel not only have I done all this great stuff for Hearthstone, but individually I’m able to learn and grow.

Zeyil: I can start off, I mean, for me because I’m a lead I’m probably in a lot more meetings than I would like to be, but typically, at least at the lead level, a lot of these meetings are very important because you’re making decisions about kind of the long term future of the game, and about our interactions with the other teams. So that takes up maybe like 50% of my week, and then a lot of my time is also spent making sure that the people on the server team (my team) are engaged and doing interesting work, and they’re enjoying that, and then I probably spend like a couple hours a week coding, but that’s about it at this point.

Nick: I can add a little to that as well, since I’m sort of on the other end of that right now, where I’m an individual contributor on our team. So I have a lead, and he tries to keep me engaged with the good work. So, a lot of my time is dedicated to just solving those problems that are sort of given to me and discovered as well. Obviously, we have the autonomy to be able to discover if something needs to be done, and get that done; and you will hear this a lot probably throughout all this talk, but it’s sort of different for Game Team, it depends how they decide it’s best to manage their people and how they work best.

Ryan: I also think it’s important to say that we also spend a little time playing video games with our coworkers, absolutely.

Andrew: That is a great question. On Battle.net, for example, we have dev ops embedded with a lot of our teams. Operationalizing our platform is the key to how we are continuously striving to keep our systems up. How do we keep them up, How do we keep them fast, and if it’s something where you just toss it over the wall. It’s never going to work right, so we actually have them embedded with the team. We actually have a lot of our team, Quality Assurance team is embedded, dev ops team is embedded, system liability engineers. So when we’re going and pushing something new, it’s to get all of those people involved as early as possible so we can do the right roll out solution. Is it something that we can auto-scale, or is it something we can’t? s How do we push it live? So… right, they are there from the start.

Zeyil: I have some pretty strong feelings about this whole topic, because a previous company I worked at had a strategy that I don’t necessarily agree with. I think it’s really important (like Andrew mentioned) to not throw things over the fence, and the common phrases “eat your own dog food” as well. We need to make sure that we are experiencing the same problems that everyone else is experiencing, and then they will get the attention they deserve; and so for us, for Team 5 (the Hearthstone team), we don’t actually, we don’t actually have a dev ops team per se. There is a live ops team, but everyone on the team is responsible for what happens with everything, and that’s really important to us.

Nick: Yeah, I want to second that as well. It is the same on the Diablo team, it is very important. You’re not just pushing a game out for people to enjoy, and then just sort of forgetting and moving on to something else. That game is living and breathing, and it’s everyone’s job gladly to keep it running as best as it can, and everyone pulls their own weight on that.

Ryan: I feel that we have a lot of freedom to do some exploration on what’s the best. I know with the mobile team, especially now the platforms are changing very rapidly, so we’re always looking for what is the next best thing, whether it’s the best language — iOS is the best native language moving forward, or if there’s any way that we can share code between the different platforms; and a lot of times people actually do their own research and bring their case for whether or not this is going to be something that we should be using as a team, and we decide on that as a team.

Andy: I was just going to add that we try to strike a balance, and sometimes it makes a lot of sense for things to be more consistent and for it to be more standardized, but then sometimes it’s also important to also look into the future and what else is out there.

Andrew: We also probably use pretty much any programming language or stack you can think of. We have literally everything. It may be a particular team. This is our framework. This is what we do, because if you have a one off that doesn’t work well, but we have C#, Java, C++, Swift, Node… like we have everything, and use the right tool for the right job.

Zeyil: So, I think one of the most important things is to do stuff, whatever that stuff is, like we talked earlier about projects or open source initiatives, and complete them, because during the process of completion you’re going to learn that there are sacrifices to be made, and compromises to be made; and once you learn those sorts of decision-making skills, those are really really valuable. And then, the second thing is just understand the technologies, right, so we’ve been listing off a whole bunch of different technologies that we use, and the more familiarity you have with them… the better.

Derek: As a hiring manager, I would prefer to see one language really well or one skill really well, then a plethora of languages all very superficially. So I think I would add that.

Andy: I like to follow popular blogs and news channels for all the technologies I am trying to learn. It is also really common for conferences to publish their talks online, and you have to look and dig a little bit; but there’s lots of free content out there. And another thing that’s really helpful is that you have to really geek out about the stuff you’re learning, like, if you’re not geeking out while you are learning it, you need to take a step back and just figure out what about that technology makes you enjoy it, and then go from there.

Andrew: Don’t be afraid to try something new. Like learn it and make a ton of mistakes, but learn that at home, and make a ton of mistakes. At work, already have an idea of like, hey maybe we could use this, but at home, I mean I have so many projects where it’s like “man, I made a mess of this” but I learned something out if it. So just keep trying new things.