BlizzCon 2017 CodeCraft: Under the Hood Panel Transcript

Andrew: Code archaeology.

Nick: When I was first hired to work on Battle.net, and at the time we did a weekly rotation for people to handle sort of those legacy issues, tickets that would come in for those legacy systems, which by this point I think there was one person who knew how it all worked. Probably not that bad, but it felt that way anyway. So, part of this was to rotate everyone through it so everyone, we could spread this knowledge around and everyone could know.

So I had to work on a ticket for Diablo II, which meant that I had to go into the Diablo II server code, which I was not familiar with at all; and it was very enlightening to go through that code. The particular issue was some people would login, and it would say they were banned, and they were not banned; and so they were like what’s going on, and going through we eventually figured out we were trying to– by this point in time, we had made more registration keys (CD keys) for the game such that there were more bits associated with that CD key than the code was assuming, and so it was trying to shove in a higher precision value into– it was trying to shove 64 into 32 essentially. And so, you get those lower 32, and it would be like: “Oh this guy is banned because it is lower 32-bit match, this other random guy that got banned: somewhere else.” So, it was fun digging through that old code and figuring those kind of issues out.

Zeyil: Way back when before Hearthstone was actually integrated into Battle.net, our previous tech director, his name was Bob, created kind of a pseudo Battle.net type system for Hearthstone to be able to work, and he called it Bob.net; and so all over the code there are comments about Bob.net, and like nowadays no one knows what that is, but that is from many many years ago.

Zeyil: We use a ton. So our BI guys (Editor: Business Intelligence), first of all, they’re amazing, and they do all these cool graphs. I don’t know data science very well, so I’m just going to use Layman’s terms, but graphs. Cool graphs that tell us things like we have like a meta report that tells us in Hearthstone what sorts of cards are popular, what sort of cards are doing well, the correlation between two different cards and stuff like that. So I mean the amount of data that we consume is just phenomenal, and the pipes that this data comes through are very very large, but there are some really amazing stuff that comes out of that.

Ryan: Yeah, mobile is my familiarity and we have to put a lot of analytics in our applications because we need to know how people are using the app. It’s really important to understand how they’re using the app, and what they may or may not be doing a lot, because usually we only hear from like the crashes or the big asks, but we just want to know from like the common user who may not actually directly contact us; and so analytics are a big part of it, and we have some very smart people analyzing that, and setting up how it’s going to be put in place.

Derek: From a mathematical standpoint, what I really enjoy working with our BI team with is on the matchmaking for Hearthstone, and that’s where I think the rubber meets the road; because they actually look at all the numbers, they do the analysis, and then they say: “Okay, in order to adjust our matchmaking so that we have this nice perfectly formed curve that we want these are the numbers that you need to put into the system.”

So, as a software engineer you understand how to craft code, but the more higher mathematical understanding of what’s going on in the meta, it’s very important that we have them to help us do our jobs.

Nick: I do want to say something up top about this — which may not directly answer your question, but I do want to get it out there which is: always apply, like I can’t speak to so many times before I got my job here where I psyched myself out, like seeing what the positions were, just saying, oh this is Blizzard, I’ll never get in, oh this position requires too much experience where I don’t necessarily have XYZ skill so I shouldn’t even bother applying. I cannot stress enough, always always always apply. It’s their problem to figure out if you have what they’re looking for, but if you don’t… you know, it’s that old saying: “You miss a 100% of the shots you don’t take,” and so there’s more to it than that, but at base level I cannot stress enough just put your name in the hat, apply, just apply.

Andrew: My team is actually hiring entry level people. We are always looking for smart people. If we put a framework out there. or something we want someone to be familiar with “such and such“, that’s something we prefer, that’s something we’re using, that’s something where if you know “what” off the bat, that’s extra; but really we want smart people, people who aren’t afraid of challenges, people that aren’t coming in with: “no, we’re going to do this this way.”

There’s a lot of times reasons why things are done a certain way. So having an open mind, and kind of having a tool chest to sort of go, I have solid programming, maybe I’m not solid in Java, maybe I’m solid in C#. Could you learn it? I’m sure you could if you’re smart at that. So apply anyways. We’ve had multiple people where we’ve hired, where on my team that didn’t use Java, but that’s our main platform.

They picked it up and they’re pumping out solid work regularly. So yeah, apply like absolutely apply. There are a ton of entry level positions and if you’re in school please check out the internship program. We’ve had some of our brightest people that are moving up in the company come from the engineering team. Hearthstone has had people. We have somebody who came through engineering program who is on my team who’s doing great. So apply.

Derek: I don’t think it’s so important that you go to a game development school if you want to get into Blizzard, I think what’s important is that whatever school you go to that you get the most out of that experience you possibly can, which starts with you. So I personally, I went to a public higher education, I didn’t benefit from going to an IB (Internet Business) League or anything; and I’m sitting here on the stage with my esteemed colleagues, and I’d like to think that everyone here can do that; and actually there are a fair number of people who don’t even go to college, they came right out of high school, they’re very smart, they’re very passionate. So, I think passion and determination is a very important attitude to have, and ultimately that will carry you very far in life, not just here at Blizzard; and also I think you should write a good cover letter because I know at least one of the guys here read them.

Ryan: Because this is an important topic, I want to add too. I think it’s really important to have something to show, I mean whether that is a side project or something that you’re passionate about, something that you can kind of showcase that you’ve been a part of in the past; and also that there is a lot of hope, Blizzard hires a lot of new grads and so it’s not just a bunch of senior engineers, and so yeah there’s a lot of hope; and then we have a big internship program (as you stated) and new guys get hired often.

Derek: Thank you for that great question.

Zeyil: Loud voices. (audience laughs) Actually, I will answer that seriously. I mean so, I like to say sometimes that there needs to be a healthy conflict between different disciplines, and maybe conflict is too strong of a word, but ultimately you have to understand that everyone wants the same long term goals. So for myself for example, for my team everyone on the Hearthstone team wants Hearthstone to be successful, but different teams will have different views of how to accomplish that; and so for example, when I’m in a meeting with production, and design, and art, and so on, we’re all going to have our own views of what is the most important thing in order for Hearthstone to continue to be successful, and no one person is right or wrong. It’s just different perspectives, and so at that point it’s a matter of learning to negotiate and compromise; and so for myself oftentimes, I’m into those meetings saying well wait we don’t want to do that, because our system is not going to be able to scale, or it’s too late in development cycle and you add too much risk; but other people are going to say other things, and eventually you hope to be able to come to some sort of consensus, and do what’s right for the game.

Zeyil: So, when I’m hiring for the server team there’s a couple things I’m looking for that’s outside of maybe some of the other engineering disciplines. One of them is you always need to be thinking about scalability.

So if that is not forefront on your mind, then maybe a different discipline is more appropriate for you, but scalability is always going to be a super important thing for us. And then, the second part of it is oftentimes when we get software engineering applications, most of the time they’re for gameplay, and for obvious reasons — like I said earlier — there’s this kind of visceral feel good moment when you see gameplay stuff work; but when I’m interviewing for server people, one of the things I look for is: Do you get joy out of seeing a well running system? and I do, and I want to look for other people that get that same joy: Wow, all these servers are running, we’ve got X number of people logging in at the same time, and it’s all going smoothly.

To me that’s really exciting, and I’m looking for people who find that exciting as well. So I think that those are a couple really important aspects, at least for me.

Andy: Yeah, I would like to also add, it’s important when you’re designing your software and your architecture to horizontally scale as much as you can, and also it’s really important to know how to monitor the pieces that you’re responsible for, so that if there is something like a BlizzCon, there is high load events coming in that you couldn’t monitor it well, and see where the bottlenecks are, and address them by adding hardware and configuring accordingly.

Andrew: Data segregation is another important thing. If one team is having maybe a large event, you don’t want to cause an outage on another teams. So you want to be able to architect your application in a way where– when it’s appropriate, you can segregate data. It could be maybe something stored in different places, it could be you have a separate pool of servers. If you’re going to build your application that way, the last thing you want is to have an application requires just bigger and bigger and bigger servers. So you’re going to run into problems with that. So the best thing to do is write code that is going to scale horizontally, and not cost a ton to do it; and then when you do keep growing, how do you deal with the fact you’re going to end up a bottleneck, with bottlenecks somewhere? So how do you deal with that bottleneck in a way that’s not going to require something to be completely rewritten?

Nick: Something else on top of, part and parcel of horizontal scalability is having good fault tolerance with your horizontal scalability, because if your system is set up such that when one server goes down it kicks off 20 thousand people or something like that, that’s maybe not as ideal as what you would hope for other than like maybe when it crashes or shuts down, or anything else, other things just sort of take up the responsibility. Maybe something like that.

Zeyil: Actually, there’s one other thing that I thought of, is: graceful degradation of services. So for example, if some part of the game has a problem, the ability to shut that part of the game off, but allow people to still enjoy the rest of the game is really important as well, you don’t want to kick everyone off just because there’s some small portion of your system that isn’t working correctly.

Derek: So it’s a great view of what we’re doing internally to be able address those issues, but I think we’d be a bit remiss if we didn’t acknowledge that we have some great partners that we work with outside of Blizzard. From our data provider, our internet providers, and other companies that manage the traffic of the internet; and we actually work with them when we’re having problems.

Say if somebody comes in banging at your door, and they’re trying to get in, what do you do? Well, you hopefully will call the police because they can help you from the other side. So we do partner with those companies, so we do our part and we ask them to do their part, and I think that’s how we’ve been able to manage these issues in the past.

Zeyil: So, first of all, I’m going to mention this for obvious reasons, there’s a summer camp called Girls Do Code, and that is actually something that Blizzard is involved with as well, and I think we’ve been seeing a lot of good success coming out of that. In general, I think it’s really important to find out what you’re passionate about. So I mean obviously software engineering is kind of a very broad topic, so it’s important to try a lot of different things, and see what speaks to you; and I think that’s going to be one of the more important things to do at least at this point.

Andrew: Are you taking any CS classes or anything like that?

Andrew: If you can, enroll in CS. If you can’t do that, check out Khan Academy, check out like anything, just try, and just start writing code for things. It doesn’t need to be perfect. Early in my career, I spent a ton of time just trying to make the perfect thing. You end up with one beautiful file that does almost nothing.

So when you’re starting out just try and write lots of little things. You’ll learn to get better at it over time, but just right basic things. One of the things I find is trying to do something visual where you can get rapid iteration. Your first programming… a lot of times you’re writing a basic loop that spits out: “1, 2, 3, 4, 5.”

Figure out… maybe there’s a way you could move that onto the screen, maybe you could have it spit out something on a web page, maybe you could have it do something that’s going to continuously keep your interest to where something you thought you’re going to spend an hour on and you spent a whole evening on, just make sure to do your homework and stuff like that too.

Derek: I think being here in the community that we have here, there’s also a big help, because people here I think will support you in your goals and your desire to learn how to code; and maybe make games or maybe do something else; but there are a lot of people, a lot of diverse experiences here, and I think you should reach out to them, ask them if they’re coders.

I think if you see any of us around on the showfloor, go ahead, stop us, ask us how we got into it. But I think it’s ultimately, as we’ve said, do your homework, your studies, take classes, go on the Internet — a lot of resources, and I think there’s a lot out there to help you. Thank you.

Nick: I can mention a little about that. I haven’t myself on this, but I know a number of engineers in our company that are contributing have made proposals and such to the C++ standard. It’s escaping me which explicit ones they have done, I think Modules might have been one, but I’m sorry that I don’t have more detail on it, but I know there are, again, a number of engineers who are heavily involved with– Premake is another one. That’s a very good point, Premake… there is heavy contributions to that project.

Andrew: RabbitMQ, some Spring. I think web and mobile teams recently had some contributions, I can’t, like there are people who do things all the time on that. Qt is another one that I know we’ve contributed patches upstream to, like there’s open source, there is you know GDC.

Before I got to Blizzard, I love just going and finding out like what was on GDC, like what are some of the really cool presentations, like I love the GDC talk for StarCraft II for how screen space ambient occlusion was being done. That’s one of my favorite papers when I first was trying to get excited about the game industry to just sort of read how that stuff… so there’s a ton of things out there. Just got to kind of take a look for it.

Zeyil: So, one of the initiatives that we have actually is game data, so I believe there’s some data that we publish there for consumption by third parties. I am not that familiar with other things.

Andrew: dev.battle.net is where the game data API is. We’re always striving to try and put out more and more stuff. There’s nothing really that we have to really talk about now, but like it’s the thing that we’re constantly like: “hey, how could we expose this to people who want to build things on our platform.“

Zeyil: I think, so first of all this is probably a question that’s better asked for designers, because ultimately they’re the ones that consume a lot of the analysis that BI does in order to decide what to do next.

In terms of the Pirate Warriors specific part of the question. There were a lot of them, but I don’t think they were doing so well compared to everything else that they were necessarily a problem, and I mean this is less of an engineering question, but there’s a world for a kind of relatively cheap decks that people can play and pirate warrior kind of fit the bill for that.

Derek: Well, thank you very much. I think we’re about out of time. So I just want to thank everyone here in the room and also on the virtual ticket for joining us.

I think for myself, I think also for the gentlemen here, that what makes all this worth it “for me” is: all of you. So thanks for coming again, and we will see you hopefully tonight at the concert, and enjoy the show.