Oriol Vinyals: DeepMind AlphaStar, StarCraft, and Language | Lex Fridman Podcast #20

The following is a conversation with Ariel Vinales. He's a senior research scientist at Google DeepMind and before that he was at Google Brain and Berkeley. His research has been cited over 39,000 times. He's truly one of the most brilliant and impactful minds in the field of deep learning. He's behind some of the biggest papers and ideas in AI, including sequence to sequence learning, audio generation, image captioning, neural machine translation, and of course, reinforcement learning. He's a lead researcher of the Alphastar project, creating an agent that defeated a top professional at the game of StarCraft. This conversation is part of the artificial intelligence podcast. If you enjoy it, subscribe on YouTube, iTunes or simply connect with me on Twitter lexfriedman spelled F R I D. And now here's my conversation with Ariel Vinales. You spearheaded the DeepMind team behind AlphaStar that recently beat a top professional player at StarCraft. So you have an incredible wealth of work and deep learning and a bunch of fields. But let's talk about StarCraft first. Let's go back to the very beginning, even before AlphaStar, before DeepMind, before deep learning first. What came first for you? A love for programming or a love for video games?

I think for me it definitely came first. The drive to play video games. I really liked computers. I didn't really code much, but what I would do is I would just mess with the computer, break it and fix it. That was the level of skills, I guess, that I gained in my very early days, I mean, when I was 10 or 11. And then I really got into video games, especially Starcraft. Actually the first version, I spent most of my time just playing kind of pseudo professionally, as professionally as you could play back in 98 in Europe, which was not a very main scene like what's called nowadays, esports.

Right, of course, in the 90s. So how'd you get into Starcraft? What was your favorite race? How did you develop your skill? What was your strategy? All that kind of thing.

So as a player I tended to try to play not many games, not to kind of disclose the strategies that I kind of developed. And I like to play random, actually, not in competitions, but just to. I think in Starcraft there's, well, there's three main races and I found it very useful to play with all of them. So I would choose random many times, even sometimes in tournaments to. To gain skill on the three races because it's not how you play against someone, but also if you understand the race because you play it, you also understand what's annoying then when you're on the other side, what to do to annoy that person, to try to gain advantages here and there and so on. So I actually played random, although I must say in terms of favorite race, I really like Zerk. I was probably best at Zerk and that's probably what I tend to use towards the end of my career before starting university.

So let's step back a little bit. Could you try to describe StarCraft to people that may never have played video games, especially the massively online variety like starcraft.

So starcraft is a real time strategy game and the way to think about Starcraft, perhaps if you understand a bit Chess, is that there's a board which is called Map or the map where people play against each other. There's obviously many ways you can play, but the most interesting one is the one versus one setup where you just play against someone else. Or even the built in AI Blizzard put a system that can play the game reasonably well if you don't know how to play. And then in this board you have again pieces like in chess. But these pieces are not there initially like they are in chess. You actually need to decide to gather resources to decide which pieces to build. So in a way you're starting almost with no pieces. You start gathering resources. In Starcraft there's minerals and gas that you can gather. And then you must decide how much do you want to focus for instance on gathering more resources or starting to build units or pieces. And then once you have enough pieces or maybe a good attack composition, then you go and attack the other side of the map. Now the other main difference with chess is that you don't see the other side of the map. So you're not seeing the moves of the enemy. It's what we call partially observable. So as a result you must not only decide trading of economy versus building your own units, but you also must decide whether whether you want to scout to gather information. But also by scouting you might be giving away some information that you might be hiding from the enemy. So there's a lot of complex decision making all in real time. There's also unlike chess, this is not a turn based game. You play basically all the time, continuously and thus some skill in terms of speed and accuracy of clicking is also very important. And people that train for this really play this game at amazing skill level. I've seen many times this. And if you can witness this life, it's really, really impressive. So in a way it's kind of a chess where you don't See the other side of the board, you're building your own pieces and you also need to gather resources to basically get some money to build other buildings, pieces, technology and so on.

From the perspective of the human player, the difference between that and chess or maybe that and a game like turn based strategy like Heroes of Might and Magic is that there's an anxiety because you have to make these decisions really quickly and if you are not actually aware of what decisions work, it's a very stressful balance. Everything you describe is actually quite stressful. Difficult to balance for amateur human player. I don't know if it gets easier at the professional level, like if they're fully aware what they have to do. But at the amateur level there's this anxiety. Oh crap, I'm being attacked. Oh crap, I have to build up resources. Oh, I have to probably expand and all these the time. The real time strategy aspect is really stressful and computation I'm sure difficult. We'll get into it. But for me, Battle Net so Starcraft was released in 98, 20 years ago, which is hard to believe. And Blizzard Battle Net with Diablo in 96 came out and to me it might be a narrow perspective, but it changed online gaming and perhaps society forever. But I may have made way too narrow of a viewpoint. But from your perspective, can you talk about the history of gaming over the past 20 years? How transformational, how important is this line of games?

Right? So I think I kind of was an active gamer whilst this was developing the Internet, online gaming. So for me the way it came was I played other games, strategy related. I played a bit of Command and Conquer and then I played Warcraft 2 which is from Blizzard. But at the time I didn't know, I didn't understand about what Blizzard was or anything. Warcraft 2 was just a game which was actually very similar to starcraft in many ways. It's also real time strategy game where there's orcs and humans, so there's only two races. But it was offline and it was offline, right. So I remember a friend of mine came to school saying oh there's this new cool game called starcraft. And I just said oh, this sounds like just a copy of Warcraft 2 until I kind of installed it. And at the time I am from Spain, so we didn't have Internet, like very good Internet, right. So there was for us StarCraft became first kind of an offline experience where you kind of start to play these missions, right? You play against some sort of scripted things to develop the story of the characters in the game. And then later on I start playing against the built in AI and I thought it was impossible to defeat it. Then eventually you defeat one and you can actually play against seven built in AIs at the same time, which also felt impossible. But actually it's not that hard to beat the seven built in AIs at once. So once we achieved that also we discovered that we could play, as I said, Internet wasn't that great, but we could play with the LAN basically against each other if we were in the same place. Because you could just connect machines with cables. Right. So we started playing in LAN mode as a group of friends and it was really, really much more entertaining than playing against AIs. And later on, as Internet was starting to develop and being a bit faster and more reliable, then it's when I started experiencing Battle Net, which is this amazing universe, not only because of the fact that you can play the game against anyone in the world, but you can also get to know more people. You just get exposed to now this vast variety of. It's kind of a bit when the chats came about, there was a chat system you could play against people, but you could also chat with people not only about Stacker, but about anything. And that became a way of life for kind of two years. And obviously then it became like kind of it exploded in mean that I started to play more seriously, going to tournaments and so on, so forth.

Do you have a sense on a societal, sociological level, what's this whole part of society that many of us are not aware of? And it's a huge part of society, which is gamers. I mean every time I come across that in YouTube or streaming sites, I mean this is a huge number of people play games religiously. Do you have a sense of those folks, especially now that you've returned to that realm a little bit on the AI side?

Yeah. So in fact, even after Starcraft, I actually played World of Warcraft, which is the main sort of online world on presence that you get to interact with lots of people. So I played that for a little bit. To me it was a bit less stressful than starcraft because winning was kind of a given. You just put in this world and you can always complete missions. But I think it was actually the social aspect of especially starcraft first and then games like World of Warcraft really shaped me in a very interesting way because what you get to experience is just people you wouldn't usually interact with. Right. So even nowadays I still have many Facebook friends from the area where I played online and their ways of thinking is even political. They just don't. We don't live in like we don't interact in the real world, but we were connected by basically fiverr and that way I actually get to understand a bit better that we live in a diverse world and these were just connections that were made by. Because you know, I happened to go in a city, in a virtual city as a priest and I met this warrior and we became friends and then we start playing together. Right. So I think it's transformative and more and more and more people are more aware of it. I mean it's becoming quite mainstream. But back in the day, as you were saying in 2005 even it was still very strange thing to do. Especially in Europe. I think there were exceptions like Korea for instance. It was amazing that everything happened so early in terms of cyber cafes. If you go to Seoul, it's a city that back in the day StarCraft was kind of. You could be a celebrity by playing StarCraft. But this was like 99, 2000, right? It's not like recently. So yeah, it's quite interesting to look back and yeah, I think it's changing society the same way. Of course, like technology and social networks and so on are also transforming things.

And a quick tangent, let me ask. You're also one of the most productive people in your particular chosen passion and path in life and yet you're also appreciate and enjoy video games. Do you think it's possible to do to enjoy video games in moderation?

Someone told me that you could choose two out of three. When I was playing video games, you could choose having a girlfriend playing video games or studying. And I think for the most part it was relatively true. These things do take time. Games like stack. If you take the game pretty seriously and you want to study it, then you obviously will dedicate more time to it. And I definitely took gaming and obviously studying very seriously. I love learning science and et cetera. So to me, especially when I started university undergrad, I kind of stepped off starcraft. I actually fully stopped playing. And then World of Warcraft was a bit more casual. You could just connect online and I mean it was fun but as I said, that was not as much time investment as it was for me in StarCraft.

Okay, so let's get into AlphaStar. What are the. You're behind the team. So DeepMind's been working on StarCraft and released a bunch of cool open source agents and so on the past few years. But AlphaStar really is the moment where the first time you beat a world class player. So what are the parameters of the challenge in the way that AlphaStar took it on and how did you and David and the rest of the DeepMind team get into it? Consider that you can even beat the best in the world or top players.

I think it all started back in 2015. Actually, I'm lying. I think it was 2014 when DeepMind was acquired by Google. And I at the time was at Google Brain, which was in California. Still in California. We had this summit where we got together the two groups. So Google Brain and Google DeepMind got together and we gave a series of talks. And given that they were doing deep reinforcement learning for games, I decided to bring up part of my past which I had developed at Berkeley, like this thing which we call Berkeley Overmind, which is really just a stacker of one bot, right. So I talked about that and I remember Demis just came to me and said, well, maybe not now, it's perhaps a bit too early, but you should just come to DeepMind and do this again with deep reinforcement learning. Right? And at the time it sounded very science fiction for several reasons. But then in 2016 when I actually moved to London and joined DeepMind, transferring from Brain, it became apparent that because of the AlphaGo moment and kind of Blizzard reaching out to us to say, wait, do you want the next challenge? And also me being full time at DeepMind, sort of all these came together. And then I went to Irvine in California to the Blizzard headquarters to just chat with them and try to explain how would it all work before you do anything. And the approach has always been about the learning perspective. In Berkeley we did a lot of rule based conditioning and if you have more than three units, then go attack. And if the other has more units than me, I retreat, and so on, so forth. And of course the point of deep reinforcement learning, deep learning, machine learning in general, is that all these should be learned behavior. So that kind of was the DNA of the project since its inception in 2016, where we just didn't even have an environment to work with. And so that's how it all started, really.

So if you go back to that conversation with Demis or even in your own head, how far away did you. Because we're talking about Atari games, we're about talking, talking about Go, which is kind of, if you're honest about it, really far away from starcraft. Well, now that you've beaten it, maybe you could say it's close, but it seems like starcraft is Way harder than go philosophically and mathematically speaking. So how far away did you think you were? Do you think in 20, 19 and 18 you could be doing as well as you have?

Yeah, when I kind of thought about, okay, I'm going to dedicate a lot of my time and focus on this and obviously I do a lot of different research in deep learning, so spending time on it I really had to kind of think there's going to be something good happening out of this. So really I thought well this sounds impossible and it probably is impossible to do the full thing, the full game where you play one versus one and it's only a neural network playing and so on. So it really felt like I just didn't even think it was possible. But on the other hand I could see some stepping stones towards that goal. Clearly you could define sub problems in Starcraft and sort of dissect it a bit and say, okay, here is a part of the game, here is another part. And also obviously the fact. So this was really also critical to me, the fact that we could access human replays. Right. So Blizzard was very kind and in fact they open sourced this for the whole community where you can just go and it's not every single StarCraft game ever played, but it's a lot of them. You can just go and download and every day they will. You can just query a data set and say, well give me all the games that were played today. And given my kind of experience with language and sequences and supervised learning, I thought well that's definitely going to be very helpful and something quite unique now because ever before we had such a large data set of replays of people playing the game at this scale of such a complex video game. So that to me was a precious resource. And as soon as I knew that Blizzard was able to kind of give this to the community, I started to feel positive about something non trivial happening. But I also thought the full thing like really no rules, no single line of code, the that tries to say, well I mean if you see this unit, build a detector. All these, not having any of these specializations seemed really, really, really difficult to me intuitively.

I do also like that Blizzard was teasing or even trolling you sort of almost. Yeah, pulling you in into this really difficult challenge. Do they have any awareness? What's the interest from the perspective of Blizzard except just curiosity?

Yeah, I think Blizzard has really understood and really bring forward this competitiveness of esports in games. The Starcraft really kind of sparked a lot of something that almost was never Seen especially as I was saying back in Korea. So they just probably thought, well, this is such a pure one versus one setup that it would be great to see if something that can play Atari or Go and then later on chess could even tackle this kind of complex real time strategy game. Right. So for them they wanted to see first obviously whether it was possible if the game they created was in a way solvable to some extent. And I think on the other hand, they also are a pretty modern company that innovates a lot. So just starting to understand AI for them to how to bring AI into games is not. Is not AI for games, but games for AI. Right. I mean both ways I think can work. And we obviously at DeepMind use games for AI, right, to drive AI progress. But Blizzard might actually be able to do and many other companies to start to understand and do the opposite. So I think that is also something they can get out of these and they definitely. We have brainstormed a lot about these. Right.

But one of the interesting things to me about starcraft and Diablo and these games that Blizzard has created is the task of balancing classes. For example, sort of making the game fair from the starting point and then let skill determine the outcome. Is there. I mean, can you first comment. There's three races, Zerg, Protos and Terran. I don't know if I've ever said that out loud. Is that how you pronounce it? Taran?

Yeah.

Yeah. I don't think I've ever in person interacted with anybody about starcraft. That's funny. So they seem to be pretty balanced. I Wonder if the AI, the work that you're doing with AlphaStar would help balance them even further. Is that something you think about? Is that something that Blizzard is thinking about?

Right. So balancing when you add a new unit or a new spell type is obviously possible given that you can always train or pre train at scale some agent that might start using that in unintended ways. But I think actually if you understand how StarCraft has kind of co evolved with players in a way, I think it's actually very cool the ways that many of the things and strategies that people came up with. Right. So I think we've seen it over and over in starcraft that Blizzard comes up with maybe a new unit and then some players get creative and do something kind of unintentional or something that Blizzard designers just simply didn't test or think about. And then after that becomes kind of mainstream in the community, Blizzard patches the game and then they kind of maybe Weaken that strategy or make it actually more interesting but a bit more balanced. So this kind of continual talk between players and Blizzard is kind of what has defined them actually in actually most games in Stacker but also in World of Warcraft they would do that. There are several classes and it would be not good that everyone plays absolutely the same race and so on. Right. So I think they do care about balancing, of course, and they do a fair amount of testing. But it's also beautiful to also see how players get creative anyways. And I mean whether AI can be more creative at this point, I don't think so. Right. I mean it's just sometimes something so amazing happens. Like I remember back in the days, like you have these dropships that could drop revers and that was actually not thought about that you could drop this unit that has this, what's called splash damage that would basically eliminate all the enemy's workers at once. No one thought that you could actually put them and really early game do that kind of damage and then, you know, things change in the game. But I don't know, I think it's quite an amazing exploration process from both sides, players and Blizzard alike.

Well, it's, it's almost like a reinforcement learning exploration. But the scale of humans that play, that play Blizzard games is almost on the scale of a large scale DeepMind RL experiment. I mean if you look at the numbers that's. I mean you're talking about I don't know how many games, but hundreds of thousands of games, probably a month.

Yeah.

I mean so you could. It's almost the same as running RL agents. What aspect of the problem of Starcraft do you think is the hardest? Is it the, like you said, the imperfect information? Is it the fact they have to do long term planning? Is it the real time aspects so you have to do stuff really quickly? Is it the fact that large action space so you can do so many possible things or is it, you know, in the game theoretic sense there is no Nash equilibrium or at least you don't know what the optimal strategy is because there's way too many options. Is there something that stands out as just like the hardest, the most annoying thing?

So when we sort of looked at the problem and start to define the parameters of it, what are the observations, what are the actions? It became very apparent that the very first barrier that one would hit in Starcraft would be because of the action space being so large and us not being able to search like you could in chess or go. Even though the search base is vast. The main problem that we identified was that of exploration, right? So without any sort of human knowledge or human prior, if you think about Starcraft and you know how deep Reinforcement learning's algorithm work, which is essentially by issuing random actions and hoping that they will get some wins sometimes so they could learn. So if you think of the action space in Starcraft, almost anything you can do in the early game is bad because any action involves taking workers which are mining minerals for free. That's something that the game does automatically sends them to mine and you would immediately just take them out of mining and send them around. So just thinking, how is it going to be possible to get to understand these concepts but even more like expanding, right? There's these buildings you can place in other locations in the map to gather more resources, but the location of the building is important and you have to select a worker, send it walking to that location, build a building, wait for the building to be built and then put extra workers there so they start mining. That feels like impossible. If you just randomly click to produce that state, desirable state that then you could hope to learn from because eventually that may yield to an extra win, right? So for me, the exploration problem due to the action space and the fact that there's not really turns, there's so many turns because the game essentially ticks at 22 times per second. I mean that's how they discretize sort of time. Obviously you always have to discretize time. There's no such thing as real time, but it's really a lot of time steps of things that could go wrong. And that definitely felt a priori like the hardest. You mentioned many good ones. I think partial observability, the fact that there is no perfect strategy because of the partial observability, those are very interesting problems. We start seeing more and more now in terms of as we solve the previous ones. But the core problem to me was exploration and solving. It has been basically kind of the focus on how we saw the first breakthroughs.

So exploration in a multi hierarchical way. So like 22 times a second exploration has a very different meaning than it does in terms of should I gather resources early or should I wait or so on. So how do you solve the long term? Let's talk about the internals of AlphaStar. So first of all, how do you represent the state of the game as the input, right? How do you then do the long term sequence modeling? How do you build a policy? So what's the architecture like?

So AlphaStar has obviously several components, but everything passes through what we call the policy, which is a neural network. And that's kind of the beauty of it. There is I could just now give you a neural network and some weights. And if you fed the right observations and you understood the actions the same way we do, you would have basically the agent playing the game. There is absolutely nothing else needed other than those weights that were trained. Now, the first step is observing the game and we've experimented with a few alternatives. The one that we currently use mixes both spatial sort of images that you would process from the game, that is the zoomed out version of the. Of the map, and also a zoom inversion of the camera or the screen, as we call it. But also we give to the agent the list of units that it sees more of as a set of objects that it can operate on that is not necessarily required to use it. And we have versions of the game that play well without this set vision that is a bit not like how humans perceive the game, but it certainly helps a lot because it's a very natural way to encode the game is by just looking at all the units that they have. Properties like health, position, type of unit, whether it's my unit or the enemies. And that sort of is kind of the summary of the state of the game, that list of units or set of units that you see all the time.

But that's pretty close to the way humans see the game. Why do you say it's not? Isn't that you're saying the exactness of it is not similar to humans.

The exactness of it is perhaps not the problem. I guess maybe the problem if you look at it from how actually humans play the game, is that they play with a mouse and a keyboard and a screen and they don't see sort of a structured object with all the units. What they see is what they see on the screen. Right?

Yeah. So remember that there's a. Sorry to interrupt. There's a plot that you showed with camera base where you do exactly that right around. And that seems to converge to similar performance.

Yeah, I think that's what we're kind of experimenting with, what's necessary or not, but using the set. So actually if you look at research in computer vision where it makes a lot of sense to treat images as two dimensional arrays, there's actually a very nice paper from Facebook. I think I forgot who the authors are, but I think it's part of kming's group. And what they do is they take an image which is this Two dimensional signal and they actually take pixel by pixel and scramble the image as if it was just a list of pixels. Crucially, they encode the position of the pixels with the XY coordinates. And this is just kind of a new architecture which we incidentally also use in Statcraft, called the Transformer, which is a very popular paper from last year, which yielded very nice result in machine translation. And if you actually believe in this kind of awe, it's actually a set of pixels, as long as you encode xy, it's okay. Then you could argue that the list of units that we see is precisely that, because we have each unit as a kind of pixel, if you will, and then their XY coordinates. So in that perspective, without knowing it, we use the same architecture that was shown to work very well on Pascal and imagenet and so on.

So the interesting thing here is putting it in that way, it starts to move it towards the way you usually work with language. So what? And especially with your expertise and work in language, it seems like there's echoes of a lot of the way you would work with natural language in the way you've approached Alpha Star.

Right.

Does that help with the long term sequence modeling there somehow?

Exactly. So now that we understand what an observation for a given time step is, we need to move on to say, well, there's going to be a sequence of such observations and an agent will need to, given all that it's seen, not only the current time step, but all that it's seen. Why? Because there is partial observability. We must remember whether we saw a worker going somewhere, for instance, because then there might be an expansion on the top right of the map. So given that, what you must then think about is there is the problem of given all the observations, you have to predict the next action. And not only given all the observations, but given all the observations and given all the actions you've taken, predict the next action. And that sounds exactly like machine translation. And that's exactly how I saw the problem, especially when you are given supervised data or replays from humans, because the problem is exactly the same. You're translating essentially a prefix of observations and actions onto what's going to happen next, which is exactly how you would train a model to translate or to generate language as well. You have a certain prefix, you must remember everything that comes in the past, because otherwise you might start having non coherent text. And the same architectures, we're using LSTMs and transformers to operate across time, to kind of integrate all that's happened in the past. Those architectures that work so well in translation or language modeling are exactly the same than what the agent is using to issue actions in the game and the way we train it. Moreover, for imitation, which is step one of AlphaStari, is take all the human experience and try to imitate it. Much like you try to imitate translators that translated many pairs of sentences from French to English. Say, that sort of principle applies exactly the same. It's almost the same code, except that instead of words, you have slightly more complicated objects, which are the observations and the actions are also a bit more complicated than. Than a ward.

Is there a self play component then too? So once you run out of imitation.

Right, so indeed you can bootstrap from human replays, but then the agents you get are actually not as good as the humans you imitated. Right. So how do we imitate? Well, we take humans from 3000 MMR and higher. 3000 MMR is just a metric of human skill. And 3000 MMR might be like 50% percentile. Right? So it's just kind of average human.

What's that? So maybe.

Quick pause.

MMR is a ranking scale, the matchmaking rating for players. So it's 3,000. I remember there's like a master and a grandmaster. What's 3,000?

So 3,000 is. Is pretty bad, I think. It's kind of goals level.

It just sounds really good relative to chess, I think.

Oh yeah, yeah. No, the, the ratings, the best in the world are at 7,000. I remember 7,000. So 3,000, it's a bit like Elo. Indeed. Right. So 3,500 just allows us to not filter a lot of the data. So we like to have a lot of data in deep learning, as you probably know. So we take these kind of 3500 and above. But then we do a very interesting trick, which is we tell the neural network what level they are imitating. So we say this replay you're going to try to imitate to predict the next action for all the actions that you're going to see is a 4000 MMR replay. This one is a 6000 MMR replay. And what's cool about this is then we take this policy that is being trained from human and then we can ask it to play like a 3000 MMR player by setting a beat saying, well, okay, play like a 3000 MMR player or play like a 6000 MMR player. And you actually see how the policy behaves differently. It gets worse economy. If you play like a gold level player it does less actions per minute, which is the number of clicks or number of actions that you will issue in a whole minute. And it's very interesting to see that it kind of imitates the skill level quite well. But if we ask it to play like a 6000 MMR player, we tested of course, these policies to see how well they do. They actually beat all the built in AIs that Blizzard put in the game. But they're nowhere near 6,000 MMR players. Right. They might be maybe around Gold level Platinum perhaps. So there's still a lot of work to be done for the policy to truly understand what it means to win. So far we only asked them, okay, here is the screen and that's what's happened on the game until this point. What would the next action be? If we ask a pro to now say, oh, you're going to click here or here or there. And the point is experiencing wins and losses is very important to then start to refine. Otherwise the policy can get loose, can just go off policy, as we call it.

That's so interesting that you can at least hope eventually to be able to control a policy approximately to be some MMR level. That's so interesting, especially given that you have ground truth for a lot of these cases.

Right.

Can I ask you a personal question?

Sure.

What's your MMR?

Well, I haven't played Starcraft 2, so I am unranked, which is the kind of lowest league.

Okay.

So I used to play StarCraft, the first one and.

But you haven't seriously played.

No, not Starcraft. So the best player we have at DeepMind is about 5000 MMR, which is high Masters. It's not at Grandmaster level. Grandmaster level would be the top 200 players in a certain region like Europe or America or Asia. But for me it would be hard to say. I am very bad at the game. I actually played Alphastar a bit too late and it beat me. I remember the whole team was, oh, Oriole, you should play. And I was, oh, it looks like it's not so good yet. And then I remember I kind of got busy and waited an extra week and I played and it really beat me very badly.

How did that feel? Is that. Isn't that an amazing feel?

That's amazing. Yeah. I mean, obviously I tried my best and I tried to also impress my. Because I actually played the first game, so I'm still pretty good at micromanagement. The problem is I just don't understand Starcraft 2. I understand Starcraft and when I played Starcraft, I probably was consistently like for a couple years, top 32 in Europe, so I was decent. But at the time we didn't have this kind of MMR system as well established, so it would be hard to know what, what it was back then.

So what's the difference in interface between Alphastar and Starcraft and a human player in Starcraft? Is there any significant differences between the way they both see the game?

I would say the way they see the game, there's a few things that are just very hard to simulate. The main one, perhaps which is obvious in hindsight, is what's called cloaked units, which are invisible units. So in Starcraft you can make some units that you need to have a particular kind of unit to detect it. So these units are invisible. If you cannot detect them, you cannot target them. So they would just destroy your buildings or kill your workers. But despite the fact you cannot target the unit, there's a shimmer that as a human you observe. I mean, you need to train a little bit, you need to pay attention, but you would see this kind of space time, like distortion, and you would know, okay, there are.

Yeah, yeah, there's like a wave thing.

It's called spacetime distortion.

I like it.

That's really like. The Blizzard term is shimmer.

Shimmer.

And so this shimmer, professional players actually can see it immediately, they understand it very well, but it's still something that requires certain amount of attention and it's kind of a bit annoying to deal with. Whereas for Alphastar, in terms of vision, it's very hard for us to simulate sort of, are you looking at this pixel in the screen and so on. So the only thing we can do is there is a unit that's invisible over there. So Alphastar would know that immediately. Obviously still obeys the rules, you cannot attack the unit, you must have a detector and so on. But it's kind of one of the main things that it just doesn't feel there's a very proper way. I mean, you could imagine, oh, you, you don't have hypersensation. Maybe you don't know exactly what it is, or sometimes you see it, sometimes you don't. But it's just really, really complicated to get it so that everyone would agree, oh, that's the best way to simulate this. Right.

You know, it seems like a perception problem.

It is a perception problem. So the only problem is people, or you ask, oh, what's the difference between how humans perceive the game? I would say they wouldn't be able to Tell a shimmer immediately as it appears on the screen. Whereas Alpha Star in principle sees it very sharply. Right. It sees, it sees that the bit turned from 0 to 1, meaning there's now a unit there, although you don't know the unit or you don't know, you know, you know that you cannot attack it and so on. Got it. So that from, from a vision standpoint, that probably is the one that is kind of the most obvious one. Then there are things humans cannot do perfectly, even professionals, which is they might miss a detail or they might have not seen a unit. And obviously, as a computer, if there's a corner of the screen that turns green because a unit enters the field of view, that can go into the memory of the agent, the lstm, and persist there for a while and for whatever, for however long is relevant. Right.

And in terms of action, it seems like the rate of action from AlphaStar is comparative, if not slower than professional players. But is there. But it's more precise is what I mean.

Right, so. So that's, that's a very like. That's really probably the one that is causing us more issues for a couple of reasons. Right. The first one is starcraft has been an AI environment for quite a few years, in fact, I mean, I was participating in the very first competition back in 2010, and there's really not been kind of a very clear set of rules, how the actions per minute, the rate of actions that you can issue is. And as a result, these agents or bots that people build in a kind of almost very cool way, they do like 20,000, 40,000 actions per minute. Now, to put this in perspective, a very good professional human might do 300 to 800 actions per minute. They might not be as precise. That's why the range is a bit tricky to identify Exactly. I mean, 300 actions per minute precisely is, is probably realistic, 800 is probably not. But you see humans doing a lot of actions because they warm up and they kind of select things and spam and so on, just so that when they need, they have the accuracy. So we came into this by not having kind of a standard way to say, well, how do we measure whether an agent is at human level or not? On the other hand, we had a huge advantage, which is because we do imitation learning agents turn out to act like humans in terms of rate of actions, even precisions and imprecisions of actions. In the supervised policy, you could see all these. You could see how agents like to spam. Click to move here. If you Played especially Diablo, you would know what I mean. I mean, you just spam, oh, move here, move here, move here. You're doing literally like maybe five actions in two seconds. But these actions are not very meaningful. One would have sufficed. So on the one hand, we start from this imitation policy that is at the ballpark of the actions per minute of humans, because it's actually statistically trying to imitate humans. So we see this very nicely in the curves that we showed in the blog post. Like there's these actions per minute and the distribution looks very human like. But then of course, as self play kicks in, and that's the part we haven't talked too much yet. But of course the agent must play against itself to improve, then there's almost no guarantees that these actions will not become more precise or even the rate of actions is going to increase over time. So what we did, and this is probably kind of the first attempt that we thought was reasonable, is we looked at the distribution of actions for humans for certain windows of time. And just to give a perspective, because I guess I mentioned that some of these agents that are programmatic, let's call them, they do 40,000 actions per minute. Professionals, as I said, do 300 to 800. So what we looked is we look at the distribution over professional gamers and we took reasonably high actions per minute, but we kind of identify certain cutoffs after which, even if the agent wanted to act, these actions would be dropped. But the problem is this cutoff is probably set a bit too high. And what ends up happening, even though the games, and when we ask the professionals and the gamers, by and large, they feel like it's playing human. Like there are some agents that developed maybe slightly too high APMs, which is actions per minute combined with the precision, which made people sort of start discussing a very interesting issue, which is, should we have limited this? Should we just let it loose and see what cool things it can come up with?

Right, interesting.

So this is in itself an extremely interesting question, but the same way that modeling the shimmer would be so difficult, modeling absolutely all the details about muscles and precision and tiredness of humans would be quite difficult. So we're really here in kind of innovating in this sense of, okay, what could be maybe the next iteration of putting more rules that makes the, the agents more human. Like in terms of restrictions.

So, yeah, putting constraints that. More constraints, yeah, that's really interesting. That's really innovative. So one of the constraints you put on yourself, or at least focused in, is on The Protoss race, as far as I understand. Can you tell me about the different races and how they. So Protoss, Terran and Zerg, how do they compare? How do they interact? Why did you choose Protoss? In the dynamics of the game, seen from a strategic perspective.

So in Starcraft there are three races. Indeed. In the demonstration we saw only the Protos race. So maybe let's start with that one. Protos is kind of the most technologically advanced race. It has units that are expensive but powerful. So in general you want to kind of conserve your units as you go attack. So you want to. And, and then you want to utilize these tactical advantages of very fancy spells and so on, so forth. And at the same time they're kind of. People say like they're, they're a bit easier to play perhaps. Right. But that I actually didn't know. I mean I. I just talk to now a lot to the players that we, we work with, Team DLO and Mana and they said oh yeah, Protos is actually. People think is actually one of the easiest races. So perhaps the easier. That doesn't mean that it's, you know, obviously professional players excel at the three races and there's never like a race that dominates for a very long time anyways.

So if you look at the top, I don't know, 100 in the world, is there one race that dominates that list?

It would be hard to know because it depends on the regions. I think it's pretty equal in terms of distribution. And Blizzard wants it to be equal. Right. They wouldn't want one race like Protoss to not be representative in the top place. So definitely they tried it to be balanced. Right. So then maybe the opposite race of Protoss is Zerk. Zerk is a race where you just kind of expand and take over as many resources as you can. And they have a very high capacity to regenerate their units. So if you have an army, it's not that valuable in terms of losing the whole army is not a big deal as Zerg because you can then rebuild it. And given that you generally accumulate a huge bank of resources. Zergs typically play by applying a lot of pressure, maybe losing their whole army, but then rebuilding it quickly. So although of course every race, I mean there's never. I mean they're pretty diverse. I mean there are some units in Zerg that are technologically advanced and they do some very interesting spells and there's some units in Protos that are less valuable and you could lose a lot of them and rebuild them. And it wouldn't be a big deal.

All right, so maybe I'm missing out. Maybe I'm going to say some dumb stuff, but summary of strategy. So first there's collection of a lot of resources, right? So that's one option. The other one is expanding, so building other bases. Then the other is obviously building units and attacking with.