Reflection AI’s Misha Laskin on the AlphaGo Moment for LLMs | Training Data

I think someone needs to solve the kind of depth problem. The field as a whole, I think, or large labs have been really working on the breadth. That's amazing. And there's a big market for that and a lot of very useful things that get unlocked. But someone needs to solve the depth problem too.

Hi everyone. Welcome to Training Data. Today we're hosting Misha Laskin, CEO and co founder of Reflection AI. Misha is a former research scientist at DeepMind and his co founder Yanis was the creator of AlphaGo, an RLHF lead for Gemini. Together they are building universal superhuman agents. We'll chat about why we're so far from the promise of AI agents, even with best in class models today, what we need to truly unlock agentic capabilities for LLMs and what we can learn from those who have built both the most powerful agents in the world, like AlphaGo and AlphaZero, and the most powerful LLMs in the world, like Gemini. So, Misha, to kick things off, we'd love to learn a little bit more about your personal background. You were born in Russia, moved to Israel when you were one, then the United States, in Washington State when you were nine. Your parents were pushing forward the field of technology and research in chemistry. And I think that inspired a lot of your love for pushing forward the frontier of technology and getting into the world of AI today as well. Can you share a little bit more about what inspired you to get into this field and what has inspired you throughout your childhood and adulthood so far?

Yeah, definitely. When my parents, we immigrated from Russia to Israel, it was when the Soviet Union collapsed and they came to Israel basically. I mean, with, with nothing. They had, I think $300 in their pocket, which was then stolen from them as soon as they landed because they put down a deposit for an apartment and well, didn't get that, just disappeared. They, I, I don't even know if there was an apartment. So they, and they didn't speak Hebrew, so they decided to kind of pursue a PhD in chemistry at the Hebrew University of Jerusalem. But that's not because, it's not because there was kind of some kind of internal passion for kind of academia at that time. It was more that Israel was giving stipends to Russian immigrants to get further educated. And it was interesting asking my parents about this because they kind of grew to love their craft as they got excellent at it. So I think from them that might be the thing that I took away most. It's not that they are particularly impassioned about chemistry to start, but as they kind of learned about it, got curious about it, and really went deeply into it. I think they became kind of masters of their craft. And that's something that I found really important myself. Moving from there to the States. Well, my parents promised that we're moving to this kind of beautiful state with all these mountains, this Washington State. And I remember, like, we're taking the plane flight. I mean, I bragged to all my friends in Israel, and, you know, I was really excited. So, yeah, we're flying. I do see mountains in the distance, but then the plane does a sort of U turn. And if many people don't know this about Washington State, but it's a kind of half desert and half, you know, mountainous and forests. And the plane turned to the desert. And so I see it kind of landing in this kind of middle of nowhere. And I asked my parents, like, where are the mountains? Like, well, you saw it from the plane. The reason I'm saying this is because I basically moved to a very boring place. Which city. There's this area in Washington state called the Tri Cities, and it has some pretty interesting history. The reason it exists is because it was one of the sites of Manhattan Project. So this is where the plutonium was enriched at this place called the Hanford site, which is a sister site to Los Alamos. So it's a town that was basically built for that in the 1940s and is in the middle of nowhere, kind of like Los Alamos is. And there's not much to do. I remember seeing my first tumbleweeds. You literally saw tumbleweeds kind of rolling across the highway. I found myself in a place where I didn't really speak the language that well, English. I was in this kind of very rural place that was different from where I grew up. Didn't have many friends and had a lot of time on my hands. And the way I got interested in science at the time it was physics is after, you know, after getting sort of video games out of my system, I got bored again. And I found these Feynman lectures that my parents had on. The Feynman lectures in physics and the. They were so interesting because Feynman had this way of explaining incredibly complex things in a way that basically a person who's. I mean, not that educated mathematically at the time can really understand something fundamental about how the world works. And that is probably the thing that inspired me most. I just got really interested in this idea of understanding how things. How things work at this sort of root level and working on problems that are Sort of root node problems and that, I mean, there are all these examples I was reading about, like the invention of the transistor, which was invented by Joe Bardeen, a theoretical physicist, or how GPS works. It turns out you need to understand, you need to make relativistic calculations, which is coming from Einstein's theory of special relativity. And I wanted to work on things like that. So that's why I got into physics. I pursued it for a while, got educated in it, got a PhD in it. And I think maybe the critical bit of information that I had not, did not have in my context then was that you don't just want to work on root node problems, you want to work on root node problems of your time. You want to work on the things that are really can be unlocked now. And it's no surprise, you know, when you're being trained as a physicist, you're doing like these really interesting problems and learning very interesting things about how people thought about these, you know, about physics. Basically 100 years ago, 100 years ago, physics was the root node problem of our time. And that's why I decided to not pursue professionally. I kind of did a 180 and wanted to do something very practical. So I started a startup. But as I was working on that, I started noticing deep learning as a field taking off. And in particular AlphaGo, like when AlphaGo came out, there was something, something just felt very profound about it. That how, how do they get this system that is like a computer to not just perform at a higher level than a human, but do so creatively? In Alphago there was this famous move called Move 37 where the, well, where the neural network made this kind of move that looked like, it just looked like a bad move. Lisa Dahl was really perplexed by it. Everyone was perplexed by it. It just looked like a mistake. And it turned out that 10 moves later, this was actually the optimal move to kind of put AlphaGo in a winning position for the game. And so you could tell that this is not just, this is not just a brute force thing. This is not a thing that is just. It's obviously, I mean the system does a lot of search, but it's able to find creative solutions that people hadn't thought of before. And so that kind of made me feel pretty viscerally that solving agency, this was the first real large scale superhuman agent. Yeah, that seemed profound. That's why I got into AI and got into AI to build agents from the first day. So there was a kind of non linear Path where I was an outsider, I wasn't really. I mean, it was competitive back then, too. And OpenAI released these requests for research. At the time, this was maybe 2018 or 19. Their requests for research were just things that they wanted other people to work on. I think by the time that I was looking at this list, it was actually already stale. So I don't think they really cared about those problems. But it gave me something concrete to work on. And I started making progress against one of these problems. I felt like I was making progress. I don't know how much progress I actually made. I was kind of peppering a few research scientists from OpenAI with questions and kind of, I mean, effectively cold emailing them until maybe I got too annoying and they started, you know. Well, I guess they responded rather, I'd say graciously. And I built some. Some relationships there. And one of them introduced me to Peter Abeel, who is a PI at Berkeley. One of the greatest, I think, researchers of our time in the field of reinforcement learning and robotics. But his lab kind of does everything. They have some of the most impactful generative model research as well. One of the key kind of diffusion paper. Diffusion model papers came out of there and honestly got lucky. He took a chance on me and kind of brought me into his group. He had no reason to there. After I was on the other side and looking at applicants coming into the group, there was really no reason for him to take someone who was not vetted. So he kind of took a chance. And that, I think, was my foot into the field.

You and your co founder, Yanis, have worked on, I think, some of the most incredible projects out of DeepMind and Google, maybe. Can you give the folks here a taste of some of the projects that you both worked on, like Gemini and AlphaGo, maybe. What were the key learnings from each and how they propelled your thinking forward to the present day?

Yeah, so Giannis was basically the reason I got into AI. He was one of the key engineers on AlphaGo. He was there in Seoul when they played against Lee Sedol. And he's actually, you know, before AlphaGo, he worked on this paper called the Deep Q. Deep Q Network is dqns. And this was actually the first successful agent of the deep learning era. This was an agent that was able to play Atari video games. And it kind of catalyzed this whole field of deep reinforcement learning back then, which was AI systems that were autonomously learning to act in, I mean, mostly video game and robotics environments. But it was the first agent it was kind of a proof point that you can learn to act in an environment in a reliable way from just raw sensory inputs coming in. This was, I think, a big unlock that was completely unclear at the time. The same way that neural networks working on Imagenet was an unlock in 2012. And then, yeah, Yanis worked on AlphaGo and the kind of subsequent line of work. There's AlphaGo, there's AlphaZero, there's a paper called MU0. And I think that really showed how far you can take this idea. It really scales relative to the models we have today, large language models. The AlphaGo model is actually really small. And it was so smart at this one thing. I think the key lessons, at least for me, from AlphaGo, were kind of encapsulated in this famous essay that Rich Sutton, this reinforcement learning researcher, or I guess, kind of a sort of father of a lot of reinforcement learning research put forth, which is this idea of the bitter lesson in that essay. He basically says that you want to. If you're building systems that are based on kind of your internal heuristics, those things will likely get washed away with systems that kind of just learn on their own, or rather systems that leverage compute in a scalable way. And he argued that there are two ways to leverage compute. One is by learning. So it's training. When we think of language models today, they're leveraging compute mostly through learning by training them on the Internet. The other way is search, which is leveraging compute to unroll a bunch of plans and then pick the best one. And AlphaGo is actually both ideas in one. I still think it's the most profound idea in AI that combining learning and search together is the optimal way to leverage computer a scalable sense. And those things together are the things that produce a superhuman agent at go. The issue with AlphaGo was that it was only good at one thing. And I remember being in the field then, and it did feel kind of stuck. The field of deep reinforcement learning, because the goal everyone set out for themselves is to build general agents, superhuman general agents. And where the field landed was superhuman, very narrow agents. And there was no clear path to how do we make them general because they were so data inefficient that if it takes 6 billion steps to train on one task, then where are you going to get the data to train on all these others? And that was the big unlock of the language model era. Like one way to think about the Internet is. Or all the data on the Internet is a collection of many tasks, like you have like Wikipedia is a task of kind of describing some historical events. Stack overflow is a task of Q and A on coding. You can think of the Internet as like a massive multitask data set and that.

Interesting.

Yeah. And that the reason we get generality from language models is because it's an. It's basically a system that's trained on tons of tasks. Those tasks aren't particularly, let's say, directed or, you know, there's no notion of reliability or agency on the Internet. So it's no surprise that the language models that come out of that aren't particularly good agents. I mean, they're obviously incredible and they do incredible things. But one of these fundamental problems in agency is that you need to think over many steps and you have some error rate over each step and that error accumulates. It's called error accumulation. And so it means if you have some percent chance that you're wrong, the first step, they'll compound very quickly over a few steps to the point where it's basically impossible for you to be reliable on a task that is meaningful. The key thing that I think is missing is that language models are systems that leverage learning. They're not yet systems that leverage search or planning in a scalable way. And so that I think is kind of the missing piece. Okay, now we have general agents, but we have general agents that are not very competent. And so you kind of want to move up the competence. And the only existence proof for that has been AlphaGo, and it's been done through search.

I really love that. And the encapsulation of how you just shared that, I think that sets the stage wonderfully for reflection. Can you share a little bit more about the original inspiration, this problem space that you're going after, and your long term vision for reflection?

I mean, the original inspiration came very much from working. Giannis and I collaborated very closely on Gemini. Giannis led the Arla Tref effort and I led reward model training, which was kind of a key part of our lhref what we were working on and what everyone is working on. These language models is in post training you align them for chat, so you align them to be good interactive experiences for some end user. So that was through products like ChatGPT or Bard, which has now been named Gemini. These language models, like the pre trained ones, are very adaptive and so you can, if with the right data mix, you can adapt them to be like highly interactive chatbots. And I think our key kind of insight from working on that was that there's nothing specific that was being done for chat. It was just, you know, you're just collecting data for chat, but if you collect the data for anything for another capability, you'd be able to unlock that as well. Of course, it's not so simple. A lot of things change in the sense. I mean, one key thing is that chat is subjective. So the algorithms that you train are different than the algorithms that you train for something that has kind of an objective like this task was done or not. There are all sorts of issues. But the main thing was that we think the architectures and models work. A lot of the things that I thought were bottlenecks have been kind of washed away with compute and scale. Like long context lengths is something I thought would be something that would be, you need a research breakthrough. And now all the players are releasing models with extremely long context lengths relative to what we thought was possible even a year or two ago. The methods for training these things and aligning them and post training them are pretty stable. And it's really. Yeah, it's a data problem and it's a data problem and a problem of how do you enable planning and search on top of these objects. And we thought we'd move faster against this problem if we did it on our own. I think we just wanted to move very quickly against it.

So you've described agents as kind of the dream, both for you and Yanis as researchers, but also for reflection. Can we pause on the word agents for a little bit? Because now it's become, now that it's become the term of 2024, everybody is calling themselves an agent and the word is starting to lose its meaning a little bit. And I imagine that you have a, probably a more pure definition of what an agent is. Maybe. Could you just explain that? How do you think about what an agent is? And when we look at some of the agents that everyone's gotten really excited about recently, it seems like they're still very early in terms of being reliable enough to be kind of colleague level true agents. So where do you think we are on that curve? What is an agent and how do we kind of get to the promised land?

Yeah, it's an interesting question since I. The term agents has been floating around within the research community for a while, I think. Well, since kind of the start of AI, But I primarily have been thinking more agents in the context of the deep learning era. So starting with DQNs. And the definition is pretty simple in that it's. It's an AI system that is able to reason on its own and take however many steps it needs to to accomplish some goal that's been specified to it. That's kind of it. And now the way that goal is specified has changed over time. In the deep reinforcement learning era, the goal was usually specified through a reward function. So for AlphaGo, it's whether you won the game of go or not. There's no, no one wrote like go, win the game of go, like via text. So that's how people usually thought about agents. They thought of agents as things that are optimizing a reward function. But there's a whole area of research even then before language models, on goal conditioned agents. So these would be either in robotics or video games. Or you set a goal for the robot, which could be you give it an image of an apple having been moved somewhere and you ask it to kind of reproduce that image and it has to act on the world and pick up an apple and move it somewhere in order to accomplish the goal. So as a short definition, it's AI systems that have to act in an environment to accomplish some goal. That's an agent.

And then I guess as a follow up, if you take for example, coding agents as one potential domain of agents, where there's been a lot of activity recently, you could say the goal is create a calculator app for me and the agent has to go and accomplish the task. When I look at what SUI agent and what Devin have done, is that in your mind, is that agentic reasoning and does kind of scaling that up get us to the promised land, or do you think there are kind of different approaches that you need more on the RL side or more on whatever other techniques you might need to use to get us to the promised land? Because I think those agents are still in the 13, 14% kind of task completion rate range. And I'm curious how we get them to 99%.

They're definitely, by the definition of agents, these are agents, they're just on a spectrum of capability, maybe not at a high level of reliability yet. I think the way most people think about agents today in context of language models is prompted agents. So you take a model, you prompt it, or you set up some flow of several prompts to get it to accomplish a task that allows anyone to kind of take a language model and take it from zero to something that's kind of working somewhere. So I think that's quite interesting. I think it can only go so far. So this I think is actually like, it's a very. I mean, this is a Kind of example of what I think the bitter lesson would apply to, because prompting things and kind of really directing them to go in like these specific ways, that's exactly the kinds of heuristics that we have that we're baking into these models to kind of try to achieve higher intelligence. I mean, every major advance in agency since the deep learning era has been kind of showing that with learning and search, a lot of that gets washed away. I think that the purpose of a prompt is to specify the goal. So you'll always need a prompt. You always need to tell an agent what to do. But if once you start deviating from that, and the purpose of the prompt is actually to put the agent on rails and where you're kind of doing the thinking for it, you're kind of telling it, okay, now just go here and do this thing that I think is going to disappear. I think that's a local thing that's happening today. Future systems, I don't think we'll have that.

So the key is that the thinking and the planning needs to happen in the AI system, not in the prompt layer. In order to kind of not hit a wall.

I think you want to offload as much as possible to. Yeah. To the AI system itself. Again, these. These language models have not been trained. They were never trained for agency. They were trained for chat interaction and predicting things on the Internet. So it's almost a miracle that you can prompt your way to getting something that kind of works. But what's interesting is that once you're able to prompt your way to something that kind of works, that's actually the best place to start for a reinforcement learning algorithm. A reinforcement learning algorithm, all it does is it reinforces good behavior and it kind of downweighs bad behavior. If you have an agent that is doing zero, that is just doing nothing, then there's no good behavior to upweigh, and so the algorithm doesn't work. This is known as a sparse reward problem. If you're not hitting your reward, like, if you're not accomplishing your task, then ever, then there's nothing to learn from. But if you've prompted your way to an agent that is kind of working like, you know, sui agent or something like this, it's getting 13% or something like this, then you have something that is like minimally capable where you can reinforce really good behavior. The challenge becomes a data challenge of where do you get the set of prompts to train on? Where do you get the environment to run these things through? I guess sui Agent does come with an environment, but for many problems you need to think about that. And then perhaps the biggest challenge is how do you verify that a thing has been done correctly or not in a scalable way? And if you can solve where the tasks come from, which usually that's through products, that's solvable, what environment you run them through, what algorithm you use, but it's really kind of what environment you run them through. And then critically, how do you verify if a thing has been done correctly or not in a scalable way? I think that's a recipe for agency.

I think that gets to the crux of the problem space in AI agents today. Just to set the stage a little bit for the problem that reflection is going after. What do you think is the current state of the market broadly in AI agents? I think many assume that we are capable of more than we actually are with the models that exist today. So what do you think the problem is and what do you think is why do you think the current attempts around AI agents are failing us today?

One way to categorize or classify what it means to be a general agent, and maybe I'll use the term universal agent, since I'll use the term generality to apply to breadth. So a universal agent needs to be abroad. A very general agent that can do many things, can handle many inputs, but it also needs to have depth in the kind of task complexity it can achieve. And so examples are AlphaGo is probably the deepest agent that has ever been built. It can do one task, so not that useful. It can play go but not tic tac toe. Yeah, the current systems language model systems like Gemini, Claude, ChatGPT, the GPT series of models lean the other way. They're very broad, they're not very capable in a depth wise sense. They're extremely impressive and capable broadly. And I think that's one of the things that's been honestly miraculous. Like as I said, the field felt like we did not have an answer to generality and then these objects came along. But now we're in the opposite end of the spectrum where we have, I think more or less de risked as a field progress towards breadth. That's especially evident with the latest generations of models like GPT4O and the latest family of Gemini models that are multimodal in the sense that they just understand other modalities at the same base layer that they understand language. You don't need to translate one modality into language. So that's, I would call that breadth. But nowhere along this Process where things train for depth. The Internet doesn't have real data around how to kind of think sequentially. The way people try to solve this problem is work on data sets that might have this structure and hope it generalizes. So math data sets, coding data sets, kind of what people refer to reasoning, which usually is reasoning along the lines of can you solve a mathematical problem? But that's still not really addressing the problem head on. I think we need methods that you can take, recipes, let's say that are general in that you can take any task category, have a bunch of prompts for it for your training data and make a language model iteratively more capable on those things. I think someone needs to solve the kind of depth problem and the field as a whole I think is or large labs have been really working on the breadth. Well that's amazing and there's a big market for that and a lot of very useful things that get unlocked. But someone needs to solve the depth problem too.

I think that takes us really nicely into the unique insight that you and Ioannis have From working on AlphaGo, AlphaZero and on Gemini and the importance of post training and data. Can you share a little bit more about how those experiences have shaped the unique perspective that you have that gets us to the unlock with the gentic capabilities.

One of the things I found very surprising about language models is how close they are to, you know, how oftentimes, even if they're not working on something that you want them to, they're actually quite close. They feel like a nudge away. They feel like they need to be grounded in the thing a bit better. And that's, I think that was the insight that led them to be good in chat. Like you could play with them and they're yeah, a bit unreliable and they kind of go off the rails sometimes but they're almost good chat companions. And so then the recipe there, there's a recipe for how do you take a pre trained language model and make it a reliable chatbot? So by reliability there it's just a. The way you measure that is with human preferences. Do people interacting with this chatbot prefer it over other chatbots or other versions of the previous versions of itself? So if the current version is much more preferred than the last few iterations ago, then you know you made progress and that progress is made by collecting data for it. So it's collecting data for the kind of queries that users input into a chat box, the outputs that the models provide and effective ranking between those outputs. So that you push the model to index over on the kind of more preferred outputs. So when we say ranking, where does that ranking come from? Well, it comes from humans. So there are either human labelers or it's something that's embedded into the product. You sometimes might see thumbs up or thumbs down. In ChatGPT, it's harvesting your thumbs to kind of know what your preference is. And that data is used to kind of align the model with the user preferences. That's a very general algorithm. That's a reinforcement learning algorithm. And that's why it's called reinforcement learning from human feedback or RLHF. You're just upweighing the things that human feedback is expressing preferences for. There's no reason why the same approach would not be possible for enabling more reliable agency. There's a whole sequence of other problems you need to solve. I think that's what the reason this is so hard is because as soon as you go into kind of agent territory, you have more than just the language outputs. You have the tools that they interact with. And the tools being, suppose you wanted to send an email or work in an IDE or anything that an agent does, it does in an environment. And that requires tools. That requires the environment. And everyone who's deploying agents is deploying agents in different environments. And so there's a challenge of how do you integrate with environments and how do you onboard agency onto them. So I think that's why it's a bit of a schlep. If you get into this kind of line of work and you have to be careful about the environments and kind of the way you structure it because you don't want to overfit to some particular environment. But conceptually it looks very similar to aligning a model for chat. There are just some more integration challenges that need to be solved along the way.

Since you view AlphaGo as kind of like the pinnacle of building an agent that was truly capable. And I imagine you're trying to usher in an AlphaGo moment with LLMs. What do you think are the differences? To me, with gameplay, you have a very clear reward function. You have the ability to do self play. Is doing kind of the reinforcement learning from human feedback. Do you think that's enough to kind of get us to an AlphaGo moment in LLMs or like, I guess, how should I think of the differences here?

I think what you said around not having a ground truth reward is a key. And maybe the key thing, what we learned from the previous era of reinforcement learning research is that if you have a ground truth reward, you're kind of guaranteed success. Like that's kind of. There have been so many very impressive project that showed this at really unprecedented scale. I mean you think aside from AlphaGo there was OpenAI's Dota 5 or AlphaStar and let's say AlphaStar and Dota 5 are a bit more niche in the sense that you kind of have to play those games to understand. But as a former Starcraft player, I was, I still am completely blown away by AlphaStar. Like the strategies that the AI discovered were. It just looked like a very smart, like a smarter than us alien came like upon the earth, decided to play this game and completely out competed us. So that's due to the existence of a number of things. But a ground truth reward is really like is extremely important for tightening that behavior. Now both with human preferences and for agency, these are very general objects and we don't have ground truth rewards for whether something is accomplished or not. Like even for a coding task. What's the ground truth of whether this was done the right way? It could pass some unit tests, but it can still be wrong. So it's a really hard problem and I think it's the, I think it's the fundamental problem for agency. There are others as well, but this is kind of the big one. The way you get around this problem for chat is again through RLHF. Well, you train reward models. Reward models are. It's a language model that predicts whether something was done or not correctly. The challenge with that, so first it works well the challenge with that is that when you don't, in the absence of a ground truth, when you have this kind of noisy thing that it can be wrong, your policy or the agent quickly gets smart enough where it finds holes in the reward model and exploits them. To give a concrete example in chat, suppose you notice that your chatbot was outputting, was outputting some, let's say harmful content or like there are some topics that you don't want it to talk about because they might be sensitive. And so you put in some data into your data mix saying like where it's examples of the chatbot kind of ignoring or not ignoring but saying I'm sorry, as a language model I cannot answer this. What can happen is that okay, you now train a reward model against this. And suppose in your data mix you really, you only put in like data points that you know, showed instances of like this kind of this happening, but not instances of a chatbot taking something like kind of sensitive and Actually like, you know, answering it. What that means is that the reward model could think that it's actually like a good thing when you just don't answer the user's query ever. Because I've only seen positive use cases of that. And when you train against that, the policy with language model will at some point get smart enough and discover that this reward model gives me high reward whenever I just don't answer, whenever I punt the question and it can collapse into a language model that just never answers your questions. And this is why it's very finicky and it's very difficult for this reason. I'm sure a lot of users who have interact with ChatGPT or Gemini or these kinds of models, probably through interacting with them, found sometimes that they kind of degrade and they all of a sudden don't answer questions as often as they used to get slightly worse at something or are politically biased in some way. And I think a lot of that is, well, it's artifacts of the data, but the artifacts and the data get amplified by bad reward functions. So that is the hardest problem.

I think if I view the rough kind of large model training pipeline or large AI system training pipeline as pre training and post training, I kind of think pre training seems largely like a solved. Like we're in the, the techniques are solved and we're just kind of in the race to scale moments. On the pre training side, post training still feels a little bit like in the kind of research phase of market where people are still figuring out what techniques will work in a general way. I'm curious if you agree with that. And in an ideal state, what is pre training responsible for doing? How should we as laymen think about it and what is post training responsible for accomplishing and how should we think about that from the perspective of a five year old?

Yeah, I would generally agree with that statement that pre training has become. There are a lot of details I need to get right and it's by no means easy. So it's a very hard endeavor, but it's a better understood endeavor at this point. And one way that I think about pre training is I actually think thinking about it through the lens of something like AlphaGo is quite simple and clear because it kind of, you know, rather than thinking about this massive Internet thing, you just think about a very clear setting which is clean setting, which is this game you can think about. The Pre AlphaGo has two phases. It has an imitation learning phase where a neural network imitates a bunch of expert amateur like expert Go players. And then it has this reinforcement learning phase. And you can think about pre training as the imitation learning phase of AlphaGo. You're just kind of acquiring the basic skill of learning to play the game. You're not, maybe your neural network then is not the best in the world, but it's pretty good. It goes from zero to pretty good. And pre training for a language model is getting going from zero to pretty good on everything, which is why it's so powerful. Post training is, I think about it as hardening good behavior. What that means is with AlphaGo, you start off with you did imitation learning. You start off at a place where you have a neural network that can do something, it can play a game pretty well. Then you apply this other recipe to it, which is reinforcement learning, which is then the network starts generating its own plans and kind of acting through the game, getting feedback and basically good actions get reinforced. That is, I would say that that's post training and you can think about from a chat perspective, you're hardening the model like good behavior along the chat axis. So it's actually quite interesting that the high level recipe for training AlphaGo and for training Gemini is actually the same. You have this imitation learning phase and then you have a reinforcement learning phase. The reinforcement learning phase in AlphaGo is just much more sophisticated than what we have now. And the reason comes back to reward models. If you have a reward model that is, you know, that is fairly noisy and exploitable, then there's only so much work, there's only so much you can do before the policy gets smart and finds a way to trick it. And so even if you threw the fanciest RL algorithm at it, like Monte Carlo tree search with AlphaGo, it may not be that effective because it collapses into this kind of degenerate state where the policies hack the reward model before it could even do any interesting search. Like suppose you're like thinking about, like if you're playing chess and you're thinking about what to do multiple moves ahead, but your kind of judgment is really bad at every move, then there's no, like, there's no point of planning 10 moves ahead. And I think that's where we are with RLHF today. There's this wonderful paper that I think is very over or underrated called Scaling laws for Reward Model Over Optimization. This is a paper from OpenAI studying this phenomenon. And what's interesting about, I mean a number of things, but it showed that this phenomenon happens at all scales. And I mean it tried a Couple of different rl href Algorithms and it happens at all scales for all algorithms that were tried in that paper. And I think that's such an interesting paper because it's the, it's the kind of fundamental problem of post training. It's that paper.

Yeah.

Just to pull on the thread a little bit. If you follow the results from Alpha 0 though then we may not need pre training at all. Is that a fair conclusion of what to make of this?

I think that at least my mental model is that the alphago part, the imitation learning is necessary more from a practicality standpoint. When, when we went from or DeepMind went from AlphaGo to AlphaStar there was no Alpha 0, Alpha 0 of Alpha Star. There's no Alpha Star 0 that was, you know, released after that or anything like this. And AlphaStar had like a big part of it was imitation learning across a lot of games. I think with AlphaGo it was like this kind of special place where you have, you don't only have a zero sum game but you can get to the end of that game fairly quickly. And so it's again like you can get that feedback about whether what you did was right or not. Got it.

Okay. So it's just way too unconstrained of a problem to throw it out generally.

Yeah, I think in practice like yeah, AlphaZero would work generally for everything if we had ground truth reward functions for everything. But because we don't, you need to do the imitation learning piece as almost this is just like a practical, like we need to get into the game somehow.

You described earlier the importance from a technical perspective of having an agent in its environment, also from a product distribution and getting the product in users hands perspectives. It's important to think about what the right task categories are for users to first interact with the most powerful agents. What are some of the task categories that are on your mind and what do you imagine are some of the possibilities that users could use these in their daily workflow?

If you want to make progress along the depth axis you could go for like AlphaGo first which is like a really hard thing or you could kind of expand concentrically in the sort of complexity of the tasks you're able to handle. And we are focused on kind of enabling depth but in this sort of concentric way. And we care a lot about having a general recipe that does not kind of inherit heuristics that are special to some tasks. So from a research perspective we're building general recipes for this now you have to ground those recipes in something to show progress. And at least for us, it's important to show diversity of environments. And so we're thinking about a number of different types of agents, web agents, coding agents, kind of OS, computer agents. The important thing for us is to just show that you can have a general recipe for enabling agency.

Switching gears a little bit, you've attracted a seller team already. Who else are you looking to recruit on your team?

Yeah, we've been fortunate to be able to draw some talent from the top AI labs in the industry. And I think a lot of that has to do with. Well, with both of the work that Giannis and I did. But definitely, you know, I think a lot of credit goes to Giannis and his reputation. You know, that there is this. I was watching the Michael Jordan documentary, and Michael Jordan was. Well, one of the reasons he was so effective is because he was such an incredible kind of individual, like, basically contributor to the game, maybe the best that he really inspired people on his team to get to his level, even if they couldn't get there. And Giannis has this effect on people. Like, I worked very closely with him on Gemini and he had that effect on me. I don't know if I ever got to Yanis level, but I aspired to, and I definitely became a much better engineer and researcher through the process. And I think that's a lot of the draw, is that you get to learn a lot from him. We're primarily continuing to look for. So we're not hiring out quickly, we're hiring out, I think, more methodically. We're looking for. Definitely interested in other researchers and engineers joining us on this mission. I'd say a commonality between everyone who's joined is we're all very hungry. Maybe that's how I'd put it. Giannis and I could have stayed and tried pushing agents at DeepMind. And as I said, I think the reason we decided to do it in our own way is because we think we can move quickly and much faster against this goal. And some of this urgency is driven by a real belief that we are three or so years away from something that resembles a digital AGI. And. And by that, that's what I've been referring to as universal agents. Something has both this kind of breadth and depth of knowledge, and that means we're actually on a very accelerated timeline. You're a few months in, you're kind of 5% away from hitting that timeline. And maybe some of this is also driven by how quickly AlphaGo went from experts in the Field doubting this is possible. Things kind of decades, human level or expert human level. Go play was decades away and how effectively they were able to solve that problem within months. I think we're seeing a similar kind of acceleration happening with language models. There's. I think one viewpoint you can have is that we've saturated a lot of what we can. We're on the sort of at the tail end of an S curve, and we don't. We don't view it that way. We think we're still. We're still on an exponential. Part of the reason is that these things are so bulky and slow to train, that there's no way collectively, as a, you know, as a field of researchers and engineers, that we've optimized it yet. Like, if it takes a few months to run and like a few months and a few billion dollars to run the biggest model, then how many experiments can you really run? So, yeah, we kind of see things going at an accelerated pace, and we think solving the depth and reliability problem is something that is not getting the kind of attention it needs. Like, there are groups that are following this, as I would call it, more like side quests within these big companies, but I think you need a player that is focused entirely on it to solve this problem.

Yeah.

I love the framing of main quest.

Versus side quest, and I love the hunger and the zero complacency and impatience in a healthy way that you and the rest of the team have. And the only other thing I'd highlight is the revered reputation that you described for Giannis with inspiring and motivating other people. I think is true for you and Giannis, from everyone we've known, we know at DeepMind.

So three years until I have an agent that will write my memos for me. Hopefully last, I think, three years.

Yeah, I think the memos might be.

Coming sooner because that was one of my burning questions. Is this, like, decades away? Is this months away? It sounds like you're closer to the, you know, months or small number of years away.

I think small number of years.

Wow.

Yeah, it. Yeah, it's honestly kind of alarming. I think the speed at which the. The field is moving and part of. Yeah, part of depth and reliability. It's also like. It is. I mean, reliability is safety, so you want to. You want these systems to be safe. I think that there's a lot of very interesting research in terms of. There's a recent paper from Anthropic on kind of mechanistic interpretability, and that whole line of work is really interesting and I think starting to kind of get to the point where there's utility in it as well in terms of finding like, neurons in the model that are like, you know, lying neurons or, you know, that you can kind of suppress. But to me, safety is, is reliability. If the thing is kind of running around your computer breaking all sorts of things, that's an unsafe system. Maybe it's like a utilitarian safety. Like you just want these things to work and, and do what you intended them, but what you ask them to do.

So I have a few years to find another hobby other than my memo writing then.

Yeah, well, or maybe you'll just have an army of AI interns that, you know, will do all the research work for you.

Can't wait.

Wrapping up our topic around reflection. If everything goes right. What is your dream for reflection?

I think the. There are two angles at this question. One is we're working on this because this is the kind of scientific root node problem of our time. We're scientists. That's why we're so kind of interested in committing it. And it's really. There's a world where you get to be part of one of the most exciting journeys in science ever made. And you've accomplished your goal of building universal agent. You have highly safe, reliable digital agents running around on your computer, basically doing things that tedious work that you don't necessarily want to do. I think rather than people going and spending less time working, I don't think the human need or like the human need to be productive and to contribute is going to change. I just think the capacity of each human's ability to produce and, you know, impact the world is going to dramatically increase. As, you know, in my line of work there as a researcher, there are so many things that I spend time on that a smarter AI could help me out with to make faster progress towards our, towards our own goal. I mean, this is kind of a circular. But if we had something close to a digital AGI, we get much faster to solving the problem. Digital AGI, that's one angle. I think the, the other angle is from. I guess, I guess we've kind of moved on to the other angle. It's from the user perspective of a lot of the things that we do on a, on a computer are, you know, are. Maybe you can think about the computer as like the first digital tool that we've been introduced to as a, you know, as people in the same way that there were, you know, hammers and chisels and sickles that people used. And I think we're moving towards the kind of layer beyond that where instead of you having to kind of learn how to use all these tools with great precision and spend all your time on this, which actually is kind of time taken away from achieving whatever personal goals people have, that you kind of have these incredibly helpful AI agents that can help you bring kind of any goal that you have to fruition. And I think it's very exciting because I think the kind of ambition of our individual goals, it's already increasing in this local sense. A software engineer can get a lot more done today with these tools, but this is just the beginning. And I think we'll be able to really set dramatically more ambitious goals for ourselves and for these sorts of things we want to achieve, simply because we can offload a lot of the work that's needed to get there to these systems. So these are some things that I'm really excited about.

We'll close it out with a few questions that we like to ask everybody about the state of AI. First question, what are you most excited about in AI, in your field or more broadly in the next one year, five years and 10 years?

I think there are a number of things that. One of the local one that comes to mind is, because the paper is fresh, is this kind of work on mechanistic interpretability in that, I mean, these models are largely black boxes, and. It's really unclear how to study them as, like, what's the neuroscience of language models, like, if you think about them as brains? And this seems like a really interesting line of work that is now starting to see kind of signs of, well, signs of it working beyond toy settings. So maybe like, the sort of neuroscience of language models is, I think, kind of a really interesting field in AI to get into. And more generally, if I was in academia, I'd probably be looking a lot at the science of AI. So this is. The neuroscience of AI is one thing, but there are all sorts of things one can investigate in terms of, well, what really determines the scaling laws that these models have, both from a theoretical perspective and from an empirical perspective, how you change data mixes. Maybe kind of taking a step back, we're. We're basically in the, like, equivalent of what the late 1800s looked like for physics. Electricity was being discovered. No one knew why it worked or how. There's a lot of, like, there are a lot of empirical results, but there was no kind of theory behind it, which just meant that they were not very well understood. And then this, like, very Rich set of theoretical models were developed that were very simple to understand these phenomena and that gave rise to actually basically the next wave of empirical breakthroughs. And so I think the science of AI is kind of in that state right now. And I'm very excited to see where that goes.

So interesting. Who do you admire most in the world of AI?

I think most people, when getting this question or some people might kind of put someone. Yeah, maybe I'll take that back and say, like, I want to emphasize people that I admire based on having worked with them and kind of see how they operate. Because through my last kind of number of years in AI, I think there are a handful of people like this who've inspired me. And one of them is so Peter Abeel is certainly one of them. He is, I think I've never seen anyone, I think, almost operate as efficiently as Peter. That was something that. And to date, like since meeting him and to date, like, I think there's a sort of, you think a lot about research as a creative pursuit oftentimes. And I think what I learned from Peter is just sort of operational competence and efficiency around this. He's very creative as well. And like his lab does a lot of creative work, but I think it's brought like these things are very hard and they need to be pushed hard and with, with great focus. And he, he ran his lab like it's the tightest trip that I've ever been on and really helped focus like all the projects. And so, yeah, I think I look up to him a lot both in terms of his, the work that he's done. Obviously it's remarkably cross field. Like he's both done well, incredible breakthrough work in reinforcement learning and unsupervised learning, generative modeling. And a lot of it has been, I think from kind of recognizing and enabling like talent. So there was this very much. It was like the, the group was, it was a bunch of independent thinkers, students, PhDs and people are kind of pursuing like what was interesting to them. But I, the way I saw like Peter is a sort of like great amplifier.

Yeah.

Like he kind of helped people amplify and focus on the thing that really mattered within, within their pursuit. I think in a few other people that come to mind. So my, my manager at DeepMind, his name is Vlad Mini. He's. Yeah, I think also like a very, an incredible, very creative scientist. He was first author of the Deep QN paper and then there was actually, there are actually two papers at the time. There's a, like this, a 2C a 3C papers. These are basically the two algorithms that defined reinforcement learning and he deep reinforcement learning. And he kind of really pioneered both. Yeah, I think his strength is like he was extremely kind and people oriented. Very humble despite his accomplishments. Giannis as well. I mean definitely. I mean Giannis has the Michael Jordan effect. I think he really. Yeah, he just, he just wanted to be the best you could be when you work with him. And there are team was like quite small and people pushed really hard in order to like largely I think inspired by. By him. Yeah. These are some people I really look up to.

Thank you for sharing that. It's so interesting to hear you say that about everyone. And one comment on Peter Beal. Uh, I tell him all the time that he's also just created a mafia of founders in the last couple of years and it's probably because he's taught them how to do many things and there's a self selection that's naturally happening. The creative thinkers and the independent thinkers who come into his lab. But he's also taught them a lot about how to run a tight ship and how to focus incredibly well. Um, so I'm sure that doesn't come without intention on his part.

Maybe. Last question. Any advice you have for founders building an AI, you are, you're just starting your journey right now and I'm sure you've asked others for a lot of advice. What advice would you pass on to the next generation?

I think one, I think I'll be in a better position to answer that question in a few years in a way that is much more meaningful. But I'll actually provide a piece of advice that I lived through through my previous startup has nothing to do with AI. To just work on things that are internally really matter to you in a way that is almost independent from what's happening around you in a way that when things go bad, it's still interesting to you. There's just some fundamental drive around this problem that independent of everything else that's happening, is just really interesting to you. Maybe I say that about AIs because there's this. It's such an interesting, highly capable, cool technology and so there's this sort of, I think, appeal of taking it and just kind of like, well, let's just see what we can do. I think you inevitably find yourself in a hard place without having a very strong internal compass of independently of AI. The what it is that's important to you and what you want to do. And so having been in that position previously that's kind of what I've done, what I would have done differently and what I would advise people to do.

I really love that.

The line that I like to think about is play in your own stadium and don't get distracted by the glitz and glamour of someone else's stadium. You need that internal drive and grit and obsession with the problem to get you through all the tough times.

Yeah. And I think there are things that come with it. Like if you really care about some problem, like you will care about the customers who you're solving the problem for. Like, like having customers that you don't care about is like a terrible place to be. So I think, yeah, it has to come. And it's not like, I think it's kind of actually hard to control like who you care and who you don't care about. That's like a personal thing. Yeah. So you can't, it's actually really hard to force yourself to like care about something out of necessity if it's kind of not aligned with something inside you already.

So no more shopping and retailers for Misha.

Yeah. So, you know, I was building software for kind of inventory prediction for retailers and for some people, you know, they really care about that problem. Like there's a reason they would. They've kind of seen that problem. They've, you know, maybe if you're a merchant at like a, at, you know, one of these retail companies, you've really felt it viscerally. And in my case, you know, I hadn't. It was like we were trying to make, you know, just trying to build, build a revenue generating business almost kind of independent of like an internal compass.

Misha, thank you so much for joining us today. You are working on the most ambitious problem of our time. I love your framing of the root node problem of our time and I think that is today agents. And it's very clear that both your and Yannis's experiences make you the very best team at what you do. Yannis, obviously from a RLHF perspective and yours from a reward model training perspective and the insights and experiences that you've both had working on AlphaGo, AlphaZero and Gemini. We're so excited for the future of reflection.

Yeah. Thank you for having me.

Sa.