One Brain, Any Body: Google DeepMind's Keerthana on Gemini Robotics 2, Cross-Embodiment & Humanoids

Hello, and welcome back to The Cognitive Revolution. Today, I'm excited to welcome Kirtana Gopalakrishnan, staff research scientist at Google DeepMind and research lead for Gemini Robotics, back for her 4th annual appearance on the show. One of the biggest questions in AI today is how soon will general-purpose robots become broadly useful? AI is already affecting the world in major ways, even in its purely digital form, But the most sci-fi forecasts for the AI future predict that robotics will soon hit key tipping points. AI 2027, for example, predicts that humanoid robots will become useful sometime in mid-2027, and that by 2028, a whole robot economy could take shape, where robots build more robot factories, which in turn produce more robots, leading to unprecedented exponential economic growth, but also creating serious risk So how does that vision line up with the reality of robotics research today? We begin with a discussion of the extremely viral Robot Olympics held in China this summer, as I was really curious to get Kirtana's take on what it means that humanoid robots can now run faster than the fastest humans. Her point of view, as you'll hear, inverts the usual US-China dichotomy in AI. Yeah, I think that's a really interesting point. She says that while the videos are obviously impressive, skills like running on a flat track are actually relatively easy to train in simulation. And more to the point, foot speed is not really a limiting factor in the utility that robots can provide. As such, her team at Google is more focused on practical value. From there, we get into Gemini Robotics 2, a suite of 3 models that Google released this summer. Gemini Robotics ER2 for embodied reasoning, which Keerthana describes as a System 2 for robotics control, is based on Gemini Flash and is available via the API, allowing developers to define the affordances available to the models as tools, just like we do with digital agents. I played around with it and found it remarkably accessible. The other 2 models, Gemini Robotics 2 and Gemini Robotics On-Device 2, translate these higher-level tool calls down to the level of robot action and are now capable of controlling the whole robot from fingertips to feet on a wide range of form factors, but are currently available only to trusted testers. Having understood how these models work, we again zoom out and I ask Kirtana how she understands progress in robotics overall. On the one hand, like so many other AI researchers recently, she says that she has been surprised by the pace of progress. But at the same time, she still feels that robotics remains in its GPT-2 era. We have seen a number of recent demos of robots learning new tasks from just a few or even just a single human demonstration. But Kirthan argues that the range of tasks you can teach this way and the generalization profile of robotics models across different robot bodies still isn't strong enough to deliver the versatility and reliability that that real-world use cases demand. Toward the end, we move on to talk about recent progress in hardware, where robots are likely to be deployed at scale first, whether capabilities or safety, alignment, and robustness will ultimately be the limiting factor for the consumer market, and we consider the future of the field, including whether we can continue to run LLM-derived robotics models in a loop, as today's systems mostly do, Or we'll need some sort of predictive world modeling, as we humans use, to create systems that can anticipate and smoothly interact with the world. On that question, and on the timeline for key utility milestones, Keerthana isn't one to speculate too wildly. For her, these are all empirical questions to be resolved by research and experimentation. But what is clear to me coming out of this conversation is that robotics will either need to hit key tipping points soon, or some of the most aggressive timelines for AI transformation will have to be pushed back. With that, I hope you enjoy this very well-grounded update on the state of robotics with Keerthana Gopalakrishnan from Google DeepMind.

Keerthana Gopalakrishnan, staff research scientist at DeepMind and research lead for Gemini Robotics. Welcome back to The Cognitive Revolution.

Hey, nice to be here.

Great to see you again. It's been about a year. Obviously a lot has happened. I wanted to start off with something fun. There was recently a Robotics Olympics in China, and clips were obviously flying all over the internet and people were having a laugh. And at times, I think a little empathy for robots, which is something we've talked about in the past. I'd love to just start by getting your Observations and takeaways, given the depth of knowledge that you have on the subject, what stood out to you in watching clips of the Robot Olympics?

I hadn't until my dad was talking about it and he was like, wow, did you see those humanoids running? And then there were all these videos about the humanoids running and catching fire. And then I also saw my friends in India who have now entered politics and they're doing like various student protests there. And they were talking about Wow, look at that. In China, they're building the robots that can run faster than the fastest humans. So I think in a way that competition, even though from a research perspective, it's like, it's not very surprising, but I think it has really captured the public imagination in a very emphatic way. And also I think it's like a benchmark, right? Running as a benchmark and trying to see where humans are at and then where robots are at. And I think If you look at the videos from— people had made these collages where the DARPA challenge back in the— I think it was now 20 years ago, I think, those robots would just walk, open a door, and then like really fall down. And now they are— here we are. So it has been amazing progress, and it's also progress that's very much in the public's imagination.

Yeah, the fact that a robot can run faster than Usain Bolt, On the one hand, it's like crazy progress. It's crazy impressive. But one question I did have for you about it, is it valuable? Are these Robot Olympics, is it just a, you know, is its main purpose to capture the public imagination and just have something cool to show off? Do we want robots that run 20 miles an hour in humanoid form? Are, in your work, do you think, oh, I wish my robot could run faster, I need to work on that, or is that just a sideshow?

So I think one is that there are people working on research and then there are people trying to push how fast that research can go. And for better or for worse, locomotion is the research. There's locomotion and manipulation, and locomotion has advanced quite a bit. And especially it is the type of field where simulation can make a lot of difference. And a lot of these models are like trained in simulation. And because of that, locomotion has advanced quite far. So it's like a display of where the state of the art of humanoid locomotion, like bipedal locomotion, is. I think, so there are many ways of looking at it. I'm pretty sure maybe for some defense applications it's useful, but I think we are very much focused on doing robots help people and do useful things in the physical world. And I think I'm a very productive human. A lot of my friends are also very productively employed, but we don't run faster than Usain Bolt. I think it's useful, but I, It's the frontier is so vast, right? Different people are trying to push in very different directions.

Tell me more about, it's like running into a wall at their sprint and get smashed into pieces. Clearly they are not training them like that on an ongoing basis because they'd run out of physical robots to do the training on awfully quickly. So from that alone, I was like, okay, clearly they must be doing a lot of training and simulation. Tell me more about why these kinds of tasks are amenable to training in simulation and what kinds of tasks are still difficult to train in simulation?

Yeah, so sim-to-real is the thing to notice. And sim-to-real in robotics context means the gap between if I train something in simulation, can I replicate that in real? And as we know, especially with agents and stuff, the things that we can learn in simulation are probably the first ones to get solved. In that sense. So in locomotion, if you look at it, you can often model the ground or the wall as like a, you can model it and there is contact with that. And often these are like very flat surfaces. And so that makes this task a very good task for in simulation. And then you can do society of the entire robot. But for manipulation though, you really— What you care about is contact, right? And when contact happens and how objects behave, that's where current simulations start to break a little bit. It's harder to do cloth folding because the cloth have friction, but even in simulation, a lot of pick and place can still be like very solved in simulation because there also the contact dynamics are often somewhat predictable.

So basically the more rigid the body is, whether it's the floor or like here's a wooden cube, the less deformation those things are gonna have, the easier it's gonna be to do simulation.

Yeah.

Makes sense.

The co— yeah. Contacts are basically, the physics is harder to model in simulation. So where the physics is easier to model, simulation gets in.

In terms of the AI side of these robots, how, what would you infer from what we saw? Obviously, I don't know about obviously, but it sure seems like there's not reasoning components to what's going on. I don't have a clear sense of what architectures were and were not allowed and were all the models that were used to control the running robots, were they on the device or maybe some of them off the device? How would you imagine that they were typically doing that? And how high up the stack do you think they were working? Clearly they're doing a lot of stuff with stabilization, making sure it's not keeling over at any given point in time.

Yeah.

But is that it? How would you kind of, on the, because we're gonna get into Gemini Robotics coming up and there's a much higher level part of that architecture too, reasoning and understanding scenes. I'm assuming that just all of that is kind of skipped for the purposes of these Need for Speed demos.

I didn't work on these demos, so I'm only speculating, but there, there is very widely pub— public, published body of work around how to do locomotion and simulation. And I would imagine that it's like very good RL control that are probably trained from like motion imitation and then fine-tuned with like good RL that suits the robot's body and for that specific task where it's targeted towards running really fast. And then maybe they have some navigation, although I would think that just running a race is quite easy.

So you mentioned the goal of having robots do useful things for people in the world, and obviously foot speed is not typically the limiting factor on one's ability to do that. In today's world, you live and work around robots all the time. Is there anything now that you actually delegate to robots on a practical level where you're just like, they're actually good enough at this that it is abstracting away from my research interests and all these other things, but just straight up convenience, robots win on these tasks? Are there any such tasks?

Yeah, the weird thing is that I'm actually a researcher, right? So if the robots win in some tasks, then I'm going to work on the things that robots cannot. The frontier is moving. And I remember back in the day when I started working on manipulation, even pick and place was very hard to do. And at the time we didn't have the right algorithms, but now with the foundation models, this sort of stuff is quite easy to do and therefore Like the research frontier has moved into, like humanoids are a newer thing. If you look at it back in the day, our demos were just like gripper robots and now it's with humanoids and hands. So the frontier is constantly moving because the things that used to be easy, the things that seem easy kind of get solved and then the frontier moves into harder stuff. So if you look at the robotics tool release, right? Like it's whole body control, like whole body manipulation. This was not something that—

Yeah.

That anyone had shown before, and especially whole-body manipulation with generalization. So it's not just running really fast, but being able to take steps, being able to squat, and manipulating objects while doing that, and in a closed-loop sort of way, and then also generalizing and reasoning about different types of objects. I would say like maybe a year ago, if you had told me we would do this by now, I would be a bit surprised. So you can see over time, I do feel a bit proud about it because we started working on humanoids like 2 years ago. And at the time it was like just beginning, right? And they're like trying to just grab an apple and even that was really hard. And now you can prompt it. And I put out this video on Twitter where like I'm like playing with it with a watering can and it's squatting in different locations and trying to pick it up. So yeah, things have quite a bit.

Maybe another way to frame the question is in terms of the meter curve. I'm sure you're familiar with the famous meter curve of task length and the, it's typically plotted as over time, the models can do increasing task length as measured by how long it takes a human to do the task. But then there's also this factor that's like the rope, the, the AI in that case can do it at 50% success or they have an 80% success version. Obviously one big difference between LLM usage and robotics deployments is 50% success rate in the real world is probably not going to cut it unless you have a really failure tolerant task. Whereas on my computer, I can be like, eh, you kind of missed some stuff, whatever. But the robot drops something, it breaks, we have a problem. So maybe, you know, if you tried to describe where we recently were and where we are now on a meter-like curve of how hard are the tasks that robots can do well enough that they could actually be deployed? Or what are the things that they can maybe do 99+% of the time such that, hey, for these tasks, they can actually do the job. And then what would be where they're at like 80%? What would be where they're at like 50% today?

I think a lot of pick and place is close to getting very good signal and close to deployment. I do think though that, so there is like a balance that we need to strike in robotics and generally AI in general, right? You could build very narrow AI, just very good at one thing, but then you lose general— then you're not working on generalization. So you can see generalization and mastery as somewhat orthogonal axes. And working on generalization is like lifting the boat, lifting the wave for all the boats. So if you imagine like the tasks are like sitting along this manifold, Having very general common sense, very general understanding is making it much easier to get to mastery on kind of all the tasks. And this is also like very much part of our thesis in thinking about how to approach this. And you can build like, even today you can build very narrow AI that's just like very good at one or two things. But then the cost of doing the nth thing becomes exactly the same as the cost of doing the first or the second thing. But building a very general baseline then makes it much easier to quickly take it to tune it towards mastery. I think we have seen this even in LLMs, right? Like initially people had specialized models that do one thing, but now you know that ChatGPT and Gemini are like very good at a broad range of tasks. And then from those general baselines, then you can climb really high. So I think the same thesis might hold true for robotics as well.

Yeah, that's interesting. So last year when I saw your talk at The Curve, the, one of the interesting points, as I recall, was taking the base model that you had at the time and then doing some task-specific fine-tuning and showing gains from task-specific fine-tuning. Recently there's been, I don't know, at least a couple Probably several demos that I certainly haven't been able to interrogate myself in a hands-on way, but you see videos online of different robotics companies saying, hey, look, we've got few-shot generalization, or we've even got one-shot generalization. Here's a person doing the task one time, demoing it to the robot, then they can go do it. Obviously, that's great if you can get there, but what do you think is the actual likely path that people will follow? to deploy robots? Which, are they gonna do 100 demos of a task? Are they gonna have to do 10? Could it just, do you think we are on the verge of getting to a lot of things working on just a one-shot basis? What sort of investment should people be prepared to make to get their humanoid dialed in on whatever they most care about?

Yeah, I think it's going to be a spectrum, definitely. Being able to ICL or in-context show a robot how to do a task and then it doing it, Definitely reduces the time to deployment and the time to pick up the task, right? And you don't need any specific fine-tuning. You don't basically need to train it. You can do it in context. And so that definitely is a very exciting development, although you can think of it as video prompting, right? Even the older models, like when I say, let's say, pick up the object, right? And it is a very unseen object. What am I doing? I'm sending in language and then I'm getting a very general behavior out of it that I did not show it before. So that's generalization. Now instead of prompting with text, you can prompt it with an image. You can say, here's the thing, and then draw a circle around the thing that you want manipulated. And that's prompting with image. And this is prompting with video in some sense. And so I would imagine, I think the ICL results currently are on the spectrum about of how do you prompt a foundation model and you can prompt it with text, video, you can prompt it with language, you can prompt it with image. And then the question is, so now what is the generalization that you can get? Now these models are also very subject to kind of memorization in some sense. So if you really show it exactly this is what to do, they will copy it. But the question is, can they generalize, right? Can you now, if you change the scene and stuff, can they do, can they now adapt? And then the second question is, so for the same task, and the second question is, what is the level of difficulty of the task that you are prompting for? So a lot of pick and place, like the models have a lot of data and it's also like easier to comprehend. But can you show a robot to tie a trash bag? And then can the robot tie a trash bag after you show it? So I think it's still fairly early and we need to see how that evolves.

If you had to score, this is a silly question, but it might be useful for just calibrating. If you had to score robotics today on 1 to 6 GPTs, are we like in earlier conversations, I think we were, yeah, we're maybe hitting like a GPT-2 kind of moment. I think of GPT-3 as being really synonymous with in-context learning, but we're following a somewhat different path with robotics, obviously, where we have like instruction following built in, maybe in many cases even before like good in-context learning happened. So that just exposes the flaw in my question. But if you have to analogize to how many GPTs we are along the way in robotics, where would you score?

I think still GPT-2, and here's why. I think for GPT-3, we need firstly few-shot learning to work really well for a lot of different tasks. And secondly, also robotics weirdly is still very subject to cross-embodiment, right? If it just works on your robot with your specific setup, is it really a generic brain? Now, can I put that brain on my humanoid or my another robot or some other robot that I just bring in? It should be. And if it's completely helpless, in that setting, is that a generic brain? I think GPT never had this problem, right? Like we, my phone or your phone, my computer, Mac, Linux, it doesn't matter where you run it, it kind of behaves, you can expect this very similar behavior. But here I think we are very subject to which robots that you act on. And so I think there is a lot of work still needed to be done to make very generic brains that are, that can discount those factors out.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by Athena, the executive assistant company on a mission to improve how people work and live. If you want to increase your impact, you have to free up your time, and that's what Athena does best. They match you with a dedicated, full-time, top 1% executive assistant. Who can take over your inbox, calendar, travel, and everything else that's quietly eating up your week. Athena is SOC 2 Type 2 certified, so you can rest easy knowing that your sensitive data is in good hands. And as a former AI advisor to the company, I can personally vouch for how much they've invested in AI tools and training. In fact, one of the very best AI users I've ever met is an Athena client who delegated the exploration of AI tools and the development of AI workflows to his EA. Athena clients report saving an average of 15 hours a week, and the average client refers more than 2 friends a year. That, to me, checks out. I was an Athena client while running my startup, and to this day, I continue to refer friends. Go to athena.com/cognitive right now and get matched with your EA. That's athena.com/cognitive. Give yourself back a few hours this week. Go to athena.com/cognitive and see who they'd pair you with this month. Today's episode is sponsored by Parallel, where agents find answers. Most engineers today closely follow new model releases, but don't pay nearly as much attention to their agent's most important tool: web search. If you're like me, your agents are often conducting hundreds of searches per day. And while the unit cost is small, over time this does start to add up. Before starting to use Parallel, I calculated that my agent search bill would total roughly $500 this year. The good news is that I just had my agents conduct a systematic test, and I found that Parallel's Fast Mode, which costs just $1 per 1,000 queries, worked just as well as my previous default provider. And switching to it will save me roughly 80% of my search bill going forward. Parallel's infrastructure is enterprise-grade, and their suite of APIs offers a range of Pareto-optimal options that allow you to choose the right balance of quality, cost, and speed for your needs. So whether you're building voice agents that need 200-millisecond latency or long-horizon agents that need deep research, adding Parallel to your agent's toolkit Is a no-brainer. Get started for free at parallel.ai/tcr. That's parallel.ai/tcr.

Lynn, maybe that's a perfect transition to Gemini Robotics 2. How about for starters, just give us an overview of the architecture? There's like, when you go to the webpage, there's 3 models released. There's Gemini Robotics ER2. There's Gemini Robotics 2, and then there's the on-edge, on-device model. Describe how those relate to one another. Just give me kind of a broad lay of the land, and then I'll dig in with some more specific questions.

Yeah, so Gemini Robotics ER, you can think of it as like a System 2 brain that can do like very generic reasoning. It's based on the Flash line of models, but maybe more tuned towards robotics. It can do a lot of robotics tasks, and it's also widely available that you can query for like general image and video understanding and semantic understanding. Now you can think of Gemini Robotics as the VLA that is like controlling the actions model, right? How the robot moves and what it should do given a certain intention from either a user or another assistant tool model. And you can think of the Gemini Robotics on-device model as a much smaller version of that that kind of fits on the robot's computer that doesn't live in the cloud and do It, it is comparable. It's essentially doing what Gemini Robotics can do, but it's a, it's a smaller model that fits on device.

So one question I have around just how this is made. So it's based on Gemini 3.5 Flash, and with that comes a lot, right? Like you have chain-of-thought reasoning, you have the multimodal inputs, all that kind of stuff. Is this the long-term architecture that you expect robotics to work around? One obvious challenge is just language models, especially with their reasoning traces, they take a while to respond, right? So the, wait, I did a little, it was actually pretty fun. So this model's in the API, spoiler, that's another topic to touch on, but because it's in the API. I was able to have a coding agent go in and use it and expose some tools to it. And I created little just 2D demos actually, where I like feed in a, a little image of a scene, but it's actually like a little canvas in the browser. It's amazing how far I was able to get just with a few prompts in terms of helping me understand what the model can do, how easy it is to use. But the, I was also getting timing traces back from that. And so Some of the times it would be like from prompt of move the banana to the plate to the tool call is 6 seconds or something. That seems like a long time for the real world. So that makes me wonder if this is a base of convenience that you're building on now, or if this is like the paradigm that you ultimately think we will see perhaps just sped up or otherwise somehow modified that will actually have the robots working in a responsive way in our environments?

Yeah, I think it all depends on the application specifically, right? Like certain applications, like if you're on an assembly line, you can't loiter and now you need to do things really fast. But let's say if I'm at home or at some point and I'm like, hey robot, can you go fold my laundry? I don't really care about how fast it is folding laundry. I care about how well it is doing the task. I can wait. And then maybe there are also Scaling is a very important factor in LLMs and in robotics. And I would imagine that a lot of the very frontier capabilities are going to appear in the very large models. And those models are going to be naturally slower than the smaller models. So I think there is definitely a trade-off and a spectrum to be made. And I think I would imagine that the Flash and the Probe paradigms would also exist in robotics. And even in robotics, maybe you could even see like a, maybe like an on-device paradigm, like even the Flash models, they still run on cloud, right? And they, and you need very good network connectivity to talk to them. And robotics deployment could be like very distributed, right? You might wanna run it on some remote robot in like Michigan and stuff. So I think the paradigms will evolve and, but I imagine all scales to be relevant and useful for different types of applications.

Yeah, it's interesting that you contrast, 'cause typically I think Robots will go first to warehouses, factories, commercial spaces where there's a lot of control and there's a lot of consistency to the tasks that they will need to do. And there's of course professionals who can maintain large numbers of them. All of those things historically have made me think these are gonna, they're not gonna come to my home before they're like pretty well deployed in these commercial settings. But what you said just there was a little bit pushing me the other way. Where it's like they may not be fast enough for the factory, but they could definitely be fast enough to do all your chores overnight. And that could be an inversion. Do you still, I think last time we talked about this, you had said you would expect commercial settings to be the first places. Is that still the thinking? And how does that relate to the responsiveness question?

I definitely think that commercial use cases are probably more controllable, therefore, to try out different new intelligence compared to like people's homes, which are more complex. And where you cannot exert control. And even there is higher bar that you need to meet for safety given toddlers and stuff. So that still, I still believe that. However, the speed question, I don't think it's going to be like we make the slower models first and then make them faster. I think we are gonna simultaneously make the fast and the slow models and then distill them to one. And although it is very potentially possible that some capabilities are going to just emerge in the larger models, and then we'll need to figure out how to bring it to the models in the smaller scale.

So on the, just the difference between the Gemini Robotics ER2 embodied reasoning model and the base model, the Gemini 3.5 Flash that it's trained on, one of the things that my coding agent did without even me asking actually was just compared the 2 and gave both of them like, here's a scene. Here's the tools that you have. The tools were like fairly high level in the little test that I did of grab at, and it would grab the item and then place at. And it even did one where the inspired by famous videos, like the thing was moved right before the, so it tried to grab and it failed. And then it got another image, but the thing's in a different place. But this was all like relatively simple stuff. And on the tests that I ran, Flash and ER2 basically performed the same. They actually just— Both did fine. What would— how would you describe the frontier of embodied reasoning tasks specifically that ER2 can do that Flash can't do?

Yeah, so I think a lot of the benchmarks that we are tracking are like instrument reading. It's like the way that we think about is like ER is working much more closely with, for especially for robotics use cases. And that is like inspection, instrument reading, a lot of pointing. But it is a very, like, it's a collaboration, right? It's not like ER and Gemini are at competition. There's a lot of data upstreaming. So it is a very collaborative effort to how to make the Gemini models really good at robotics generally. And if it's not, but the way to appear, the way for the capabilities to progress would be like, even if the Gemini models are more generally intelligent, you can still create specialized models that were Let's say if a specific robotics use case needs like more specific engineering or fine-tuning, then you have like specialized models that are slightly better in that use case. So I think it's a question of, it's a question of what you want and what do you wanna engineer.

The model has 128,000 context window. How much is that in the context of robotics? I can I have done enough that I have an intuition for how many pages of text that is and whatever, but I don't really have a great sense for how the history of an episode gets encoded and how many images per second. How should we think about that in terms of time? Like what's, what does 128K translate to in terms of the sort of maximum length of an episode for an ER2-powered robot?

It's about, I think about 3 minutes of memory and it depends on how you tokenize and like how many like tokens you represent and what is the other information that you're given. However, I think regarding the memory, you would see that like the Pro models are much larger context length and they have, they are also like, they are getting a lot better at like fitting much larger things. I would think ER right now, you can fit like a very long episode of a robot doing a lot of things and try to get an output out of it. But as working on context length is like a continual problem, so I imagine that especially for these like offline analysis use cases, the Pro models are going to be more useful there.

So 3 minutes would be like if you're basically packing context very densely. If you were to offer any advice to API users who are like, Okay, I wanna aim for higher than that. I can obviously downsample video and have fewer frames and do all kinds of things, right? I've got context engineering for robotics. Maybe the question I really wanna ask is, what have you learned about context engineering for robotics that people should know? How should they think about like what parts of the history of a rollout they need to keep in order to have coherence and Maybe it's similar to LLMs, but I imagine there's probably some things that are distinct about it.

Yeah. So I think very dense memory. So there is like, I don't know, I think Daniel Kahneman or someone had The Fast and Slow, and but maybe you can think of it even for like memory aspects, right? Especially for like something that I did in the long past, is it worthwhile to keep it in a highly dense modality like image, or can I also do a lot of text summarization of like what I'm doing as I'm doing it? So let's say if I'm cooking, right, it's not sure in my In my brain, like, if I want, I can go back to the time when I was, I don't know, flipping my egg or something. But also a lot of the time, the information or the narrative of how I'm doing things and summarization even in text is fairly useful or fairly informative about like what I did. So I think context is, it depends on what you wanna maintain is the relevancy of the information for the task. And obviously, If you can remember everything, then that's great. But as practically, you need to be more efficient with how you manage your like RAM, right? And you kind of wanna have the most important information represented. And sometimes that can take the form of like also representing information in like text, which is a little bit more compressible. And this is also now related to the like the hardness between like ER and the VLA. Like the ER tries to parse the intentions from a human, and via text and other modalities kind of talks to the LA. And so they're like, the bandwidth there is much more interpretable modalities.

Hey, we'll continue our interview in a moment after a word from our sponsors.

I've got a text.

The summer might be almost over, but a new bombshell has entered the villa. Meet the voices of Deepgram Flux TTS. I'm Drew, low-key casual.

I'm Alexis, an Upbeat morning person.

I'm Haley, always cheerful. Flux TTS has some real personalities that are ready to speak. You can interrupt, pause, and keep talking without losing the plot. Fancy a chat?

Come try all the voices free through September 12th at deepgram.com/keeptalking. Terms apply. This episode of The Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. OutSystems is helping their customers build, modernize, and operate enterprise systems. Starting from any coding tool in a governed agentic engineering model. It's time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide trust OutSystems for their mission-critical apps. And we're not talking about theory. We're talking about massive operational wins. Take, for example, a leading global retailer that was struggling to keep up with their rapid expansion. They used OutSystems to build AI agents that automated their invoicing and translation processes, shifting 95% of their daily store operations to a new digital app, saving thousands of hours of manual work. Or look at a major global logistics giant that was drowning in technical debt. Their engineers were spending their weeks manually firefighting by reviewing platform logs. Using OutSystems, they built an AI log review agent in just 5 days, and that one agent now saves them over 1,600 engineering hours every single year, letting their team focus on real innovation instead of reactive maintenance. Even in modern digital banking, companies are using this platform to standardize and automate complex financial workflows, delivering secure, personalized experiences at a pace their competitors can't match. OutSystems gives you the flexibility to deploy how you want, And the governance to do it safely. Stop chasing the hype and start owning your agentic future. Learn more at outsystems.com/tcr. That's outsystems.com/tcr.

Today's episode is brought to you by Anthropic.

By now, you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable-5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept, and Fable writes some amazing verses. I give it feedback on its misses, and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days, I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai/tcr.

Just before getting deeper on the VLA, Why did you and DeepMind as a whole decide that this was the first robotics model to put out via the API? And what have people built with it?

Yeah, I think it's definitely ER is much closer to Gemini itself and in terms of the capability. So it is more mature. Actions, I think, is like farther out into the frontier. And so it is still like very much, there's a lot of research to be done about how to make them very useful, how to serve them more widely and so on. And I think it has been quite surprising to see how people use it. I've seen it in so many— it has such wide appeal and the team has done a really good job there. So I think maybe one thing we learned was like there was a lot of testing that we were doing with trusted tester partners and stuff, but it was once we put it more widely available on AI Studio and stuff, then the usage is on a very different scale than whatever the information that we were finding from deeply working with trusted tester partners. So that was surprising, like how much things once you put on a more widely platform. But second is, I think obviously there are like robotics companies using it, but then also I think for me as a researcher, I was quite surprised by how academics, people in academia use it. And I think some of the things that I've seen is on like definitely benchmarks where it was either using VLA or even trying to do even low-level control. Now that there is a lot of conversation going on around like how agents will impact robotics, so I think how ER has been used in that context has been quite— it was quite nice to see. And maybe thirdly, now that ER is more widely available, it also makes it very easy to be benchmarked. So it's a kind of like free information for us about what the model is doing well, what it's not doing well, what could be like the next frontiers that we may need to push into.

Are there any consumer products or any just users you would particularly want to Spotlight that are using ER2?

So there are a lot of demos out there. I think I saw the Boston Dynamics Spot robot going around reading instruments and like handing out snacks, the dog foam, and I thought that was something I liked.

Is it— I know that you can, I, I, cuz I did it, or my coding agent did it for me, you can define any tools for ER2 to use in a similar way that you can define any tools for Uh, regular Gemini model to use. What, what kind of additional guidance would you give on what kinds of tools it can use well? Maybe you could describe like what the Gemini Robotics 2 VLA model exposes and then how people can go toward more abstract or more low level than that and what works and doesn't work so well.

You mean how the Is the question about how ER controls the VLA?

Yes, but also my understanding of using ER too is that I can give it user-defined tools with a, just a little tool description, right? And then it chooses what tools to call based on its general purpose reasoning ability. But I'm assuming that some tools it can use better than others. And so I'm wondering what tools does, do you expose to it in the form of the What are the sort of ways that it interacts with Gemini Robotics 2, the VLA? I'm guessing you probably have kind of the sweet spot of its ability, but then people could go toward higher level concepts or toward much lower level concepts. And I'm wondering, what does it do well and what does it not do well as you depart from Gemini Robotics 2 as kind of the canonical VLA that it presumably was trained most directly to use?

So I think one definitely is the Gemini Robotics ER is built on top of the Gemini tool use and other capabilities. And as we are very much working on how to improve its relationship to the VLA and how to instruct that better, maybe one is I would imagine that there are— robots are like very different and each one has very different APIs and stuff. I would imagine that it's much more friendlier to the APIs of the robots that we commonly use that are more widely known. But then if you probably subject it, and here I am I think we would need to test to see how these things are. And maybe there is no standard for robotics APIs and the model protocol equivalent. So I think that would be something where I would imagine that more widely using it could face some challenges. And definitely it's a very good model that's very useful for language instructing VLA. But you can think of VLA itself as like a It's a mapping between joint space language to joint space language, image space to joint space. But there are a lot of demos now and people are thinking about, can these like higher-level models directly control in the joint space? And so I think there's a lot of work still needed to be done. I think current results are still that specific robot foundation models are still better, although the larger brains are showing impressive capabilities in orchestrating these robots. So I think that would be something where I would imagine A lot of push in the future.

So when I did my little demo, the tool that my coding agent decided to expose to ER2 was a grab at, place at, and it had to just, based on the image itself, pick XY coordinates of exactly where to grab and place. It's obviously been trained pretty well on that cuz it did a pretty good job. It, again, in 2D. How, what, what sort of, with Gemini Robotics 2, what kinds of commands do you have ER2 giving to the VLA? Are they like, it sounds like not so low level as move hand to XYZ coordinates. Sounds like that you were saying is like more experimental. What you're, the interface that you're actually working with is more pick up this item, or pick up the yellow, like, In my demo, there's a lemon or a banana. So there you could say, pick up the yellow item, but that would be ambiguous on banana or lemon. How would the ER2 model disambiguate for the VLA model? Would it give coordinates? Would it like use the term banana? Would it say, describe the shape of like oval versus crescent shape? How, how many concepts does the VLA have to work with? And conversely, the opposite side of that is like how specific.

Yeah.

Does the ER have to be in its commands?

I would tell you an answer, but it's only true at this point, right? It is like an evolving— so this field of work is called steerability, and people are looking at various ways of steering. So VLA steerability, and then the ER is essentially using whatever modes that you can steer the VLA with, right? And VLAs right now are steerable in language. You can point to objects. And so here, It's not like an API that I'm giving where I'm like saying, put this pixel location. I'm literally just doing pointing and then I'm pointing to an object and then it's, can the VLA do it? And then the ICL results are like, can I prompt it with video and stuff? So I think it is like a spectrum and eventually you would imagine that the way that you want to interact with robots generally is very wide, right? You wanna say if the modality between the ER and the VLA is just text, a lot of things can be lost in there. So We want that contact interface to be also as rich as possible over time.

Yeah, so the VLA itself is multimodal in the sense that it can take all these different domains or all these different modalities.

Yeah, and I think like the ER kind of, you can think of it like as orchestrating or giving feedback to the VLA to do the task. And there is a range of things that, like instructions, like even while we don't generally train it to do a lot of like low-level stuff, I think like it says things like turn around, look to your— put your head there or look to your left and you can get all that information and the VLA responds to it. And sometimes, often the thing about foundation models is that like sometimes you can be quite surprised at what they do and because like even you didn't know that they could do that. And then sometimes you can see that coming out of just the harness as well.

So has it been disclosed or can you tell? It sounds like the Gemini Robotics 2 model is also created from a VLM base, but then, and obviously 4. So really interesting. You, you mentioned earlier that it has whole body control. Tell me, what does whole body control mean exactly? Does it include, 'cause stabilization typically needs to run on like a pretty high cycle time, right? A pretty, a pretty high frequency. And if this is a post-trained, especially post-trained VLM that has such high-level concepts as you're describing, it seems hard to imagine that it would be able to run on the frequency to do lower-level stabilization. So maybe the robots themselves bring that kind of stuff, and that's not part of whole body control. Like, what is in whole body control? And is there some stuff that's left for kind of even lower-level systems?

Yeah, so I think you can think of as like the Chinese humanoid race as an example, right, where there are definitely controllers that can control the body, but then eventually what you need to make sure is like, can I do the task that the person is asking to? And the very low-level controllers often don't have a semantic brain there. And now you need to like control like, and in I doubt a lot of these things cannot be abstracted away to, oh, this system will just do this or this system will do that. I think, so here the VLA is controlling the whole humanoid body from like fingertips to the feet. And only when you reason entirely about how the body is and also all the sensory information about how the environment is can you do that. I think that is the right design to do a lot of different tasks. So the VLA is controlling the whole body, and, but there are more decisions being made about how to stabilize and how to achieve the targets of the VLA.

So this one is not available via the API, but you do have a program. What, I guess that's, you mentioned it's mostly because it's just more frontier, less reliable technology as it stands today. Does that mean that If we were to break down failures, the majority of the failures would be at the VLA level still, as opposed to the reasoning level? Like the reasoning is working and the VLA is where failures— it can't quite actually do the thing?

We are working with partners to develop this VLA. So we have partnerships with Actronic, Agile Robots, Boston Dynamics. And so that's where we build or how we are building the VLA together with them. I do I do think in general robotics has this embodiment problem, right? PLAs are also still training on data from robots. And while we see a large leap being made in cross-embodiment, I think that taking a new embodiment and completely zero-shotting it to a lot of tasks with very high reliability is still something that is— I don't think there is a very strong precedent for that. And there is also like a trusted tester program for the Gemini Robotics on-device model where we work with a lot of different labs and they have their specific robots. And I think the on-device release showed like it controlling. This was one of my favorite parts of it. You can, with very, very small number of examples, with 200 examples, you can learn a lot of different tasks on a lot of different bodies because you already have the foundation to do a very generic understanding of the physical world and also like how to control it from, and the data for how to control it with many different embodiments. I would think, I think a lot of the challenges with how the orchestration works is still split. There is surprisingly, there is the latency component, right? Because when you have 2 models, like one model is like doing the task and another model is also like looking at how to switch the task, like when it is done and how now, how do you progress to the next task? I think a lot of failure cases often also come from how to do the orchestration really well. And yeah, so I think, and it is also compounding of errors, right? Like now if you have 10 tasks in sequence and now you have the ER model, which has a certain X success rate and the VLA model, which has a Y success rate. So the, if you have a sequencing task, you are basically compounding the error. Does that make sense?

So if I ask a, if I ask an AI robot to go fry an egg, how far does it typically get? Can, you know, are we getting some successes on eggs fried, plated, and served to the table, or would that be beyond what one of your robots today could realistically do?

It depends on whether you have data or not. I think that if you, there are ways to make, like if you specifically just wanna fry an egg and how to do that reliably, there are algorithms that can make it very reliable just for that task. But if it's like a very generic model which has not seen how to fry eggs and now you bring it into your home, which is, to be fair, I think models have made a lot of progress on visual generalization. They're now not distracted by like lighting conditions or the different setups of different homes and stuff. So I wouldn't, but I think it's like the task semantics and the context, right? And the ability to make mistakes. I think especially for frying, it's like you can overcook an egg and no one would like that. And there the room for error then becomes small. Like—

Well, I overcook my own usually too. So that would, we could count that perhaps as a success for me.

Tasks that allow retries are much easier. Like if you are doing like Lego assembly, it's fine if you make a mistake. You can go back and fix it.

Yeah.

Yeah, that's a great distinction.

If you crack an egg and then it's on the floor, then now you're in a mess.

Now you've got a mess. Yeah, you might slip in it too.

What are the—

so one of the value propositions of the Gemini Robotics 2 model, if I understand correctly, is it can work with any embodiment. What are like the most exotic embodiments that have come your way via the Trusted Tester program?

Surprisingly, not a lot of exotic embodiments. I think maybe the weirdest thing I've seen is people putting arms on drones and stuff. I think it's just that once you have, if you have data, you can learn anything. I think that is the principle. So I do believe that you can fit like very weird embodiments to it. But in general, I think the embodiments kind of look very like a normal distribution, right? Arms are very, if you look at like Like the eigenvectors in the embodiment space. Like there are a lot of like bimanual arms, there are a lot of like hands, there are like humanoids, but, and then like everything weird is like a long tail in that sense.

Any kids' toys that you've seen that might be in demand this holiday season?

Kids' toys, like for robots, you mean, or?

Yeah, I don't know. I just can imagine. I'm surprised honestly that I haven't heard more demand from my kids just based on what their friends have and whatever for some kind of AI-enabled little robotic toy that might, I don't know, honestly walk around the house, but move around on wheels even and talk. And I'm not sure I want them to want this because I'm not sure I want it to be their life, but I'm surprised that I haven't seen more. So I'm just wondering if the—

I'm also surprised.

R&D pipeline is there.

I think it's probably because the market is a little bit burnt. I remember when I was in school, there were a lot of companies trying to do my tabletop, like a gaming robot. And I think I remember when I was in CMU, there was a lot of conversations around these. But maybe what a lot of people learned from that was like, people buy these things for novelty reasons, but they don't go back to using them. I don't know how that is. None of those robots were machine-learned robots, which was connected to the large models. And now things are very different. I have. I speak to Alexa and Siri and stuff, and sometimes I wonder it would have been a lot more fun if they were much smarter.

I'm still waiting for good Siri, for God's sake.

Yeah, I've been talking to the Gemini, the live user interface, when I'm doing tasks or just talking about like my personal life problems and trying to get its perspective, like I think you shared. And yeah, it has been very useful in having an expert go over your problems where you're probably not very skilled So yeah, I'm a big voice mode fan too, actually.

In a previous conversation that we had, I asked you what you most needed from hardware makers, and you said, I need good hands. How are hands coming along? How is hardware coming along generally? Are you pleased with the progress? Is it gonna become a bottleneck for you relative to how fast the models are improving? Where do we stand?

When I last spoke to you was probably in the May— March of last year, right? So here's the interesting thing, right? We had in the March of last year, we came up with Gemini Robotics 1, which showed that a lot of dexterity with grippers. And in the summer of this year, and it's only like a year and 4 months, right? 3 months, a year and a quarter, we showed Gemini Robotics 2, which does multi-fingered dexterity, which can tie trash bags and stuff and do very fine control of multi-fingered hands. So it is, here's where my internal model is also updating, right? All the tasks that the gripper robots could do and that they were at the frontier of dexterity, the hand robots can do now. And the hand robots can do more things that the gripper robots could not do. So in a way, hands are now at the frontier of dexterity and grippers are Maybe not. And I think this has been, I think I was quite surprised at how much the hardware has come along. There are a lot of hands on the market that are quite good that people are doing a lot of research with. There is still more work need to be done about how to make them very reliable, very repeatable, and also make them cheaper.

How strong are they? If my wife needs me to open a jar, would the robot be able to substitute for me and open the jar for her? Or would it be like she would need to open jars that the robot can't open? Like, where does it fit on the kind of the, in addition to the dexterity, the ability to really torque something?

I think there is a spectrum and it depends on which hands that you use. I think especially even off the shelf, there are a lot of variations. The Wuji hand is closer to, I think, Maybe like a 10-year-old, I would say. It's, it's, it's a bit weaker. And the Sharpa hand is, I think it can lift like 20 kgs. I don't know how much the torque stuff, but I've seen it open jars, so I'm pretty sure that's possible. Though maybe not a very tight one, I don't know. So there is a lot of spectrum and some hands are built for lifting more. The Sharpa hand is also much bigger than my hand and the Wuji hand is maybe close, more closer in human size.

It'll be funny if the last job for humans is opening stuck jars that the robots don't quite have the wrist strength for. Random aside, it's fine if the answer is nothing or it's all fake, but what is soft robotics? What do you know about soft robotics? What should I be paying attention to? I recently talked to a founder in this space and honestly, I came away more confused than I went into that conversation.

What are they building?

Basically, it was funny. They basically wouldn't really say anything. I was like, why did you schedule an interview if you don't want to say anything at all about what you're doing? So that was a random kind of strange experiment or experience, but I did nevertheless come away thinking, what should I be paying attention to, if anything, in soft robotics?

Regarding soft robotics, there is a wide spectrum and I have seen things from people building soft bodies to people building very soft hands. And what's relevant for AI-driven robotics is what's repeatable and what's durable. I think one trend that I'm very interested in is like the UMI type of efforts for a lot of gloves and stuff and for a lot of like tactile modality. I think that type of how to do them really well and how to build that really well is still an area of research. And part of it is like soft robotics. I think I think, yeah, I'm also not fully— roboticists are of different kind. There are also like a lot of roboticists who are like very mechanical engineering focused and then who see the work that I'm doing and, you're not a real roboticist. And I'm like, sure, okay. Computer science roboticists and the mechanical engineering. Oh, you should go to a robotics conference. It's all kinds of people. ICRA is a good example. There's every kind of roboticist there.

I did. That sounds fun. I did have a chance to go to WAIC in Shanghai this summer, and there were a ton of robots there. I mostly felt like the demos seemed highly staged. I didn't feel like they were really demonstrating a lot of generalization ability in most of the things that they were showing off. It's like a CES-scale show with multiple giant venues and every AI company in China basically there. It was a cool experience. The funniest robotics specifically experience I had there was sitting down at the Tencent booth. They had a sort of half humanoid. I don't think it had legs and it was just mounted upper body and it was giving massages. So I got a very brief robot massage, some sort of like pressure point thing. Was it good? It was too short to really score. It didn't hurt me. It did apply some like non-trivial pressure. And I walked away with a smile on my face. So I guess it wasn't too bad, but I'm not sure. I think I still would be highly confident that a human massage would be better. They're letting robots touch people over there now. It was, that's something. Speaking of robots touching people and potentially crossing other lines, everybody, of course, this summer is talking about Open Face, the scandal, the incident, the, I sometimes call it a lab leak. out of OpenAI. And it just has me thinking like, maybe the research is all going to work and the alignment and the safety questions will ultimately be the bottleneck on robotics deployments. You could analyze that in multiple ways. One way to think about it is just like, when do you think if we put the alignment and safety questions aside, I might get my domestic servant robot and then people can think for themselves. Oh, I think that's like before or after I would expect alignment to be good enough. to actually want one?

Yeah, I think this is definitely an area that we think very carefully about. And I think of safety as like a capability, right? Like, there is a lot of discussion around like safety and capabilities being at odds with each other, but people are not gonna use an unsafe robot and unsafe agents. I think the most useful agents are going to be ones that are safe, that know what they need to do and they're not overriding human instructions and causing chaos or breaking laws and stuff. So I think safety needs to be front and center in the design for AI and also robotics, and especially in robotics. Like in robotics though, I think there are different types of safety. Maybe one thing that is not relevant to AI safety is operational safety, right? Like how do you make sure the robot is not like injuring people. And this is not injuring people because you're malicious, but just not injuring people because you're dumb. And a humanoid being stable enough, like there, it's not doing a wild plot to take over your house, but it's just if it falls, it can still be quite dangerous. So there are like a lot of different safety. So that's maybe like one type of safety research that's like very specific to robotics, but not yet, I don't see it being a big thing in the AI space. But then there is also things like, what should I do if there are, like, obviously, like the higher-level goal stuff, right? Like how do you make sure you do the task that the human is asking but without breaking guardrails? And then there is also the stuff about, and maybe this is also specific to robotics, like what happens like when some sensors go wrong and how should I react in those situations? This one very funny demo that we showed was like the humanoid is doing something and then someone goes in and puts a basket on top of its head. And now what should you do? You can continue doing the task with your eyes covered, but the right thing to do there would be like really seeing that, hey, like something is obstructing my vision. Can you please remove it? So I think safety, and safety is like a full system thing, right? Like it goes all the way from like the design, the mechanical design of the robot system safety, to all the way to like the ER and the high-level brain. That's how I think about it. I do think as AI becomes more capable, Like we need to think, spend more flops thinking about safety. Yeah.

Your example of somebody messing with the robot is an interesting one. I mean, we've seen this with Waymos and stuff obviously, right? Where people like go out of their way to cause trouble for AI systems.

In fact, it's something that I was very surprised of. It's like maybe because you, and it's hard, right? You have 2 brains in you. One brain is the researcher brain who knows that they're machines and now you don't really treat them as human-like. But then there's the other brain which is like, because they look so human-like, it's a bit duplicitous, right? Because when it's talking and stuff, you feel kind of very human-like around them. Now, so as a researcher, I'm like very careful a little bit, but I think when we see new people interacting with the robots, they very— people need to be more aware that these are machines. And I think sometimes when new people interact with the robots, because it looks very human-like, it's very— that boundary can blur. I also think this also creates a higher bar for humanoids. If you have a gripper-based robot, like it makes a mistake, like people don't expect it to be smart, but a humanoid just like grappling around and creating a lot of failures would be judged much harshly because even if you say, okay, this is the state of the manipulation research or whatever, people still expect humanoids to act smarter. Than robots that look anything unlike it.

Yeah, that's interesting. I don't want to go down too much of a rabbit hole on consciousness, but I do think it's incredibly striking to me over the last year how much analogous structure has been found in LLMs. And personally, my willingness to entertain the possibility that AIs actually are somewhat conscious or sentient or something is up a lot just based on how much analogous structure we can now point to with things like functional emotions and functional wellbeing and the J-space and so on and so forth. If you have any comments on that, I'm interested to hear them. If you want to pass on that, you can just pass on AI consciousness for the moment.

Well, I, I've, I want to highlight one thing. So something that really So what's really happened for Gemini Robotics 2 that was not in our previous releases is the human-robot interaction component. So this is where it creates like very natural gestures and it's not like pre-programmed. And previously, if you look at our videos from our last releases, it would do all these nods and stuff were a bit more like pre-programmed HRI. And now it's like very natural HRI. And there now you see like very funny things happening because the robot itself is deciding what gesture should I use as I talk to the person. And something that was very interesting was as we were like filming for the Gemini Robotics 2, there are, if you look at the videos, there are also interviews with the robot. So people ask the robot like, hey, how do you think you did the thing? And then the robot would be like, yeah, it was great, but maybe I made a mistake on this part of the task. And so it's— And also we had the filming crew and they were shooting the robot. And initially the filming crew, they hadn't spent that much time around the robot. But at the end of it, they were like, action robot, and then the robot will do the thing. I think, I do think as we become more comfortable around humanoids and also as these models become very smart, I think, yeah, the lines are going to blur. I'll say one funny incident. So this was like the robot working in the basement and the garage and then trying to pack things and stuff. And then there was an interview about what was the hardest part of you doing that task? And then the robot says, The videotape, which was my favorite object, I think it was very hard to put it in the basket instead of holding onto it. And then we are all sitting in the back and we are all like laughing at the robot's response. And yeah, it's, I don't know, would a robot like a videotape?

I guess so. Self-report is at least we're seeing somewhat reliable, although not always.

Sure, there have been some very, very adorable moments. Yeah.

Where are the gestures coming from if they're not pre-programmed? What is it that is causing the model to be inclined to gesture to humans?

It is being prompted. There is a model that is using HRI to come up with gestures.

Gotcha. So like ER2 is saying to the VLA model, raise your hand in a gesture as I give this response or something along those lines.

No, I think it's like following along in the conversation. Like, you can— it's not very explicitly prompted to do this or this, but it's like coming up with the gestures on the— so just like how the VLA is, right? No one is asking the VLA, put your hand out and then go grab the thing. So it's more natural than that.

Fascinating. Would you call that emergent? Not quite emergent. It sounds like you designed for it somewhat, but not. Super specifically. So it's semi-emergent maybe.

Yeah. And this is also a field of research that we started thinking more about as you work on humanoids. I think in the outside world, there's a lot of conversations around why use humanoids for, why not just do armed robots and stuff? I think there are certain fields, entire fields of research that you would not touch because you would not be exposed to those problems until you you look at what is the peak form factor that I wanna target? And I really like that maybe DeepMind is probably the only lab here in North America where you can study humanoid intelligence in a cross-embodied way. It's not, we are not specifically building a brain for any one body, but it's very general and there are multiple humanoids in our lab. So it's thinking about like, how do you think about the intelligence substrate, but then also abstract away the body a little bit? And intelligence is, in some ways a little bit abstracted away from the final form factor. And there are multiple fields that only arise that when you do different humanoid bodies. One is human-robot interaction becomes a much bigger thing. Whole body control becomes a big, bigger thing. Multi-finger dexterity becomes a new frontier. And to get exposed to all these 3 frontiers and research problems, you need to work on HUTS.

So one big question I have about kind of the future of the field, shout out to Dr. Jim Fan from NVIDIA for an outstanding talk he gave that inspired a couple big questions for me. He makes the case that basically model, the robotics models in particular, need to think ahead, right? They need some sort of forward-looking simulation. Which is not exactly synonymous with world modeling, but certainly highly related to world modeling. But the key idea I think that he has, going back to my earlier question on the response time of a classic reasoning LLM backbone system, is you can't, in his telling, you can't just have the model look at the present all the time, reason about the present, take action, and run that cycle over and over again. You need to have some way of kind of projecting into the future how things are going to go so that I can move and measure the delta between my expectation and what's happening. And that's like the path. Certainly that seems to be from what I understand of how our brains work. That seems to be what we're doing. Do you think— I think this is, you could look at LLMs and say, they've come awfully damn far on a token-by-token prediction without explicit world models. Maybe robotics can do the same. Maybe it's different. Where do you come down on that? I'm sure a pretty central debate.

Yeah, I think the jury is still out there, right? Ultimately, the thing about working as a roboticist is that you are not emotionally attached to one method or the other method. You are emotionally attached to the problem itself, and whatever way you solve the problem, you will pick that. And yeah, there are many schools of thoughts. Definitely world modeling is a very interesting area for robotics, and not just for as policies, but also simulation and other things. And I think the jury, I say the jury is out there still because there are obviously the VLAs, the VAMs, but also look at how the agent robotics are going. And I think we'll see. And this is maybe one of the most exciting parts of working in robotics is that it is still very early that the recipes haven't stabilized. Different people can have very different schools of thoughts about how I'm gonna go about and solve robotics. And that's what's exciting. And we'll see who's right and all of them could be right. And it might be the future until we have solved the problem. It's very hard to very definitively say this is the solution or this other thing is the solution.

Yeah. Interesting. Okay. So it sounds like you're open-minded and obviously Google DeepMind has world modeling efforts that are pretty well established as well. So to the degree that you need to pull a world model off the shelf and just start simulating into the future, you have a pretty good foundation right at hand to do that kind of work. He had a couple other interesting points I'd be interested in your take on. One was basically he thinks that we are going to go to egocentric video as kind of the main data used for training. He thinks that like teleop will go to a tiny scale, a tiny portion of data. And even the sort of hand, we've seen a lot of innovation. I think it's been pretty cool around just like wearable robot hands that you can use. But he thinks even that will ultimately just give way to first-person POV videos. And then beyond that, it's just going to be simulation. And he thinks basically like all the problems of simulation will just get solved. And that will really be the way that we'll get to enough robotics data to solve robotics. Any thoughts on how that data, that vision for the future of data compares to yours?

I think everything needs to be backed by results, right? And it's, I don't think there is much point in making a prediction. And usually you have a prediction, you run an experiment to validate that prediction, and then now you know that is the right prediction. And these things can also change over time. And I think, as Jim suggested, it's great that he has very strong views on certain types of data. I think it might also very much look like it's a mixture over time. Like, you look at LLMs, they're trained on all kinds of data, and different data has different purposes. Like, the teleop data is definitely useful in grounding into the robot's controls. UMI data is definitely very useful in learning much scaling without robot in the loop. So much cheaper scaling. And human data is also very, like the egocentric human data is also very useful in getting broad semantic understanding about how, what to do and stuff. But they also have their failure modes, right? If you imagine that you're controlling very fine hands, you don't get very fine signals about where the hands are unless you're getting, you're estimating all that from vision and vision has a lot of errors. And so like each data modality has different strengths and different drawbacks. Like the UMI data, like with sensors, you are getting the action labels not from vision, but from sensors. So it's a bit more accurate. So there is, I would imagine each data source is on the x-axis, you can plot like scale and the precision and maybe like teleop, not very scalable, but highly precise because because it's exactly the robot's data. But it's also not very future-proof, right? Because as your robot evolves, your teleop data starts becoming a little less useful, even if you have very good cross-embodiment. Then there is UMI data, which is more scalable, but because of all these sensors, it's more precise than human data, but it's less scalable because of all these sensors. And then there's human data, which is very scalable, but can also be very noisy 'Cause you don't have very precise end effector actions. And you also need to like now put that on the robot. So there are errors that come from, and each human is very different. Some humans are small, some are large, and due to that, how they do the end effector tasks is very different. So I think it's going to be like probably closer to a mixture. And the thing is hardware is changing each of these paradigms differently, right? Like as very good UMI hardware comes on the market, like that, how to scale that, now goes up. So, um, I don't think it's wise to take a very principled view about this because it, all of this will change as new evidence emerges and better hardware emerges. And you probably need all of, all types of data.

Then maybe one other aspect of his talk that I thought was interesting, it wasn't a main focus, but it got me thinking a little bit, was he used the term physical API. And it got me thinking like, how do I want to interact with robots in a world where I'm interacting with like multiple different robots? Do I want them each to present as an entity unto themselves? Or maybe it turns out to be more of like a HER model where I have one entity that I like interface with the most and it kind of deals with all of the robots as a fleet. And therefore the robots are not like individual entities in my life, but they're all sort of extensions of this like single personal superintelligence or whatever you wanna call it that can run all these other things. Do you have an intuition for how you would want to interact with many robots in your life?

Yeah, I, I don't have the context of this physical API conversation. But it also ties a little bit into what we were discussing before, right? I think the surface area of our interaction with robots needs to be highly multimodal, and I, the user, needs to have both high-level and low-level access to robots. I think that is probably the only way that we can build towards a very safe and flexible future. And anyway, from language, and if robots code, and if a task requires multiple robots, then I need to be able to to be able to hand it off and then it can handle the collaboration. I think the Gemini Robotics 2 kind of had this where both the Duo robot and the Apollo was doing the task. One was doing one and then it stops and the other one takes over. So there is, I think, yeah, maybe, I don't know, this is a boring answer which I've said before. It's probably going to be a spectrum, right? If there are tasks which require multi-robot collaboration and a lot of agent and then there I need to be able to hand off and they need to figure out how to do things. And then there are tasks where I'm only interacting with one robot and I either want to give it text instruction because I'm just talking to it, or I wanna give it more fine-grained specific feedback. And there I need to go even more lower level. I think for some, a lot of safety and interpretability reasons, maybe we need to still maintain much lower level access to robots.

Yeah. What you're saying there reminds me of what I'm increasingly thinking of as like the universal UI for software products, which is like, there's the product itself and I can click on it and edit fields and drag stuff around. And then there's an agent or assistant that kind of sits alongside that and has basically all the same affordances that I do. And most of the time I probably just want to tell that thing to do it, and I don't really wanna concern myself with the details. But I also have found that it's really annoying sometimes if the app doesn't let you get to that level of detail, because sometimes the AI is just like not getting it on its own. So I can see a, a way that paradigm ports over to robotics.

Yeah.

Especially let's say like, I cannot tell a robot to clean my desk because I like my desk done in a certain specific way. So I need, I would like the ability to give it more feedback and even personalize it to my own, like.

What's up these days with, for lack of a better term, I'll call it circuit breakers on robots. What I mean by that is I saw one example where the company had made a sort of inflated airbag in the torso of the robot. And the idea was like, if you poke that thing, It is immediately disabled. The pressure of the bag that you could easily puncture as a human immediately disables it. I thought that was just interesting new idea in terms of like how you could worst case scenario have a, an off button locally. But more broadly, there's like what happens when it encounters resistance it's not expecting or whatever. How has that evolved over the last—

Yeah, so I think for the longest time, this air bubble that you break, there has been a version of that. It's called e-stop or kill switch. So robots, most robots have a button that you can press and then it will depower. And there are many phases of it, right? There is the hard e-stop where you stop all the kind of power to the body and then it can just collapse, which can then be dangerous on its own. There is also the soft e-stop where you just freeze the robot. And this is often safe for humans. Bipedal and other platforms which are not stable by themselves. And recently, I think especially in manipulation, there are a lot of new features kind of emerging to control forces at the end effector, right? Imagine that you wanna stack 2 chips together. You can, if you press too hard, you crumble the thing. And, or if you are handling very delicate material, you need force control and backdrivability at the end, compliance there. So I think sensing is improving quite a bit that you can have very, you can read what the end effector forces are and then the models can take much better decisions than they would if they didn't have that information.

Last question, kind of a galaxy brain or maybe a fly-brained one. So there's been all these demos I'm sure you've seen recently of people using the fruit fly brain to do all kinds of things, including seemingly to control some small Actually not seen them.

Maybe our, like, Twitter feeds are still fairly personalized. I don't know.

There's a, yeah, there's one. I'll send it to you. There's one where somebody has trained this thing to control a little, I think it was called a Beast Strand, one of those like beach walker robots, but a real small one. The connectome of the fruit fly brain was fully mapped and put online in digital form, and people have just started downloading it and training it to do different things. I'm not 100% sure how real it all is. But there's enough of it going on that I think some of it's real. It sounds like you don't have necessarily a take on that. I'd welcome any comments, but maybe zooming out, I wonder, far future, do you envision yourself with some sort of robot body? Is there like a neural brain-computer interface where you imagine having a hybrid form yourself?

Yes. Yes, and it already exists in some sense, right? Like when people are teleoperating, their physical body is somewhere, but their robot body is the one, just teleop itself is like that. And then there is people who are building remote teleop that works across the Atlantic and stuff. So the person is sitting in Europe, but the robot is in the United States, like stacking, I don't know, stocking shelves or something. So definitely this is very interesting. I would imagine someday that you get to explore Mars or something with robots because you are teleoperating them. Yeah, I think the future is exciting. One of my friends, the thing that he wants is he wants to walk around. He wants, you know, the guards in India, they have like all these arms popping out from their behind, but now you can build them, right? You can put those trust arms on your back and then those arms can go pick things and you can just walk around in SF with those. So that's the thing that he wants to build as a hobby. Sometimes I also think about like my dog. I looked up how many parameters she has in her neural network. Apparently it's 25 trillion. So she has more parameters than a lot of the models out there, even though her language skills are not that developed. Yeah, I think the future is very interesting.

Yeah, that is a modest way to put it, I'd say. And that might be a great note for us to end on. As always, this has been a fantastic conversation, and I appreciate your help in catching up with everything that's going on in robotics. Kirtana Gopalakrishnan, thank you for being part of the Cognitive Revolution.

Thank you so much for having me.

The floor was flat, the door was wide, and I never once got burned. Then your hand came down on mine and the whole thing came undone. All that practice in an empty room and I forgot step one. It's only hard where we touch. Everything else I can see coming. It's only hard where we touch, and I never wanted easy. A stone stays where you set it down. A wall stays where it stands, but cloth has got a mind of its own. It moves inside your hands. The glass, the egg, the skin of things, that's where the trouble starts. Everything soft gives way. And the soft things are what stay. It's only hard where we touch. Everything else I can see coming. It's only hard where we touch. And I never wanted easy, so I'm done. Practicing in a room where nothing ever gives. Give me the spill, give me the slip, give me the egg on the floor. I don't learn it in my head, I learn it in my hands. It's only hard where we touch. Everything else I can see. It's only hard where we touch, and hard is where I grow. It's only hard where we touch, so that's where I'm gonna go. I know, I know, it's only hard.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, Check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.