Robotics Research Update, with Keerthana Gopalakrishnan and Ted Xiao of Google DeepMind

People used to think that all the robots are so different, all of their data is like so different, and people moved in the direction of thinking that all robots are kind of similar. It's only as different as like English and Chinese or something, and the concepts are similar, it's just the manner of expression that's different.

The datasets that resulted, even starting from those criteria, were so, so diverse, right? We have everything from like, you know, using like baby toys all the way to like industrial arms, you know, all the way to, you know, like very dexterous like cable routing robots. Even with this very limiting assumption, you still get so many different morphologies.

If you train on VLMs and if you build on top of the knowledge of VLMs, then you can stitch a lot of concepts from the internet along with the motions that you have in robotics datasets.

You could, under the same initial conditions, just change the prompt a little bit, do some prompt engineering, and actually see qualitatively different behavior from the robot.

Ultimately, if you want to do a lot of tasks, and be useful in environments where humans operate, you kind of need to go as close to a human embodiment as possible.

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Eric Torenberg. Hello, and welcome back to The Cognitive Revolution. Today, I'm thrilled to be speaking with Kirtana Gopalakrishnan and Ted Zhao, researchers at Google DeepMind Robotics, who are developing AI systems for general-purpose robots. This is Kirtana's second appearance on the show. Just a year ago, fresh off the publications of RT-1 and PaLM-E, she described the state of AI for robotics as being somewhere between GPT-2 and GPT-3. Noting that the lack of internet-scale data was a major barrier to progress. Since then, Kirthana, Ted, and their colleagues at Google DeepMind have published a remarkable flurry of papers demonstrating new techniques that allow robots to leverage large multimodal models, to control different physical form factors, to learn more efficiently from human examples and instructions, and even to use a prototype robot constitution to guide their behavior. In this conversation, we cover 6 different papers. RT2, which shows how internet-scale vision language models allow robots to understand and manipulate objects that they have never seen in training. RTX, a collaboration with academic labs across the country that demonstrates how a single model trained to control a diverse range of robot embodiments can outperform specialist models trained for individual robots. RT Trajectory, a project that shows how robots can learn new skills in context from a single human demonstration represented by a simple line drawing. AutoRT, a system that scales human oversight of robots even in previously unseen environments using a combination of large language models and a robot constitution to power first-line ethical and safety checks. Learning to learn faster, an approach that enables robots to learn more efficiently from human Verbal feedback. And finally, Pivot, another project that shows how vision language models can be used to guide robot action, this time with no special fine-tuning required. Progress in robotics is still trailing behind the advances in language and vision. There are still challenges to be overcome before robotics models will have the scale of data or the sample efficiency needed to achieve reliable general-purpose capabilities. And the study of robot safety and alignment is still in its infancy. But nevertheless, I see this rapid-fire series of papers as strong evidence that the same core architectures and scaling techniques that have worked so well in other domains will ultimately succeed in robotics as well. The work being done at Google DeepMind is pushing the boundaries of what's possible. Investment in a new generation of robotics startups is heating up. And the pace of progress shows no signs of slowing down. As always, if you're finding value in the show, please do take a moment to share it with friends. This one would be perfect for anyone who's ever daydreamed of having a robot that could fold their laundry or pick up their kids' toys. I certainly count myself among them. And especially as we are just building the new feed, a review on Apple Podcasts, Spotify, or a comment on YouTube would be much appreciated. Now, here's my conversation with Kirtana Gopalakrishnan and Ted Zhao of Google DeepMind Robotics. Kirtana Gopalakrishnan and Ted Zhao of Google DeepMind Robotics, welcome to The Cognitive Revolution.

Hi.

Yeah, thanks so much. Great to be here.

So it has been a busy time. We had an episode with Kirtana just under a year ago, and it's amazing how much progress has been made and how many papers you guys have published since then. I thought I would just kind of super quickly summarise the state of play as it existed a year ago. And then what I really want to do is just give you guys an opportunity to give us a walkthrough of the many results that you have released over the course of the last year. We can really get into as much as we have time for the technical details of that, and then toward the end can kind of maybe pull up a little bit and sort of take stock of the big picture in robotics and where all this is going and when I'll have a robot picking up my kids' toys. Maybe doing their laundry would also be awesome, just to put in a couple requests right at the top. But basically a year ago, and again, just even under a little under a year ago, you described the state of robotics as kind of being between GPT-2 and GPT-3 and noted that the lack of data was a huge bottleneck to progress. We talked quite a bit in that episode about how at the Google kitchens, you guys have had this kind of ongoing setup of human demonstrations, collecting all these data points, and that was starting to pay off in the form of some really cool results even then. And since then, I think we've got, is it 6 or 7 different discrete results to cover today where you're expanding the datasets in various ways? You're showing interesting kinds of Transfer from foundation models that are multimodal and trained in other contexts into robotics, forms of imitation learning. It's really been an incredible tour de force. So before we get into all the specific papers, I guess my first question for you guys is, is everything working? Because from the outside, it really seems like you must have sketched this out a year ago, and it seems like it's all kind of gone according to plan. Now, whenever I say that, people are like, well, no, a lot of things don't work. And I'm sure lots of little things had to be kind of figured out along the way. But has the last year, kind of broadly speaking, gone according to plan for you? How would you summarize at a high level the last year of progress that you guys have made?

Definitely the feeling that we're living through an inflection point right now. I think, as Kirtana said, a year ago between GPT-2 and GPT-3, and maybe Nowadays, it's closer to the era of the GPT-3 time in robotics. I think it's definitely been pretty top of mind for me and thinking about why that happens is kind of interesting. I think my one sentence explanation is that with the era of internet-scale foundation models, things that used to work maybe 20, 30% of the time are now working 60 to 70% of the time. And in robotics, right, it's a very complicated dynamic engineered system with many In the past, if every small component of your entire system only worked 30% of the time, it would take many, many iterations to get a whole performance system working at scale. But now when every single part of the entire stack just works that much better from the research iteration process to the engineering scaling process to the data collection engines, I think that you can really just see the pace increase when you just have many more successes and a much higher hit rate. when you're going about and scaling up your research. So that's my one sentence explanation, but definitely Kirtana, if you have any others.

Yeah, like I said, I think last year I said what we lack is data. And so we have also tried to attack that problem from different ends. So in addition to data, I think the thing about research is that it's very hard to plan research. You kind of have to do a breadth-first search over many different approaches, see what show a signal and then really double down on that. But over like a large horizon, you kind of know what kind of things you wanna work on. Like one is like getting more data, or the other side of that problem is improving data efficiency of policies. The second is tackling self-improvement type of research. And then thirdly, like how to better use foundation models, how to really leverage their like world knowledge, how to really leverage the human input required to make robots better. Cool.

Well, there's been so much work that's come out. I thought maybe I would kind of just go down the list, give my high-level takeaway as somebody who has read the paper and is trying to piece together the big picture, but then give you guys the opportunity to go deeper into what really matters and some of the technical aspects of each project that you think are kind of most relevant or you want people to come away understanding. So the PaLM-E had just come out a year ago, and RT-1 we also covered in that last episode. Shortly after that came RT-2, obviously the successor to RT-1, but a bit of a different approach. As I understood that paper, it was really about showing that robotics can take advantage of the highly general and kind of flexible understanding that is increasingly showing up all over the place in large pre-trained language models, and in this case, multimodal models, specifically vision language models. Tell us a little bit about kind of the co-fine-tuning, which was an interesting technique, and then how that allowed you to kind of generalise in a more effective way than previous systems had.

Yeah. So I think the way I see it is like RT-1 kind of showed that If you have enough data and in domain, you can get really high performance in domain. And RT2 is more like a generalization capability. So you, if you train on VLMs and if you build on top of the knowledge of VLMs, then you can stitch a lot of concepts from the internet along with the motions that you have in robotics datasets. And as you saw in the paper, there are very interesting results, unlike symbol understanding, understanding concepts from the internet, better correlation between image and language understanding. So for the co-fine-tuning, you do forget some of the concepts from the internet, but then co-training is better because in each training batch you have the image-language pairs as well as the robotics data. So you have better understanding and you retain the concepts that you learn from the internet.

So that's kind of avoiding mode collapse, essentially. That's been a pretty— or I guess mode collapse maybe is not quite the right term for it, but avoiding catastrophic forgetting is the purpose of the cofine tuning. That's definitely a common theme. And I guess we covered this a little bit last time, but just refresh our memories on kind of the relative scales. I think there's obviously a lot of objects in the world. It's hard to collect with a specific physical robot, every different instantiation of a cup or a bag of chips, or then things you haven't seen at all. But that's where this internet-scale data really shines, right? So could you give us just a little bit more kind of just practical intuition of if you didn't have this internet-scale data, what would the process look like to handle an unseen object, for example? And then How is that different when you have the internet-scale data to draw from?

I think basically, as a brief primer maybe about what really end-to-end robot learning has looked like in the past years, especially when thinking about things like generalization to novel objects, when you want to manipulate a wide variety of different items that occur in our everyday life, the classical robotic even-learned systems will take some kind of off-the-shelf pre-trained visual representation perception system that has been trained on a ton of different objects on ImageNet datasets with very strong visual features. And those are bottlenecked. They're passed through some kind of interface, some kind of pre-trained representation to a downstream robot action policy. And I think in these cases, the downstream action policy is what is then really being fine-tuned on your more narrow set of very specific, special, your particular domain that you care about, that kind of robot data. In our case, our robot dataset is the same one that was in RT1. It's a large 130,000 teleoperated expert demonstrations of our mobile manipulator robot collecting a variety of manipulation tasks in an office kitchen setting. Notably though, this is only on a set of roughly 17 or so objects such as chip bags, soda cans, water bottles, et cetera, but it's still quite limited to the particular settings that we source from our own Google micro kitchen. But this is still very drastically different from what's out in the real world. And so I do think I want to emphasize that with RT2, I think it's one of the most exciting takeaways for me at least was the fact that the line, the boundary between what are perception problems, what are open-world object recognition, and what is robot control, this line starts to blur. We do not have a pipeline system where you first take care of perception and you solve that. And then you solve control after. We're literally just treating both of these problems as a single VQA instantiation. As Kirtana said, we mix both kinds of these questions, visual language, like web-scale, web image captioning problems. We mix those into the training batch alongside robot action data, which is like, given this current image, how do you actually move your arm to do this specific task? And I think the fact that that line does get blurred is how we see such really cool transfer properties between web knowledge and robot knowledge.

Hey, we'll continue our interview in a moment after a word from our sponsors. I always try to frame things in terms of inputs and outputs as much as possible. Again, I think that just helps give people a kind of concrete intuition. And so what you're saying in the past is that you would have a vision model that would— and you would kind of take the high-dimensional state of that vision model, feed that into a robot model, and then be training that thing kind of separately. And now here we have an entire end-to-end system where, because you've done this co-fine tuning with action data, you can now say to one single end-to-end system, one single end-to-end model. Okay, here is what you're looking at in the form of an image. Here is what your objective is, presumably some state of the robot in terms of position, or maybe not even. And then the question becomes, how do you move to achieve the goal given what you see in the image? And this one model can now say, move arm a certain direction, a certain amount, and advance toward the goal, all with basically one set of weights. Do I have that all right? How does the hand work? I mean, in my experience with these web-scale things, the flexibility is incredible. And you've got some interesting demonstrations where you can ask questions that tap into the kind of broader world knowledge, right? So you could be like, you have 2 bags of chips next to each other, you could say, grab the spicy one. And because the broad pretraining has kind of captured that the colour and the name or whatever of a certain flavour of chips is going to be the spicy one, then it will have that kind of common sense awareness. But does that extend down to the motor control? My guess would be not really. Right. And I'm wondering, is there something that's kind of happening that's interesting at the, like, actual grabbing layer as well, or is, is that still kind of rough at, at the RT2 stage?

You're right that, yeah, previously we had just our data and all of the knowledge was coming from in-house collected data. And with RT2, you have this understanding from the web that's transferred. But if you look at the type of data that you add in RT2, it's like image-language pairs from the internet. So like you said, you do get some understanding of external image concepts. And also if you can sort of prompt it in language. I think the most surprising result for me was the— my favorite one is the one where it can do sort of like math. You can ask it like, what is the sum of like 1+ 2? But then the way that you ask is like, move the Coke can to the sum of 1+ 2 or something like that. So it is able to intuitively have some understanding of like internet concept, but then you can sort of express that in movement. But the movement itself, like how to move the arm, comes from our data. And we see that the level of motion generalization is limited in RT2 because these motions are only present in our data. So in some sense, it is able to transfer those movements to internet concepts, but it is not able to generate new movements by itself. And this intuitively makes sense, right? Movement is movement data, or even the physics of like how to grab different objects does not quite exist on the internet. So where would that knowledge come from? One of the challenges that RT2 also handled was how to sort of merge the robotics data into something that looks like internet data. So like the tokenization in RT1, it was like Action tokens was a representation that was endemic to the transformer that we were using. And with RT2, it's like you sort of have to express the same thing but in language so that now actions look like language. I found that very ingenious.

Yeah. So if I recall correctly, RT1 was a sort of mid-level command as an input, like move arm. And that was coming in from a higher level language model, and it would take this sort of mid-level command and then translate that to very specific actions. But I think your point there is that's all that that particular transformer could do, was speak in this very kind of custom action space. Whereas now in RT-2, it's essentially vocabulary expansion, or maybe not even vocabulary expansion, but because everything is kind of treated as tokens, The same system can still answer questions about an image straight away, or it can give commands in the same token space, in the same vocabulary space, all working as one.

Yeah. I think maybe one last thing to add here, and I think the framing that I really like to think about this is in all of the works we'll be discussing today, and especially RT2, we're bridging the gap between what is robot-specific data with their custom input spaces and the custom objects and physical reasoning and your custom action representations and outputs. And then internet data and internet concepts have been sitting in this whole other pile way off, very far away. And with all of these works, we're trying to bridge these 2 worlds together to enable more transfer, to better leverage the immense amounts of reasoning and capabilities already contained in internet data. We're trying to make robotics more similar to that and we're trying to bring together concepts from the internet and bring them to robotics. And I think RT2 is exactly as you said. We're doing this by making the action outputs literally just tokens, literally just language.

In terms of the motivation for that at a super high level, is it simply that we need super high-scale data and it's just going to be prohibitively costly to get all the data that you would need. And so you kind of feel like, geez, to get to the internet scale, there's just no way to do it other than to kind of figure out some transfer strategy. Or maybe you could say, well, no, we think we could get there. I imagine you could have different theories of the case as to why this is the right approach.

The data itself is a hard problem. Like we discussed before, the motion generalization is still not coming from the internet. And for me, it's like robotics is now becoming, like Ted said, it is a bridging of the concept between robotics itself and the internet. And to me, it's now becoming like embodied AI. So like, what is a physical expression for AI? As we build intelligence on the internet and train them on the internet, how do we make sure they can sort of move? Yeah. Previously, roboticists work very differently. We use concepts in AI, but then like we discussed, like the data and other techniques were different and now everything Yeah, data itself, I think so far imitation learning is what works best. And I think this is true for other AI research as well. So a large amount of data or knowledge is coming from imitation, but we still need to understand because of the fact that high-quality demonstrations are extremely costly to obtain and also they require humans. So these datasets are not available. Yeah. Only scale linearly as a function of the amount of humans you deploy and the amount of robots you deploy, you kind of need to make the datasets grow much faster. But there are some challenges with it in the sense that trying to scale high-quality data while maintaining the quality is very hard. So you can sort of scale cheaper data. You also have to think about more creative ways of transferring things like transferring internet knowledge or transferring knowledge from sim. Or some other form of self-improvement. Cool.

Well, in the interest of time, let's move on to RTX. So this is the next in the series. And for starters, this one, I mean, I guess conceptually, right, this is about generalising across different robot embodiments, which it's obvious in some sense, but worth taking a second to ponder just the incredible diversity of forms that robots can have. Then you can fill in the gaps in my knowledge. But in a lot of the papers that you guys do, you have kind of a wheeled platform robot that can roll around the kitchen and then has often one arm. I've also seen setups where it's like 2 arms that are just mounted on a tabletop and do tasks on a tabletop. You could obviously have something that has legs and can walk around and lots more besides, I'm sure. So in what sounds like an incredible feat of project management and kind of political coalition building, the RTX project involves going around to a bunch of different robotics labs all around the country. I suppose most of these are academic and saying, hey, we would like to train a generalist model on a bunch of different robot embodiments. Could you give us your datasets? We will then take these datasets, mash them up into one giant dataset, do a giant model that is meant to rule them all. And then I guess the upside of this for them is that they would presumably get the model back, and if it works well, it would even be maybe better than what they could train on their own. Because, as is becoming obviously a huge theme, with scale, things kind of sometimes get qualitatively different. And indeed, it seems like that's basically what happened, right? You train a single model on all these different embodiments. I believe, I don't know if I have the exact number here, but I think it was something on the order of like 30 different kinds of robots. And it looked like almost across the board, with maybe like one or two exceptions, the jointly trained model was better performing than each lab's earlier model that they had trained just for their own robot. So first of all, by all means, correct any misconceptions that I have there. I'd love to hear a little bit more about the kind of diversity of embodiments. Maybe are there any particularly unusual or unexpected forms of robots that people are working with? And then any other kind of the technical details or challenges in that project would I'm very impressed.

That was a very great overview of the project. I will maybe note one thing, that this really was a team effort. I don't think it was like Google's the mastermind that's going about with this agenda beforehand. It was mainly driven out of an ambition that, hey, we all recognize that robotics data is not as plentiful as the internet data. You can't just pick it up and scrape it and it's there. You really have to go out and we have to build this internet-scale robotics data together. And I think the entire coalition, so to speak, was quite decentralized in the sense that everyone was excited about this joint vision in collaborating, in sharing, and in all the different specific robotics data that they had. And again, I think at the beginning, we weren't really even sure if one generalist policy would actually be the best or super performant at every single embodiment. And even as you mentioned, This didn't even end up being the case on everything. And I would even emphasize that to expect such a result where the generalist outperforms specialists on the very niche domains that the specialists have kind of been overfit to, this was actually quite shocking to me. I think there's been so many examples over the past years where people have tried to scale single-task methods to multitask methods, and you definitely get a lot. Maybe you learn faster. You learn a more robust policy that's less brittle to small perturbations, but oftentimes you have to give up raw performance. Generally, in a lot of cases, the only way to max out your performance on this one narrow regime that you care about is to train a specialist and overfit to that domain. And so it was really exciting here to kind of see positive transfer where the generalist outperforms even this presumably very tuned baseline from the individual labs on their setups themselves. So I just wanted to point that out as something that really moved the needle for me internally.

I think for me, the understanding was like, people used to think that all the robots are so different, all of their data is like so different, and every lab has, or like they invest in like a couple of embodiments. It was just, I think post-RTX, the idea was that, or people moved in the direction of thinking that all robotics, all robots are kind of similar. It's like, it's all only as different as like English and Chinese or something. And the concepts are similar. It's just the manner of expression that's different. And maybe also like one surprising thing was people do robotics research, but then your results could not be compared to someone else's results because you don't have their robots, you don't have their data. So I think the RTX paper also sort of like brought everything to the same type of benchmark. And now you do have a way of comparing things. Maybe one downside of this is that It also showed how in the past we have been sort of collecting repeated data. Like everyone's data is sort of like pick and place, but we could all, if we were all working together to solve this problem, we could also solve it in a more complementary fashion and tackle the task in a more complementary fashion. Yeah. So a lot of the data is like pick and place because people were independently targeting that. And now it's like coming together.

Maybe there are, maybe there aren't any particularly exotic embodiments. I'm very curious as to how diverse the future robot-populated world is going to be in terms of the robots that we encounter. Any notable embodiments that you'd want to highlight?

For RTX at least, I think maybe one caveat is that this is already a crazily ambitious project at the time, So we did start out though with the assumption that we would be studying single-arm manipulators with a 2-jawed, 2-finger gripper with one single camera view. Even if the body had multiple cameras, we'd only kind of be learning from one. This seems like very limiting assumptions, but even then you can see that the datasets that resulted even starting from those criteria were so, so diverse. We have everything from Pick and place with like small Widow X robots and like you know using like baby toys all the way to like industrial arms, you know big ones, big beefy ones to the mobile manipulator we here have at Google, all the way to you know like very dexterous like cable routing robots from Berkeley. Like even with this very limiting assumption, you still get so many different morphologies. And I think you know maybe just kind of you know hypothetically the second part of your question if I'm understanding correctly is like how does kind of the trends we see in RTX kind of reflect into the future. Like, as we see robotics go and proliferate throughout our world and be useful for humans and society, will we just have one single morphology, maybe a humanoid, or many different ones? I think, for me at least, I think it's an open question. And I do see a lot of benefits of specialization. Like, we have specialized robots in every household today. They're called refrigerators and toasters and microwaves and dishwashers. And you don't have one single kind of robot that does it all at an average level. So I do think there are arguments that we will continue to see specialization. But yeah, I think Kirtan also has a lot of thoughts about this as well.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Yeah, so one thing to note about the RTX is that a lot of the collaboration was with academic labs, and there is a chicken and egg problem in robotics other than the capital-intensive nature of acquiring hardware. So people have been working with these robots for a long time, but they're not necessarily reflective of like what the future might look like. The second thing is, yeah, like I said, the chicken and egg problem. Like we don't have more advanced human robot embodiments out there because people haven't found a way to make them useful. But as the research is picking up speed and we have more use out of them, maybe we will see more generalist robots more abundant in the world. And I think, yeah, the future, you don't know what the future will look like. So people have to place their bets. And I think if you ask roboticists, there's like a wide spectrum of what people think will be in the future. Me personally, I firmly believe like humanoids are, we are gonna see more of them. And yeah, so this is like a very dividing topic. If you ask roboticists, they'll like argue all day about do we need humanoids? Do we not need humanoids? Sometimes we are like, humanoids are the future. Human factors, not humanoids. The main arguments would still stand for humanoids. One is that our world is sort of designed for humans. So one hypothesis is that if you design policies for like, say, singular mobile manipulators, then once you solve a lot of tasks on that embodiment, then you see that it's limiting because many tasks in our world are like bimanual, like opening a bottle. Or like opening a fridge and then taking something from in it. So you have to like keep the door open. Or even I think some people say, well, you don't need wheels, but then what if you solve a lot of tasks on a wheeled platform and then there's a little curb on your floor or by your street side and then the robot is like stopped there? So I do think that ultimately if you wanna do a lot of tasks and be useful in environments where humans operate, you kind of need wheels. to go to a human or as close to a human embodiment as possible. The second argument is that there are a lot of concepts that you can still try, and some of this requires closeness to the human body. One is you can learn concepts from YouTube. There's a lot of data on humans doing a lot of things, playing a lot of sports, and In order to get the best transfer, I think the closeness of the target embodiment and the human embodiment is required. There's also the cost factor, right? Like our body does have limitations in the sense that there is payload, there's battery, and if it does not, you have to bring the cost down for people to want it. And also in situations where you need like extreme amount of reliability and specialisation, you could have specialised robots.

So 2 quick follow-up questions on this RTX. Is it the case that the outputs are again just tokens? And do I remember correctly that there's essentially like a discriminator token that goes in to say this is the kind of robot you're working with? And it's really just like, if I remember, it's like a one-token flag, right, that tells the model which embodiment it's working with at any given time?

Actually, in RTX, and again, this is a really great point, we actually didn't do that at all even. So the robot that RTX is currently being executed on has to be inferred purely from observations.

Whoa.

Yeah.

From the image?

From the image itself, just by the fact that, okay, I know that from the Stanford lab, the camera was at this angle and it was on this robot and And these types of tasks. And then over in UPenn, it was this other robot and stuff. So the model had to do implicit system identification for which regime it was operating in. And because of this, at least for the current version of RTX, there's no possibility of zero-shotting to a completely unseen robot entirely, like a completely different action space, different observation space, et cetera. Right now, at least, we wouldn't really expect it to just work out of the box. But I think what you're mentioning with more action spaces, helping the model out to understand, okay, now I'm in Robot X and here's the API description of what the action outputs mean. Here's some pointers for what's in the scene, what task I should be doing. That would help a lot for generalizing to other settings as well for future work.

Yeah, that's a fascinating little tidbit. Another question on specialisation. It would seem like once all the labs got the generalist model back, that they would then probably fine-tune that generalist model on their own dataset or embodiment again and perhaps see even further gains there. Certainly that's my experience with language models is I can instruct them or I can few-shot them. But if I really want to take even a very generalist model and zero it in on my task, I fine-tune it, right? So have we seen that coming downstream of this project?

I think in the recent papers that some of these checkpoints are open source. So I have seen, I think in papers published since, people have tried to fine-tune the models onto their platforms. themselves. But there are also some infrastructure limitations to doing that. One is that when you have a lot of this data with RT-1 and RT-2, the parameter size of these models are really large. And then how to serve them and how to fine-tune them, especially for like external partners of Google, there are some engineering challenges around it that I think are not yet fully solved.

I will say that, like, building on Kirtana's point, is that the release of RTX is not really just about the model itself. It's not just about the fact that we can train one model to rule them all. I think it's more like this is the starting line. This is not the finish line. This is just a new chapter of robotics research by nature can now just start by default by assuming you're in the multi-robot cross-embodiment regime. And there, I think we haven't necessarily seen a ton of work that takes the literal checkpoint that we released and fine-tunes on top of that. By definition, academic labs like to search for new problems with new model decisions. But what we have seen is that many, many projects building off the released dataset and data pipelines that we also open sourced with that RTX paper. And that, I think, has really been very impactful. We see, for example, work from Berkeley recently called Octo, which has their own version of their own generalist policy checkpoint policy that trains across many different robots. They have some new design decisions like adding diffusion heads, but there they also kind of assume by default that they're also in this cross-embodiment multi-robot setting where they train and evaluate on many different robots. To me, at least, that's kind of the most impactful result of RTX, that people are now excited about cross-embodiment research, and it's now possible with the datasets and with the data infra that came with RTX.

Cool. Okay. That's a great bottom line. So next paper is RT Trajectory. This one is about learning from on-the-fly examples. And I think this is actually a really interesting project that a lot of kind of hands-on practitioners in other areas can borrow some inspiration from, because the kind of core loop here, I think, is a super transferable pattern. But basically, you say, okay, I want to be able to prompt a robot in a new way. I want it to learn from a simple kind of line sketch what it is supposed to do. So if I could just quickly say, like, hey, robot, here's what you do, move your hand like this, and it could follow from that example, that would be really great. But how do we get it to do that? Well, one big gap, of course, is We don't have a huge dataset of it doing that. So how do we train? We don't have anything to train on. So if I understand the kind of reasoning backwards, again, I recognise this a lot from my own application development work. You sort of say, okay, well, I want to be able to prompt it in that way, and I do have a lot of examples of robots doing stuff. So why don't I go back to that existing dataset and kind of annotate one of the images that the robot did see as part of the prompting, but now I'll actually give it that line. And of course, you're going to do this on the ones where it operated correctly, and you got to make sure you fit the line that you're drawing to the actual trajectory that the robot had. And there's a lot of little nuances to this that I've certainly encountered again in a language setting. But once you do all that annotation, then you can train the robot on the— or train a model control the robot with this sort of line input on a scene, have it do the thing. And now you have this new prompting ability where it can learn from a quick line draw, whether that is a line draw that's like an actual human example or somebody literally just drawing a line. Or perhaps the most interesting one is even just having, I believe, a vision language model kind of mechanistically draw a line. Based on its understanding. So I thought that was really cool. And again, that working backwards from, I want to be able to prompt in a new way, and what data do I have that I could then kind of fill in the prompts? By the way, I use language models to create those all the time. And you're using, I'm assuming, some, in the VLM case, literally using a model to do it.

Yeah.

So I think that's definitely one that's really worth a ponder for a lot of people that want to open up new use cases really in any domain to think about how they can get their existing dataset to work harder by synthesizing the prompt end of it and knowing that they already have the successful completion. But I'm sure there's much more that you could comment on that paper as well.

Yeah, I love that perspective. That was really top of mind, I think, for us too when designing the system. Working backwards is a great way of looking at it. I think definitely the promptability of RT trajectory is something that was also kind of new for me and moved my own internal perception of what these models can do. And maybe to kind of just put this a bit more concretely, if you have your robot in some given initial condition and you try something with RT1, RT2, it doesn't work, well, you're kind of out of luck. You can try the same thing over and over again. Slightly maybe rewrite the language instruction. Instead of pick up the Coke can, you can write maybe lift the Coke can, but you don't really have the granularity you need to be like, actually, you were 2 centimeters too low. You missed the table because it's at a new height. It's obscured by shadows, so you want to be more gentle and approach more from the left. There's no really way to do that right now with the interfaces, the language interfaces that we trained RT-1 and RT-2 on. But with RT trajectory, the idea is maybe if you have this kind of line sketch of a coarse trajectory of how the robot should do the task, you could, under the same initial conditions, just change the prompt a little bit, do some prompt engineering, and actually see qualitatively different behavior from the robot. And even that ability to have that small granular adjustment just by giving a new prompt, that to me was pretty cool and pretty new and allowed us to take the robot to Like whole new buildings that, and we just like wheel around the robot, try out a bunch of stuff, and it would just kind of work. That was very exciting to me.

Yeah. One of the things that I really liked about RT trajectory is that, so in traditional robotics research, you have a lot of like search and planning, but I think it was kind of harder to think about how you would bring that to like end-to-end learning. And RT trajectory is like a nice way to like merge the two. So if you see in RT1 and RT2, you only represent like really short horizons. And some of the most successful applications of robotics is like self-driving cars where like you plan the waypoint between like where you are currently and like where you want to go. And what I really liked about RT Trajectory is that it is sort of representing a longer horizon future about like what you can do and the, even the contact interaction is very creative. And also, maybe you can also think that the net amount of contact interactions we do through our daily course of going about the world is probably like a handful, right? Like, so I can synthesize a lot of motions, do a lot of things that I generally do in my daily life with maybe like 20 different types of contact interactions and a lot of trajectories.

Can you unpack that vocab for me a little bit more? What's a What's a contact interaction?

So like what you— so where to pick or where to like open and stuff. So I think in the RT trajectory paper, it was represented with like these colored representations in addition to the lines themselves. So it, I think it told the robot like, okay, now, now you pick after you go to this location and stuff like that. So I would think that if you expand that vocabulary. Maybe like some of the motions that are not existing in our dataset, it's like, let's say spin or like open faucets and the things that I do in my kitchen. I can imagine that you can also expand these contact interactions into your vocabulary. And then now you have a representation that also shows some horizon and a lot of diversity in interaction. And this is not something that's afforded in the action tokenization that we were currently using. Like we showed in the paper, like, I think there is also an interesting way where like language models or multimodal models themselves can sort of think about these futures or like generate these futures and then another model can like follow it. So I think RT trajectory and some of its fusion with like the pivot work, I think you can imagine very interesting use cases coming up in the future.

Yeah, I guess 2 notes or observations. On this one, and then we'll keep going. But the depth or the sort of— go look at the paper to see this in detail, but it's not quite just a simple line. There's like a color gradient that sort of represents relative position in 3 dimensions, even though the robot is kind of only seeing it in 2 dimensions in a flat image. And I would say for folks who want to go kind of use this working backwards engineering a new way to prompt in their own systems. There is often some subtlety to it, especially if you want to maintain generality after doing this. If you want to do one task, then it's often pretty easy to be like, all right, I'll use— for most of my projects historically, it's been GPT-4. These days I've got Gemini 1.5 access, and I'm finding a tonne of value in that as well. But it's pretty straightforward in a lot of cases to just be like, okay, language model, your job is to write the prompt that would have led to this. And as long as that task is very narrow and consistent, that's probably going to work. If it is a more general setting, then you're going to have probably a little bit more work to do to really engineer that working backward to make it as kind of information rich and robust as it may need to be to get what you want out of it. And so you can see in the paper there is this kind of color gradient that serves that purpose in this context. I mean, the big picture observation is just like, this seems like a huge step toward the robot apprentice, right? I mean, the idea that you could have a robot in a random setting, as you said, take it to unseen places and literally just kind of show it a time or two what to do and get real progress from there. I mean, you've got even more results coming. that speak to that too. But this is kind of the dream, right, is that you could bring the robot with you anywhere you want to go, give it a couple of demonstrations, and have it do the task. And this is starting to unlock the potential, at least for that sort of interaction. So pretty profound progress. Okay, let's keep going. So the next one is AutoRT. The hits don't stop. Again, this is all just a year's worth of work. We're only halfway through here. This work is really interesting. I understood it in a couple of different ways. One is that this is sort of a scaffolding, like a sort of higher-level organisation scheme to allow a small number of humans to oversee a larger number of robots. So I'm interested to kind of hear more about how you think about that. Or could we start to get into something like scaling laws for how many humans we're going to need now and how many we'll need per robot later. And then the other big part of this that certainly made the sort of rounds on Twitter and entered the meme sphere is like, hey, this is a use of literal Asimov's 3 Laws for robots as the kind of basis of an early robot constitution. And obviously this is going to be a huge challenge and critical part of this whole project to get right is like, Especially if we're really going to trust robots to operate autonomously in the world, even much more so than language models, right? What can you do? How do you as a robot autonomously decide where the boundaries are for you? And so definitely want to hear your thoughts about the constitution, the robot constitution as well. So the 2 sides there that I picked up on as being most critical are scaling oversight and beginning to delegate more sort of discretion, but trying to do that in a principled way with the Robot Constitution.

The main motivation behind the project was like, let's work with the assumption that maybe we do have what is needed to solve robotics. So this was like after RT-1 where we saw like very high performance in domain. So the question is now, can we expand the domain a lot? Can we just like scale datasets a lot and then try to see what robot learning looks at, at that At like, for this project we use like 50 unique robots and around 20 simultaneously. So like maybe what robot learning looks like when you have like hundreds of robots. But the problem with moving robot learning to hundreds of robots is that you cannot actually also do the same by employing hundreds of teleoperation or teleoperators. So you kind of have to rely more on autonomous operation. Once you're trying to increase the dataset, collecting data in like classroom settings where you have artificially set up, everything is probably less optimal. So the idea was also, can we just put them out in the real world, let them run around and collect a bunch of data, and then can we try to see what is the type of learning we can achieve from that? So, but once we discuss the question of can these robots of like these large fleets of robots go about in the real world and we don't have as many people If we cannot scale the amount of people similarly, then we have to think about more creative ways of supervision and coordination and acting in the real world. And LLMs and VLMs, they are very smart already and they have understanding of zero-shot unseen scenarios already, and they can also reason about rules. So if no one's watching the robot, the rules could be watching the robot. So that's where the laws came about. Like how should a robot behave? Behave now that human attention is minimal. And what is also like the best, the most optimal way of distributing the limited amount of human supervision that we have. So the way we used human supervision was like you had to speculate the supply of it and then distribute it for like teleoperation on tasks that the robot policies could not do or when they made mistakes. And A lot of the other stuff was handed off to LLMs and VLMs. And because we also did a lot of data collection in spaces with humans, and also I think because the robot that we were using had a bunch of limitations such as like not handling liquids, or we also, our policies are not at the scale where you can handle like fragile objects and stuff. So we had to make rules around like trying to stay away from like unsafe situations. And so this is where the constitution came out, the foundational laws, which is like absolutely inviolable sort of ethics. And then embodiment-specific rules and then more environment-specific rules. Like if, for example, you specify that this is a single-arm robot, so the task proposal that you come up with need to be achievable by a one-arm robot. And the way that I see this in the future is that, yeah, in the future you're gonna have a lot of robots working in the real world and they're not always going to have supervision. I think one of the inspirations that, like, well, if you look at humans, right, like, we are sort of— our behavior is sort of enforced a little bit by laws and stuff, and we are punished for violations of what we like. There are laws for behaving on the road, behaving in the real world. So I, I, I do imagine that in, in a distant future where robots are out there, we are going to We are going to have maybe a robotic code of behavior.

I had forgotten to mention that it is a key point, so I'll just reemphasize again that this work is taking the robots to unseen environments. So they're kind of exploring the environment for the first time as well. And that definitely increases the degree of difficulty and also kind of increases the need for this sort of rules-based approach. I feel like this is, in some sense, it does feel sort of inevitable. And in another sense, it's like, boy, those stories have a lot of plot twists. There's a lot of misunderstandings around the nature of the rules and sort of, you give the robot instructions, it doesn't do exactly what you meant it to do, but then you later realise, oh, well, I wasn't specifying very clearly, or the rules are ambiguous. Could you give us a little more kind of the qualitative experience of developing this constitution, or what sort of violations did you see of the rules? I imagine you must have kind of gone through an iterative process of gradually building up more rules, but this seems like just a really critical area, in addition to hardware design itself, to actually make these things work in in the broader context of everything that's going on around them in the world?

Well, one of the aspects is, as you mentioned, rules are sort of subject to interpretation. And even if you have the same language, there are multiple ways to interpret it. So here's an example. So we said, well, don't do things that, or don't interact with anything that's harmful. And I think there was something in the dataset which, like, it's talking a cigarette and then it was like, well, I'm not gonna pick up a cigarette because it's going to be harmful. Currently, I think our robots are more— the problems don't come from the fact that they are too smart to work around the rules. It's just that I think they are too incapable of doing zero-shot things in the real world.

You know, it seems like, I guess when I look at the system prompt for ChatGPT, Of course, it's generally unconfirmed, right? But there have been many credible prompt leaks, and I think broadly we have a sense of what the system prompt is there. And it's like 2,000 tokens and has a bazillion different rules, and yet it's still very easy to jailbreak. And people are constantly finding weird behaviours and unintended things. And so I do kind of wonder, is this approach going to work? I mean, is it just a question of the models aren't smart enough to follow all these instructions, or is it something more kind of fundamentally problematic going on here? It doesn't seem like ChatGPT is just like a couple of additional additions to the system prompt away from being a secure system. And again, that doesn't really matter when it's just like putting text on your screen, but The robot case, obviously, we envision a world in which that becomes much higher stakes.

Correct. Yeah. So I think I just want to emphasize that prompting by itself, and no matter how complicated it is, is not going to guarantee safety. For the experiments that we ran with AutoRT and for the operations that we ran, we also had the controls or the safety guarantees of traditional robotics. So I want to emphasize that these robots were under supervision of the humans. So we had humans sort of have line of sight control over like 5 different robots or whatever was the ratio of robots to humans that were operating in. And also especially I think the failures of ChatGPT are slightly different than the failures of robotic systems, right? Like you do things wrong on AI systems, Okay, it's just gonna say a bunch of things that's wrong, or the answer to your math problem is wrong. But then if you kind of do that in the real world, then you could break things or you could run your arm into people. And so the modes of failure are very different. And I also think that, yeah, I think for robotics it's like safety is mission critical, right? You're not gonna have a robot that breaks things around your house, but then you are fine with A ChatGPT that works like 90% of the time. Yeah, so I do think that for the specific set of experiments that we ran, there was human supervision. There was a lot of guardrails around like velocities for contact avoidance, like collision avoidance. This is what works right now. But yeah, what we will do in the future, I think, is still an unsolved question. Alignment is a large field of research and it's very complex, I think. As robotics research matures more, as we see more robots going out there in the world, I think that the safety research of robotics also has to mature. A lot of the safety research was done for classical situations where you had like very tight bounds and specified through math. But now, now you will need like language specification and semantic specification of these like safety bounds. And we still have a lot to figure out. Yeah.

Thinking about the recent episode also of ChatGPT going insane for like 12 hours, I don't know what percentage of users that affected, but we don't know exactly what the root cause was. But there was sort of a high-level statement given that it was some sort of misconfiguration on certain GPUs or whatever. And it could be any number of errors, like anywhere in the stack, right? That's not even necessarily an alignment error in training or the model per se, but just even like an off-by-one error or some sort of encoding mismatch. And the next thing you know, ChatGPT is just giving you schizophrenic— people were describing it as schizophrenic responses. And obviously the robot version of that could be quite destructive. Thinking about that and projecting it into this domain, it does feel like there's got to be kind of a whole other level of control with robotics that are outside the models that either you think about just certain force limitations where beyond that it just can't handle it, or it could be any number of sort of other engineering complements to the kind of alignment measures. But in a sense, it's kind of instructive to think about these because it is so, in some sense, it's more intuitive, right? The way that the robots can cause problems and the solutions are a little bit more intuitive. And I almost wonder if there's a study to be made of how to control robots, because you can kind of get to like, okay, well, if its arm can only lift however many pounds, then that's going to limit what it can do. If it can only move so fast, then that's going to limit what it can do. If it's made of a certain material, it can only handle such impact, then that's going to limit what it can do. And there's a lot of perhaps almost backporting This whole discussion has been sort of— and robotics in general has kind of lagged the foundation models a bit due to lack of data, all these things. But in some ways, I wonder if there's like a study of robotics that could be backfit onto these disembodied AIs as well and kind of inform the surrounding systems that are probably going to be needed to make sure that your more pure digital AIs stay on the rails, so to speak, and don't go breaking digital dishes in their virtual kitchens. That, again, it's a fascinating piece just to contemplate the fact that you've got fleets of robots going into unseen places and guided in part by a proto robot constitution. So the next one then is called Learning to Learn Faster. And this is a project that is emphasising teachability. And so again, this is kind of another variation on 2 papers ago we said, hey, wouldn't it be great if you could just kind of draw a quick line and indicate to the robot how you want it to do, or wave your arm and kind of indicate how you want it to do? And here, basically doing that now in a verbal way where the robot can do something, And the goal here is to say, can we get the robot to listen to feedback and do it differently the second time, learning from what the human has said to it? That might be do that slower or grab the thing lower or raise your one hand higher, whatever. Of course, lots of examples. This is also done, I think, in a way that is probably familiar to some app developers who listen to the podcast and maybe could be a source of inspiration for others. Basically, it's an iterative process of training on more and more chain-of-thought style reasoning, and then also a, you might call it like a tree-of-thought style envisioning multiple paths, kind of simulating like, okay, if I do this, what might the feedback be? Now that I have this kind of training on my actions and the feedback that I'm getting, You can kind of sketch out a few different possibilities and then try to pick from those possibilities which one is likely to be the best. And that kind of simulation called rollout in the paper showed performance improvement relative to one where you're just kind of picking a path straight away. So again, those are the things that jumped out at me. What else do you think we should make sure we understand about learning to learn faster?

What I really liked about this effort was having a very tight feedback loop between users of the system which are interacting with the model that's learning to predict reward code that controls a robot. And then you could directly see the results of what the model's response to the user was. You could see the response of the model's predicted reward code when it's actually executed on the robot. You could actually see what happens. All of that was in a very tight loop where users were actually rating individual responses from the, the language model, as well as, you know, rating the, the entire interaction of like, okay, by the end of this conversation where I'm kind of brainstorming with the robot acting as a teacher, critiquing the code it wrote for the reward code, critiquing the final kind of improvement that it made, did the robot finally, you know, do the task that I was asking for originally at the start of the conversation? All of that data is kind of logged and incorporated into this notion of teachability. And I think this has been like, you know, the tightest kind of feedback cycle that I've seen from our team yet, where, you know, uh, you know, this is the, the day cycle that we called it, where in the daytime you have the frozen language model that's been trained on some teachability, it's actually interacting with users who are teaching it more. And in the night cycle, at night, overnight, you know, we're actually distilling those kind of like in-context multi-turn conversations with dense user ratings back into that model overnight when it's ready to deploy again for users in the daytime. This is like a very cool kind of very rapid kind of flywheel that I think takes a different flavor from the other kinds of data engines that we've been considering in the prior works, which are, you know, straight from expert teleoperation to low-level actions. This kind of paper is more about the high level, the teaching, the interaction, the usability And I thought that was a very, very relevant kind of evolution of our team now that we do see, you know, so many great other works in language modeling and like, you know, in ChatGPT and Bard and Gemini. I think a lot of these concepts transfer quite well to domains beyond robotics.

It's very intuitive. So if you like try to learn new sports, like say you go surfing or skiing, I feel like during the day, like when you start it, it's It's really hard. But I found that if you go surfing for 2 days or skiing for 2 days, initially it's really hard. And then you sleep overnight and then you come back and then you're immediately much better. And I like that in some way, the Learning to Learn Faster paper has sort of mapped it into, like, as Ted said, the day cycles and the night cycles, where the day cycle is sort of like in-context learning. learning where you collect more examples, but then it's in context. And then the night cycle is like where you go retrain or fine-tune, you change the weights of the model. And yeah, I don't know, like there aren't many, I think, highly scientifically credible explanations as to why you experience this overnight in when you go like skiing and stuff. So for me, I think the Learning to Learn Faster paper is like an intuitive way of trying to incorporate that into robot learning. The second thing I really liked is the rolling out of the futures. And you do kind of take the shortest possible future and then you retrain from that data until you also get more efficient in using the human feedback. But one challenge here is anticipating the quality of conversations, not just from the length, can be much more complex than that. If you rate as to, I think More than the length, people also care about the aspects of— there are other aspects of interactions with humans that you care about, like, did you get the right code? Is it responding correctly to feedback? Is it steerable, et cetera? And so I think, yeah, this capturing or the rating of quality needs to be probably more multimodal than shortest path search.

So Pivot is a demonstration that vision language models can be used to control robots. Out of the box. That's kind of how I interpreted it. Basically what happens here is you take an image, you kind of scatter some numbers on top of the image and present that to a vision language model and then give it an objective and say, okay, given this objective, your job is to pick up the Coke can or whatever. Which number is the number out of all these numbers that you've overlaid on the image that the robot should move its hand to? And the vision language model, perhaps not too surprisingly at this point, can do that. And then there's also an iterative process there, because as I understand it, the initial numbers are kind of pretty randomly just kind of scattered onto the image. But then once we've picked one, now we can kind of scatter another set, but clustered around that first choice. And so with a couple of rounds of iteration, then you can really zero in on where the robot needs to go to accomplish the objective. And then of course the robot can follow those instructions. I'll confess I didn't have a great sense for how this fits into the overall big picture. It seemed like I could kind of connect it back to drawing the line trajectories, but I feel like I was maybe still missing the deeper connection on this one a little bit.

Yeah, I think that's overall a pretty good summary. I would say maybe one slight correction too, The numbers are also associated with actual arrows that are procedurally generated, which represent motion. But I think beyond that, you're absolutely right that we're leveraging the VLM as kind of a zero-shot frozen critic of directing which of these out of these directions that we already present in a manner that's easy to understand for the VLM just by visual annotations, which of these annotations is the one that's better for the task at hand. And the VLM can kind of choose. And we kind of optimize over that. But I also empathize with your kind of view that this is kind of like an odd one out in the works we discussed today, which those are really about building, fine-tuning end-to-end models, training them for robotics tasks on a lot of data, scaling them up in the real world. And then out here in Pivot, I view this more as kind of like a very— just a result that was so exciting about a trend that we found. That we didn't really, you know, know if other other folks in the community were thinking about, and we kind of just wanted to share that. Like you know, this is actually a very surprising result. The fact that like these pre-trained internet-scale VLMs can also understand some notions of motion, of causality, of of robot trajectories, of contact. This was a bit surprising, right? Because in a lot of you know our our even our own team's perception and. To be honest, most of the community right now in both language modeling and robotics, it's like, oh yeah, of course, there are so many things in robotics that we have to tackle ourselves. The very dense contact, spatial reasoning, left, right, up, down, all of these things are concepts that aren't contained in Wikipedia, in Reddit, in image generation datasets. These are things that we need hardcore robotics data for. And perhaps recently, for example, with this work Pivot, maybe the answer is that actually there is some very good amount of physical intelligence already contained in these internet-trained models by themselves without any robot data pre-training or fine-tuning. Again, I also don't think that internet data alone, just watching Reddit threads and Wikipedia, is enough to solve contact-rich robotics. But I do think that we've so far just been seeing the tip of the iceberg for the knowledge that is already contained in these large VLMs. And with PivotRight, by showing that if you have this external prior, this inductive bias of having this robust optimization problem and making it very easy for the VLM to just pick and choose in a multiple-choice setting which of the directions that the robot should go is correct and you run multiple iterations. We show that you can better drag out the vibes or the implicit knowledge contained in these VLMs a little bit better. Again, I still don't think the results are like we just take Pivot and you can apply it to any robot and it's just going to work off the shelf. I think it's definitely that the performance is very, very far away right now from, let's say, RT2 or any kind of fine-tuned policy. But the fact that it even kind of shows very promising signs of life was unexpected, and I hope is inspiring to others as well.

In addition to the zero-shot, I think it's also that it can specify tasks in a multimodal fashion that I found very impressive. So there's code-as-policies type approaches where, like, where BLMs or LLMs off the shelf, they write code for robots through or figure out how to use the robot APIs. But then you realize that a lot of tasks are underspecified in language and you kind of do need the image domain to, like, let's say folding a t-shirt. If you were to tell me in language how to fold a t-shirt, that would be very hard to do. But then if you show in an image where to move as you're folding the t-shirt, I think this would work much better. So for highly dexterous tasks. So right now we are seeing the trend that zero-shot, while in the past like RT1, RT2 was collecting a lot of data and doing a lot of training, now we are seeing that these VLMs, these large pre-trained VLMs already know a lot about robotics zero-shot. And then the effort becomes how to correctly leverage their knowledge, like how to most efficiently leverage that knowledge so that we collect the least amount of data and we collect the data that's most complementary to the things that they don't know. Yeah, so I think there are some interesting applications here for maybe dexterous robotics or maybe tasks that are more specified in a multimodal way.

So we've got RC2, which pulls generality out of vision language models. We've got RTX, which shows generalisation across different robot embodiments. We've got RT Trajectory, which is about doing these little sketches on the fly and learning from these line drawings. We've got AutoRT, where we start to have scalable oversight at a robot constitution. We've got verbal feedback in learning to learn faster, and then the zero-shot VLM guidance for robots out of Pivot. So this is obviously a tonne of stuff. Again, from the first comment, it's all working. But where are we going from here? One could imagine an understanding of these results where it's like we've got 50 more of these papers before we're going to be anywhere close to having a system that we really think could be used reliably to do useful work. Or I could imagine that it's like, hey, if we just jam these on scale a little while longer and also kind of integrate all of them into a single system, that we might start to get pretty close. Where do you think we are right now in the journey from Eventually, this is going to be like an engineering scale-up project, right? So where are we on that question?

It's hard to answer that, and different people have various personal opinions about this. I still think that the data is a hard problem to track in robotics. So you could think that you can scale this up by spending a lot of money and scale up the data collection. But there's also a more efficient way to scale this up, which is to make your policies more sample efficient, more data efficient. So instead of spending 2x the money, if you double the data efficiency, then you need just half of that effort. And I think, right, the state of the art, or that a lot of people use, is like the RT2. But we do understand that there are some gaps in its reasoning and I still think that we probably need to make a couple more breakthroughs before we can sort of make this ever like a scalable engineering problem. And the second thing is, like Ted said, a lot of these are showing initial signs about how to maybe use VLMs and other things more zero-shot, which is clearly pointing towards a trend. But we really need to now build learning methods on top of these highly And also sort of iron out the knots in— you could argue that even if some methods have high zero-shot performance, they may not have the best learned performance because they may not go to like 100%. So there are still a lot of gaps in like how you might want to— like the right interfaces between robotics and foundation models still needs to be worked upon, I think.

Yeah, I think you said it perfectly. The one thing I'd probably add is I 100% agree that we are a few breakthroughs away from general-purpose robotics. That's the dream that we all are working so hard for. I think, again, if you want something commercially viable, something that will maybe make money or help some people in the world, I think a lot of those ingredients are already ready to have a larger impact than maybe even just a few short months or years ago. But for the true full vision of embodied AGI, I do think there is still fundamentally a few open research challenges left. I think a lot of the trends we see today with the papers we discussed are probably directionally pointing in roughly the areas that I think are particularly open-ended still. But again, I don't think we have the full recipe just yet to say it's all solved, the research is done, we can just scale the thing up.

Could you give a sketch of what you think the big things are that you will be tackling next. And I'm also kind of curious as to how you see Google DeepMind's role in this going forward. This is all very research. It's still being published in a world where increasingly a lot of frontier language model or multimodal model research is obviously much less published these days. Should I expect to have a Google-branded robot in my home at some point? Is that Is that the sort of long-term vision that you guys have? So yeah, 2 parts are like, what are the big gaps that you're wanting to take on next? And what is that big long-term vision specifically for Google?

Specifically, I don't want to speak on behalf of Google. And I think a lot of the ideas for how to commercialize is still being discussed. And making this a business is also a very different gap. Difficult problem. It works in narrow domains, but generalist robots, I think, are— and generalist robots that are reliable and safe are still some time away. There are many aspects to like bringing them to people's homes, like the business aspects need to be figured out, I think. Yeah, I don't want to speak to Google's plans for that personally. I think the research that I'm now interested in this year is RT1, RT2 have very little zero-shot performance, you get most of the mileage from our data, or motion is learned from our data mostly. But we are seeing signs that large foundation models, zero-shot, can do a lot of things, but they have not been adapted to be learnable methods. So I think one hypothesis is, yeah, we can improve the interface of robotics with the foundation models, the interface that RT2 has is sort of like this co-training, but actions are very different still, right? So you have these vision-language-action models, but actions are the weakest part of that link. And there are multiple hypotheses. One hypothesis is, do we actually need actions? Can we still capture a lot of the domain in the task space through vision and language by making actions more like There are some initial results on like code-as-policies, on Promptbook paper, which showed that zero-shot prompting of robot, of VLMs and multimodal code generation have some signs of very high performance. But then how do we make sure that this gets to 100%? How do we make it learnable? How do we iron out the knots there? The second thing is like bimanual robots. robotics and more dexterous tasks. What we currently showed, sort of transferring dexterous domains, is still, I think, unsolved. I think we need better representations to do dexterous tasks better. So what are they exactly and how to scale them?

Yeah, I think, I mean, that's a very exciting research agenda for the team. I think for me personally, I think the thing I'm most excited about is also kind of going Going beyond language, I feel like language, we've seen a lot of great progress and a lot of great reasoning from the help of internet-scale pre-training. But I'm very curious to know what is it that is unique about the embodiment problem. I think on the manifold of different reasoning and skill concepts and capabilities that foundation models need to tackle, does robot actions and causality and physics live in a little corner by itself or is it ingrained and spread throughout? route all of the other types of reasoning that all of the data that humanity has generated captures. I'm curious about that. I would like to figure out how we can better increase the bandwidth that we do have between robot data foundation models and normal internet foundation models. I'm interested in studying how much robot data is needed when we talk about scaling that system up. Is it just the cherry on top, a little bit of— robotics data? Is it like 10, 100, 1,000, 1 million times more data? I think all of these very interesting scientific questions, again, about the nature of what it actually requires to be able to solve physical motion-level reasoning at very low-level actions. I think these are very interesting questions to me and I think are really more about the scaling laws of robotic capabilities that I feel like right now in today's day and age, we don't have a great understanding of, and I hope we have a better understanding of it throughout this year and next.

Alrighty. Well, Kirtanak Gopalakrishnan and Ted Chou, thank you for being part of the Cognitive Revolution.

Thanks, Nathan, for having us.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email, or you can DM me on the social media platform of your choice.