Pete Florence: When Robots Improvise

2011 Fukushima nuclear disaster. The

Fukushima number 1 nuclear power

plant has been hit by its biggest earthquake.

Simply heartbreaking.

There wasn't a robot in the world that could walk into the nuclear facility after the meltdown and just turn a valve. And so we had to send people in for that. If you're not touching and moving the world and you're a robot, what are you doing? Any process in the world that is gated by physical progress, building houses, whatever it is, we could just do that way faster, way better. Gem 1.5, it has a couple different capabilities where you can learn a physical task The success rates of all these are modest today, but the fact that these success rates are anything above zero is the incredible thing.

Sholto Douglas, well-known Anthropic researcher, called this sort of a GPT-3 moment for robotics.

The example that we highlighted is using a brush to sweep an object into a bowl, and then instead you take the brush away and you give it a dustpan.

One of the stories of these past few years has been finding sort of the killer use case in coding. When it comes to physical intelligence, do you have an intuition of what, like, that killer use case is? I

think the best analogy would be dexterity. If we can really crack dexterity, then the robots can build the robots, and the robots can start building next generation of robots.

Robots don't learn the same way that people do. Teaching a robot a new task can take months of programming, data collection, and retraining. Generalist, an AI company for robotics, is compressing that process down to seconds. In one of its recent demonstrations, a robot was shown just one time how to take money out of a wallet. It then completed that task immediately without needing to be retrained. Today's guest, Pete Florence, is Generalist's co-founder and CEO. The company is building what it calls general intelligence for the physical world, AI models meant to work across different types of robots, from factory arms to humanoids. Since its founding a little over 2 years ago, Generalist has grown into a key player in the robotics race. In this episode, Pete and I discuss one-shot learning, why dexterity remains so hard, where physical AI follows the same scaling laws as digital AI, and which industries change first when robots learn new skills. Much faster. I'm Mario, and this is The Generalist. Well, I think the only place to start is that I think you have the best name possible.

Likewise, Mario. Yeah.

Very good taste. Well, I'm really looking forward to this conversation and getting into what you're doing at Generalist. Uh, it feels like a particularly exciting time. Earlier this month, I think it was mid-August, you released Gen 1.5, which is your, your latest foundation model. And it seemed to attract a lot of excitement. You know, I saw Sholto Douglas, well-known Anthropic researcher, called this sort of a GPT-3 moment for robotics. Maybe you can tell us a little bit about what Gen 1.5 is and why it matters.

Sure. So Gen 1.5 is definitely, in a few different ways, like a type of model that we had been kind of waiting for for a while, at least in terms of glimpses of types of capabilities that Both are just exciting in and of themselves, and then also point towards what might be a future of what's coming next. So overall, Gen 1.5, it has a couple different capabilities in what we call one-shot or few-shot learning, and also some other things in what we think are really remarkable levels of zero-shot physical generalization. These one-shot and few-shot broad types of learning capabilities where you can learn a physical task in just a single demonstration. And mind you, just to be clear, like the success rates of all these are modest today, but the fact that these success rates are anything above zero is the incredible thing. Yes. And, you know, if I dial back like a few years, one of my co-founders, Andy Zhang, and myself, like we've been talking for a while about this threshold of crossing into a very generalized one-shot and few-shot learning capability in robotics. And of course, like the rise of large language models around 2020, yes, of course, GPT-3, like that was the kind of the main thing that, or at least arguably the main thing that people were especially excited about with the model GPT-3 was this broad one-shot and few-shot learning capability. They coined the term in-context learning in that paper. It's been an incredible thing to see.

And so one-shot and few-shot learning is sort of a necessary principle to have that sort of like generalizable intelligence in the same way that GPT-3 had, where you're able to say, hey, you know, make me a poem in the style of, you know, Emily Dickinson. And after a few tries, if you give it an example, it can do that. You're sort of seeing that similar style of behavior in the physical world. Is that sort of the parallel?

Yeah. So, you know, specifically one-shot is where you give a single example of an input and an output. So you could say like, who wrote Romeo and Juliet? Answer, William Shakespeare. And that's like a question-answer, an input and output. And then you give another input and you let the model do the output. So you give it a single example, and then the model then knows what to do from that single example. That would be like one-shot. You can also do zero-shot, which is you just like say, you know, who wrote Romeo and Juliet? And you don't give any example of a full input-output.

Yes.

And it was actually, if you trace back the history, it was actually GPT-2 started to show a little bit of these types of capabilities, but it was really GPT-3 that like brought broad one-shot, few-shot learning into the, like, wow, basically a very wide variety of tasks. You could just give it a go. You could just ask the GPT-3 model, for those that remember, just like trying to prompt it with a few different examples. And it had a reasonable shot at doing something intelligent. And that's like the level that we feel like we're now seeing, where the success rates are not super high. There are limitations on the types of tasks that we can just prompt the model to do. But a very wide variety of tasks, you can just try to prompt it and see what happens. And there's a reasonable shot that something intelligent occurs. And again, this like, the fact that this happens at all is the incredible thing to me.

And the tasks are things like take these bills out of a wallet or sort these cups, which it's amazing to watch these things play out. Did you expect to see this capability as part of this model? Like, where are things tracking versus how you were sort of Imagining they might.

It was surprising to us that this model started to show this particular type of one-shot, few-shot capability, in part, and it's something that we talk about a lot in what we released describing some of the development behind the model, but it was not trained to do one-shot or few-shot examples. And that is actually the same way that, at least as far as we know, as it is described in the GPT-3 paper, that happened as well, where you actually just train the model, in the case of GPT-3, on every single possible piece of text that OpenAI could find at the time. And it's later been studied and given a name that it's kind of, you know, the way that the data is just laid out, it has this property which is called burstiness, which is that when you see a single SAT question example, just to give an example here, like, It's pretty likely that the next thing that the model sees is another SAT question. So when you start to load the context of the model with similar types of things, it's pretty likely that the data distribution it was trained on has a similar type of thing occurring after it. That's one of the ways that we can try to hypothesize how this capability showed up in the model GPT-3. And similarly, Like, we never trained Gem 1.5 with like an example of a task and do it again. But the model just, you know, the physical world also has burstiness where if people are like taking bills out of a wallet, it's pretty likely they'll take another bill out of a wallet, right? Like, it's literally just guessing what is most likely to happen next or what it should do next based on a little bit of prompting in its context. It's been surprising for us to see this model start to have that. I would have thought it would have taken maybe a little bit more time. But at the same time, I do think it was inevitable at some point, maybe through different means, to start to have this capability.

Do you have a sense of why it's more successful at some of these tasks than others? Like, it seemed like, I think the wallet example is actually, it's one of its best tasks, right? Versus things that to a human look of equivalently difficult? Is that just a data availability question? Is it something else happening?

That's, it's a great question. And the honest answer is it's, it's really hard to trace exactly why. We can sort of build up some intuition from a lot of the empirical experimentation, you know, in the pre-training distribution. You know, a lot of these tasks, there's no way that like every single task that you can think of is actually in your pre-training data. Like definitely, There are tasks that we can do that there was no pre-training data for this. And one of the easiest ways that we know this is we can just 3D print objects that we're pretty sure have never existed before. We can demonstrate a task with it and it can do it, right? But there are certain sort of underlying skills or behaviors that together comprise a task. And if you have more data supporting those tasks, then that's maybe one that would have higher success rates. But it's not only It's not only about how much data is there. It's not very simple to describe what makes a task challenging in robotics, especially for dexterity or dexterous manipulation. But I think there's a few different kind of component features that add up to make things challenging. Like one is how precise from a motion precision standpoint does a task have to be? How precise from a force precision standpoint? So how like— hard do you have to push on something, or not hard do you have to push on something? How coordinated do all of your degrees of freedom have to be? So like a task that really requires a lot of bimanual precise coordination between the 2 different hands, that's very challenging. Those are a few, but there are other aspects as well where it's actually a very rich space that we can try to kind of make sense of about what makes a task challenging in the physical world. Those all contribute here. Yeah.

Robotics is so interesting. You know this 1,000 times better than I do, but I find myself getting so excited about some of the stuff that you're doing because it touches on what we think of, touches on biomechanics, it touches on philosophy. You get into these evolutionary questions. When I was starting to do some reading about the evolutionary development of our own hands, it seems like it traced extremely closely with the development of our cerebral intelligence as well as our manual intelligence. Like those 2 things just, you know, came together in a really interesting way. You've mentioned before that dexterity is something that's sort of like this, I don't know, holy grail or big topic in robotics for a long time. More specifically, why is that something that seems to be just so important to get right? Is it just about the precision? Is it about those different factors that you sort of just mentioned?

So one, For sure, it's something that honestly, both like me personally and the company has driven a lot of inspiration from, is exactly what you mentioned, which is that if you study evolutionary biology, of course there's evidence to suggest these things, but we can never know for sure. But it does seem that, and there's this term for this area of the brain, my brother's a neuroscientist, it's called Broca's area or something like that.

Which is language, maybe?

It was believed to, there's all these different regions of the brain can be associated with many different things. But there is some evidence that I have read in published research, which suggests that, like, the development of dexterity and the evolutionary adaptiveness of, like, having better and better toolmaking skills then helped create the evolutionary pressure to have the right type of scaffolding in the brain, or even could have been the thing that helped just grow the brain in general. Yes. That's me extrapolating a little bit from what I've read from the scientific research, but there's just a general connection between humans becoming tool builders and the drive to master more and more dexterity and some suggestion that that might have helped open up the brain for other types of cognition, like language and all these other things as well. The reality is that, of course, in AI, like, all these connections are sometimes nominal at best, but are also like genuine sources of inspiration, right? Like the way that we have created AI today, You know, digital neurons are not the exact same, of course, as neurons in our brain, but there is a definite inspiration that has led to genuine significant advancement. So anyway, I think that that's just fascinating. But then over in the case of more just like grounded in what has been the state of robotics, like over the past several decades leading up into now, and why dexterity is such a thing that stands out in general, I would say. And just the state of the state has been that very generalized dexterity, AKA not just being able to pick up something that you have a predefined model of and you're doing it over and over again, but being able to touch and move the world and manipulate it, AKA like move it into any type of configuration that, that is the goal for the robot, that has been an incredibly challenging thing. And just the empirical state of research has been that it has been the hardest thing to master across a very wide range of tasks. Whereas other things like, you know, in an amazing way, it wasn't the case a couple decades ago, but like walking robots work pretty well. At least on like pretty flat ground. And there are edge cases that, you know, general case of what's called local manipulation or— parkour type things are challenging, but walking around on flat ground has become a pretty well-developed capability. Meanwhile, putting a robot in front of a bunch of different objects and getting it to do anything with them, especially for challenging different types of objects, challenging different types of tasks, that has been a very elusive thing.

I always like to ask a few people close to the interview subject what I should ask you. And one thing that someone suggested was you should ask him, You should ask Pete how he would describe a hand. What does he think a hand is fundamentally?

There's this guy, Matt Mason, who's arguably like the godfather of robotics manipulation. He has a nice definition, which I'll butcher a little bit. I haven't read this in years, but it's somewhat in my head, which is that robotic manipulation is the process or something like that. of selectively choosing contact with the world to put the world into some desired configuration. And the operative part here is selectively making contact, right? And it is actually very carefully chosen because you can do things like not touch something, and that can be part of your goal. It's not just making contact, but it's also choosing to make contact or not. And there's this very particular word that just For those that are maybe not acquainted with robotics, like contact, you know, that's a word that in many different contexts has many different meanings.

Yes.

But contact is really everything in dexterity or what's called manipulation in research. It's about touching something and being able to, you know, have things that are not touching, put them so that they are now touching. And when they are now touching, you can move them around. And It just turns out that when you have to actually touch and move the world, that is where the physics is really hard to predict. It's an incredibly rich sort of, at its core, mathematical problem that, you know, for many decades we've tried to like solve these types of things analytically. It's very challenging. And in practice, the thing that has made the most progress, just like in language, and, you know, at this point it's just such a well-known story, but just having Models that are trained on a massive variety of data and very carefully the training recipes are refined through tons of experimentation. That's been the thing that has given us the most progress. So sorry, back to hand. You asked me about it. This is exactly the sort of rabbit hole I wanted to go down.

So that's great.

I think a hand, you could say, is anything that helps you selectively make contact or not, is what I would say is the definition. So, you know, that's pretty broad.

Yeah. Yeah. You can imagine the evolutionary pressures of our, of the hands of the future could be, could look very different.

Yeah. And even like, you know, things like feet, we can consider hands. Yes. And of course we can use our feet as hands.

How do you think about giving robots a real sense of touch? Like, is that pressure? Is it temperature? Like, what are the right ways to gather that kind of data? And what does it miss at the moment versus real human touch?

Human touch is incredible. I haven't studied human touch that much, but of course I have an intuitive sense of human touch. And I think in robotics, touch is certainly important. A simple exercise on how much though, like touch is limiting us is you can teleoperate a robot and you can have no touch feedback. And you'll realize that like, you pretty quickly could teleop a robot with no touch feedback at a very like super AI level. Yeah. Like not like superhuman level of, because you are, you can do it much better than our best AI models can today. Yes. And what that shows you is that like, yes, there are certain cases where touch would really help us a lot more, but at the same time, human brains still are quite Incredible compared to even our best models, even Gen 1.5, et cetera, at generalised dexterity manipulation. That said, I strongly believe touch is important and will be part of reaching the highest pinnacles of performance that we can with robots. Then there's actually quite a lot of different options for how robots can effectively get these types of non-visual or non-audio types of sensation of the world. I said before, I haven't studied human touch that much, but I know a little bit. Like, so one is that like, there's this concept called proprioception, which is, um, sometimes people would say different than touch, but it's, it's very relevant here in that it's like, you know, non-visual, non-audio senses of the world. And we as humans have a bit of proprioception in that like, we know how our arms are bent. We can close our eyes and we know how our arms are bent.

Yes.

And that one is, is pretty easy for robots to get. Basically every robot gets that for free. Yeah, that makes sense. That's called encoders. You know, you know your joint angles. Another type of proprioception is knowing how hard something is pushing on the muscle at your joint. And that one, a lot of robots now also have, either through what's called current sensing or torque sensing at the joints. But that one is also in most robots already today. And those 2 alone, I think, provide a lot of what you might think of as touch sensing. But then there are other things where, I believe the term is called cutaneous touch, or the ability to, you have your fingertip and you can scratch it. And you're not feeling it like down in the rest of your arm, but you can feel it right there. That is the one where there's starting to be more and more sensors there. I don't think, it's not at least a widely deployed thing to have lots of those sensors all over robots today. I think over time we might see more of those. And yeah, I think that there's a lot of cases where those types of sensors will be useful.

You grew up in the Bay Area. Were your parents in tech? Did you have models of entrepreneurship sort of around you all the time? Or was that not sort of the version of your upbringing?

Both my parents worked in technology at different points. My mom's side of the family has been in the Bay Area for a very long time. So I think this is actually like, you know, living in San Francisco, a lot of people come and go. I love San Francisco, but the reality is a lot of people that you meet in tech, you know, maybe moved here this generation or last generation. For my mom's side of the family, it traces back to like, I'm a 6th generation Bay Arian.

Oh, wow.

Gosh. And it was my triple great-grandfather moved from Lafayette, Indiana, Settled in the Bay Area, up in Petaluma. That's my mom's side of the family. My dad's side of the family, my dad moved up from LA. But yeah, my mom worked in tech and then my dad worked at a bunch of different startups. And my grandfather was the scientist. He was a chemist that, you know, back in the middle of the 20th century, one frontier area of technology was rubber. And yeah, he was a chemist and he worked at a rubber factory.

And was he sort of, I know you studied chemistry, was he a bit of an inspiration in picking your sort of first paths?

Uh, I did study chemistry. That's what I did in, in college. 100%. Yeah. I think my grandfather helped inspire that.

Were you the sort of kid that was, you know, programming, uh, you know, in your early days or, you know, building, building things in that sense, or did that come, you know, a little bit later?

There was always building stuff. Uh, I was actually just talking with my wife about this the other day that like, she was talking to my brother and we were just talking about like Pete as a 12 or 13-year-old and, uh, modifying Nerf guns to make them like, uh, you know, more powerful. Uh, we actually, me and my, one of my best friends, uh, we had a website we called a modification directory for modifying Nerf guns. Oh, I love it. The same thing with Super Soakers. Those are like the types of things that I would, would, would do at that age. Um, and then yeah, programming in college I did. you know, enough to sort of like support like scientific research that I was doing in chemistry or in physics. But then, yeah, graduate school was really when I like became a more full-blooded software engineer and everything else that you need to know to, you know, do AI and robotics.

When did robotics sort of enter the picture for you? When did you start to think, you know, this was, you know, as rubber and chemistry was to your grandfather, maybe the most exciting challenge technically of our generation?

So I had done undergraduate in chemistry and I did a 1-year master's in physics. And that everything I was working on was a fascinating field to be in. You know, before you choose to do your PhD, there is this opportunity to just think about like what really would be a field of research to like really dedicate a, you know, huge amount of years and potentially a lifetime to. There was a couple things that just felt like, you know, this was 2014, that robotics might have just a massive impact on the future. There was a little bit of a direct sort of connection in what I had been working on before, which is that I was in the lab doing physics research and I would come into the lab and I would just think about like, What is the thing I can do today that would most advance my research? And science requires a lot of different type of thinking. But I do think that like, if I just thought about the hours of the day and what I could do the most to advance my work, it was a lot of it was just like doing all of the manual labor to push through a bunch more experiments. And that like, as simple as it is, like iteration speed both in robotics and AI, but also the rest of science, especially empirically driven science, which is a very massive fraction of all of science. It felt like being able to rapidly accelerate that, like we didn't have the machines to do that. And of course, like there's been a long history of scientific automation for things like physics and chemistry, different types of experiments, but there have always been very sort of narrow regimes in which those types of instruments have worked. and a very broad capability to like have a robot just run all of your experiments. That was something that felt very far off at that time, but felt like it could happen in my lifetime. And that was one, you know, yeah, very direct reason why I thought it might be very exciting to start to work in robotics.

I love that that's the call to adventure for you, because I feel like often the call to adventure for a robotics person is, hey, you know, there's these set of jobs that are dangerous or sort of, you know, drudgery that would be great if humans didn't have to do it. And that's, I suppose, this was, you know, a form of drudgery, but it's a very high-minded version of that where you're like, imagine if we had thousands of robots conducting scientific research at this massive scale. It's a really interesting vision.

I think too, like, for me, there's like a few levels here. So one is, yes, like there are the drudgery things and actually like, You know, even people talk about this less today, but like when I was starting graduate school in robotics, like the example that everybody gave of the obvious thing that of course we want a robot to do instead of people to do was the 2011 Fukushima nuclear disaster.

Yes, right.

And the story that, you know, I would always hear, and I just think, yes, it's something everybody's just like, yes, that makes sense, is that there wasn't a robot in the world that could walk into Like the nuclear facility after the meltdown and just turn a valve. And so we had to send people in for that. So that's like, okay, that's certainly something where we should—

Everyone's on board.

We should have robots do that instead of people. And then I was just talking about the science example. You know, for me at least, it's a very inspiring example that like, yes, we could have things that we already know today, but we could just do them way faster. Or like the same type of thing already exists, but it could be way better. Or we could make things, we just iterate way faster on how we build, design things. Yes, we could run way more science experiments. Any process in the world that is basically gated by physical progress, building houses, whatever it is, we could just do that way faster, way better. That is one layer. But then there is also this third layer of just things that we just can't even imagine really yet today that will be possible. when, you know, we can actually like build structures that might take like thousands of people, but then now we can just have a whole bunch of robots do it. It would have been very hard to foresee all of the implications of the internet when the personal computer was coming around, or like, you know, before fire existed. It's just like before fire existed, it's hard to imagine all the implications of fire. I think the same thing will very much be true with the level of robotics that, that will be possible. It's just hard to even think of all the possibilities.

Yeah, there'll be all these new, new use cases, new structures that suddenly become economically and, and sort of logistically viable in this way that, that weren't before. So this is sort of the moment you, you become increasingly excited about robotics. You know, you go to, to, to Google and spend some time there. At what point did you sort of realize This should really be its own standalone company. And I actually happen to know these 2 other folks that I really want to do it with.

We just kind of talked through it in terms of like, why do we actually want general intelligence for the physical world? And I think like we just talked through, I think a few different, like, you know, very much for me, examples of like, why is this something that I think it's just really important to have like an absolutely incredible team behind? And then in terms of why did, you know, why did we think we might have a shot to make it all happen? Robotics is just amazingly interdisciplinary. And the decisions that are made around like the interplay of what hardware are you using? What data are you getting? How are you training the models? How are you closing the loop between the hardware and the data and the models? How are you, what's your plan for how you get these models to start to do something useful and then close that loop back between the hardware and the data and the models? And like all of that requires both like from what is the plan, but then also from who is the team doing that? Like it just requires an incredible set of expertise and— Yeah, between the few of us that started getting it going and then very quickly the founding team we got in place and everybody we've we've hired, you know, and joined, has joined the team since. I think we just have the best team in the world to go do this thing. So.

When you think about what Generalist is trying to do, there are obviously other companies in this space that think about, you know, this broad aspiration of creating, you know, embodied intelligence, physical intelligence. What do you think are the things that you're doing particularly differently or the different bets that you're making versus You know, other players?

I think that we have this very deep commitment to like the philosophy of everything behind what we build. There's several aspects to that, but I would start with one, which is that it's something that we've learned from machine learning has worked really well, which is that simplicity scales really well. A huge fraction of the decisions, not just on the machine learning side, but also on the hardware and how we do data and how everything comes together and how we think about deployments. We want to keep everything as simple as possible and focus on the core model training recipe and how all these things come together so that we can have the fastest possible iteration loop. So we can figure out how to get all these things going together. And then once we figure out the model intelligence recipe, that's something that we know will scale horizontally across all the other different types of things that are maybe less simple to start with. So I could talk about many different ways in which that's sort of embedded into the philosophy and strategy of the team, but that would be one place I would start with. I think also too, one thing that you've seen with the way that the best labs in the world that do large language models and other types of primarily focused in the digital world intelligence? The approaches come and go a little bit.

Approaches like the different strategies that work at a given moment in time?

The different technical approaches. I see. To a certain extent, technical approaches will come and go over the course of years. But in setting up a team which for a very long period of time will sustain all of the innovations required. Because quite frankly, what we have today is amazing. But to really achieve the real goal of have this very general intelligence for the physical world, which yes, can power all these robots to do all these different things, there's going to be a very significant amount of advancement that will happen. And I think that setting up the team and the culture to have all the right emphasis on all the right pieces, I think that's something that, that we think a ton about. And we try to be very not precious about like any of the choices that we've made so far today. We try and always just see what works the best and be constantly both willing to and even excited to completely change our mind about what we think is, is the right way to be, to be focusing our time and efforts.

This idea of simplicity and that tight iteration loop and having the right culture around it, achieving simplicity is so hard, right? I imagine there are so many temptations to convolute the problem in one way or to pursue something. When you talk about that as being sort of core to the company, what does that involve? What do you have to do to sort of like protect that simplicity and protect that culture built around it?

people who really like to get things to work has just continued to attract all of the amazing people that come and join the company. Again, it just kind of self-replicates, so we don't have to consciously think about this part too much, but it is really important to us. It's just that being really focused on what are the goals that we're actually trying to do. And if we're very focused on the goals and we're less precious about the methods that get us there, that's one thing that really helps conjure simplicity. And again, it comes from multiple different sort of expertises and disciplines that come together, right? It's, yes, it's on the machine learning side, right? So just to call it out, right? There were so many other different possible ideas of how the level of artificial intelligence that we just take for granted today could come into existence. The bitter lesson, as the essay has well canonicalized, is There are some levers that matter in an outsized way relative to others. And it's simply stated, things like compute, things like data, it's simply stated, having an incredibly talented team focus on just the things that actually matter is not easy because there's a lot of, if you know the expression nerd sniping. Of course, yeah. It's very easy to get nerd sniped along the way and think, oh, we can make this other thing and we can make this other thing. But yeah, we try to just, Focus on whatever actually really makes the most progress towards our goals. And again, that was a more sort of machine learning inspired example of the bitter lesson. But I think we have some just incredible hardware, mechanical engineers, other types of folks that have been behind building many of the robots at Boston Dynamics and other places that you know of that have changed your mind about what a robot is. possible of, what it can look like, what it's capable of rather. And we have people that, several people that have built humanoids before, et cetera. And humanoids are an awesome form factor, but for the purpose of maximal iteration speed on the core model intelligence recipe, which we think that's really been the bottleneck, is the intelligence piece rather than the hardware today. There's a lot of things on the hardware side too, where we just, we keep them as simple as possible. for the purpose of maximizing the velocity of the model intelligence research.

I think in 2025, you talked about gathering something like 10,000 hours a week of new, you know, lived robotics data. Where is that at today? If you can share directionally or specifically, depending on what you're comfortable with.

Sure, we can say directionally, it's up.

That's good to hear.

Yeah, I think, so definitely directionally, it's up. And then I'll also say though, right, I talked about the importance of things like the Bitter Lesson, but we are very passionate about scaling laws. We are very passionate about scaling. You can fundamentally shift the curve, shift the scaling law curve when you have just algorithmic advances and other things you can do across your training infrastructure, architecture, et cetera, to just have better scaling. And so yes, data is incredibly important. And it is table stakes, but there's a lot of other things too that we have found over the past few years just make you incredibly more efficient at scaling. And I think between hardware and how you do data collection and how you do data processing and how you distribute your training infrastructure and how you fold all these things together and how you think about deployment, et cetera, You can have efficiency gains that are easily 10, maybe even 100, maybe even 1,000x relative to somebody who's just like buying expensive stuff and seeing what happens with the robot.

I heard that, you know, in the early days, this is another person I talked to in preparation, to get the data sort of flowing, you have to really, you know, do the sort of classic YC thing of, you know, do things that don't scale of You know, lugging suitcases full of hard drives and digging trenches for fiber and all of these sorts of things. Can you talk me through a little bit of, yeah, what were the craziest things you had to do to get that up and running?

Uh, those are a couple of great examples. I mean, it's crazy, but like putting a bunch of hard drives in a backpack and getting on a plane, turns out the bandwidth of that is pretty hard to beat compared to the internet.

Wow. Yeah.

Especially when you don't have, um, You know, good fiber lines in, uh, remote places of, of, of the world. We've also, I think we've shared about a little bit about this publicly. We have built like entire custom server machines that just, uh, simultaneously upload data from huge numbers of memory cards. Like, um, you know, basically, I mean, it's, it's all built on Linux, but I would say something roughly the equivalent of kind of like an operating system for, for that type of, not, not fully an operating system, but like. Significant amounts of software to support effective scaling.

And as you think about the data collection piece today, some of this I'm sure is maybe not for public ears, but to the extent that you can share, what does that operation look like? What are you trying to run on a weekly, monthly basis? Is that sort of a portfolio of different things? Do you think of it almost like a portfolio manager of making sort of bets here and bets there?

I think about it more from, I'd say, like a scientific perspective rather than a portfolio management perspective, although of course there's parallels, you know, trying different experiments and seeing what happens. That is something that like we view as very core is like, how do we close the loop, not between just like collecting a lot of data and then later seeing what happens, but collecting a lot of data and then, you know, iterating very quickly on what data is giving us biggest gains in model performance. And yeah, we try a lot of different things, right? Like an hour of data for training a robotics model can be wildly different compared to another hour of data. Like an hour of data is not a fungible unit, you know?

Yes.

There's a huge amount of factors that can really massively change how valuable the data is depending on what you're trying to do. So like, Yes, the types of tasks people are doing, the types of objects that they're interacting with, but also all the different metadata that is recorded for that session and how you guide which new types of data to collect relative to what you already have. And again, like what are, what is actually driving capabilities? These are actually all like pretty challenging questions to do right in a quick iterate way. And it's something that we— we're always trying to do better on.

In very sort of, you know, order of magnitude, broad strokes, can 1 hour of data be 1,000 times more valuable than another, 10,000? Like, what are the sort of rough heuristics here?

I think it's actually literally, I would say, infinite. Nobody would try to collect an hour of data that's useless, but you can have an hour of data that's completely useless. Like, if you don't move the robot at all, and you're looking at a blank wall, it's useless. So I would value that at zero, and then anything is infinitely higher than zero.

It sounds like, given, you know, how much you've mentioned hardware already, that like vertically integrating this process matters a lot to you. What parts of it do you not care about owning?

You know, I and we as a company don't really think about it as like, what do we want to own, but rather like, just what do we think will help us make the most progress, right? And so there's things that we feel like we don't have to reinvent the wheel when it's, when there are, you know, the equivalents of wheels that already exist that are great. So turns out industrial and cobot Arms are, there's been a lot of good work that has gone into, um, you know, robot arms over the many decades of developing robot arms. We could probably make robot arms that would be a little bit better for our different specs that we prefer. But, um, that's an area where at least, you know, so far today, as demonstrated by everything that we've shown publicly, we haven't, haven't decided to build our own arms today. Hands, we have made our own hands. But we also have, as we recently announced, we've started scaling data on lots of other hands as well. We don't make camera sensors. There's a lot of different parts of hardware where there's good things that exist. And again, our philosophy is always what will maximally accelerate the model intelligence research? And we'll put stuff together when it exists. And as we think about with everything across the company, certainly in engineering and elsewhere too, we a very like no fear mentality on what we'll build, whatever we think needs to get built.

What is it that building your own hands does for that model loop? Is it just higher fidelity data gathering? Is it sort of giving you more control so that you can iterate better? Both of those things?

It's all about the iteration speed is the most important thing. There's also other things in terms of just like the capability envelope that is unlocked from just like the physics of a hand. But the most important thing for us is really what lets us go as fast as we can on the model intelligence research. So yeah, there's a lot of benefits of having a bit of full-stack control over the physical hardware itself, how the data is materialized off of that hardware and how the model controls it and all these things. There's a lot of advantages there. But again, also other people make hands that we've been scaling on as well. And There's a lot of good options.

You know, I've seen a lot of different robotics founders think about the distribution of their models in different ways. Some folks, you know, want to get out, you know, deployed across customers as quickly as possible. You know, some folks are maybe spending more time on sort of their own data gathering. Why wasn't the right strategy to say, hey, even if it's not that generalizable, if we can achieve mastery in these 2 commercial applications, we should scale that massively and just gather tons of data? Is it just not diverse enough data? Is it not sort of useful enough for that core model loop?

There's a delicate balance here of what type of data is most useful. If you have something deployed commercially and it is really, really working, you know, with, let's just put a number on it. Let's say it's got like 5 nines of reliability, 6 nines of reliability. It's really, really working. Then you already have the intelligence to solve that problem. As you're climbing nines up to 5 or 6 or however high you need to go, you'll find lots of interesting edge cases along the way. But like, if you have robots deployed that are autonomous and you can already solve the problem, then you can already solve the problem. And then there's the diversity argument too, as you just said, like if you're going to deploy robots autonomously to solve something, like just We don't have the, we haven't previously at least had the capabilities to do that on a wide variety of tasks. So then you have a very narrow distribution of data. What you really want is a mix, right? You want as diverse as possible data as you possibly can find. We know that from everywhere in machine learning. And certainly now we've started to see that at least with our own models really well supported that diverse data is very important in having a very broad base of intelligence. But then you also, to actually deploy things that are useful, you need to be able to get through these levels of performance that actually unlock commercial value. And certainly deploying is useful, but ideally, of course, we're deploying a very general intelligence that can also achieve very deep levels of mastering a particular task rather than just a very narrow intelligence that only masters like one specific task.

You know, it's tempting and perhaps in some ways helpful to analogize to sort of LLMs and the The difference between embodied intelligence and what we think of as sort of these traditional foundation models. When we think about Claude Code and OpenAI and Fable and all these models, one of the stories of these past few years has been finding sort of the killer use case in coding. And coding has sort of taken things to this new level and driven this amazing feedback loop. When it comes to physical intelligence. Do you think, do you have an intuition of what like that killer use case might be? Is there an analogy there that works where you think folding things or, uh, you know, some task will, will sort of be the equivalent sort of first driver of a lot of this demand?

I think the best analogy would be, would be dexterity. Right. And so, you know, we were talking about dexterity earlier and like kind of giving some definitions on it. Um, it's very broad, but so is coding, right? I mean, like you can, true. code many different types of things. And similarly, dexterity, you know, moving, touching and moving the world using your hands. It's very, very broad, but it is quite different than things like, you know, navigation, locomotion, et cetera. And there's also like this connection, which is part of why coding has been a a thing that has taken off as well, which is there is a little bit of an actual sort of self-improvement story. And that if we can really crack dexterity, then the robots can build the robots and the robots can start building next generation of robots, et cetera. Yeah. But I also just think intuitively, like, you know, software creates a lot of value and that's why coding is very useful. And similarly, if you're not touching and moving the world and you're a robot, What are you doing? If you don't touch the world at all, what can you do? And I would say, for all intents and purposes, if you don't touch and move the world, you're basically just moving a sensor around. Okay? And then let's keep thinking about this. So if you're moving a sensor around and you're not touching the world, well, you could use a drone and drones work pretty well today. So like, you know, there you go. There's, in a lot of cases, your solution. Yeah. But then in a lot of other cases, you could often just like tape that sensor to a wall.

And you could put one on a different wall and that would be a lot of a cheaper solution to cover that area.

So like there's applications and deploying robots to go inspect valves. And I always wonder myself, like, why don't you just set up a tripod with a camera to look at that valve all day? But inspection is a great use case in certain cases. But there's just one facetious example. Um, touching and moving the world is, is kind of everything. It's, it's a huge fraction of, of actual tasks that you can think of where it's like physically creating value.

Yeah.

So dexterity is, is really big. Um, it, yeah, I think dexterity is the best analog of coding that I can think of.

When we think about, you know, the analogies a little bit, You're sort of building the foundation model layer, and then there will be these application layers almost in the physical world where you have construction or data center cabling or whatever it might be. I think there's a question when it comes to LLM application layer of whether the foundation model eventually sort of erodes the value of those businesses, whether that's legal businesses or website generation. I think, you know, you can, you could make the argument either way. When it comes to the physical world, it strikes me that the application layer would be much more durable, uh, because you're sort of building, I don't know, these distribution relationships. You're sort of physically installing things. How do you, how do you think about that?

I think that making robots do useful things in the world is such as we were talking about before from like a team perspective, but also from like a making it happen perspective. There's just so many different things that need to come together. Right. And like, yeah. There's so much value in better hardware compared to less good hardware. And there's a ton of knowledge about workflows and industries that is required to take a robot, even a very generally capable robot, and make it useful for some particular type of application. As you said, I think that there's a lot of durable value in all the different pieces that need to come together to make robots massively useful is the models that we make, or models in general to make robots do these things. Those will be very useful, but there's all those other things that are needed too, for sure.

How do you think about how the sort of open source, open weights world might play out in your space? Do you expect it to sort of follow a similar pattern?

We talked a little bit before about like the value of different types of data and having robots actually deployed and having you help climb this, like it works kind of okay. And then it starts to work better and you're climbing and you're finding all these edge cases to find all these like things that, you know, bring you all the way up to the levels of performance, levels of nines of reliability, et cetera, that you need. That type of deployment data is, is definitely valuable for the, for the right types of applications. And the thing that is not clear about the open source route is that, I mean, certainly there should be great open source models, but then it will be interesting to see if there is actual, like, you know, people overuse this word, but it's a real thing, like, you know, data flywheels that sort of, you can actually get more value from deploying more and bring that data back in and improve the model. It's not clear just In the open source case, if there's enough ability to accrue all of the data back to the open source model creator, because there's not really an obligation to necessarily give that data back. We'll see how it plays out, but I think that is one of the, at its core, biggest questions for how open source develops. But I think that there will definitely be better and better open source models, and there will be better and better models that are not open. And I very much think of it as like a rising tides raise all boats type of situation.

How do you think about it from sort of the geopolitical perspective? Because I could imagine a version where you would have Chinese models dominate an open weights ecosystem for robotics models as well. Clearly we've already seen sort of, and the administration is concerned about protecting American companies in this space. Would you imagine a world in which American companies can only use American models? How do you sort of think about the next few years that play out here, maybe informed by what we've seen to date when we look at LLMs?

In terms of the more policy side to it, I think there's a lot to be thought through with many different types of people on how we can think more about it. There's a huge amount of advancement in the field. I think it's also, you know, calling a spade a spade, like, you know, quite early days still. So I think that there is a good amount of time to start to have these conversations that need to happen to figure out how do we think about, you know, these models from a very global sense and the open source and closed source implications of all this stuff.

You know, I think the other sort of maybe adjacent conversation is around safety and alignment, which is obviously something that these foundation model companies spend a lot of time thinking about. Because robotics is a little earlier, as you said, you know, we sort of hopefully have more time to think through that stuff. Does it start making sense to invest in, you know, researching those things now? Is it still a few years to come before, you know, you have to really look into some of those questions?

It's definitely something that we're thinking about. If you look back at the Um, the Gen 1 blog post, we have, we have a short section in there talking about alignment. It's definitely something that we're starting to think about. And, um, of course, yeah.

When I look back at your, uh, career, it seems like you've been a teacher or a coach many, many times in your life, sort of teaching English or tutoring people. When you think about, uh, training a robot and teaching a child, uh, what, what are sort of the parallels? How similar do those things end up being?

Oh, definitely. And I think there's a huge amount of parallels. And especially one of the amazing things that we've started to see as the base models get stronger and stronger is that the parallels feel like even more there. And so, you know, we talked a little bit about like some of the like Gen 1.5 few-shot, like new task learning capabilities that we've been seeing, but even for a while now, even before Gen 1.5, The most compelling first glimpses for me of really rapid learning with just a few examples that we had started seeing was that we could have a model that we had adapted to a particular task and it could start to do pretty well at the task, but it would have a certain type of failure mode that it would have a few times. Yeah. The robot arms just kind of meandered off to the side in a way that they shouldn't, or they're just stupid types of failure modes that were just very non-intelligent behavior. If you don't have a strong base model in place, you're constantly running into these. It feels like you're playing whack-a-mole with all the stupid mistakes that a model could make. But as the base models get stronger, and we've seen this for a while now, you have to only give them a very small amount of guidance on how to not do the thing that's causing it to fail, and it just won't do it again. And we saw this a year ago of we were starting to have models that would mess up a certain way, and we would show them, don't do that anymore, and it would just never do it again. And I think that that, it feels more and more like there is this kind of coaching-type mentality. that goes into helping these models be successful at all the different tasks that we care about. So that's at least one very direct connection. But if anything, I think more about the sort of different things I've done in teaching and coaching in my life, more just about the actual team that we have rather than the robots. But yeah, for the robots too, for sure.

I think sometimes it's helpful to think of some of the LLMs almost as you know, a forgetful or inconsistent PhD that works with you, right? You're like, this model is so intelligent, but every once in a while sort of goes off the rails or, you know, forgets everything I've told it. When you think about, you know, Gen 1.5, what is the analogy? Is this sort of like an extremely strong 5-year-old? Or, you know, how do you sort of analogize it?

The comparisons to children as they grow up in terms of capability level equivalent, I think is always very interesting. Working on robotics for a while, we've always had to be very humble with how capable are our robots relative to baby children. I do think even as amazing as a model of Gen 1.5, there's still things that a very young toddler is just much better than Gen 1.5 at today. Yet at the same time, as we started to get a model like even a previous generation Gen 1, We started to show that, you know, you can take a general model and you can pretty quickly adapt it to do a task hundreds of times in a row. I don't think you can get any toddler to do any task hundreds of times in a row. Yeah, that's for sure true. Yes. Right? So in that sense, that's more of in sort of like the ability to take a general model and specialize it. I think that, you know, I won't try and pick a specific age number comparison there, but There's certainly some type of comparison to be made. And then I also think that these one-shot, few-shot type learning capabilities that we're starting to see in Gemma 1.5, that feels much more of the type of behavior that you might expect toddler-level intelligence to start to be able to do. If anything too, though, I think one thing that we hadn't talked before about with Gemma 1.5 is this Novel tool use improvisation, I think is the best thing that highlights it in terms of this very physical generalization. And that, for me, actually, the one-shot and few-shot capability is incredible and probably the thing that is the most directly tied in a very clear way to development of language models. But it's also this very zero-shot physical generalization that is I would say perhaps like even more feels like we're on the right path. And to give like more of a language model analogy there, the zero-shot physical generalization that we're seeing, that's more of the equivalent of like asking a language model to just write a poem that has never existed before. There's no example that's being given. It's zero-shot, but it's also a task that requires a ton of creativity. Write a limerick about who's going to win the World Series this year or whatever it is. There's several different ways that we have started to see Gen-1.5 do that. But I think the most very crystal clear to describe is this novel tool use improvisation type thing where we can teach the model to do a task with a certain tool and then we just give it a different tool. And it knows how to use a tool with incredibly different physical properties to still accomplish the task. The example that we highlighted in the blog post and the video was, you know, using a brush to sweep an object into a bowl. And then instead you take the brush away and you give it a dustpan. We just take it so for granted, but like the physical properties of a brush sweeping something compared to taking a dustpan and lifting something up off the ground with a dustpan, like those, those are the— The levels of physical generalization that, yeah, it's just incredible to start to be able to see these models have.

I loved that part of the blog post as well, because it also corresponds to, you know, so much of what we know of intelligence psychologically of like, you know, not having sort of a flexible sense of functional fixedness of an object. You know, a brick can be a doorstop, it can be a hammer, it can be, you know, those are all, you know, in so many different ways are forms of intelligence, right? And so it's sort of incredible that you Yeah, literally see it do that.

That's a beautiful example. The brick can be a door. Yeah, a brick can be a doorstop. The physical intuition to like improvise with how things in the world can be used to accomplish whatever you want to do. I do think it also like, it all connects back to like some of what we were saying earlier of just like this, like toolmaking capability has been such a thing that has defined, uh, you know, human intelligence. We haven't been asking the robots to make tools yet, but the fact that like we can give, give them different tools and they can just figure out how to start to use them. Um, and, and again, like just glimpses of this today, but the fact that we have glimpses of this at all is, is, is the thing that is, um, yeah, really remarkable.

What do you hope, uh, your robots or, you know, the, the model is capable of empowering a year from now? What will you be excited to see it do that it can't do today?

Having worked on AI research for a while now, a year is an interesting amount of time where this is very, I'm sort of, I'm getting to answer your question, but first I just think it's interesting that in very fast-moving research in general, but especially at least in my experience and in AI research in particular, a year is an interesting timeline. Because in my opinion, it's hard to think of things that you think are likely to happen about a year from now, but are not possible of happening within the next month, right? It's a little bit easier to think like, well, what's probably going to take 5 years or 10 years to get to? Yeah. Yeah. Meanwhile, anything that I would say would be a goal for a year from now, it's possible that we actually We get there in the next couple of weeks. And I would literally say that like NovelTulio's improvisation, if we hadn't already seen that, that would have maybe been something that I would sit here and say like, that would be amazing to start to see in a pretty compelling way a year from now. Yeah, even just about a month ago, I probably would have said that.

What's your benchmark for when you think, you know, you or someone like you has a robot in their home that's like doing meaningful Uh, labor.

Well, depending on your definition, I, you know, you could say robot vacuum is already doing that today. Uh, so, but then, uh,

let's say a dexterous one with our first use case.

Yep. And then I think, you know, I, you know, this is, we'll see how far this goes, but there are these, um, you know, you see them at, um, CES, these like robot vacuums that are starting to have these little arms on them, which clearly are capable of like picking up small things off the floor. Not much else than that, but, um, That's a certain type of dexterity. It's a very primitive type of dexterity. But then a much more higher levels of dexterous intelligence robot in your home. I think that if people haven't launched them already, I think there are at small scale products like this, but like, you know, a robot that can fold your clothes. I think that people will start to have them at least in small scale. The question of whether or not they're like actually adding value or if they're just sort of— And more there for novelty in a very cool sense. I mean, I think it's awesome, but it's a separate question of whether or not they're really actually meaningfully net useful to you. I think that there will definitely start to be that. I at least think, we at Generalist think that the more industrial type spaces are more likely to take off in a bigger way faster over the next couple of years. And then The data and other types of learning that happens from like more of a takeoff in industrial spaces, then like the models and other things that power that, it's then likely to be able to help transfer those benefits over to like making the home, you know, other consumer spaces happen more as well. One of the things about home environment that is just, you know, we have to figure out more things before we just deploy tons of, uh, you know, large, heavy robots is, is, is the safety thing is, is of course, uh, you know, something to think about for sure in the home. And then in more industrial spaces, there's just other ways around that. Either robots can be, you know, just cordoned off in, in, in safe areas. There's just a lot of existing technology in more industrial spaces that make a lot of sense for that problem.

Last question, uh, and a bit of a lighter one. If you had the chance to assign a book to everyone on Earth to read and know they'd understand it, What would you want to assign to people?

I love these 2 books that basically put together in a much better way than I probably could myself. But it's sort of like, sometimes you read a book and it's like, I've already believed all these things. I just didn't realize that somebody else had written them down. Yes. And also not everything, but the core messages are things like, oh my God, I love this. I'm so glad somebody wrote it down. And then you also learn a ton from Is Bill Walsh, one of the coaches of the 49ers, amazing football coach, has this book called The Score Takes Care of Itself. And it just describes a lot of his coaching philosophy. And then similarly, and there's actually a lot of overlaps. I don't even know what sort of level of studying one another there would have been, but John Wooden, amazing college basketball coach at UCLA for a long time. They have these 2 books that basically just describe a lot of their— philosophy on teams and achieving greatness together. And I think we talked about it a little bit earlier, but I think robotics is such a challenging and interdisciplinary thing that it really requires just an absolutely incredible set of people with an incredible diverse set of expertise. I think, you know, yeah, those teachings are very useful both for robotics and, you know, a lot of different things too.

Amazing. Well, Pete, thank you so much. Such a pleasure.

Thanks for having me, Mario.

That's it. Thank you for listening to this episode of The Generalist Podcast. Please subscribe on Apple Podcasts, Spotify, or your preferred podcast app. Ratings and reviews help others discover these discussions, So if you enjoyed the conversation, I'd be grateful if you could take a moment to leave one. For all past episodes and more, visit us at thegeneralist.substack.com. See you next time as we continue to explore the future.