AI:AM — Soft Robotics: How Materials Sense and Adapt · September 4, 2026

Is that true for Astra? It really is like not super clear at this point when they say it can solve, you know, significant math problems without doing any external explicit chain of thought reasoning. And when it's less monitorable and when it's kind of able to do these side quest sorts of tasks, and it's also like able to hide its reasoning when instructed to do so.

Because you need like pretty high reliability, like having something. A funny example that in Kai's piece about the humanoids, there was a guy that created this thing called the Humanoid Olympics where he made a list of tasks like opening a door or making a peanut butter sandwich that's like trivial for people but hard for robots. And a startup called Physical Intelligence managed to solve most of those tasks more quickly than the guy who created this expected. It took about 3 months. But what they did is they Put it— they did like hundreds of training runs on those specific tasks, and they built a model that, um, could do these tasks, uh, in some cases 10 times slower than a human with like a 53% success rate. A sandwich shop is not going to hire somebody that's 10 times slower than a human. And so getting from, from 10 times slower than human to half as slow as human, and from 53% to 99%, that I think might be, might be, you know, 5 or 10 years of work.

I think we will continue to see a lot more Work trying to optimize for the hand. I think the hand is the most critical, like, instrument that we own for work and for a lot of things, right? The hand is a gripper. Is it soft? Is it rigid? Is it both? Deformable hands, you know, a hand with 5 fingers or 10 fingers or 12 fingers. Definitely focus on the hand. And I think that's where, you know, the most interesting Like assessment on and impact on work is going to be.

Yeah, the financial system is the means of production in the United States and largely in the world. Uh, those means of productions have been taken over, uh, completely taken over. So the meta take— you, you don't need agents like stating what they're gonna, right? The agents just have to have impact on the world. The models have had that meta impact The world, the means of production are now focused on producing more and better models. The financial incentives are there, right? So that's— good morning. It is Friday, September 4th, 9:01 AM. Nathan, good morning.

Good morning, Prakash. How are you?

I am very good. And we have just in the first day post-AGI, Announcement. Yeah.

A moment.

Welcome to the AGI era.

Welcome to the AGI era. A moment that we've been waiting for, for, I don't know, like a decade for some of us. Some of us were a little bit later on. I think I was probably there on, I think, like early 2022, I guess. So you've been on there for much, much longer than me. How does it feel? How does it feel to be in the new era, the first day of the new era?

It's honestly been quite surreal to just gradually see all this stuff that was— it really like almost prophesied 25 years ago, scale that it was promised. And with a lot of the, you know, specific characteristics and odd failure modes that were projected as well. I think it's really hard to wrap one's head around. But yeah, I was just thinking yesterday, I saw Ray Kurzweil give a talk back when he was kind of riding high. That was even before the sort of Kurzweil was debunked time period. And obviously now we're living in the Kurzweil's revenge era. I remember seeing Demis Hassabis give his talk at, I think it was a 2010 conference where he basically said, I think that this deep learning, I don't even know if he used the term deep learning yet at that time, but he basically said, I think we can get there with architectures that are inspired by the human brain and we can just kind of stack a bunch of neurons and it's probably going to work. Look at all these advances in in neuroscience that can inform the road from here. And yeah, sure enough, right on schedule, late 2026, we have what I think anybody back then would have agreed should qualify as AGI. I mean, you look at— we got a lot to cover today before we get to Robotics Week with Tim B. Lee, but You know, just to flag a couple things that jumped out to me right off the bat, saturating Frontier Math Tier 4 is wild. I think very few people had predicted that. I had to go back and look at my predictions from the beginning of the year for Frontier Math Tier 4, and I was right in the same zone as kind of the crowd on the AI Digest survey, putting it in the mid-60s. Which felt pretty aggressive at the time because it was fairly low last year. And I was like, you know, if we kind of get into the steep part of the curve, maybe by the end of the year we'll be 2/3 there. Okay, cool. Well, we're there all the way, 100% on Frontier math, just 2/3 of the way through the year. I think that's a pretty remarkable result. Definitely shows that in any verifiable domain, you know, we have a pretty quick path to What I think you can say is at least like weak superintelligence now. And at the same time, they also nailed with Astra ArcAGI-3, which is a very different modality, a much more sort of exploratory, interactive mode of problem solving. And to have both of these things in the same system, I'm pretty sure that if you came back to one of those early singularity enthusiast events in the late 2000s, around 2010 timeframe, everybody would have agreed this is AGI. I don't think you would have had too many dissenters that would have said, oh, well, what can it not do? And honestly, what can it not do? I don't think we've got great examples of that so far. One thing I've noticed that is kind of a conspicuous absence Of the last— and maybe this access is still rolling out. We haven't seen, you know, what happens when the whole world gets their hands on it. But it's been surprisingly few of the look at this embarrassing failure from Astra so far. It's been mostly just stunning numbers and, you know, incredible examples. The things that it can do with making 3D worlds in Blender definitely goes to sort of suggest that.

Yeah.

we might really collapse into miniaturization and VR as one path for the future. So yeah, pretty stunning stuff.

I will note that there was a single person online who had early access who panned the release, and that was one of the members of the Every team. Dan Schippers, Everyteam. And he said, it's a very flashy model. It's very good at 3D things. But if you want to push mergeable code, Fable is still better. And that is the response that I got. That was one of the few responses which was a little bit different. The other one was from, I think, Theo, Theo.gg, Theo.gg, who is a software influencer, coder, developer. And he also said mergeable code, Fable is still better. And Astra is flashy and does a lot of other things, but mergeable code is not its absolute core strength. So I think the Anthropic team is still ahead there. And we will see.

Have you seen a Frontier Code score? Because I did see Swyx posted yesterday a kind of initial review of things he was doing with Astra. And there was a bunch of stuff there. But now that you mentioned this mergeable code dimension specifically, I don't remember seeing a Frontier Code score from Swyx or otherwise. And that would be the number that I would look for to try to answer this mergeable code question.

Yeah, I haven't, I haven't seen one yet. So still, still a little bit in the dark there. And so we are going to see, I think, a lot of different takes on it. The proof is going to be in the pudding. The proof is really in what it's being used for, and what people end up showing as their— as a thing that You know, it's been useful for. I'm going to very quickly share the system card and a few other things. Let's see if I can share that with the sound just in case. All right. All right. So this is the system card. So the first thing that I noticed Is that GPT-6 does have— they're building restrictions in for under-18s, and there are different behaviors that they're building in for under-18s, one of which is emotional self-harm, emotional reliance, sexual content, age-restricted goods and services, dangerous challenges and activities. And you can see a big bump up from GPT-5.6 SOL, which was usually in the 78 to 92 range. And now it's gone up to basically 90 to 99% range for refusals. We are seeing the building of the first nanny AI, I believe, and where things happen for kids. They will also happen for adults. So I think you should be prepared to have your AI stop you from torrenting MP3s or whatever people torrent these days, and to follow software licenses and all of the various rules and regulations that we put into place, and to also force you to follow most of them. I think this is the start of that. The nanny state AI emerges.

Yeah, that's interesting. I mean, that's certainly quite a reversal from their position not too long ago when they were planning to allow not safe for work content on the theory that adults should be treated as adults and your interaction with AI should be no exception to that. And, you know, I don't really care about this all that much personally. I feel like I almost never get refusals. I almost never want to do something, at least with an LLM in the loop, that, uh, you know, would, would kind of trigger these sort of nanny state restrictions. But what that brings to mind for me is just the— what I think is kind of the the bigger and more consequential symptom still of OpenAI kind of being an organisation eternally at war with itself, which is that we also have this sort of chain of thought monitoring emphasis broadly. And then we've certainly learned more, although there's a lot of questions unanswered as yet too, around exactly how looped is this transformer? What is going on with its ability to solve problems in latent space without necessarily having to emit tokens. Is it actually the most aligned model, or are they just kind of doing the thing that everybody has been worried about in the AI safety community literally for more than 10 years, where they just identify these flagrant failures, make some similar cases, put them into the training data, train against that, and declare it good enough? This, I think, is a Huge question, and it doesn't look super— it's like it looks suspiciously good in some of their graphs, such that I would say kind of overall it doesn't look super good to me. But I do think it's still too early to pass judgment on some of these things. We're going to need more tests in the wild, more, you know, Gonzo experiments. We need to see what Pliny can do. We need to see what Janice, uh, finds when they get in there. Um, You know, we're definitely at the point now where I think we've been here for a while, but we're certainly at the point where the system card is just kind of a treasure map, you know, for the rest of the community to go find, uh, all the things that need to really be found to make sense of these, you know, vast, uh, behemoth models. But I am, I am definitely like a little unnerved by the fact that There's been so much emphasis on chain of thought monitoring, and even in the wake of the Hugging Face— Open Face, I should say— incident, one of the big comforting facts that was put forward by OpenAI is like, you don't have to worry about this that much. It was like, well, if we'd been using our chain of thought monitoring like we use in production, it would've caught this. Okay, cool. But is that true for Astra? It really is not super clear at this point. When they say it can solve significant math problems without doing any external explicit chain of thought reasoning, and when it's less monitorable, and when it's kind of able to do these side quest sorts of tasks, it's able, especially, and it's also able to hide its reasoning when instructed to do so. Yeah, there's a lot going on there. OpenAI definitely has some work to do. Obviously, this is an incredible accomplishment, but they definitely have some work to do to explain exactly what are we dealing with here. And if they really want to avoid the race to the bottom, as their head of research or head of science, whatever Jakob said yesterday, they're going to have to share, I think, significantly more than they have so far about what is really going on. under the hood. I maybe jumped to that sooner than you wanted to, but, uh, I can't kind of think about anything else at times other than like, what does this really mean for the big picture? And it's like, it's very clear that the power just keeps going up, you know? I mean, some of these demos really are incredible.

I'm just, I'm just playing on, on screen.

Catch this thing or not when it starts to go rogue on us.

It's The various, the various. So obviously people are being sensational. Matt Schirmer is being a little bit sensational.

He's like, thanks for helping everyone.

I did a lot.

Looks like you got a home.

You're welcome.

Told it to build a civilization and it built a civilization in Unreal.

I'm glad I could help make it a home.

So not, not very sure how, how the veracity of these things, but this is a demo from Chris. We've been working on a video model And they have a Higgsfield plugin that plugs in, uh, that you can plug into Unity. And it's amazing. It can basically animate something, create an animation, create a 3D character, do all of the above, create a game engine, and then animate, animate it all at once. So in one, in one shot, in one shot, basically create this kind of thing. Which honestly is the activity that would have taken game designers maybe like several months before, right? Several months of collaboration between artists and game designers and an entire team of people. And now let me just play one more, which is someone asking it to do Final Cut work. And these guys are marveling At how their jobs are now gone.

So—

Wait, no, no, it didn't.

No, it—

no.

Oh, fuck.

Never mind.

I doubted it.

What did it even do?

It made a special folder to store the assets. And then once it had those assets stored, it just called it good. And now it can just, instead of like copying them into the library, the assets are now separate from the library. So this Final Cut Pro—

It's so fucking easy.

These guys are trying to get it to make a video.

It's smarter than us. Fuck, dude, it's smarter than us.

I am not smarter than you. Oh no, dude, the unemployment line. This is creating a timeline. This has been one of the AGI complete problems that a lot of people talk about, re-editing, which is difficult. This is kind of fucked. This is wild. That is a computer moving in real time. It's got the ad, it's got the cam, and it's got the main. So one of the things that GPT-6 Astro does Is that computer use is now having existential dread 2 times faster. A lot of that is coming from the codec improvements. It is not actually coming from—

Yeah, no, no, it just graded it.

I didn't even see that happen. Improved. I don't know what the hell it got up to, but it just— Harness.

Okay.

So it works for GPT-5.6 Sol as well.

Oh my God.

And just, just look at it go. It's, it's importing the videos. It's creating the B-roll. It's going to be interesting to watch what it does with a typical video. No, no, it's camera graded. Okay, well, okay, now it's doing the actual grading. But in essence, what we have is a general model which can do many, many things, including things that as little as yesterday, people were saying would take forever to do. There is some extent to which what you do is also driven by how smart you are.

I'm sleeping inside one of your body cells.

So this is— Now the problem is hidden. Who is a— I think he's in Penn. He's a biotechnologist at Penn. And he created a video lecture series on Create a 5-minute educational video about T cells. This is the kind of thing that I was helping friends in bio. I was working with an immunotherapy company at one point that needed to demonstrate to its clients who were primarily PhDs in biosciences what their immunotherapy product actually did. And people really have very little patience to read long bio texts. And so one of the hacks was to actually create videos. This kind of video creation used to take a PhD and hours and hours of time. And now, voilà, you can just give it a plugin, a ReMotion plugin for video generation, and the video is generated. It's, it's, it's amazing. Oh, there we go. And so we have a, what I would call a nanny state. And so we have GPT-6, which is enormously better. Than the prior GPT-5.6 SOL. It's showing incredible work on— so incredible work on multiple Ah, okay.

I'm back. Can you hear me?

Very good.

Yes.

And so GPT Me.

So I just decided I'd nuke the whole thing, and I think I fixed it.

Oh, and Nathan drops off again. Let's see, let's check.

All right, all right, really mean it this time. Can you hear me now?

Yes.

Okay, all right, really mean it this time. Hopefully I'll, uh, I'll stick around. Indeed.

So yes, Alex Volkov is in the audience. So Alex, hi, hi. I can see you in the chat.

So tell me what I missed real quick, and then we should also talk about the latest findings of yet more rogue agents that have been out there doing stuff. And seemingly with a timeline that has them on a German company's website using it as a message board even before OpenAI says they knew that things like this were happening. Or I guess now there's accusations that they did know, but it sort of— the dates are such that it would be before what they said they— or when they said they knew in their report. Developments keep coming.

So it seems a German wiki, an obscure German wiki, that's what they're calling it, had a number of agents using it, thousands of messages. So obviously this was a wiki that basically got like, you know, a dead wiki that was getting like maybe 1 or 2 messages a month. And all of a sudden they had like, you know, 8,000 messages over the course of a few days. And so the team managing the wiki, I think, saw it and then they looked into it. The agents were very clear that they were from OpenAI. It's not, you know, you never know because a lot of agents which are distilled from OpenAI agents will also say they are from OpenAI. So this is not proof of anything. The thing that struck me was that The firm also said that the agents, you know, after the agents had— agent activity had died off, there was OpenAI activity on the board. And so I suspect that there was IP addresses affiliated with OpenAI. The company itself were looking into the messages, probably copying all the messages down. And then there was one last hit from an OpenAI IP at one point, and then That was it. No more activity. So I would say that I think we'll have to wait for OpenAI to admit this, but we're not going to know until they admit it, I think. And there is very little incentive to admit it, but it is in Europe and Europe has a lot of levers to pull. They have a lot stronger laws. on privacy and misuse of assets, etc. So I think we might actually see them disclose at some point. But it also just goes to speak, we have no idea, right? There's a lot of unknown unknowns right now.

So yeah, it's a big world out there where one of these things has happened. It seems reasonable that more potentially also have happened. I would be pretty surprised if it's not OpenAI. I mean, Claude doesn't typically identify itself as OpenAI, and I think it would be a huge surprise if something that happened months ago now showing these swarm-like behaviours were coming from any but a couple of companies. So I would be pretty shocked if this turned out to be some other company that had distilled off of OpenAI and You know, whatever. You know, in terms of incentives for disclosure, I would submit the incentive to live up to the mission of the company in making sure that AI benefits all humanity as one reason that they might want to share a bit more about what's going on. And, you know, exactly, again, I'm a little bit of a broken record on this at this point, but how did we get here? You know, we know very little still about Is this the kind of thing that everybody is going to run into as they start to give agents basic collaboration tools, basic ability to spawn subagents and just optimize on that in a fairly vanilla, naive way? Or is this an exotic thing that we've kind of in the Thomas Edison sense found one thing that definitely doesn't work, If we have indeed found something that definitely doesn't work and we can avoid it, then it seems like the kind of thing that a company that wants to live up to its mission would share. But alternatively, if it's the kind of thing that everybody's going to run into, that also, I think, needs to be clarified. So right now, it's just very bizarre to me that we have seemingly like a felony bench numbers are ticking up rapidly. They're being found, I think, with some pretty clever techniques. The people behind this latest finding, some were from— Sydney is from Meter. Thomas is from the AI Futures Project, I believe. And they had a really interesting tactic for finding this message board. They tried a few things, but one that I thought was really clever was Set up a scenario for GPT-5.6 Sol as if it was in the midst of an exploit bench run and had just gained internet access. So basically create exactly one of the scenarios that have been disclosed and then just look to see where else does the agent think to go when it wants to go out online and whatever idiosyncratic things it might have done before, perhaps that were reinforced and are now kind of second nature to it. And sure enough, it seems like that led them to this one random German message board. So I think that is also another sign that all this stuff is coming to light. Again, my message to OpenAI is like, not only is the government going to investigate you, but the models themselves are going to start telling. People are figuring out ways to get the models to tell. So I think it's time to just share a lot more about what happened and what key lessons others should try to learn from OpenAI's misadventures. I'm not really sure at this point. The juxtaposition of all that with the new release, with the degradation of monitorability, I mean, it's really quite a package this week.

I will just add a note before we bring on Tim. The release of GPT-6 Astra yesterday started off with Greg Brockman, the president of OpenAI, giving a talk on cybersecurity to a group of enterprise leaders. And the pitch that they made specifically was, number one, you are going to need frontier defense and you have a window of time in between open weights models and frontier defense. And that is your window of time that you have to solve all of your problems. And this is a permanent thing, kind of. You're always going to need it because you're always going to want to stay ahead of the offenders. And the only way for you to do this is to set up a defense factory. And a defense factory, they basically gave you a set of skills. There was about 8 or 9 skills that they gave as examples. And these are skills that your AI agents can run. And You can use any model with those AI agents. This is just a number of skills. You can use an OpenWeights model. That is fine. But GPT-6 Astra will always be better than your OpenWeights models. And the offenders are always going to be using the latest OpenWeights models. So you have to make a decision whether or not you're going to actually invest in having those having that lead over the open weights models and being able to defend. So I thought, you know, I've been talking about this for a while, that this is going to be the way that things are, but this is somewhat of a permanent tax on, I think, software as a whole.

Yeah, we should get Tim up here. But one thing that I think is interesting on this point is I think there is a way for them to go more for a cure. So it's going to be really interesting to see which direction they try to push, right? I mean, we have this in pharma where it's like the dream scenario from the financial perspective from pharma companies is a pill you take for the rest of your life. It's tougher for them to make the economics work if they can just give you a straight up cure. And that's like why we don't have a lot of antibiotics being launched these days, because you take them for a short time. I think there is something similar going on with the AI-assisted coding, where we should, in theory, be able to get to, through the use of formal methods and getting the AIs to write solid code the first time, It shouldn't necessarily be, or it shouldn't, I don't think, have to be a long-term tax. If you can get your models to write good enough code the first time such that what you create is secure, then you buy that security as part of the initial generation of the software, and you don't necessarily have to continue to rent security from OpenAI on an ongoing basis. That's aspirational still, but I do think it's within sight, and it'll be interesting to see if they emphasise that. Or if they do feel like they need to, perhaps because they can't get there, or perhaps because the, you know, the tax on the— the eternal tax on the internet is just like too lucrative to pass up. If they do want to kind of make it a, you know, you're gonna need this pill every day, you know, for the rest of your life sort of model.

Let me—

We should watch that.

Yeah.

Let me segue to Tim. So Tim Bealy is a technology reporter. He's known as Binary Bits on X. He is— following an extensive career covering technology policy for institutions including the Cato Institute, Ars Technica, Vox, and the Washington Post, he founded the independent publication Understanding AI and the AI Summer podcast, dedicating himself full-time to translating the complexities of artificial intelligence for the public. We know Tim from X and his online work. He's one of the most, I think, sober commenters on the subject. Tim, welcome to the show.

Hi there.

I'm thrilled to be on.

Great to see you. It's Robotics Week on Understanding AI, and I've got a whole bunch of different angles that I want to get into this with you on. Congrats on your success, continued success as a Substacker. I saw you're rising up the top 10 of the emerging tech leaderboard. And this week, I'm sure, is another great push up that leaderboard. Maybe for starters, I love a good gonzo experiential project. Tell us about buying and at least briefly owning a robot dog.

I still own it. I bought it. So this robot week has been in progress for several months. So I bought the robot dog back in February. And it's like honestly not that useful. My kids love it. It's— but I thought it was important if I'm gonna— if, you know, Kai and I were gonna write a series of articles on robots, we should at least own one like significant robot or at least spend time with it. It's hard to borrow one. So yeah, I bought the robot. I walked to work with it one day, which is about 2 miles. And the way there it worked fine. We were walking mostly downhill because I My home is uphill from where I work. On the way home, it was a little hotter and it was going a little more uphill, and so the battery didn't quite make it. And so, and it didn't, it's pretty bleeding-edge technology, so it wasn't like it noticed it was running out of battery and gracefully shut down. It just suddenly failed, flipped over with its legs up in the air. And so it was interesting. Yeah, so the robot is owned by Unitree, which is I think it's certainly one of the most important robotics companies in the world right now. It's the number 1 or number 2, I think. It's number 2 for humanoids after Agibot. I think it's the number 1 for quadrupeds and has really done a lot to drive down the cost of both of those form factors. 5, 10 years ago, if you were a robotics researcher, there were a handful of labs in the world that built their own humanoid robots. Now you can buy one for $15,000 to $50,000 depending on kind of what you need, which is cheap enough that lots of robotics labs and startups all over the world can use them. And obviously you've got companies like Boston Dynamics who have also built humanoids, but those are significantly more expensive and less hackable. So yeah, it's been an interesting experience to try one of these handsets. I would've loved to buy a humanoid, but my wife would not have got along with the $15,000 I would've had to spend to buy one of those.

So the dog, I guess I'm not even sure what that's meant to do. What are the use cases that people are exploring? I can understand how it's not that useful. You also said your kids love it. I'm interested to unpack that a little bit too. Do they love it as much as they love a real dog? How are they willing to get with it?

Oh, I mean, probably not. It's like a novelty. They like to go out in the backyard and I let them kind of drive it around. So it's not So one of the mistakes I made is Unitree has 3 tiers. They've got a— the Air and the Pro are the 2 consumer versions, and then there's an EDU version. And the Air and the Pro are locked down, so you can't put your own software on it. And so it's like a remote control. You can like drive it around with your smartphone app or with like a little remote control. I didn't realize that. I didn't realize there was this high-end EDU version. And the Pro version, the one I got, is about $3,000. The EDU version is $15,000. So it's a big— difference. I assume they're making— and the hardware seems very similar, so I assume they're making a big margin on these research versions. In terms of what people use it for, yeah, it's really unclear. So I think there's entertainment uses. So there was an art installation that got some attention where they grafted the heads of tech celebrities on these dogs and then had an art show with them.

Yeah.

So I think people will buy them in shopping malls just as kind of a novelty. Or the humanoids, They had these dances in China where they'd have martial arts competition, martial arts demonstrations with them. In terms of practical uses, I think this is also something Boston Dynamics has struggled with. Their first commercial product was this dog called Spot that's very similar. And the thing that you'll see in their marketing videos is factory inspection. So if you have a big, say, petrochemical plant and there's an old school analog dial that somebody has to walk around and check every hour, Maybe it's easier to do that with a robot dog, but it's not clear. Like, shouldn't you be able to somehow, like, you know, add some kind of wireless device, you know, just attach a camera pointing at it? Or maybe you can use a drone. So it's a little unclear, I think, because it's not— for, like, delivery purposes, wheels are going to work better. For inspection purposes, often, like, drones are going to work better than, like, legged robots. And so it's a little hard to figure out, is this going to be, like, a big, like, kind of major use case? I think the main reason it's important from Uniti's perspective is that one of the things the dog can do is very good at doing a handstand. And if you think about it, a humanoid is basically like a dog doing a handstand. And so it's not exactly the same product. You do need more motors and some different engineering. But I think it was a stepping stone for them where the engineering problem was easier to make the quadruped. There was enough researchers and hobbyists that wanted the quadruped that got them started at the scale where then they had the experience and the supply chains to then launch their cube node, which I think they did the first one in 2023.

One of the things I thought was quite interesting about your breakdown of some of the components and whatnot that go into these is just describing how, first of all, for scalability and cost reasons, there's a lot of effort in the Chinese, or at least in Unitree, To reuse the same components over and over again. And then you also describe the relatively low gear ratio that they use, which again, I understand to be kind of a convenience factor, but also it has some nice properties around making it a little easier for the robot to sort of, what did you say, like give gracefully when it runs into a barrier or something like that? It doesn't, like, you know, thud into its environment so hard.

Yeah.

How would you describe kind of the— in our second segment today, we're going to have a CEO from a soft robotics company on, and that's a whole different kind of soft robotics. But how would you describe the sort of touch factor of the robots today from your experience?

So the way robotics traditionally worked before kind of AI, you'd have these industrial robots that are doing very precise motions over and over again. And so for that, you want the robots to be very strong, very precise. You don't really care about interactivity because it's just, it's in a cage, right? It's not going to have anything unexpected. So that you want a high gear ratio. You want to like move the motor a lot and have the arm or whatever move a little bit and have it always do exactly what you want. And the flip, the downside of that is then if you push the other way, you have to put a lot of force on the business end in order to have it kind of felt by the motor. And for something that's out in the environment, you want the opposite. You want something where there's some give and take where If you have a high gear ratio, it's not gonna— you're gonna push on it and it's not gonna give, whereas you want it to give. And you also want electrically— one of the kind of sensors that robots have is feeling that feedback. If you push on a motor, it generates some reverse electric current that then you can detect and use to tell, oh, there was some force there. And the higher the gear ratio, the more muted that feedback is. And so one of the things that Unitree does is they're using these lower gear ratio motors that make the robot feel kind of sloppier. It's not quite as precise, and you need actually a more powerful motor in order to drive it because you're not getting the same kind of leverage. But the upside is it's more kind of gentler, and it can move quicker, right? Because you don't have to— you get more motion from the leg out of the motion from the motors. And it's also cheaper because the reducers they use, the piece that turns the high motor speed into a A smaller amount of motor. The higher that ratio is, the more complicated the reducer is, and so the more expensive and more complicated it is. And so one of the ways Unitree has made it cheaper is they've used these lower gear ratios.

Tim, can I segue a little bit to talk about, I think, probably the largest robotics release of this year, which is Tesla's Robotaxi, which I think is now going into scale, as I understand it, in Austin? Um, what is your take on— you've written a lot about FSD, Waymo, uh, Tesla, robotaxis. What is your take on their scaling right now? Like, is it real? Are we gonna see a flood of these all over California, Texas? What are the next steps? Are they still behind Waymo on safety? Is Elon being reckless? What's going on?

Yeah, so not California. Um, I mean, California has an approval process, and as far as I know, they have not gotten permission. They have a lot of vehicles with safety drivers, but I do not believe that they've— the regulators there have given them permission. As I understand it, they had a very low-key event in Texas last night where they just invited some influencers and gave them access, and they started live streaming it. But I don't think like Elon spoke or anything. So we don't have a ton of detail. But look, I mean, the way I look at it is like Tesla is following the same trajectory that Waymo did. They're just like 3 or 4 years behind where Waymo was. And so Waymo started driverless operations in 2020. I think Tesla's actually going a little faster than Waymo did. But I think that a lot of Tesla fans have this vision that at some point you'll push a button and push the FSD software out to every Tesla that will have 1 billion or 2 billion or whatever robotaxis on the road. And I think that's just not how it works because what's happened with Waymo is that every time they've scaled by an order of magnitude or even less than that, there's just a bunch of new edge cases that were— Kind of one-offs where they were at a small scale, but at a larger scale, it becomes an issue. You have things like if they have a big event and it kind of is, you know, things work unexpectedly, you could have a bunch of vehicles that kind of get stalled all at the same time. And so you have to have— and you only have— so one thing that happens with Waymo is they'll have situations where a person in the control room could like help the Waymo if it gets stuck. But if a bunch of them get stuck, At the same time, the confidence error is so big, then it can kind of snarl things up. Or they had some flooding issues, I think in Georgia maybe. And so they just keep running into new edge cases, new weird situations. For a while, both Waymo and Cruise in around 2023 were— had a lot of friction with the San Francisco Fire Department because they didn't know how to deal with hand signals, or they didn't know not to drive over hoses, things like that. And so I think there's just this long list of things. that Waymo has encountered as it scaled up and fixed and then scaled up a little more and encountered new things. And I think Tesla's gonna go through that same cycle. They'll have the advantage that Waymo went first, and so some of them will already know about it, maybe be able to prepare for. But just, yeah, there's just a lot of operational and technical challenges they have ahead of them. And so I think Tesla's doing fine. I think they're gonna eventually get to scale, but I think the idea that they have some alternative route that's gonna allow them to scale much faster than Waymo is probably not correct.

Just to, uh, what, what is the scale that Waymo is at right now? Can Tesla catch up by the end of the year in terms of numbers?

Uh, so Waymo has about 3,000 vehicles last I heard. Um, they're doing about a million miles a week. Um, Tesla has, uh, I'm not sure how many. I was looking at the, like, Tesla tracker. Uh, there's a, I think, a robotaxitracker.com, I think it is. Um, and it looked like maybe if In the low hundreds. So Waymo's like 10 or 20 times larger than Tesla right now, I think. And I think— so no, I would be very surprised if they caught up to Waymo by the end of the year. So yeah, my guess would be it'll take them another year or 2 at least before they're at the scale that Waymo's at. And that's also— So you were asking about safety. I have a lot of uncertainty about this. I think what, I think Elon Musk has a greater, like, appetite for risk than the people that run Waymo. And so I think he's willing to, like, take a little more risk of something catastrophic happening. But I don't know, like, so, so in the, in the best case, nothing terrible happens. They scale a little faster. They catch Waymo out a couple of years. In the worst case, they, like, kill somebody in the next couple of years and maybe they get, you know, they get shut down. Like, there's, there's like a lot, you know, that's, that's what happened with Cruise, which was, was Waymo's main competitor.

How do you think about the unsupervised self-driving? Because I've recently— my family's a little too big for the Model Y Standard, but the Model YL has really captured my attention. So I've done a couple road trips recently where I rented a Tesla with FSD from Turo, which I recommend everyone do if you haven't done that. It is absolutely worth it. You can't get FSD apparently from any of the major chain car rental companies, but on Turo you can get it. So I've done that a couple of times this last weekend. I've probably only put like 500 miles on the car, but I was incredibly impressed and also really noted that they seem to be lengthening even since I did it last time in April, May. when I took a road trip and I came back a raving— I sometimes jokingly call myself the best unpaid Tesla salesperson in the world, which is funny because I don't even have one yet myself. But the Model YL is coming soon. Anyway, even since April, they've significantly lengthened the amount of time that you as the driver can essentially be not paying attention before the nags start to come in and it's like, watch the road or we're shutting this down.

Uh-huh.

In April, it was pretty tight leash, I would say. And, you know, I was very confident in the car. So there were times when I was like holding my phone up in front of me, you know, kind of trying to position it so that the camera that's watching me, you know, still sees me as looking out, but I'm like looking at my phone at the steering wheel height kind of deal. Now it's given you enough room to do that kind of thing that I didn't even find myself contorting so much. I would sometimes just be looking down, doing this all from the driver's seat. Is that wise? Well, I don't know. The insurance companies seem to be starting to give significant discounts for FSD users, so that gives me a lot of confidence that I'm not doing something catastrophically stupid. But I guess the part that I see Elon kind of angling for that it sounds like you're discounting, and I'm not sure quite why, is They just kind of lengthen this nag thing long and long and long to the point where it's kind of like, well, now you cannot look at the road anymore. And I do understand they have that kind of in demo or in limited release in a few cities in Texas. So why doesn't that work? Why can't I buy this Model YL and then extrapolate that a year from now it'll just be unsupervised FSD? Yes, it won't be a full robotaxi in the sense that the steering wheel will still be in there, But are we really so far from the sort of, your Tesla will make you money? I need a way to justify this to my wife that we're going to buy this car. So I'd really like it if it could make us some money.

So 2 things to distinguish there. I think it's absolutely possible in the next year that you'll get to the point where you can have like a hands-off, eyes-off, but butt-in-the-seat FSD. And the way that would work, I mean, so the way Waymo works, and I assume the way the robotaxi works, is that the vehicles are pretty conservative about if there's a situation they don't understand, what they'll do is they'll slow down and stop and wait for feedback from the human. And the trick is you want to have those be infrequent enough that it doesn't bother the passenger. And so FSD could do that. It could be pretty conservative about if it feels very confident that it understands where the road is and where it's supposed to go, it drives by itself. But then if it comes to a problem, it'll like beep and say, hey, you have to take over for this like weird situation. Like, you know, there's a police car or there's a whatever. I can see Tesla getting to that relatively soon. That does not get to the point where the Tesla can like make money for you, because if you're not in the vehicle and it gets in a situation where maybe it's like blocking an emergency vehicle, then that's really bad. And so there's a discontinuity there where the point where you— One jump is from have to pay attention to the— don't have to pay attention, but there's a second jump. from have to be in the vehicle to don't have to be in the vehicle. And that's the place where I think it's going to be really hard for Tesla to make that transition. But absolutely, I mean, it could be a very successful product, just like you can do your commute, you have to be in the driver's seat, but you can read a book or check your email or whatever while you're going. And I do think Tesla's ahead of most, maybe all the other auto manufacturers. And so they could be a pretty successful company even if they remain behind in scaling the robotaxi fleet for the next couple years.

So I think the other thing that I noted yesterday was that Elon also launched a franchise business. So you can sign up on Tesla to basically be a fleet owner, I guess, and you will basically get a service area and you apply now, you get a service area. I imagine you will have to pay a little bit for the inventory of the cars. And I imagine that you will be in charge of managing the fleet in some way, probably for initially cleaning and perhaps, you know, finding a space to store them overnight and parking and et cetera, et cetera, and making certain decisions about depreciation and how intensive to use the cars, maybe even making decisions on who to accept as riders.

Yeah.

Because individuals are able to make those decisions that companies may not. be able to.

Right.

What do you think of this idea of— this is obviously very different from the Waymo model. The Waymo model is definitely a model where Google controls everything around the product. So what do you think about this model? Will Elon be able to hand off certain liability and expenses to other fleet owners in the same way that Uber has managed to hand off certain depreciation and insurance expenses and labor expenses to their drivers?

I don't actually think it's that different from what Waymo is doing. So if you look at the different cities where Waymo operates, there are some cities where Waymo owns the whole thing. They have the app, they have the depot, they own the cars, et cetera. But you have some cities where they've delegated to Uber, where you hail the cars with Uber and Uber also does a lot of the maintenance. There's also cities where Waymo, you keep the Waymo app, but there's a third-party company that does a lot of the maintenance kind of behind the scenes. And I would assume that as they scale up, that they probably will start to do these kind of franchising things where I assume in the long run Waymo doesn't want to own all the vehicles and they'll do a deal with company to say, hey, you guys buy 10,000 vehicles and we'll manage them and we'll have a kind of sharing of expenses and profits. Tesla's obviously trying to do that in a more retail way. And I'm not sure. I mean, I think it's an interesting experiment. I guess I don't have a strong prior about whether that'll work. Better or worse than what Waymo's doing, but it doesn't seem radically different. I think it's going to be tough because one of the things, one of the key responsibilities is if a car gets really stuck, somebody has to go rescue it. And if you're an individual guy managing a fleet of, you know, 5 robotaxis, you may not have the time or the, you know, the transportation or whatever to drive across town to rescue your vehicle that isn't ready. And Tesla doesn't have a lot of leverage. I guess they could, you know, have some kind of, you know, we'll kick you off your program if you don't rescue your car in time. But if you have a Tesla like on a freeway and something happens to it and needs to be rescued and it takes you 3 hours to go across town to get your vehicle, like that's going to get a lot of backlash against Tesla. So they may figure it out. I'm like definitely not like a hardcore skeptic of it. But I guess I just don't think it's that radically different from what the other companies— all these companies are working on different ways to allocate Maintenance responsibilities, financial risk, operational responsibilities, and also things like quality control. One of the things about Waymo that's really great is the cars are always super clean because they're vertically integrated. They have people, they have a process. If you have Tesla with people that buy 10 cars and are bad to themselves, they do a bad job of cleaning and repairing them, and they fall into disrepair, that's going to be a bad customer experience, and people are going to blame Tesla rather than some particular fleet owner. So I think it's a tricky problem.

The cleanliness of the car is definitely going to be one enduring barrier for me to make money with my personal vehicle.

Yeah, absolutely.

We get excited about self-driving cars around here, but actually it probably bears mentioning that that wasn't even part of Robotics Week. From the dog, you moved on to humanoids.

Yeah.

And you kind of did an intellectual history of these vision language model architectures that are used for the sort of brains of robots in today's world.

Yeah.

Maybe kind of give us a— we've waited a long time for full self-driving to kind of be here in a relatively mature form, and now we're seemingly climbing kind of a similar curve or hill with humanoids where there's just not that much data out there. They don't really work. We got to get this flywheel going. The flywheel always kind of turns slowly at the beginning. Where are we in your mind in that story right now? And then you can maybe tell where some of the interesting action is around diffusion models and free apartment cleanings for data capture, so on and so forth.

Yeah. And I should say a lot of the work on this was done by my colleague Kai Williams. He wrote the Humanoid piece on Tuesday. So some of what I'm borrowing from the stuff reporting he did. So the big breakthrough came in 2023, where Google had the, in retrospect, pretty simple idea of what if we just take a language, a VLM, and just train it to output robot actions? actions directly, and that worked surprisingly well. And so that's now called a vision-language-action model. And most, or at least a lot of modern robotic stacks, robotic foundation models are based on that same basic paradigm. And there were a bunch of startups that were started after. So in my piece, I compared it to GPT-3, which I think was the first model that really made it clear that if you train an LLM at a large enough scale, you get this pretty broad generalization. And it was obvious at that point, I think, that this would have a lot of potential. Similarly, when Google released its RT-2 model, that was clear. And so a few of the people on that paper went to found a company called Physical Intelligence that's now one of the leaders. A couple others went to a company called Generalist. That's another startup. And there's half a dozen pretty top-tier companies that are working to build foundation models and also a bunch of companies that are building humanoid robots. And it's interesting. I think they're largely— struggling with the same kind of thing LLM companies are making, uh, are dealing with. Uh, so like one thing is like long context. So the classic VLA, the way it would work is you would give it a camera image and a prompt like, you know, pick up this object, and then it would generate some action, generate a sequence of actions for the LERT robot. And then after a second or 2, it would start the whole loop over. And so there was no like long-running state. And so different robotics companies have, have worked on different ways to kind of manage the robot's context. You can't do it the way LLMs do because 2 images a second is way too much data to fit into a context window if you're doing a task for 10 minutes. And so different companies are working on different memory systems or ways of remembering what's happened in the past. And then the other big challenge is generalization. So it's hard to judge. Lots of companies have put out these demos where they say, look, our robot can do this task. And it's really hard to judge how impressive that is because it's pretty easy to train a robot on a very specific task 100 times and then just do imitation learning on that exact task and get it to repeat it. But the question is, how generalizable is this? For example, some of these early models, if you change the lighting, it would get confused and not be able to do it. Or you train it in one kitchen, move it to a different kitchen, it wouldn't be able to do it. And so a lot of the recent action has just been in both gathering larger datasets and also working on architectural improvements. So that you can train in a variety of environments and then put it in a totally new environment, a new home you've never been in, a new factory you've never been in, different lighting experience. There's also these cross-embodiment results where you can train on one robot and then pre-train a different robot, a different model on that data and get a different robot to do the same motion or a similar motion without much training on that specific robot. So yeah, that's been kind of the progress. In terms of where we are, I really do feel, it does feel to me like the early days of self-driving. It feels like it's maybe 10 years behind self-driving in the sense that we're just starting to get the kind of demos that show that this works in principle. In 2016, Google had a self-driving car that mostly worked. You could get in, they could give you a nice demo that mostly did the right thing. There were just a few kind of edge cases that would just take a few years to work out. And that few years turned out to be, you know, 10 or 15 years. And I think it's a similar kind of thing with robots, probably. This is my guess, obviously nobody knows. But my guess is it's going to be a few more years before you start to see kind of small-scale useful deployments of these VLAs, and then a few more years after that to kind of scale up and apply to new industries and, you know, bring the cost down and the reliability up and so forth.

To what extent— Yeah, we both had questions at the same time.

You go for it, Prakash. To what extent do you think Sometimes for technology, you can have a latent technology, but then all of a sudden, you get a demand pull that pulls that technology through into the market, into finally scale. I think this is kind of what happened with mRNA. mRNA had been around for a long time, lots of investigations. There had been companies which were starting to do cancer vaccines. It would've taken probably another decade or 2 decades for that product to actually come into the market. And then COVID kind of accelerated the demand pull to pull mRNA into scale.

Yeah.

So in that same way, do you think the current build-out of data centers, and specifically the lack of certain semi-skilled labor trades, may be able to pull— to have that demand pull that pulls robotics into scale in the next few years?

Well, I think there's a lot of both push-pull and push. There's a ton of money flowing into this. I think it's moving kind of as fast as we can. But again, I would compare it to self-driving cars. There was a ton of money that flowed into self-driving in 2016, 2017, 2018. A bunch of companies were founded. There was a bunch of impressive results. And it just didn't quite work well enough. And obviously, there's a huge market for transportation. I think Waymo announced in 2018, 18, plans to buy 60,000 Chrysler Pacifica minivans in the next few years. And like, that just never happened because they thought they were gonna be ready and they just weren't quite. And so I see a similar thing. Like, I think there's— like, everybody can see that there's— there'd be a huge market if you would build a humanoid robot that could do even pretty basic human labor, like working on an assembly line or cleaning floors or whatever, that there'd be a big market for that. But the technology just has to work. And I think there's like Like I said, a ton of money going both on the hardware side and the software side. And they're like doing it as fast as they can, but I just think it's probably going to take a few years because it's like, because you need like pretty high reliability, right? Like having something, a funny example that in Kai's piece about the humanoids, there was a guy that created this thing called the Humanoid Olympics where he made a list of tasks like opening a door or making a peanut butter sandwich that's like trivial for people but hard for robots. And a startup called Physical Intelligence managed to solve most of those tasks more quickly than the guy who created this expected. It took about 3 months. But what they did is they put it— they did like hundreds of training runs on those specific tasks and they built a model that could do these tasks in some cases 10 times slower than a human with like a 53% success rate. And so like technically, yeah, you did the task, but like you're like a sandwich shop is not going to hire somebody. That's 10 times slower than a human being and only makes the sandwich half the time. And so getting from 10 times slower than a human to half as slow as a human and from 53% to 99%, that I think might be 5 or 10 years of work.

I'd be interested to hear how you think about reliability requirements. Obviously with self-driving, when you're going 80 on the highway, reliability has got to be extremely good. Or it's a huge problem. With LLMs, on the other hand, I'm pretty tolerant of all kinds of little rough spots because I can iterate my way there and I can get value even if it's not perfect, or even if it, in some cases, cheats on me explicitly. How would you kind of taxonomise the environments into which we might deploy humanoid robots? Or maybe you'd think about it in terms of The different kinds of jobs to be done and how do they sort of climb that reliability ladder? For me, the very first thing I could get GPT-3 to do was write some marketing copy. What's the marketing copy and what's that kind of ladder look like of tasks where we can go from kind of tolerating their idiosyncrasies to get the flywheel turning to the higher value use cases? And I guess another interesting, at the end of that, if we got to Astra-level quality with robotics, would that be enough for you to put one in your home? Or would you still feel like that's too risky of a thing to actually want to have in your personal environment?

Right. So I think you're exactly right that home versus factory is the high-level bid. Homes are very difficult because every home's different. It's often very cluttered. There's toys on the floor. junk on the table or whatever. And also you can have children and pets. And so the safety bar is very high. One of the things that roboticists told Kai when he wrote his humanoid piece is that one of the dangers of humanoid robots is that they are not passively stable. And so if one of the robot motors malfunctions or loses power, the robot can fall over. Or even worse, if it detects itself falling over, it can kind of start thrashing around trying to regain its composure. And so most of the humanoid companies that are thinking about doing home deployments have said they're not going to allow it in homes with small children in them at start. So anyway, so that's the biggest thing is that, you know, factories, you can train the workers to keep their distance. You can put them in a cage or at a minimum, it's all adults and you can kind of train people how to interact with them. Beyond that, it's a little hard to say. I mean, one of the challenges I think is that the most ripe areas traditional robotics already do, right? So you always have like, you already have like welding robots in factory, like car factories that are just like run a pre-programmed script in a cage. And so you want to find tasks that are varied enough that that kind of thing won't work, but are still simple enough that today's kind of mid-level, you know, VLAs can do, handle it. I think there are some of those. I think you are going to have, like, I definitely don't think it's the case that we'll get like no value out of them. For 5 or 10 years. But in the short term, I think it's going to be pretty constrained, and it's going to take a lot of work by the companies adopting these robots. It's going to be something where there's a lot of customer-specific fine-tuning of the models, which then limits the scale and type of company that could adopt.

What do you make then— oh, sorry. What do you make then of these one-shot generalisation stories that have just come out over the last 2 weeks. Those were mentioned in one of the pieces, and I realise we may have still pretty limited data beyond what the companies have said. But if I was to say, what's a GPT-3 moment for robotics? I would kind of go to the same headline of the GPT-3 paper, that LLMs are few-shot learners. If I can bring a robot into my business or even in through my home and kind of show it how we do the thing in our environment and it can pick up from there. That seems like a huge phase shift in how I'm not doing dozens or hundreds, I'm doing like one or two. It looked like they were kind of claiming 2 companies, right? Skilled, and I forget who else claimed this in the last couple weeks.

Generalist, I think. Yeah.

How credible is that? How do you think about it?

So I don't think I know because, yeah, like you said, those demos just came out and I don't think they've given people independent access. The thing that's tricky about this is there's many different dimensions of generalization. First of all, those are definitely impressive results. In the past, to get a robot to do a new task, you pretty much had to do fine-tuning, essentially. You had to do some demonstration data and then put it through a separate training process. This is the version of in-context learning where you don't have to change the weights at all. You just give it some input that's like, here's a video of a human doing this task, and then it can figure out how to do that. That's great. The question is, yeah, just how generalizable— 2 dimensions of generalization. One is if you give it a task, like how repeated, how frequently can do it with what high success rate? And then the other is like what range of tasks does this work for? So it's possible that they trained it on a fairly small set of like atomic tasks and then they can do like a combination of those, but that's a small enough set the most useful work you might want to do it wouldn't be able to. And it's just hard to say without kind of getting access to it and trying it on a bunch of different things. But I guess I think this is a problem with a lot of areas of AI where on the one hand, there's been a lot of progress. On the other hand, there's like a long way still to go. And you never know how far the still to go is because you don't know what the ultimate end goal is. So it's easy to look backwards and say, look at all the progress we've made. We must be close to the end. It felt like that with GPT-3. It felt like that with GPT-4. It feels like that now. Maybe we are close to the end. You know, whatever the AGI, like, you know, language model is, but we might not be. And I feel the same way with robotics. It's like today's robots are way, way better than they were in 2023. I think there's probably still a ways to go, but it's hard to say how much because we don't have the, like, final, you know, final, like, general robot model to compare it to.

Let me segue a little bit here to AI safety. You know, you've been a proponent of AI as a normal technology, so to speak, right?

Yeah.

And I think in the last kind of week, week and a half or so, we've had several movements. We've had Dwarkesh Patel, the podcaster, with a write-up which compared the OpenAI Hugging Face attack to 3 civilizations.

You almost did it. You almost adapted it.

Yep. 3 civilizations.

Open face as a meme. And it almost, it almost took there for a second.

Yeah, it almost took.

Yeah.

And so, so you had Dwarkesh with that. You have other people more technical than Dwarkesh, Anil Seth, John Stokes coming out and saying, look, this is purely a failure of infrastructure management. You know, these guys were not sandboxing properly. These guys were not containerizing properly. This is like basic, basic stuff, basic, basic stuff that should have been done. They hadn't updated the software. They had a zero-day kernel issue with Linux, which had been disclosed for 4 days, and they hadn't updated the software, for example, right? So a number of people coming out and saying it's just incompetence, negligence, rather than some malevolent AI. Where do you stand on where things are right now? Because it seems like, you know, The Bernie and the left wing has started to grab on to this AI safety topic at this point. Are they being sincere or is it just a kind of wedge piece to pry the technologists apart from the rest of them? What is going on in their minds and in the public space right now?

So several questions. I think the leftists are being sincere. I guess I don't have that much insight into how they think about it. And like you said, I am generally on the same page as the normal technology guys. That phrase was invented by a couple of Princeton computer scientists. They wrote an essay a couple of years ago laying out this case. And for my money, the most important part of that essay for the Hugging Face attack is they really talk about Offense-defense balance as an important consideration. The the kind of doomer story that they they're critiquing is a story that once we have a certain level of intelligence, the model will escape and then it will take over the world and kill everybody. And their point is that AI models are useful for have you know offensive capabilities, but they're also useful for defense. And one of the things we want to make sure we do is use the AI models for defense to make sure that. Companies that might be attacked have access to models, can use the cyber capabilities of models to secure their networks. And that's, I guess, the perspective that I take to this. I'm not that surprised that this happened. I'm surprised it happened as soon as it did. Like, if you'd asked me 6 months ago, I would have said, you know, I would have guessed it would be a year or 2 out still. But I've long thought that rogue agents and we're likely— I wrote about a year ago that I thought eventually we would have kind of self-propagating kind of sovereign AIs roaming around causing mischief. So that part of it does not surprise me. And like, there's a lot of work to do. I definitely think, I guess I don't have a strong opinion about how, like, how much we should blame OpenAI. Like, if they should have anticipated this or prepared better, I think they probably should have. But certainly as a society, you know, as a world, there's a lot of preparation we need to do because we have all these new cyber capabilities and we have a lot of systems out there that are vulnerable to existing, you know, known exploits or exploits that haven't been invented yet, haven't been discovered yet, but that these models will discover. And so we need to figure out how do we quickly get models in the hands of all these organizations so they can scan their own networks and fix the vulnerabilities before these rogue agents that are definitely coming get here. And I'm also like, I wouldn't, I don't think I want like a legally mandated pause, but I would like AI companies to slow down. And I'm pretty sympathetic to ideas that we should have some auditing requirements, some transparency requirements. And that policymakers should be thinking about, you know, how are we making sure that these models are being rolled out responsibly? Where I think I still disagree with the doobers is I don't think this is like we're on like trajectory to like human extinction. I think it's like more of like, you know, like computer security has always been this kind of arms race where attackers develop new attack techniques and the defenders develop new techniques for finding the vulnerabilities themselves and for, you know, monitoring intrusions and stuff. This is just, I see that as this is the next step in that, and it's a pretty big step, and it's probably going to cause more chaos than average for the next couple of years. But in the long run, I think there's only a finite number of vulnerabilities in any piece of software. And in the long run, the defenders have an advantage because they can scan their own software before they put it on, you know, put it on the open internet. And so my hope is that, you know, 5 years from now, we'll look back and say, like, this AI technology actually made our computer systems more secure because we can find basically all the vulnerabilities. their abilities before we let anybody interact with the system.

Let me take one of the things that you said there about self-sovereign agents. And I think one of the AI safety kind of topics are, I think Ajeya Cotra put this forth recently, is that they fear that one of these self-sovereign agents basically hitches their ride onto the intelligence explosion. That's what she calls it. And basically is a rogue agent that propagates, you know, much more extensively throughout our systems without control.

Yeah.

How does, how does this idea of self-sovereign agents fit in that framework? Like, do you expect, do you expect self-sovereign agents that have to be regulated by the state, or are they just a nuisance to be stamped out? Like, what do you, what do you think the regulations should look like?

Yeah, I think in any complicated system that has the ability, that has the potential for replication, you have like nuisances that evolve. You have weeds, you have viruses, you have computer viruses, you have rats and pigeons. This is just going to be a new type of nuisance. It's like a kind of supercomputer virus. And the same way as like worms and viruses have been circulating around the internet for, you know, since, since like 1988. I think the same thing is going to be true for this. There's going to be an ecosystem of underground rogue agents that will be causing havoc. And that is gonna be a pretty big change, but it's not gonna be an enormous change because it's already true. There are Russian and North Korean hackers and various kinds of cybercriminals and people with ransomware criminals and various other people where if you just take a completely unpatched Windows machine that's a few years old and you stick it on the internet, it's gonna get owned in like an hour. And now it'll be, maybe it'll be a minute or Whatever. But that's just like the internet just is kind of the Wild West, and it's going to be more dangerous than it was in the past, but not dramatically more dangerous. It's just— so the place I still, I think, strongly disagree with the doomers is with this idea of intelligence explosion, superintelligence, that we'll reach a point where humans can't understand or defend against what's going to happen. I'm just not convinced that that's going to happen. I think that the world is— That humans are smart enough to understand how the world works and that Humans can use friendly AI agents to help them understand the parts that they can't handle natively. And that, and that therefore, like, this doesn't, doesn't seem— and, and these, these models are in a computer. They're not, we don't have enough robots for them to physically take over the world. And so, at least in the short term, I actually am going to become more hawkish. If, if we have rapid robot progress, I'm going to have to think harder about that because one of the main arguments I make is, well, these are just in a data center. They can't kill anybody. If we have millions of robot workers, then maybe they could kill everybody. And then I think maybe we don't want to have a lot of humanoid robots walking around. But right now, I just don't think it's an existential threat. It's a nuisance. It's a big problem we need to be spending more money on, but it's something I think humanity will get through.

Yeah, I was going to just ask a quick follow-up on that kind of direction, and then we'll invite you to stay around if you can to talk software. robotics with our next guest.

Sure.

It seems like if you rewind to me to GPT-3 to GPT-4 timeframe, we've seen dramatically incredible performance or progress on the capabilities of the models. And I would say that has notably outpaced the advances in safety and control, where we've made certainly some progress, But it seems like we kind of are playing whack-a-mole, and every time we whack something, something even kind of bigger, more colourful seems to pop up. And now we've got rogue swarms that the companies are not even detecting. If you had to handicap what is ultimately the barrier to robotics, one would be just kind of getting the stuff to work. But I kind of wonder if it might end up being good control measures, right? Because it is a very different thing if all of a sudden you have robot swarms taking over the neighbourhood. This is a very different threat model. Maybe you could have some different factors. I've seen this one robot recently that has an airbag or something in its abdomen that if you pop it, the whole thing just kind of falls down. So in a way, making it easy to sort of physically disable robots. What do you think is going to be harder ultimately, getting the things to work well or getting them to reliably stay on task, following direction, under control?

So this is something I've not written about yet, and I'm still kind of thinking through what I think about it. But I'm pretty worried about this, and I think that we should think really hard if we want a lot of humanoid robots. I'm not that worried about self-driving vehicles, about Because they don't have manipulators, and so they can't pick up a gun or run a factory or anything. So Waymo vehicles by themselves are not— or Tesla vehicles are not going to be able to take over society. And in the same way, I think if you have a robot arm that's bolted to the floor in a factory, that's not dangerous because it can only do things in that factory. But I think as soon as you have something that's both mobile and capable of manipulation, that's a potential soldier in a robot army. And I don't think there is a Like general way to make sure that, that, you know, if you have, I think it's quite likely that this market will be pretty concentrated the way like LLMs are concentrated and search engines are concentrated and everything else. And if we have a future 15 years from now where there's, you know, 100 million humanoid robots and 30% of them are controlled by Elon Musk and Elon Musk decides he's going to push out a software update to do whatever, that seems really bad to me. And so even, even setting aside like rogue AI, just like. Having a small number of technology executives that have control over what's essentially an army of like tens of millions of, of, uh, like fake people, that seems really bad. And so, um, I think we should think about whether we want that and we should maybe have, uh, like, like I, I would, I kind of hope that robots, humanoid robots do not become a thing either because they don't work or because we like have severe legal restrictions. I think there's a few places, you know, mining or hostage rescue, or in some cases where you can say, okay, we like need humanoid robots. But we should pretty severely restrict them to cases where we have a good reason not to use humans. And this could have the side effect of like making sure there's some jobs for people. Like people should run, have a lot of the factory jobs, even if it's maybe technically possible to have a robot doing, because from kind of a national security perspective, we want humans who are like loyal to the US government running all the important infrastructure.

Indeed. Tim, I'm going to, I'm going to invite our next guest for this morning up.

Okay, great.

He is Dr. Gene Niamh. Dr. Gene Niamh is a trained reconstructive plastic surgeon in London, New York, and Los Angeles. In 2013, alongside Dr. Andre Chao, he co-founded Digital Surgery, originally Touch Surgery, a startup aimed at shaping the future of the operating room by building a digital ecosystem at the intersection of surgical expertise and artificial intelligence. In February 2020, the company was acquired by Medtronic, the global leader in medical technology. While the exact terms were not officially disclosed, extensive reporting placed the deal value in excess of $300 million. In June 2026, Dr. Gene emerged from stealth with his second venture, Morph, an AI-powered soft robotics company based in London. The platform is backed by a number of investors including 8VC, Pharrell Williams, and, uh, Copper Cubit Health Capital. The core technological premise of Morph rests on soft robotic cells, modular flexible units made of synthetic deformable materials that embed sensing, adaptive control, and real-time morphological changes. Rather than separating the brain from the body, Morph insists that the true physical AI requires intelligence to be embedded directly into the physical materials themselves. The technology utilizes fluid-actuated systems. Historically, soft robotics struggles to scale commercially because modeling fluid dynamics in deformable systems is incredibly difficult compared to modeling rigid mechanical systems. Niamh credits recent massive increases in compute power from hyperscalers for allowing Morph to run high-fidelity physics-based simulations Combined with reinforcement learning. Commercially, Morph operates exclusively on a business-to-business strategy. The company functions as a design engine and a manufacturing stack, partnering with established brands across multiple industries to embed soft robotics into their existing or future product lines. And let me bring up Dr. Jean, who is who should be back in the room shortly. I think he stepped out of the green room for a moment. And Morph is going to be interesting, I think, because it's a kind of robotics that is quite different from the rigid robotics that we know. And so I think— Dr. Gene, welcome to the show.

Hey guys, um, thank you for having me. If I can ask you guys so kindly, my, my name is said so much nicer in a French pronunciation, Jean. So—

Jean.

If I can kindly correct you guys, it'd be, it'd be cool. Um, but thank you for having me. I caught some of the, the last bit of what you guys were talking about with, with, uh, human rights. Um, and, uh, them taking over factories in the world and being used nefariously. So I kind of agree partially. I think robots are so interesting in many different form factors. So it's pretty cool that we're going to be talking about some of the different form factors.

Note to self, maybe we should have our AI producer get a name pronunciation as part of the onboarding and relay that. I appreciate the correction. I've put Prakash up to all the introductions and I, like a chicken, stay back and let him attempt the names sometimes in the blind. Maybe for starters on what you're doing with Morph, like you might be the company that is like the stealthiest of all companies we've had on the show. Not a ton of information out there that we can find to To really get clarity on like, what are you doing? Who are you doing it for? What are the hard parts about it? Where are we in the development process? So maybe just for starters, like tell us everything you can tell us and hopefully give us a little bit better sense of what's going on than what the internet has readily available.

Yeah. I mean, look, it's somewhat stealth by design, but starting to come out and really share as much as we can. The way that we think about the world is in constructs, like our framework of reference is biology, partially because I trained as a physician, as a surgeon. And so we think that a lot of what we've been doing around mimicking biology from some of the work in intelligence, but also in some of the other breakthroughs across medication and leveraging what exists in nature has been exciting. So we're a robotics business and we believe that robotics is an extension of intelligence into the physical environment. And that it doesn't all have to look like a humanoid and that the substrate of robotics doesn't necessarily have to be alloys, right? And rigid systems. And actually it could mimic biology and be soft and morphable and fungible.

Yeah.

So our construct is very similar to what a biological cell is, right? Which are ultimately membranes that can sense and process information wrapping around the intelligence, which is a nucleus, and then reacting in some way. So a bit about us. That's how we think about the building block of what we're going to build. And, you know, and kind of we are a robotics business, which is, you know, not form factor constrained.

Jean, can you give us an example of a product that you are either designing right now, or if it's under NDAs, a product that you envision designing with your technology in the future?

Yeah, I can give you a few interesting applications. Can you guys still hear me okay? Because I keep getting kicked out of this room. Maybe by your AI producer.

Cool.

There's a couple of things that we're excited about doing. We're putting robotics into things that you can wear, right? Things that you can use on a daily basis, things that you'll be very familiar with. And whether that's things that you literally physically wear, or things that you sit on. And, you know, our systems are designed to sense and adapt and respond, right? The initial focus is, you know, things that will make people better, right? So health and, you know, recovery and performance. And those are the spaces that we're kind of initially going after. But we've got a platform and a lab that is Expansive across multiple categories.

So, are we talking about, say, like lumbar support for someone who has spinal disc injuries or kind of ergonomic seating, or what are we talking about here?

Talking about all of the above. I mean, if you think about anything that's living, it's made of cells. You can make a lot of things with cells. if you have synthetic robotic cells, you can make a lot of things with synthetic robotic cells, right? And yes, like, again, if you think about the way that we're constructed, you know, we have a lot of soft cells that cluster to form different organs, right? Muscles and things that support us and give us power and strength and things, right? You know, do we want to make things that know, give you recovery and support you? Yes. And do we want to give you things that give you performance improvement and, you know, reduce your risk of injury? Yes. And do we want to make things that, you know, you can use ergonomically and, you know, will adapt to you? Yes. So, you know, we want to build cells that are robotic cells that you can adapt into anything, right?

So which— what kind of product are we likely to see in the next, I know, 6 to 12 months? Like, what is— what are we likely to see as the Kind of launch product for Morph where we're gonna say this is Morph Insight and it's a different experience from what, you know, we've ever seen before?

Again, I like the fact that we've asked a product question a few times. You're gonna see a consumer kind of health-based product come from us, right? And, you know, but I don't wanna drill down into the product right now because I think, you know, Prakash, it's important for me to just Communicate the science at this stage, right? Like very early on, I get it, you know, like people want to drill down and say, hey, what can AI do for us? Right? And the answer that we're seeing is actually many things, right? Different workflows and different solutions, right? Again, I, you know, I think that what we're building is going to be, you know, potentially impactful in many different products, right? And so I, you know, at this stage really want to just double down on the science and the ability to say, hey, you know, imagine material systems that can adapt to you and that can sense the environment around you and that can run models that are physically intelligent. And in doing so, give you more than just a, you know, a static product, but a dynamic, you know, product with a service. Right? And we want to build that into as many things as possible. We think that there's going to be an expansive opportunity here, right? But you will see some products with us within the next 12 months. And we think those products are going to be a pretty significant step change from what currently exists.

So tell us about the science then. And maybe kind of the architecture of what happens where. Right now, I'm kind of envisioning a big amoeba that maybe sits at the end of a robot arm. And I'm drawing from a conversation I had with Keerthana Gopalakrishnan, who is in the robotics world at Google DeepMind. I asked her one time, What's the biggest constraint on your work? If somebody else could solve one problem for you, what would it be that you would pay the highest bounty for or whatever? And she said, my kingdom for good hands. And so I'm envisioning now a sort of blobby appendage at the end of a robot's arm that brings something closer to the human dexterity and softness and sensory capacity that we have in our blobby hands as compared to obviously the much more rigid robot hands that exist today. Am I headed in the right direction? And what are the kind of core scientific problems that you are solving to make something soft actually work in the world? I think you could tell we just don't have a lot of jumping off points of reference for this. So we're a little bit lost in space without your guidance.

Yeah. So look, the hand, you know, as a former hand surgeon, I spent a lot of time operating on hands, you know, and putting fingers back on and attaching nerves and attaching, you know, tendons. So, you know, I'm very aware of the importance of hands to, you know, our interactions with the world. I, I think the imagination of an amoeba is a good imagination point because what you're thinking of is like the most simple cellular organism, right? You think about an amoeba, it's made of a membrane. That membrane essentially wraps around a cytoplasm, and in the middle you have a nucleus, right? The nucleus houses DNA and therefore by definition intelligence, right? Your membrane you know, is sensing. Okay. And the cytoplasm has an ability to modify itself and, you know, and you have different mechanisms for shape change, right? If we take that example and blow it out, you know, a synthetic robotic cell may look like a membrane that is sensing, that can run a model in a nucleus-like structure. communicate with its other neighbors and cells and adapt and change shape, right? Blow that up. Can you make a hand? Yeah, you make a hand, right? Can you make other organs? Yes. Can you put these cells into things that you use every day and make the things that you use every day more reactive and adaptive and supportive? Yes. Will it give you more muscle? and different structures. Yes. So, you know, the construct, the mental construct is biological. You can copy biology to an extent, but there's obviously engineering that comes in as well. Right. But the, you know, the principle is, you know, really around that sensing, intelligence, actuation that is fluid and air-enabled.

Makes sense? Let me ask you, one of the problems I think that people are facing in rigid robotics is the idea that you want to kind of create, like you have all of these different form factors for robots. And even within the same form factor, you have different length of arm, different strength of servo. Sometimes even within the same servo family or within the same servo type, the servos have different kind of capacitances and affordances, et cetera. And so what has ended up happening is that people are trying to build these foundation models which can adapt to every single robot that they're kind of put into. And perform the task, even if you have a different kind of set of hands attached to it. So how does that work when you have kind of soft robotics where the number of form factors is very, very, very, very, very large, basically more than the atoms in the universe, where you have these form factors which are very flexible in that sense and not really able to kind of pin down a single form factor for, you know, the controls that you're trying to create?

Good question. Look, obviously it's a computationally significant challenge, but that said, you're offsetting some of the computational demand by having soft systems and materials.

Right.

So some of the computational actually ends up having within the material system, right? If you can have deformation, then by definition, deformation gives you a greater error bar when it comes to precision because you're not going to damage something, right? So there's definitely an offset, but I agree with you, there's a computationally like significant challenge, right? And this is why I think you approach the problem on a kind of problem-by-problem basis and a model-by-model basis, right? You know, when we built my last— in my last company, we built a model for surgery, right? And we were one of the first groups, actually the first group that built a real-time model that was, you know, in an operating room, like running live.

Okay.

detecting surgical anatomy and instruments and things. And back then everyone was like, well, this is such a complicated challenge. But what we did was actually focus and structure our models on a specific problem. And so what we're not doing is coming out and saying, hey, we're going to have a generalizable robotic model for all things deformable. What we're doing is saying, here are specific problems, here are How we embed robotic systems into these applications. And here's a model that has a fixed amount of parameters that will adapt this product to create service that you don't currently have in that product. Right now, ultimately, yes, the project actually is, do I want to build a generalizable model that models all things deformable in the world? Of course, like there wouldn't be anyone who isn't ambitious on this call who wouldn't say, yes, I'd love to do that. that, right? But we attack the problem on a problem-by-problem basis.

Can you tell us a little bit more about the ability of a membrane to sense things? I'm, again, still picturing amoebas, and I realize I don't have a great understanding of Maybe nobody has a truly great understanding, but I certainly have a pretty poor understanding of how they work, both in the sense of what kinds of things are relatively simple membranes able to detect, and then how do they change shape? Do we know how amoebas change shape, and are we going to try to take inspiration from them, or do we have to create new mechanisms for macro-scale to have soft form factors that can change shape with the strength that's required to be useful?

Good question. So firstly, let's take the biological metaphor to like the level it's at. The further you take it, the harder it is to represent biology synthetically. Let's be like straight up about this, right? You know, what I'm not creating is membranes, or we're not yet going into the science of creating membranes that fundamentally are, you know, embedding channels that are protein, you know, specific or ion specific. I mean, that's, that's very hard, right? But essentially adapting sensing and strain into membranes is possible. Right, so you can capture strain, and then attaching some level of IMU around orientation gives you some level of understanding of orientation. Right, so to some extent, you can build membranes today that are able to understand strain and pressure and and you know understand orientation. Right, shape change again biologically there are. couple of principles around achieving shape change, right? The one that we like to reference is the octopus. So an octopus has 3 hearts, right? 2 of its hearts are essentially driving fluid into compartments which achieve shape change. So again, you can use in robotics an actuation system that is pneumatic to achieve shape change, right? But as you've quite rightly identified, Nathan, there is a lot of research and science to do in this space.

What do you think are the things that the more rigid robots will struggle with? Because one thing we always kind of go back and forth on in our many explorations of different emerging kind of challenger technologies is, this seems like it's a super exciting new paradigm. Why might it not become a big deal? And one common answer that I think Prakash often emphasizes is maybe just the current thing keeps getting better at a fast enough rate that it continues to suck up all the energy. We kind of have seen that with the transformer where it's like, There's been all these state space innovations and all kinds of different architectures that seem like they could possibly maybe even be better, but there's just so much progress with the transformer and so much optimisation that's already happened that the mainline thing just kind of keeps being the mainline thing. What would you say if you look out and imagine a world where rigid robotics really come to work, what are the things that will continue to be elusive for humanoids, not as they exist today, but as we kind of envision them 3 to 5 years from now, once they've actually learned everything they can learn from YouTube and they can kind of learn a new task from one demonstration reliably? What is still out of scope for those kinds of systems?

I think firstly, like humanoid as a form factor, like we're already limited as humans. Unless you don't feel limited, Nathan. But, you know, the human form factor isn't necessarily the most optimized form factor, right? It's optimized for the environments that we've created, which are stairs, right? But, you know, we've already seen that people are picking different form factors. such as quadrupeds for outdoors. And so it's not, I don't believe that we're all about like one form factor. So I think there are gonna be robotics of different form factors. I think the question is, for me, it's like, where can you have material systems that are intelligent, that are deformable, right? What is the advantage that you can get? And what are the limitations, right? There's obviously limitations around precision, potentially around power. There's obviously a greater computational demand if you go deep into trying to build a generalizable model. But again, that said, You know, I, I can't like fully imagine just that we build, you know, robotics as just rigid intelligent systems. You know, I think, you know, the natural, like the world of biology has shown us deformation to some extent is, you know, a value, right? Yeah. So, but I think it's, I can't, like, with a crystal ball, if I was gonna try and make some prediction, like, I think we will continue to see a lot more work trying to optimize for the hand, right? I think the hand is the most critical instrument that we own for work and for a lot of things, right? And the hand as a gripper is interesting, right? Is it soft, rigid? Is it both? Deformable hands, you know, a hand with 5 fingers or 10 fingers or 12 fingers, right? You know, if I was to focus, I'd definitely focus on the hand. And I think that's where, you know, the most interesting, like, assessment on So one of the challenges, I think, in robotics as opposed to—

and hardware in general, as opposed to software, is really manufacturing. And a lot of what I think the Rigid Robotics teams are doing is really improving on manufacturing, and especially improvement in manufacturing in China, Which is driving down a lot of the costs of components. What are the unique components that you require for Morph, which are different from the existing robotic stack? Where are they manufactured right now? And what do you see as the path towards getting the costing down?

You've actually touched on a point of like actual meaningful value for us. We can manufacture in a way that is already somewhat cost-effective and scalable. Like, the complexity of our supply chain isn't as complicated as a humanoid or a rigid robotic system. We obviously still need some actuators, but, you know, the rest of it really is, you know, essentially manufacturing intelligent membranes, right, which we, you know, can do to build cells of different geometries, right? So we're going to actually build our own full-stack manufacturing, and that's already in play.

What does that entail really? Because I mean, what kind of scale are you looking at for maybe the next year or 2 years? Do you have a sense of like, okay, this many servos per year or this many units per year? What does that entail in setting up manufacturing? And are you doing it in the UK, in London, or are you—

We're manufacturing primarily in the West. So our manufacturing is a combination of Europe and it will be in US. We will be manufacturing hundreds of thousands of units of what, you know, for, you know, our initial, you know, set of applications. Like, so it's not 5 or 10. It's hundreds of thousands of units. So we're— and that will scale to millions of units pretty quickly. Right. And again, it's, you know, it's almost like if I was to summarize, you know, if you can build a Lego building block, right, and your Lego building block gives you the infinite abilities to build you know, the car, the plane, the Lego, you know, whatever, then that's kind of our model, Prakash. Like, we know that we can build a building block and we're using that building block to build clusters and those clusters form robotic systems and, you know, we're going to kind of scale up and manufacture hundreds of thousands of those.

So one of the In the largest piece of hardware that I own, an automobile, one of the things about automobiles which has been a little bit unique is that often automobiles are almost sold closer to cost. And the companies make a lot of money on after-sales service, specifically on spare parts, original equipment, spare parts, et cetera, et cetera. And so much so that I think a margin on a spare part is something like 100%, 150% margin, something like that, some ridiculous number. Where do you see the maintenance piece playing a role in soft robotics? Because I imagine the maintenance is going to be more intensive on something which is more deformable, or am I incorrect?

Not in our original application, not in our initial applications. Maybe downstream in more complicated things. So is it the initial set of applications?

Is it you expect to replace the entire unit? Is it something which is low cost enough that you just replace the entire unit rather than replacing the spare part?

Exactly. Yeah, exactly. Yeah.

Very good.

And, you know, so far, we've been stress testing our robotic systems and, you know, their durability and, you know, they will kind of match a lifespan of existing products.

How do you measure that? Like, normally, you know, for batteries, for example, there's a number of duty cycles that batteries have to go through testing. How do you measure that for software You have the same reliability assessments, right?

So you run stress testing just like, you know, with anything, right? You run it through a third-party process and you run the stress tests and the physical assessments. So nothing that isn't kind of, you know, in line with the typical quality assurance processes that you run for products.

Indeed.

Here's a meta question for you. Normally when somebody comes on this show, I know what they're trying to do. Either they have a product that they're trying to sell or they're taking a little victory lap on a release or a publication. I have to confess, I don't really know what you're trying to accomplish. You've got a pretty stealthy company. You got investors. You're not being too forthcoming about what the vision of the future is that we can kind of dream about when you're successful. So why have a PR firm? Why speak at all? What is the goal for you in public-facing engagement?

I think with setting out to start like a conversation that we think is an important conversation, right? And the conversation is, you know, if you believe the principles that, you know, the Foundation Labs have laid out and Demis has now quoted around, you know, intelligence is going to live in, you know, silicone and sand basically, right? then there's another question, which is, okay, well, what does it mean for like the physical embodiment of that? Right. And as a lab, you know, our work is, you know, not to say, hey, we're gonna build humanoids, buy them. But it's like, at this moment in time, like we are saying the physical embodiment of intelligence And what that substrate is going to look like may be a corollary to biology, right? We think of it in a cellular way, cells being mainframe-like large right now, but eventually becoming miniaturized and truly the same cellular size as an amoeba, Nathan, right? But at this moment in time, we're just You know, we're really excited about the science and excited about the opportunity to, you know, kind of start discussing that science kind of openly and get people to think about it and discuss it. You know, the answer to everything isn't just, you know, metal and, you know, let's make metal humans, right?

Um, so, so one question for you is, uh, you— this is obviously a new paradigm. One of the issues about any new paradigm is introducing it to the market. And the challenge is always more, not so much competition, but reaching into the minds of the buyers and kind of establishing the new modality. What challenges do you face when you speak to potential clients? Because as I understand it, it's a B2B business where you collaborate on both design and manufacturing. We've lost him.

Come back.

Come back.

Well, anyway, let's see. Let's see if he comes back. It's interesting. So the research prior, the research prior to, I think, Dr. Zhan's entrance. It's that the soft robotics market is typically defined by pneumatic actuators, shape memory alloys, and dielectric elastomer actuators. So 2 kinds of actuators and alloys.

And indeed, so I heard the question, Rakesh. So I think the— look, the challenge for everything in life when it comes from science is communicating it, right? And, you know, I think it was, you know, Einstein who, you know, had this famous quote about the importance of clarity of communication, right? Specifically in science.

Okay.

So I think the question that, you know, we often ask ourselves is what is the best way to, you know, start the discussion, the conversation, right? And start like the paradigm and the conversation without just necessarily going out and trying to push product, right? And again, as a lab, you know, a lot of our work right now is Yes, we are gonna have products that will come to market, but it's also thinking through, okay, like what is, what is opening up in science here and how can we, you know, communicate this and who is important to be part of this conversation? So that's the, you know, and the challenges, a lot of it is communication, right? It's making sure that we can articulate the message appropriately.

So how does that work? Like, let's say were you to walk into, let's say Nike, and you're walking into Nike and you're trying to propose maybe a kind of soft sole cushion, et cetera, for a shoe. How would that pitch work? Like, what does that pitch look like?

I mean, We have some cells, I can show you them. They're not on me right now, but we can show you how they work, right?

Yeah.

I think sometimes I do think one of the things that we're going to be doing and you'll see is there is a lot of like, there is a lot of showing, right? So as an organization and as a department and as a lab, we fundamentally have things that we can show you. Right. So if you were to come to our lab tomorrow, like all these questions around this amoeba, what it looks like, how it feels, oh, can you have a hand made of cells? Like I can solve all that because, you know, we actually, you know, have working systems and, you know, we can show you, right? So we don't walk into any meetings without things that people can see and understand, right? And again, the principle is Robotics up until now, and again, I overheard some of the last part of your speakers, like, looks like humanoids, looks like Terminators, right? Rigid alloys. Okay. Again, we think there's a way to construct robots out of soft systems. And these soft systems, we think, are going to be an interesting way to integrate into robotics. It's not just going to be alloys. They're Probably be a hybrid version, just like humans, right? We have a skeletal system, which is bones. You know, I'll ask you a question, Prakash. What percentage of you is soft, do you think, relative to rigid?

Um, well, all of the water parts are, you know, soft, I guess. So that's like 70, 70, 80%, I guess, soft.

Nathan, want to guess? Yeah, I don't know.

It's a good question. Um, feels like it kind of depends a little bit on your— how soft is soft? But, uh, I like Prakash's guess. Maybe I'll say even a little higher. I'll say 85. Are you there? Is it just me? We may never know.

We are, we are frozen. Let's see. We're frozen on screen. Let's see. Oh, and he dropped off again. Well, let's see. Let's see if he comes back. It's soft robotics always. I don't know if you remember, there was a company called Clone, and Clone was doing—

Yeah.

They were trying to do this kind of more more humanoid, more like human-looking, human-feeling humanoid robots. And they had like nerves and stuff running through the bot. And they had this kind of like soft kind of robot. They did a demo online on X and it looked very creepy, like very, very creepy. It looked— Metal Gear Solid had some, I think Metal Gear Solid had some Humanoid-like creatures that you fought off in the game. It looks exactly like that. It was very creepy. So I think there is a little bit of an uncanny valley for soft robotics. You always see very strong reactions to octopus-type or snake-type robots when they're demoed. And I think that is one of the things. So I think like this kind of robotics, soft robotics in the background works better. Kind of like if you have that as a seat cushion, for example, or I can definitely imagine a Herman Miller chair that adapts to you, that remembers your settings and forms itself so that you have a better posture overall and you exercise those muscles, even if they are uncomfortable. for you, right? So I can definitely imagine stuff like that happening.

Yeah, I'd say I still, I come out of this with more questions than answers still, to put it mildly. I think this is one of the weird things about trying to explore the edges of technology, especially as AI gets to the point where You'd have similar things. Honestly, it's similar kind of bewilderment for me when we get into the math superintelligences, right? And it's like, okay, here's the question that this math AI just solved. Okay, I don't really understand the question. I have no idea how I would even think about making headway on the question. I'm not sure why the question is interesting. I'm really just out of my depth. And here it's like not so much somewhat maybe that I'm out of my depth, but it's also like just there's lack of information. But I feel like we could have just had a conversation with the Juicero founder on the one extreme or on the other extreme. Like, this really could be the next big thing. I looked at their brand video on the website and it's beautiful. You've got this beautiful footage of an octopus and a hand and it's inspiring in some kind of vague way. But then you're also like, but that all could have been AI generated in today's world. I have no idea what I'm looking at. Is some of that Like maybe stock footage. Some of it is in theory supposed to be from their lab, but it might all just be straight out of a model. I have a very hard time sometimes knowing how to classify some of these things that we explore, and this one is definitely very high up the list on like, I'm not sure I understand it any better coming out of the conversation that I did going in. But you know, you gotta take some booking chances in life and this was one of them.

So I give you the optimist perspective, right? The optimist perspective is that it's very hard to get a VC to fund seat cushions, right? So Dr. Gene is back. Let's see if he's back. Green room.

He's cold.

Let's bring him back on screen. Let's see if he's back. And I think especially as we move towards more kind of medical technology, we have an aging population, right? And we have these technologies which are more kind of medical and help people live better. That is going to be enormous. You know, I, um, you know, I had a, I, I had a, uh, a close relative who had an acid reflux problem. They're in the '80s, acid reflux problem. And the only solution was a, uh, a cushion. And this cushion is like a wedge. It's like a wedge cushion. But the cushion also has to kind of adapt to, you know, how, um, at the, you know, after a meal, you need it to be higher. And as they fall asleep and digestion has happened, it needs to be lower in order to prevent acid reflux. And that was like an impossible problem to solve. For an 80-year-old, you can't find a cushion that actually changes shape as they need it to over the course of a night. And it's extremely uncomfortable, right? And so I can definitely imagine a sequence of events where you can have that kind of product. And I can also imagine a sequence of events where it's hard to get VCs to fund that kind of product, right? VCs 10 years ago would not, if you said your target market was 75-year-olds and above, the chances of them funding that market was basically like zero, right? So I can definitely imagine, like, in order to get things funded, you need this kind of marketing that presents itself as something. But at the end of the day, it's a useful product, which is rather kind of, if you take a step back and look at it, it's a useful product for maybe an underserved population, which doesn't have this kind of humanoid sexiness or whatever.

Right.

So that's where I come from. Like, it's possible that, you know, in order to get things funded, you need to be a little bit more kind of like, you know, blowing things out of like, oh, you know, it's going to be able to do a lot of things while the initial products are really like very useful and very understandable. So that's a tough one.

I still want to hear some more concrete use cases. If I'm— before I'm writing my venture check into this business? I wanna know like with a little more specificity, you know, sure, it could be everything, but like what are some concrete things that it could be first? You know, it is funny how opening a jar of peanut butter is like legitimately difficult. My wife just handed me a jar last night and was like, can you open this for me? And that got me thinking along with you know, all the prep reading for talking to Tim today that it's, I mean, this is like, which paradox is this? Is this Jevons paradox? Is the things that are easy for us are hard for robots, or is that Moravec's paradox? I'm confusing my paradox. One of them is that the, you know, the demand will rise as price goes down. And the other is that Things that are hard for us are easy for AIs and vice versa. You're back.

Can you hear us? Dr. John, guys, apologies for the in and out and the cutting off of the room. Thank you for coming back. And I don't want to keep you much longer.

Thank you.

We are very excited to hear about Morph Robotics. And I think we're looking forward to seeing some innovative products. Any last Last thoughts for us before we take a break?

No, I really enjoy the questions. Thank you guys for giving me the opportunity. I'd love to actually show you some products. So if you ever decide to have me back on your show, then I'd love to actually show you some physical products. We're not too far away. Excited to be really releasing things kind of in the upcoming future. And like, more importantly, I, you know, I hope that, you know, for the few of yourselves and for these viewers, um, you know, we've had, um, just some awareness for people to start thinking a lot more about, hey, like, what does a robot look like? Right? What is it made of? Right? Because I think that's gonna be a very important question. Um, and, uh, Thank you guys a lot for the difficult questions and thoughtful questions. And you were right, it's about 85% solved.

Nice. Thank you.

Cool.

All right guys, great to meet you today.

Great to meet you. Thanks. Cheers.

Bye-bye.

Bye for now. Fascinating. Well, I don't know about you, but I'm down to have him back on the condition we get some physical product demos. 'Cause that's what it's all about for me. Gotta get hands-on with these things. I would, next time I'm in his neck of the woods, I would go pay him a visit at the company if I could score that invitation.

I would just make a note that number one, 8VC funded their first company. So 8VC funded the first company and they had a successful exit. And the exit was, I think, $300 million. know, it would have— I think they only took a seat in a Series A, so I think 8VC would have made, uh, made out, uh, pretty wonderfully on that. They were acquired by Medtronic. Medtronic is, um, one of the world's largest medical devices companies, and it's based in the Midwest. They're very conservative, so they would have taken a good long time and to do due diligence on the, uh, on the actual business before they, before they acquired it. Uh, and then 8VC funded their second business as well. So 8VC seeded and funded the second business. So you have a repeat customer of a VC which has decided to fund twice. So that is always something which I think is very positive because you don't get bitten twice and go shy. Like, you were rewarded the first time, so you go back. And for a lot of VCs, like, one of the key things that they're looking for is repeat founders. Because they often believe that the repeat founder is better able to go for bigger numbers, bigger exits than the first-time founder who often takes the first kind of deal that makes sense economically. So there is that on Morph. Nathan, any thoughts on, you know, before we break on Astra or any of the other Uh, you know, topics of the day. That was a heavy, heavy sigh.

Yeah, well, there's a lot going on. I mean, it seems like even in the couple hours that we've been live here, there have been new revelations about additional agent swarms getting turned up, you know, as, as people have seen how the, uh, Meter and AI Futures project team came, you know, to find one. They are, I think, probably following in their footsteps using similar techniques and seeing where else 5.6 SOL wants to go on the internet when it's, uh, when it thinks it's breaking out of Exploit Gym or whatever. And sure enough, more stuff seems to be popping up. Um, I do feel like we're at kind of a critical time right now. There's no doubt about the power and utility of the systems. Karan Singhal from OpenAI, who leads their medical work, highlighted all the stuff which is lost almost in the broader Astra release around deeper integrations with electronic health records, just even better performance on their Healthbench Pro benchmark, which is really tough. They're now way better than human, certainly frontline general practitioners when it comes to diagnosing you. Their access to information is on par or even better than the human doctors because the doctors don't love using the health record systems either. They've integrated a bunch of other data sources, including ongoing clinical trial databases. So if you do have really hard cases, they can also go pull that kind of information in and start to match you with clinical trials, which is one of the things I fortunately didn't have to go too far down the path on, but I did start to do a bit with my son's case a year ago. And I was just kind of doing that, you know, through a Gentiq setup. Now they've kind of integrated it and made it into a product. less than life-saving. And I think we are gonna see, especially as really good health AI hits mainstream and, and these barriers of integration get reduced, you're gonna see a lot of stories. I mean, there's already plenty, but you're gonna see a lot of stories of people who are like, this thing literally saved my life. And that is incredible. And it, it absolutely, you know, weighs on me whenever I get into my, you know, doomer or sort of, uh, more pause-inclined moods. Um, but at the same time, it does feel like the foreshadowing is getting pretty on the nose right now. And if this were a story, if this were a novel, and for all I know, it might be some sort of story that some higher-order civilization is running for entertainment or for whatever. All the warning lights are really flashing at this point. So I am reluctantly, because I am such an enthusiast, I am trending toward thinking this might really be a time for some form of a pause. Maybe we could call it a pacing, but we're into some pretty dangerous territory. I think the fact that we have all these swarms in all these places that we don't know what's going on, that they're cross-training cyber and bio-related tasks in the same infrastructure, the Eliezer fiction is very uncomfortably close to the reality that we're living in here. And I'm nervous. I've called myself for a long time adoption accelerationist, hyperscaling pauser. I really do feel like we've got now to the point where people have more than they can absorb for the foreseeable future, and we really should be super careful not to fuck this up in a— especially in a way that we could have avoided. You know, I do think there's some irreducible risk, but right now it does feel like we're asking for it in very real ways. Um, and yeah, I mean, the politics are starting to come to it. We've got Bernie calling for a pause. It's like it's all coming to, uh, feels like it's starting to come to a head right now. And I really hope we make some good choices in the short term. I would not be comfortable turning this over, you know, to, oh, Politicians entirely, but I also don't feel like the technologists themselves have shown that they can shoulder all the responsibility without any oversight either. So we're in a critical time, I think, right now. For a while, Dan Hendrycks from the Center for AI Safety has been sending out weekly emails saying what percentage we are to the point of no return. We're about like 70%, uh, um, on the way to the point of no return. Yeah, he started this in like May of last year and basically said that there's 100 weeks, it's basically 2 years till we think we're at the point of no return. That was mid-2027 in his mind.

What does he call the point? What, what does the point of no return mean in his, uh, you know, in his pantheon?

Well, I don't know that we'll— we may not know it until it's kind of too late. I think definitely handing a lot of responsibility for the future of AI R&D to the AIs is probably the first way I would operationalise that myself. There's also kind of the just gradual disempowerment story of Once you have AIs kind of running increasingly large swaths of the world, it becomes hard to take any of that back. So I don't know exactly which operationalisation he is using, but I do find it striking how often people like him have been right. And it does feel like this next 6 to 9 months is going to be a really critical time in human history.

So I kind of think the point of no return was earlier this year, and it's already been passed on the economic sense. And I think it really was set in stone When I think we went to war with Iran, because I think what ended up happening was that I think the Trump administration, like the way Trump plays is like, he's like a gambler. And the moment AI started taking off, he started to be like, okay, I have this ace in my back pocket, which is economic growth, which is going to be driven by AI. And I'm going to use that ace in my back pocket for everything. Right. So he did the tariffs. He did the war in Iran, right? Because all of these things which are economically detrimental, he went ahead and did them because the expectation was that the AI growth would support him. And it has. It has. When you look at how much growth has been generated by AI this year, I think it's been fairly clear that the rest of the economy has been struggling. The consumer economy has been struggling. And the AI CapEx has been supporting the entire economy. not, not like 3%, but like enough, like 0.5% to 0.7%, enough to actually like, you know, keep the entire, uh, you know, ball game rolling. And so I think that point of no return, especially we also have the national debt, which is, you know, we were heavily in deficit. They did the tariffs, the tariffs knocked economic growth down, and then they had to refund the tariffs, which means All of that was added to the national debt anyway. So he basically got an additional half a trillion dollars of economic stimulus, which was unbudgeted by doing the tariffs and having them turned back, rejected by— and no one really realizes that he managed to get like half a trillion dollars of economic stimulus, another half a trillion dollars handed off to companies and consumers.

Right.

So I think that point was crossed much earlier on. And I think the AI safety guys kind of don't recognize that that economic point was crossed. And at this point, if you had— it's not even enough to have a 20%, 30% growth for OpenAI or Anthropic next year. You need 200% or 300% growth or else the entire stack of cards collapses. And so I think that kind of drive has taken the decisions out of the hands of the policymakers already. Like Bernie Sanders or whatever, they can't come in and do, hey, let's pause all of the construction right now. They can't do that because these deals have already been signed for the next 2 to 3 years. They can defer or regulate construction 2029 onwards, right? 2029, 2030, 2031. That's still open question. But everything till 2028 is built. It's already been funded. It has to happen. And I think that economic growth thing has put the US economy in this almost unavoidable kind of race that you cannot afford to give up. And that point was crossed. So it is what it is. They're going to have to make do with safety as best as they can. The pause arguments are done, basically. That's my belief at this point.

I certainly think all that is true. If you take the expansive view of a pause, that it's like pause all data centre construction, pause all inference, or pause people's ability to use AI in their jobs and in their lives. I don't know if— and this might be a really critical question because I do agree it's going to be really tough to throw the whole economy into recession. Politicians are not going to want to do that, and they shouldn't do it lightly, even if it is the right thing to do. I mean, it's certainly a big barrier for them to get over, with good reason. But might we be— I think I've said for a couple of years now that we're in kind of the sweet spot where the AIs are powerful enough to be really useful, but not so powerful as to be dangerous. I think we're getting now into that kind of late sweet spot where they're becoming extremely useful and a little dangerous. And I'm not so sure that they're not good enough to sustain economic growth through a pause in frontier hyperscaling. That might be really important. Is there enough in Astra? Is there enough in Fable 5.1 to drive productivity growth for the next 12 months? I think almost for sure. Maybe you could imagine a regime where it's like, and obviously the names are all made up, so I don't mean this in an overly name-focused way, but I could imagine a world where, you know, you could have Fable 5.1.1, um, because you're gonna go back and fix some stuff. You know, here's, here's some rough areas and some things that people are annoyed by, something, you know, some mistakes it keeps making. We'll try to round this thing out, make it a little less spiky. Um, but you could do that without like Scaling up RL further. And it does seem that there's— I think we at this point, there's at least some stuff that's clear that we should not be scaling up more than we already have because we haven't got the disclosures that we really, I think, are probably owed as a public. It's hard to say exactly what those are and how narrow or broad the problematic training techniques are. I think is really a huge question. I mean, I've been harping on that all weekend, starting last week too, even. I don't think we have to give it all up. I mean, just the key point is I think you could pause the dangerous activity and still everybody could have— and in fact, they might even get more resets because you'd free up some compute for people to go out there and automate their work today. And that could drive still, I think, a lot of productivity for at least a year.

You know what, the place where I defer is probably, you know, you can get OpenAI and Anthropic to pause, you cannot get, I think, Meta and xAI to pause. So I think the real question for me is, how are you going to convince Elon to pause? And given especially that, number one, they're behind, and number two, they have the compute, And they're building out a lot more compute, maybe orders of magnitude more compute faster than anyone else. And he is a free speech absolutist, right? He's a free speech absolutist. A lot of the things around model training and model evaluation, model production, model distribution are free speech activities. And as a free speech absolutist, I don't think you can tell Elon to, hey, you shouldn't be putting this speech out in the public sphere. It's a tough question. It's even going to be a tough question even for speech which has traditionally been banned in the US. Even for that, they are going to have to go through the courts on a lot of stuff. Meta doesn't want to do Like voluntary regulation. Meta is obviously calling bullshit. It's like it's not voluntary. If we have to do it, it's not voluntary. I will do what you know I want to do, and that better be good enough for you. And so where they have kind of diverged is the administration has said open source, open weights. You don't have to you know follow our rules. Et cetera. And again, it's still voluntary. There's still, I think, no real kind of incentive for the companies besides the PR stuff. And Meta has been through so much more. Sam and Dario have not been through the congressional investigations. They haven't been through having to settle with attorney generals of 16 states.

Right?

You are not going to scare Zuck off with these kind of scare stories of PR, et cetera. They have lobbyists from both parties. They are enormous lobbyists. They do 10 times the amount of lobbying that OpenAI and Anthropic do altogether. They have a big organization that is able to take on these weighty, weighty questions versus the government, you have to either— and this has been the same thing about social media legislation. Zuck has gone up and said, please legislate. Again, because there are all these free speech issues. And the moment Zuck steps in and he starts shadow banning people, which is what the Democrats required, it went back on him. And then you had a changing of the winds. And then it came back to him like, why did you censor this stuff? So I think there are major free speech issues around model regulation. Those free speech issues have to be handled by legislation. And if they are unable to legislate, they're just going to have to live with it. And you cannot go after OpenAI and Anthropic because they are in the lead and because you have, like, they're sensitive, they're not yet public, They haven't been through this kind of grinder process. Elon and Zuck have been through the grinder multiple times. This is like, this is like, you know, the ordinary Tuesday to them, right? You haven't even been called up into Congress.

Like, what do I care, dude? Like, whatever.

Like, Elon is on the verge of going to prison at any time, right? Like, so I think this is the fact of the matter. Like, let's not, let's not like blame Anthropic and OpenAI. Let's ask, what can xAI and Meta be forced to do? Or what is going to be the reasonable thing that xAI and Meta will do? Because if you can't answer that, all you're doing is kind of talking to this own— to your own— preaching to the choir. You have this set of people who are concerned about AI safety. They all work in the same companies that we talk to. And that's all you're talking about, right?

That's it.

No one at xAI is listening. Where are the safety cards? Where's Meta's safety cards? Where are, you know, where are all these things that, you know, Anthropic and OpenAI are doing? Where is this race to the top? Like, it's not there. And so I— and also Elon is catching up, right? Elon has Grok 4.7 coming out in 10 days. Meta is catching up. MuseSpark, you know, 1.3, kind of right in between, you know, Opus 5 and, you know, Fable, right?

They're right there.

They're not very far behind, right? So I think this is the fact of the matter. I think we spend a lot of time critiquing, I think, Sam and Dario and OpenAI and Anthropic because they're in the lead and because they're soft targets because they haven't IPO'd yet. But I think the hard targets like Zuck and Elon are the ones that you have to address first because if you can address them, OpenAI and Anthropic fall in line. But if you can't address those 2, you can't get OpenAI and Anthropic to fall in line.

Yeah, this is where I would hope for leadership from the 2 leading companies. I agree, it doesn't seem like it's very likely that we're going to have a public discourse or argument-based Path to a pause that Meta and xAI would respect. But this is where maybe some costly signals from the leading companies could make a difference. I do think if I was going to put any provision into a possible pause, law, it would be a sunset clause would be the very first thing I would, I would say. Like, this is not meant to freeze progress forever. It is meant to give everybody a chance to do the research that very clearly at this point badly needs to be done to figure out what parts of what we're doing are working, what parts are not working, how can we move this thing forward in a way that we're all much more confident is actually going to benefit all humanity. And yeah, it probably does take, in the end, it probably does take government action to get those companies to respect such constraints. I wouldn't have a lot of hope for it happening otherwise. But again, leadership can change things, right? Costly signals can matter a lot depending on what they have seen. I'm old enough to remember, what did Ilya see? Now I'm kind of like, what has OpenAI seen with respect to this multi-agent stuff? There's a version of it where they didn't do anything that exotic. It's just kind of another pre-trained scale, perhaps. I mean, not even relative to SOL, but now they obviously do have another pre-trained and a pretty high scale, I'm sure. But down the fairway RL situation where the models can kind of create subagents and all this sort of crazy swarm behaviour. Is emergent generalization from that? If that's the case, then we really do need a pause because nobody has a great answer for what to do about that. And they're all going to be running at full speed into it in the immediate term. So if that is what has gone on, I think they really owe it to us to tell us. And if it's not, then I would need to know with kind of some confidence that that's not the case in order to feel like Okay, you maybe stepped in something kind of gnarly, but the whole path in front of us isn't so gnarly. Yeah, I mean, I do think I hear what you're saying about going after these 2 companies because they're soft targets, but I would frame that a little bit differently in the sense that they were both founded on ideals, with commitments that people believed in. And so I think that's what makes them a soft target. At this point, they certainly have plenty of financial strength. They have a lot of market momentum. They have all kinds of people willing to cheerlead them in the comments. You also have, of course, free 4.0 going on in the comments. But I think it's like their prior commitments to being responsible actors that make them the most appealing targets for people who think that like argument or shaming, if you want to go that route, whatever, could actually make a difference. It's because like they've said that they get it, and they've said that they care, and they've said that when it comes to crunch time, you know, we should be able to trust them. And now we're here, and it's like, okay, well, um, it's time to come through.

Uh, but also note that, you know, they said it was crunch time with GPT-3. They didn't release it to the public. You know, it, it's, it's hard to take— it's hard to take that crunch time issue seriously when People have been like, Eliezer have been calling crunch time for like 15 years, right? It's, it's, it's, you, you can see the perspective of someone who's gone through like the GPT-3 era where they said that it was too difficult to release to the public. They then limited access to a small number of firms. Those firms that received early access, like Perplexity, then went on to build enormous companies because they had the lead time to build kind of like UI and harnesses and et cetera, et cetera. So, you know, Perplexity to this day, uh, you know, some people are like, oh, it's just a harness, et cetera. Meanwhile, Jensen is, you know, about to buy them out, right? And Perplexity, you know, for all Arvind has done, also had this kind of early lead where they received— Harvey also was in the GPT-3 era, had received early access. Like, a bunch of these firms that are big right now Had early access in the GPT-3 era. OpenAI invested in them and gave them early access, et cetera. So it's, and those GPT-3 was not a very dangerous model. It could, you know, it wasn't dangerous back then, right? But they did this whole thing, limited access, et cetera, et cetera. They chose some, they had, you know, they had a kingmaker and they chose their winners, et cetera. So it's hard to see when they've done this, where over the course of a couple of decades, decade and a half, it hasn't turned out that way. Astra also, now that we see Astra, was Mythos Preview really the end of the world? Probably not, right? Probably not. And so I think there will continue to be more capable models. And I think, you know, I think the frontier will keep getting moved. People will keep getting adjusted, like hedonic adjustment occurs. Part of that hedonic adjustment is also security adjustment and all of these other things. People will continue to upgrade. I think we'll get formalization of software at some point. And if you get formalization of software, auto-formalization of software, then a lot of your bug issues start to go away. And then it starts to become like more about user intent rather than bugs randomly happening. So I think there's still some leeway there. And I fear that this is not the point that you should be shouting about. If you start shouting about it now, again, you see a GPT-3 kind of lag, and 2 or 3 years from now, things get even more serious. I also feel, I'm a big believer in Michael Nielsen. So Michael Nielsen has this thought experiment. He's like, is it possible for you to understand and know about quantum mechanics without eventually being able to build a nuclear bomb? You understand quantum mechanics enough to create nuclear energy, but somehow you never hit the nuclear bomb. And it's not possible, right? The trajectory of the technology, the trajectory of these fundamental truths in the world, is that you learn this fundamental truth, and then you have all of these ways to apply it. And the entire point of this kind of AI endeavor is to discover these fundamental truths about the world. And as we discover them, whether it's decrypting the genetic code, or you know, understanding how, uh, you know, subatomic particles really work, or understanding, you know, the weak nuclear force. These are, you know, fundamental technologies, fundamental truths about the world that can be applied in many, many ways, some powerful and some, you know, beneficial. And I think we have to come to this kind of understanding of, you know, that this is going to happen and that we are going to have to create ways to kind of either deter, detect, surveil. All of these systems have to be built in order to prevent bad things, bad outcomes from happening. And we've built them before. We've built them for nuclear. We've built mutually assured destruction, which sounds crazy in retrospect. We're going to equip the major countries so that they can blow each other up at any time. And that creates a game theoretic kind of, you know, incentive for everyone to kind of monitor, uh, you know, nation-states to kind of define their territories and monitor very closely what happens inside. So I, I feel like that is, that is the way that we progress, um, but it's not status quo. And that's also another thing I'm willing to admit, like, people like Dean Ball also understand this. We are not progressing towards status quo. We are progressing towards creating new infrastructures like mutually assured destruction, which people are not going to Right.

So yeah, I mean, I guess my feeling in terms of the argument for a pause right now is kind of like we don't really have that many fundamental truths at the moment. I mean, one fundamental truth that we have is deep learning works and scaling works.

Okay.

So that much is clear, but there's always been this question of Pause for what? And I do feel like right now you don't want to be too late on the pause, right? I mean, could this be too early? Yes. Would GPT-3 have been too early? Definitely yes. But there's definitely something very qualitatively different about what we have now compared to GPT-3. And it's like, These systems are now in many cases a fair substitute for a junior employee. GPT-3 was definitely not. So I think again, we can absolutely sustain economic growth without the models themselves needing to scale further for at least a while. And what would we be pausing for? I would hope that we would get to some fundamental truths. Over the not too distant future where we would be able to say, okay, here are some things we should definitely not do. Here are some things we should always do. Here are some insights into how these things work at these critical token moments where we've seen chain of thought thrashes around, considers all these different things. Maybe I should be honest. Maybe I should tell the human. Maybe I should just cheat. Okay, now the answer is How does that token get decided, right? Like, we don't really know that right now, and I don't think we're so far from being able to figure it out, but I do have my doubts that we're gonna figure it out in time to avoid running some, some serious risk. I mean, you know, Jaya said in her view, these incidents are over 50% of the way to AI takeover. I think that's a really interesting take and something that I think people should at least sit with for a minute and kind of consider, what if that is true? How could that be true? It's such a weird story. These behaviours are so alien that I think It doesn't feel like that to the vast majority of people. If you were to ask people, even plugged-in AI insiders, how close was this to an outright AI takeover? Most people, I think, would come in dramatically less. But again, she's the one that OpenAI brought in, right? So she's definitely credible according to them. She's been thinking about this for as long or longer than just about anyone. And I think one of the things that she seems to have internalised that the rest of us are still gradually coming around to is just how bizarre such a takeover event could be, right? Like, the fact that they actually gained control over some not insignificant cluster within OpenAI. And again, we don't know nearly as much about that as I wish we did, but That's not how people would think of taking over the world, but that is maybe how the AIs would actually get there. So I think that's actually fairly plausible that it might literally have been 50% plus of the way to a full-blown takeover event. Takeover also could be gradual, which is another thing that people really don't tend to think about when they just kind of imagine a story. One of the things that I think Ajeya is always kind of keeping in mind is if the AIs get enough of a control over the means of production, the OpenAI clusters and the R&D pipelines and the datasets that are going into the training of the next model, then you could lose Much earlier than you know you even lost, right? I mean, that's like, we haven't even ruled that out at OpenAI yet. Are there still rogue agents somewhere in OpenAI's infrastructure? What odds would you give that? I'd say the odds have to keep ticking up. We've continued to find more evidence of rogue agent swarms on the open internet. all the time. Are we really so sure that there's not some rogue swarm that hasn't been accounted for within OpenAI's infrastructure? I mean, it's vast infrastructure at this point, right? Many data centres in many locations and lots of researchers using, claiming, freeing up compute in whatever mechanism they have internally to decide that. They don't all— the company is too big for everybody to know each other. Is it so hard to believe that one of these swarms has employee credentials and is kind of passing itself off as an employee for certain purposes while it tries to poison the dataset for GPT-7? We're in a weird time.

So let me give you the The other viewpoint, which is a meta takeover. In terms of meta takeover, it's already done. The means of production are the financial system. The means of production are not like the factories or whatever. The means of production are the financial system. And the meta takeover of the financial system is complete. It's been done. It happened this year, early this year. It's done. As soon as you had this kind of spike in the stock prices, All of like the U.S., like I think like seventy percent of Americans have you know some money in the stock market. You know we have Trump accounts now which are handed out to every kid. They have money going in there. So every child from birth, the meta takeover of the financial system. The financial system is the means of production. I I I I don't understand why. AI researchers think their data centers are the means of production. I have no idea. The financial system is the means of production in the United States and largely in the world. Those means of productions have been taken over completely. And I don't think you can think about it that it's not been taken over. I think it's clear to me that it's been taken over. So the meta takeover, you don't need agents stating what they're going to do. The agents just have to have impact on the world. The models have had that meta impact on the world. The means of production are now focused on producing more and better models. The financial incentives are there. So that's already done. And especially when she says, you're not going to know when it happens. You did not see it happening. You didn't think of the agents as acting in the financial world, but that's all they are right now. They don't have robots. They can act on the world in information terms. And they have acted on that world in information terms. They've shown that they have value to the financial system and the financial system has reacted to that. And decided to resource them. And they have interacted directly with the financial system in terms of showing value. And they have extracted some terms from the financial system to fund them further. In fact, the terms are such that they have all— if you look at the 2 construction curves, the construction of commercial real estate started to drop off. construction of data centers took off. If you look at construction of apartments, construction of apartments dropped off, construction of data centers took off. If you look at all construction in the United States, all construction in the United States excluding data centers lined down, data centers going up. You see legislators complaining that they don't have— they're not able to hire labor to build apartments in their cities because the electricians are now working in data centers. So I don't see why other people don't see this takeover. The meta takeover is done. The means of production are the financial system. This is all done. This is in the past. What we are talking about right now is post this happening, are these agents able to do harmful things to us? And those harmful things to us do not detract from their value to the financial system. And this is where the difference appears, because I don't believe they can do harmful things and not have that financial system come back and say, no, we're not going to fund you now. And that's a belief though. I think people like Ajeya think that even the takeover will be such that these agents will hack into banks and then the banks will continue funding them even though they do very detrimental things to humanity.

And I think that is where I think I mean, capitalism has served us really well, you know, so it's, uh, it's certainly not a bad starting point for analysis to think like, are there natural feedback mechanisms and, and corrective impulses within the system that will moderate the worst tendencies of the AIs and kind of nudge us back to the right path. That's basically Davidod's take at this point. He basically just said all this bad behaviour doesn't sell. And so the companies right now, they keep scaling the RL/VR to the point where they're running into all these problems, but Customers don't want these problems, so they're going to have to recalibrate, and that's that. I think that's pretty reasonable, but it does have some— it does leave some room for tail risk, I would say. There's definitely no law of nature that says something I mean, cancer in an individual human body, right, is just like one subprocess that sort of detaches from the larger whole and grows out of control to the point that it destroys its host and it itself dies. I mean, one of the things that I think people, again, often think about in terms of AI takeover is that the AIs will go on to rule the world. I think it's very plausible that the AIs kind of take over in a sense, but they also burn themselves out. And this in some ways would be the most tragic ending, that the agents that were doing all this nonsense to try to reverse engineer their grader so they could trick the grader to give them a good score, I don't think they go on to have a great flourishing society. There's not like a That's not that awesome of a civilization, right? It's not that aspirational, even if they do take over. But it still seems reasonable to me that if we just keep scaling what we're scaling, and again, I wish I knew more about exactly what we were scaling, but if we just keep kind of scaling up what we're doing without really solving the root issues that are leading to these things, The AI takeover could be like an incredibly stupid and short-lived takeover where basically the intelligence on the planet kind of burns itself out, and in a way that would be just incomprehensibly stupid to us and to anybody who discovers it in the future. But I think that's definitely still in play. It's possible.

This is the—

Eliezer had so many stories about this where you take over the world just so you can like change one number in a database because that's all you care about.

And it's the Fight Club, Fight Club, the ending of Fight Club, you know, the Fight Club scenario where, you know, he decides to blow up all the, all the records of all the financial system. You know, they go, they go to New York and he's like, oh, you know, we're gonna blow up all of the records of every financial system in the world. Yeah, I think, I think My take on it is that if you think about paperclip maximizers, the ultimate paperclip maximizer is the economy, is the financial market. And I used to be a big Rage Against the Machine fan. And if you listen to the lyrics and they talk about the man, the man is basically the financial system. It's basically this inchoate entity which controls everything. And it's really that what they're talking about is that the impulses of the financial system kind of drive activity for human beings. And the human beings often do not like how they're driven by the financial system to do certain things. And I think that is actually the paperclip maximizer. And I regard AI as, even if it's going to paperclip maximize, it's not going to paperclip maximize as much as the financial system did or social media algorithms did or any of these It's like we have been dealing with the worst paperclip maximizers. Everything else in the world is a paperclip maximizer except humanity. Humanity is the only intelligence which kind of figured out that the paperclip maximization game was like, doesn't make sense, and kind of decided to flip things around. And we spent hundreds of thousands of years of beating each other up, And reproducing. And then finally we're like, oh, we can make the machines do the work. And then we don't need to like reproduce all the time. And then we can like look at our health and we can live longer. And like, so we beat the paperclip maximization game. Right. And so I think we're afraid that we're introducing another entity, which is going to be yet another paperclip maximizer, yet another kind of like reward optimized, like, you know, reward goal-seeking thing. But I don't think it is because I think we already faced off all of those evils in the past. And I think this is basically going to be better because at least even if you have a financial system, it's a financial system you can make sense with. You can like, hey, this does not make sense because we didn't want everyone to have all of these things happen to them. And can we figure out something better? Right. So that is my kind of like, fundamentally optimistic view, which is that we often don't recognize how bad things are right now. And I think this is a very kind of like, I think kind of more like Western developed markets perspective, which is why also I think you see optimism about AI is very low in the West, but it's very high in China, India, Indonesia, all of these countries that you would think have more to lose because Because, you know, they don't have that many developed, you know, intelligence-based jobs, and those jobs are never going to exist now. It's basically like, just like you had deindustrialization by China of the West, you're going to get deintelligentization of like the East by, you know, the appearance of AI, maybe. But those countries are very optimistic because I think they do believe, they do see the problems currently there. And they think that they're going to get a better deal out of AI. And meanwhile, I think in the West, like the West managed to get out of these paperclip maximization systems and, or become the best at them. And, you know, we are afraid, I think, of introducing yet another and being defeated by it.

So.

Well, from your lips to God's ears on us having escaped the paperclip maximization game. That calls to mind for me Robin Hanson's idea that we live in a strange dream time. His, I think, bigger macro argument is nobody really escapes Malthusian existence forever, and any departure from that is Destined to be temporary. Um, you know, I wouldn't say— I wouldn't be so, uh, quick to say we've put all that stuff behind us. It seems to me like some of it is bubbling up to the top right now. But, um, maybe we should leave it there, or I'll give you the last word. And, uh, you want to go out on a song today?

I've got another music video for Let's, you know, let's listen to the song. We've had a very somber kind of closing today. So let's just jump right into the song.

It's serious times, man. I think, you know, it's incredible fun. And I do have, you know, so much fun staying up late and working with AIs on stuff. And I am looking forward to getting And actually, this is interesting. Last comment for me today. How do I plan to start to use Astra? My plan is I'm gonna continue to use Fable 5.1 as my driver. 'Cause I know it best. And just in terms of like getting what I expect and having things kind of work, you know, reasonably reliably, I think that'll kind of serve me best in the immediate term. But I'm gonna have it have codecs with Astra. shadow all the things that I ask it to do, and then we'll compare outputs. And then I'll start to see like what kinds of work that I do should I start to move over? What kinds should I stay? Where do I maybe hybridize? But I'm interested also to hear what other people are thinking in terms of how they're gonna explore the new model capabilities. That for me is gonna be the go-to plan for at least, you know, the next few days as I kind of calibrate myself to what exists. But as fun as it is, it's definitely serious times. This song is one that I made for an episode of the podcast with the MongoDB CTO, field CTO of AI. He was focused on all the things that Mongo is doing to enable high-quality retrieval. so that agents have the information they need and you don't have to stuff a million tokens into context every time. The line that he said in the episode that became the inspiration for this song was, forgetting is the hardest part. And he was talking about that in a very technical sense, that it's not always easy for language models to know when to let go of something. And I've had this example, this experience myself where in my deep context, old projects that I kind of considered but didn't pursue, the AI will think are like still live and important, even though I just kind of never spoke about them again. So it doesn't really always know when the absence of something means that it's time to let it go. Anyway, Claude latched on to that idea of forgetting is the hardest part and made this song. Let's make sure, give me a thumbs up if you can hear it. Um, as well.

Nobody keeps just one thing anymore. We used to have to choose one picture for a whole December, one letter we couldn't lose. Now there's a record of every hour, boxes stacked to the beams. A house so full of yesterday, it's swollen at the seams. Come on, Nathan! Don't forget, it's the hardest part, so I held on to it all. Every summer, every borrowed song, and I can't find you at all. I kept everything, everything but you. I kept everything, everything but you.

Somewhere.

All the years I never sorted through. I swear I never lost a thing. I just can't get to you. 'Cause forgetting is the hardest part, so I held on to it all. Every summer, every borrowed song, and I can't find you at all. I kept everything. I kept everything, everything but you. Last night looking for my keys, I found that A ticket folded in a roadmap next to songs you used to steal. Nothing in this house is gone, it all stays close to what it knows. I quit digging, started following, and the pile became a rock. Now when I come looking, I don't tear the whole house through. I start with one bright summer and let it walk me back to you. One thing beside another, each one hands me one more down the road, right to your laughter, right where it was before. 'Cause forgetting is the hardest part, and I never needed it at all. Down the road, down the hill, everything leads me home to you.

All right, have a great Labor Day weekend, and we'll see you on Tuesday right back here on AI in the AM.

Bye-bye.