48: Tour of a Humanoid Robot startup in San Francisco

All right, so who are you and where am I?

Hi, I'm Sinket. And you're at the offices of Foundation.

Foundation. And what are you guys doing here?

We're building humanoid robots.

That's my favorite topic.

I'll show you around.

Yeah.

So these are some of the robots the team's worked on over the years. So the team's been working on like the bipedal locomotion stuff for a long time. This one was for NASA called Valkyrie. And then this is just like another research unit that went to DARPA Robotics Challenge came second. So a lot of the stuff around, from our actuators to our control stack, often has been in the works for a long period of time. Then we took all of this technology and kind of commercialized it into Phantom, which is like our first robot. So here you're seeing it do a bunch of different pilots. Historically it's been the upper torso with like an AMR wheelbase.

And you use a term I haven't heard before, what's the amr?

Oh, it's just like this wheeled robot that goes without legs. It's not bipedal.

Got it.

It's about to be bipedal. We'll show you legs in a bit. But yeah, so those are the kinds of things it's meant for industrial use cases. So our first deployments are happening in auto manufacturing.

Okay. And then.

Paper goods manufacturing and consumer goods manufacturing soon to follow.

So kind of. So first question, why humanoids? Why? Because I keep hearing people ask why? Why not a little box? Why not a spider? Why not, you know, some weird form factor.

Yeah, well, so I think with foundation, the question we've been trying to answer.

Is.

What form factor enables you to deploy the largest fleet? Because that's how you're going to be able to train like a Foundation model to be able to do almost all generalizable work that humans do today. So I spent with my co founder, Mike, and I spent about six months from January till May in factories last year. And by and large, the thing that became very obvious to us, there's a reason why most of these tasks have not been automated. A portion of it is definitely the AI side, which is you didn't have dynamic vision based reasoning. But the second piece is the form factor. A lot of these single hand, single use robots are very restrictive in the workflows and everything. And it turns out if you actually want to do the job that humans are doing, you have to be similar to a human form factor. You don't have to be exactly Like a human form factor, you do need two arms and if you want to go up and down stairs or move around in a very tight space, legs are better than wheels. So you end up arriving at something that looks like a humanoid and then you can try very hard to make it not look like a humanoid. But at that point, why do that?

And you're aiming only for industrial kind of use cases today, which means your robot can lift a lot of weight and do repetitive things millions of times times without hopefully breaking down. That kind of thing.

Yeah, we think like the, there's lots of low hanging fruit sitting in manufacturing and logistics where the technology at this point is good enough to be able to deploy for homes. You need two things. You need it to be more safe and then obviously you need it to be more autonomous. I'm not convinced that a lot of people would allow hotel operated robot in their homes.

No massive invasion, prime stage. Yeah, yeah. My wife walks around the house naked. Ah, you can't do that. You don't want joe at 1x tech company watching it. Right, Exactly. Me, I, I would put up with that, but I'm a nerd and yeah.

Yeah, I think a lot of us are early adopters. Sometimes we think the whole world is like that, but the reality is most people won't. So I do think a lot of the use cases would be industrial. I also think that's the most pressing issue to solve as well. Well, because in the next 30 years we would not have enough humans in the labor workforce because of like the declining birth rates. So we'll still have about the same population. It's just the age pyramid would invert, so you need to be able to automate all kinds of labor, otherwise we would suffer on goods and services.

Got it.

So I do think it's kind of.

Like priority number one, which maybe while we're walking around and getting a look at your company. What, what's the state of the art of robots today? What could they actually do in a manufacturing facility? Because I know your competitor Tesla has a few dozen working over here at Fremont factory and you're in different factories. What's the state of the art right now?

Right now the robots can do most of the pick and place and move tasks with the exception of fine grain.

And when you're saying pick in place, I saw a robot picking something up, moving it over. They're not really snapping together wires or getting underneath the dash and screwing things in.

I think there are aspects of that you will probably see in next 12 to 18 months. Then the other aspect requires abduction. So like your fingers need to be able to expand like this. A lot of our dexterity comes from this. Most of the humanoid hands only do this today.

Got it.

They cannot do that. Some of some Chinese hands, new hands can. And one startup in San Francisco, X Tesla people, they're pretty much building a hand that can do abduction and everything. So that would be the next iteration in the hardware that people will do which would then enable you to do more fine tune manipulation probably like two years later from that. But you'll see definitely some kind of manipulation to your point, like connecting two things or something like that relatively soon. But again there's like so much lower hanging fruit in just being able to move things.

Yeah.

And if you can do that quite well, you have like thousands and thousands and thousands of robots you can deploy.

Interesting. And how much are your robots? Let's say I was a Ford or a Chevy and I needed a few robots to help help out in the factory. What's the state of pricing?

About $100,000 per robot per year. So we essentially charge like a subscription model. And then in that we, we do all the maintenance, repair everything with the robot. It's costing us right now about 150 to make them. We'll be able to probably bring it down to 100 in, in short order and then we'll have to probably like make some design changes to be able to bring it down further.

Got it. They're built for industrial so they're over engineered for a lot of lightweight cases.

Exactly.

I see a guy in VR back there doing some training.

He's your friend?

Yeah, yeah. I'm wearing an Apple Vision Pro. That's why I brought, I brought it because I, I had to do spatial computing at a robotic company. You know, I can't do it it on my iPhone.

So here Anant is testing this latent variable model which is essentially like a state based model that we use. So what we do is we do imitation, learning, but we combine that with this state based model that essentially compresses the physics, dynamics, kinematics of the taskbase. So it can, it can predict Anand's hand motion, where it's going to go and what his intentions are and kind of like the relationship between his hand and the cube, all these different things. So the robot can essentially start moving as pretty much a nunc is moving versus wait for none to stop moving and then the robot starts moving. This cuts down your response time by half. But then the other thing that it does, which is even more valuable is with less data, you can reason through your space far better. So you end up with more robust models than you would with just pure behavior coding. So that's kind of like proprietary piece that the teams developed. And Patrick, who essentially leads our AI research team, has been working on for about a decade.

So that's awesome. Yeah. And what's he seeing inside?

It's not, it's not what he's seeing, it's more. So currently he's using the VR cameras to give perception to the robot, but soon enough we'll just put a camera up here so we would not need the VR headset anymore.

So the robot is using the cameras in there. He's connected to the Nvidia card down there.

Yeah, exactly. He's connected to that machine. And then the robots using his camera's provision does localization to know exactly the relative coordinates of his task space and then operates within it.

And how many millions of times does somebody have to do this to train the robot to really nail it? Every time?

30 minutes. 30 minutes, yeah. But that is because of the type of model we're using. And also it's a simple task. This is a simple task, but this is all. This whole training is done with 30 minutes of data. Okay, but that's like essentially Patrick's point, which is if you essentially just add a reasoning layer with like an autoregressive transformer, which is pretty much what we're doing with chat, GPT and bunch of things. And you have this like O1 models that essentially are LLM plus a reasoning model. The only difference here is you don't want to do reasoning through reinforcement learning because there's so many reward functions for manipulation. It'd be very tedious task by task. It'll take you too long to build a good model. So you need to compress the physics, dynamics, kinematics of that specific task.

And in the past 90 seconds you used a lot of terms that I. I sort of understand auto regressive and diffusion model. GROK can actually teach you all about that. Right. So I wish GROK was here translating you to English.

I can try to translate in English.

Yeah.

So the architecture that people are going with in robotics right now is like this one end to end model that is able to take data from your perception and then spits out kind of like action tokens. Very similar to an LLM split. It takes text and spits out text tokens.

Okay.

So now you have same transformer model that takes an image and a bunch of variables in that image and they spit out what the predictive action tokens are.

Okay.

Now the thing is these models take a lot of data before they can get really robust. So usually if your training data only had scenarios, for instance, the coffee pod was in the left side of the coffee machine and then we kept on training it with that. If now the coffee pod moves to the right side of the coffee machine, that breaks your model because it doesn't know exactly how to, how to plan and how to do the task. Because kind of like trying to extrapolate based on the data, it's seen if you combine that with a reasoning stack, which in our case we use a state based model, you can also use a reinforcement learning type model that essentially has a compression of your physics of the task. So how objects move, what their poses are, what's the relationship between those objects, what their dynamics are, what their kinematics are, things like that. If you compress that in a model and then you combine it with this like, like action token prediction, it can essentially reason through that space a whole lot better. Very similar to an LLM reasoning with like an RL model that GROK has chat, GPT has now like the, the chain of thought. So this is kind of the chain of thought for robots, but in physics it's amazing.

A lot of computer science behind here. Yeah. How's. How are we going to get to the world where we're going to tell our robot, hey, just watch this YouTube video and learn how to make a new meal or I was just hanging out with people with old cars, learn how to change the carburetor on this old Jaguar or something like that for.

You to be able to get there. You're looking for something called a zero shot prediction. What that means is it's never seen that data before before, but because it's seen so many similar things, you see emergent properties where now it can do things it's never done before.

Yeah.

The precursor to that is deploying very large fleets that can collect a lot of that diverse data over time. So as these robots get deployed more and more and more, the more you deploy them, the closer you get to that zero shot foundation model where you can literally just like go, go and ask it to do something it's never done before, but it's done similar things so it can essentially like fill in the gaps and then do that task.

Tell me how you view the competitive landscape in this. Because you're building a company that's humanoid. I mean there's figure There's K Scale, there's Unitree over in China and a bunch of Chinese companies and obviously there's Tesla with Optimus. How far ahead or behind or you know or what's going to be your, your market approach? Yeah, because I know you know what the competitors are doing. Everybody gets together at parties here in San Francisco and compares notes like, hey, what does your robot do?

Well, the market kind of like splits in a couple of different facets and when you, once you split it out, like there are a few people that we compete with and others we don't, consumer or industrial. So like 1x and case scale and all of that kind of like go there inside. Also commercial. You have companies that are so focused on just the R and D, they're in love with R D. The Boston Dynamics. And there are some companies that are like have high sense of urgency. They want to ship, they want these rewards out in the field actually doing work. There are literally only two companies outside of foundation that are like that. Tesla and Figure. Yeah, so technically those are like our competitors and even Tesla is teetering to more consumer. So our real competitor is Figure. So it's like the one we're competing against is Figure. And we've taken like last year to just catch up and now the next milestone we're about to cross, we probably surpass them. My goal by end of June is to have our first fleet do full shift of work. Actual work, factory, just the auto manufacturer. If we do that where Figure is today, we'd be ahead of them. If Figure gets there, then we'd be like neck to neck at about the same time. But it's that kind of like you have to have very fast iteration and shift velocity and then you have to make the right technological bets today. But those techn, those technology bets are not going to be your most. Like eventually if what you're doing works, then everyone else is going to copy it. So your only real mode is like moving very fast and building very fast and iterating very fast. On the technology side, we do have like two big advantages. One is this model that I just showed you. It's like kind of this physics reasoning model that makes it so with less data we can build highly autonomous and robust models much more earlier on which other companies cannot do, which would enable us to deploy much larger fleets, much smaller, sooner than everybody else. Because if you're tele operating these robots, if you deploy 10,000, you need 30,000 tele operators just to be able to do three shifts. It's completely Impractical.

Wow.

You have to be able to do high degrees of autonomy. So this is our one big advantage. Second, we have a very different philosophy on actuation of the robots. So like essentially the motors that move all of your muscles and everything, all your joints figure Tesla, they're all using these cycloid disk actuators that are have been in industrial robotics for decades.

Right.

And what they're really good at is going from point A to point B. Pretty much millions of cameras.

Right.

They can just keep on doing it and there's almost no drift. It could always go to point A because it will always go to point very precisely. Very, very precisely. It was very important when all of your software stack was just written in code. If this, then that you can just have this whole thing built in a world where you have models that you can just train with more data that going from X to Y, that kind of precision isn't the most important thing because you can any kind of drift you can correct with vision, just like how humans do.

Yeah, because if I try to touch you and I'm not perfect, I fix.

Figure out how to feedback and then you do it. So now you have this feedback loop that you didn't have in the software stack before. So we don't think that's the most important thing. The most important thing is how efficient is your gearbox, which is how much is its wear and tear, how much, how powerful can you make it for a specific size? So your string wave, all these harmonic drive actuators that figure in Tesla using like 50 to 60% efficient. What that really means is about 40 to 50% of the torque that comes out of the motor is lost in the gearbox before it translates down into like, like a motion on the robot, which obviously makes your robots less strong. They can do less. But then the second piece is the wear and tear is very high. The heat dissipation is very high. The gear wear and tear is pretty high. So your replacement rates are much higher. What we've done is we've essentially invented our own actuator, which I'll show you over there in a second as well. It's a very efficient gearbox. It's 95 to 98 efficiency. And what that does for us is it gives us very high torque output because if you put a very powerful motor on one side, a lot of torque comes out the other side. So we can lift about twice to what we didn't.

Tesla can today like more than 100 pounds.

Not yet, but right now about like 20 kgs.

Okay. Yeah.

About like 45 pounds.

Yeah. That's pretty good.

Yeah. Next version, our aspirational goal is 40 kgs. We'll see if we hit it, but.

So you're designing your own motors here?

We are. We're currently designing our own gearboxes.

We're not designing our own motors.

To be able to double the torque output, we would probably have to design our own motors.

Steal some people from Tesla.

Actually, I have.

You have. That's the Silicon Valley way. Right. And who are you? Crystal.

Crystal Business operations.

Very cool. Thanks for coming on, giving me a tour. Of course. So what are we. Yeah. So what are we seeing here?

So these are the traditional kind of like figure and Tesla actuators. So this is like the harmonic drive actuator. Should try to move it so that you get a sense of.

Oh, yeah, it's hard to move.

Yeah, it's pretty hard.

It has a lot of resistance, which is great in the car because you get regenerative braking and great.

When you have very set assumptions on the.

On.

On the software side and you don't have a feedback loop of correction because they can essentially go point A to point B consistently.

Yeah.

And really good for surgical robotics as well. Well, right. Because you don't want to go a little left on your brain and then.

Correct. Yeah.

But these, on the other hand are Arist. Right. So this is. You can try to move that, the gear.

Much easier.

Yeah.

The gearbox, maybe a quarter. Yeah. Oh, yeah. Maybe even more.

Yeah.

Much easier.

So what that thing has is pretty much like. Like pretty high surface contact. So it's literally like you're moving a brick like this on the table. What we have is more analogous to like rolling contact. So it's like moving this thing on like those kinds of pins.

Yeah.

Which is what this looks like inside. So it's all rolling contact gearbox, which makes it very efficient, very smooth. Also helps solve for a lot of sim to rail gap. Because over there, like, you don't actually know the efficiency.

Yeah. You just said another thing. Sim to realize gap. What is that?

So there's one issue in robotics today where you train models in simulation and then you deploy it on the robot. They don't really work. And one big contributing factor to that outside. So there are two big contributing factors. One, modeling physics, which Nvidia and all of them are getting pretty good at over time.

Right.

It's going to take some time, but we'll get there. But the second piece is being able to model your friction in the gearbox and your efficiency of the robot and like all these different things here, that problem is easier because if your gearbox is very efficient, like 95, 90% efficiency, you can buy and just like model it at 100% and you'll get reasonably good results if you run it in sim. Over here, when your efficiency is like 50, 60% and you need a torque cell to be able to give you feedback on how much torque you're applying and what the backdrive ability looks like, etcetera, Then you have the model breaks, which is when you deploy the models. Then you have to iterate it further on the real robot before they get really good.

Got it.

So this simplifies the life of the AI teams as well, because this they can go and train in Isaac sim and then when they deploy, they'd be very close to being production ready.

Wow. Yeah. Cool.

You want to add something?

And who are you? Hi.

And where are we working on actuation here for foundation? I came from ihmc, where we've done a few different humanoids and a couple.

Different robots with NASA.

Just was here to cover any. Anything that was missing on actuation. Seems like Sam gave a pretty good overview of it.

Yeah, I'm learning.

You're learning fast. I'm learning fast, too. There's so much to learn in this new field. Yeah. What's your job like?

Every day it's a mix of work on the computer, CAD design, revising drawings, communicating with vendors, getting our techs up to speed and showing them all of the intricacies of assembling the actuation. And we glue in our stators and rotors. We have all the encoder integration stuff going on.

Lot of variety. A lot of jobs in robotics. You're hiring, I would assume? I hope so, yeah. Okay. Yeah. What's going on? No, not while doing this. I will if I watch it over again if I'm moving my head too much. So I try not to make other people motion sick. What's really cool is you can watch this in a Vision Pro again. And it's like. It's just like this. It's really cool. In fact, you can clone him and put him next to him, and I can't tell the difference, at least inside the headset, the difference between him and the coffee. And soon we'll have a virtual being walking around as well.

So those are our legs.

Christian, let's come over here.

Yep, Christian's working on bringing them up and starting to walk. Christian, if you want to tell him more about it, please go ahead right now. You See it swing, it's because the upper torso is in mounted, so it's like modeling to have more weight on top, but it really doesn't. Once you mount the upper torch, that.

Swing is going interesting. Now when I was at K scale, it was almost like hopping, you know, cuz it was very early. Yeah. Why did, why does it take so long to go from a really jerky motion to really fluid like you and I have?

I mean it's a function of different technologies.

Right.

So most of your locomotion until very recently has been built on like something called like model based controls, which is a lot of deterministic code. It's a lot of FNL type stuff. You like model in physics and try to model in different equations.

Yeah.

Which is, which is at some point it has diminishing returns to get humans to make it walk more smoother and smoother and smoother. Now the new technology people are using is reinforcement. So we'll deploy an RL based locomotion model in this in like next month, month and a half. And then go straight and walk like a human.

Let's spend some time on RL reinforcement learning. A lot of robotics people are telling me that's really starting to change the game for getting a humanoid done. What is it and why and how are you going to use it?

So the good thing about reinforcement learning is you don't really need data. What you need are like reward functions, which is what do you want a model to, what you want to reward a model to do and not do. Right. So that's the premise of it.

So like walk upstairs with a stack of pizza. Well no.

Like one reward function would be don't fall, don't fall. If you fall, I'm gonna deduct a point. If you don't fall, that's good. So that's incorrect. Then the other one's like stand straight. Right. And so on and so on. They have smooth walking, all these different things.

Right.

So you look at different reward functions that you use and then you just let your hardware run in simulation, millions of iterations of and let this reinforcement learning model just try brute forcing walking.

Right.

Because you've already given it all the reward functions. Now it knows when it's doing a bad job and when it's doing a good job. And then over time it learns Exactly. Okay. How do you consistently do a good job? And that's essentially now your RL based model. So you've trained that model with reinforcement learning where you're reinforcing it through like reward function very similar to like people training AI agents to play like Dota. It just looks at the high score and it keeps on optimizing for that and it keeps on trying different techniques and like it gets better over time.

Wow.

Yeah.

And seems like the learning rate of these kinds of robots is going up.

Because of this significantly.

What is the next, like 18 months? Look, if I come back in 18 months and see you, what kinds of things are your robots going to be able to do that they can't do today because of RL and other AI techniques?

Well, so generally I think over the course of next six to 12 months, it'll become probably six months. It'll become very obvious that bipedals are easier to build than Vero Drop.

Why is that?

Because they're just easier components like electronically. You don't have to worry about a power, unlike a powertrain and things like that. They end up being lower on cost than the wheels end up being. And with reinforcement learning. So the biggest issue with bipedals was being able to learn how to walk.

Yeah.

And because historically we use this like very deterministic code stack where you. We wrote all these if and else statements and math formulas to be able to calculate this. It was a very tedious exercise. It would take months, if not years for you to be able to get something half decent. This control stack has been in the works for years at this point. Right. Because of reinforcement learning, all of that is now out the window. So now you can build your hardware and then get it actually operational and spun up much faster than you could ever before. Wow. So next six months from a hardware perspective. Assembly, sub assembly and bring up off the legs versus the amr. The legs are easier.

Yeah.

Now, now being able to get it to walk is, is also much easier because of reinforcement learning. And that's the reason it's kind of like knock down the barrier to entry. So I think bipedal is going to be actually easier for people to build.

Interesting.

Yeah. We used to have the site what is an MVP product? And even like foundation did that. The MVP product is the upper torso on a wheelbase. If you were starting a humanoid company in the next 12 months, your MVP would be legs would not be able to wheel this.

Yeah. What do you think about the Chinese competition? They're running a marathon this weekend with humanoid robots running around. So they seem like they're ahead of us in a whole lot of ways.

But I think it's.

They certainly have good dancing robots.

I think it's slightly misleading and Some of it is true.

Right.

So the, the misleading part is a lot of these robots that are running, doing kung fu, bunch of these are like four feet, very tiny robots.

Yeah. So yeah, like the unitry.

Exactly. Like they're, it's much easier to be able to kind of like have those robots have very agile movements because they're not carrying a lot of mass around. So they can do it much easier. But practical use cases, they are close to none for those. But then there are second part of Chinese companies are doing like industrial robots similar to ours. And you know, some Chinese companies have already deployed like a thousand plus as of last year, which is ahead of any US based company.

Yeah.

Like we'll probably deploy 150, 160 this year, but still like nowhere close. So I do think in those areas, China, because they have such a massive manufacturing advantage, like they're able to go from 0 to 1, then 1 to 100 much, much faster than US companies.

Can they get motors and parts better that are better than what you can get here in America or are we fairly even? And of course then we have Trunk's terrorists and that's probably making a robot more expensive. Expensive if you're buying Chinese motors.

I mean there, there are some parts like your magnet and things like that. We only come from China because China's doing most of the rare, rare earth mineral extraction out of land. So there's no way of avoiding getting it from China. Now you can do a lot of stuff in house in like do it in the US which is around assembly, sub assembly, bring up coiling, things like that so you can do better, do that stuff. But China is very intertwined to electronics manufacturing.

Yeah.

So it's, it's going to be extremely.

I've been to China. It's insane. I mean here in Silicon Valley we have one big factory, the Fremont Tesla factory there they have thousands. Yeah. You know, it's crazy.

Yeah, yeah.

So you're up against an interesting geopolitical situation of trying to build a company and having to deal with new kinds of tariffs that are changing the costs and tensions. So it's hard to get over there and. Right, stuff like that. Yeah.

I mean the bigger issue is the uncertainty more than the tariffs, which is what if like we just stop sending it couple of these things to the US and a lot of product development for most companies to stop. So I think that's been like the bigger risk.

When you look at a robot, you have, you have the motors that are coming from China, you have legs that are how do you print? How do you make Those?

Currently, about 30% of our components come from China, over 50 come from the US and the rest of it come from Mexico, Israel, South Korea.

Yeah.

So that's kind of like.

And there's a brain of the, of the robot. What's the brain? What kind of computer are you using to run?

Sits in there. So it sits actually in the upper torso of the robot.

Okay.

Currently we're just using like a typical PC with an Nvidia GPU and an intel cpu. Very powerful computer that you like, like your workstations that you, you have one of those. It actually works quite well. Like, I don't think you need, I don't think you need like any big custom computer.

And then you got some batteries.

Well, today we don't. We just do drop power. So when we employ these, we'll do it without the batteries. The reason I did that was because I wanted to just get the AI hardware, electronics, all this stuff working really well.

Yeah.

Which without having to also have this constraint of efficiency. And then once that happens, then forcing function of efficiency to make it better and more efficient so that we can have longer range of the battery makes more sense. And industrial customers don't really mind it. They're fine with it.

Yeah. They have power on the factory floor and you can just plug in a robot. Yeah. Interesting. What are the trade offs you're having in make as you design these early robots, you know, and the entire industry probably has the same kind. Do I put a strong motor in the arm? What's that? Trade off somewhere else.

I mean, one of the very obvious trade offs right now that's a debate across board is how many degrees of freedom you have for any, any given set. Right. Like for instance, hips, like do you want forward, sideways and rotate like that or do you, do you not need all of those worth the opinion? I used to be of the opinion you don't. And now I'm of the opinion. We really underestimate human mobility. It plays a big role when you do these tasks.

Yeah.

Like being able to. Just for instance, I'll show you this thing. If you're doing this task, you have to pick this part up. Then you have to essentially go there.

Yeah. Right.

And to be able to do this, if I need to be able to use perception, I need to be able to move down like that. So now I'm using this, my hips.

Right there to do it.

So like all these small, small things that are like just nuances of doing these jobs that Actually require close to human like mobility because we take it for granted and all these jobs are modeled with around humans. So become like a big pain point if you don't have it. So I think that's kind of like one of the big debates today. I think I'm pretty much on the other side of it, which is I think you need about human like mobility across your joints and body, especially in your waist and hips. Then second, it's like hands, right? Like do you want abduction? Abduction like the stuff that I said or I just want these straight hands.

Hands.

Do you want what Tesla is doing, the tendon based actuation which is like.

So they put the motors down here.

That are pulling on, put like wires and then it's like a puppet.

Yeah.

I think that's probably the right design long term for hands. Hands for high dexterity. In the short term I don't think you'd be able to package it with, with enough reliability to be able to ship a bunch of those. So I think in the short term that would be an issue. Then other trade offs that come are like sensors, right? Like where, what sensors you're going to pick? Which ones are you going to put? Where does that look like?

Battery?

How big is your battery?

Sensors are the eyes and the hands have tactile sensors. Are you using cameras or are you going with millimeter wave radars or something crazy?

Purely use cameras.

So you, you like, you're like elon on that. Cameras are the way.

Well the issue is that sensors are a liability, right?

What do you mean by that?

Every single sensor you introduce in like a hardware system has its own error progression that happens, right? So like they all have noise. So now if you deploy a bunch of heterogeneous sensors, you would think that that would get you like really high fidelity arguable data. The reality is it gets you a lot of noise that comes with it. So it makes prediction and everything much, much harder. It turns out the base case, something you absolutely need are cameras. Cameras get you very far in terms of being able to understand the object, its pose, your contact trajectory, all these different things. Radar is much better for close range, but it's much better for close range also for like very specific things like if you wanted to not hit something, but in case of manipulation you are going and picking apart, you're kind of hitting things like by definition you need to be able to do that in a controlled way. And the highest bang for the buck comes from cameras and all the other sensors, whatever their benefits are. The downsides Outweigh the benefits.

Yeah.

For instance, these lights right now have the torque sensor sensor at its feet. And for the longest time fresh mobile stems were just like beating their head against the wall because that sensor has a lot of jitter coming. So it has a lot of noise that comes out. Makes debugging much harder.

Yeah.

So it's like, okay, let's just eliminate that sensor. So we'll just like eliminate that sensor with reinforcement learning and then we won't need it. We need it for the model based control. It was like one of the inputs, but then it's creating like such massive noise signal that you just want to like remove it. So every single sensor I've learned that you add in the hardware is a liability. So you have to be very careful and think if you actually really need it and only put sensors that you really need. Absolutely need. Otherwise don't.

This totally explains why Tesla only puts cameras in the cars now.

Yeah.

Yeah. Because mixing a lidar in with the camera introduces noise. It's.

Yeah. The first rule of hardware, sensors and liabilities. That's like the first thing that I learned.

That's the first thing you learned. That's amazing. Wow. This is fun. I'm getting a college level class in robotics here. This is awesome. Well, can't do this at Harvard. Sure.

Dan can choose this like one of the applications we're doing for car manufacturing where it's as simple as these parts. Parts come to a conveyor belt.

Yeah.

Right. And then you just go and pick them up and then you go and place them in the studies.

That's it.

And then some, some human comes and some human just like takes those stubbies with you and they take it to manufacture. It's pretty much kind of like that. And this is like a behavior cloning model. And then literally this we weekend we're going to deploy the state based model, the one that I showed you there.

Yeah.

It's going to get like really good. Then we're going to go close.

Yeah. Because you could tell it was off a little bit and it has to figure out can it drop it or whatever. Yeah. Don't. Don't kill us.

So one thing about the actuators that we're building is they're very compliant and that's because the gearbox is very efficient. So you like a human sitting right next to them can essentially push them back.

Yeah.

So they're much safer to be around. You cannot do that with the harmonic drives that the Grand Tesla used. You could by putting like a torque cell that then kind of like emulates this safely, but this is, like, safe by design. And I think that's way better than safe by adding complexity. This is reduced complexity and safe. So we're pretty proud of this piece.

Right.

With this very same.

It's really amazing just your choice of how you design hardware makes other things possible. How much you guys have been building these things for 10 years. What's the history of your company? You learned a lot of lessons along the way. I bet you did it the wrong way and then you figured out.

I started foundation in May last year, but a lot of our hardware technology came from ihmc, which is like the robotics lab in the us and then all of our AI stuff came from Technical University of Munich. So in those two research centers, the same team, because that team's here now, they've been just like, building and iterating these things for a long period of time. And then we thought now all these technologies were converging finally to something useful. So then our goal was, let's commercialize all these technologies in a way that, like, solve their problems.

What should influencers like me or journalists know about robotics? So when we see, you know, Tesla show off something or figure or you.

Yeah.

What should we be looking for? For how real is this or how important is it? Or. Because it's hard for me to figure out. Yeah. You know, who's blown smoke and who's really got.

Yeah.

An amazing robot.

I think it's. I have the opinion everybody who's trying to build is, like, not blowing smoke. It's like, more so where do people sit on the life cycle of how useful the robot is? The lowest bar is. You can record a video of it doing something. The highest bar is. You can literally deploy it where humans are and it can just navigate itself. It can do useful tasks. Now, even on that. Can you do that for teleoperation or can it do that fully autonomously? So that's like the next test. For instance, Tesla did the iRobot event. I personally thought that was pretty impressive, regardless of it being teleoperated.

Me too. I was there. Yeah.

So, like, did that whole event and these robots were like, around people.

Yeah.

And you could see that these things are real.

Yeah.

So I do think that's kind of the big thing. One thing we'll do with our robot is, well, as soon as we put a battery on it, it's just like, take it around, like, let people see it. Let it be in the environments where humans are and not in a lab. All the time.

Yeah.

So I do think that's kind of the next bit.

So you. So I'll see you walking around outside someday.

Yeah, yeah, that'll be fun.

Take it into a coffee place with.

With your robot actually the next few months. So like to your question that you asked me for a second, okay, in, in 15, 18 months, what would happen but like if I came back here.

Yeah.

One, you'd just be able to talk to the robot and ask it to go somewhere. Be like, hey, why don't you go to the actuator station and it's just gonna walk itself there and you can essentially use like voice to voice interaction and feedback with the robot.

Yeah.

Second, a lot of the inference is gonna get faster. So you're gonna see these robots moving fast because the constraint is not in the hardware. These motors can move exceptionally fast. The constraint is how, how quickly can you reason safely to be able to execute on tasks. And that is going to get efficient more and more.

So right now where it's jerky and slow and uncertain, it's going to be really fast at doing this exactly well.

So I think jerky piece you can solve for just with the models we're building.

Right.

So as soon as you move to those, you can solve for those. But the infrastructure still is slow because like your high level reasoning runs at like what, 5 hertz or something and then your low level controls run at like 1Khz. So your technical goal is could you run your entire AI stack at 1Khz? Not gonna happen in the short term, but you can really get a bunch of gains like get go from 5 hertz to 10 to 15 to 2200. The robots just start getting faster and faster and faster. So I think you'll see the robot get much faster than they are today in next 12 to 18 months and become more useful. They'll be able to do more tasks than they can do today. The third one is a function of data, but I think you'll be able to deploy some of these and get an updated to be able to do these faster.

Yeah, get more people wearing this Vision Pro on the factory floor, you'll get a lot more data. Because you can see my hands. Right? What's the challenge for you building a company at this point? Because this is a new area, it's, you know, it's rapidly expanding and the technology that you're using is changing quite a bit every year.

I mean, I think the biggest thing is you have to move at a breakneck speed.

Right.

Like, you just have to, like, the team's working exceptionally hard. Like everyone can be over the weekend and like, we're gonna deploy some new things in it and there's no other way to do it but that. Because to your point, everything's evolving so quickly. So the only way you can keep up is by just working a lot.

What are you excited about? About the bigger AI community who's developing all sorts of models and doing all sorts of R and D, not necessarily for robotics like you talked about. I'll be able to talk to the robots and. Yeah, well, that's evolving really quickly. Yeah.

I think there are pretty much like two, four technologies that I think are most fascinating and would have the highest impact on humanity. The first one are your reasoning models. Going from reasoning to agent.

Right.

Can you essentially reason through a task, but then you can browse the Internet, say, go and book my tickets for here. The goes and does all of it, does all that. And our whole interaction with machines will change drastically. Once like why we have the screen is because we essentially have to be able to do things that are going to be very difficult to do with voice.

Yeah.

Otherwise it's gonna have to describe the page to you and you're gonna be like, well, pick the first thing you said, now pick the second. It's much harder. It's much easier to just look at with your eyes in a world where you can just be like, give a very ambiguous instruction. I have to be in Abu Dhabi on Monday, Figure out, look at my schedule, figure out flights and hotels for me. And it just goes and does it.

Your.

Your relationship with your technology and your devices changes completely. So I think, like, to me, I'm very excited about a world where you can have less screens. Just generally, that's why we don't put a screen on Pantone. Like, we don't think like less screens, the better. And being able to just like voice to voice actions and then even on the robots is by and large the same thing. Can you go from right now, you have to teach it how to do one repetitive task at a time and from there can you go from, I'm just going to ask it to do something and it's going to figure out how to do it and just do it. And the instructions could be a little ambiguous and if it needs, it would ask for clarity. Otherwise it's just going to do it. Philosophically, those two are by and large very similar problems. Technically, they're different. So, like in one, you have to be able to understand how to navigate the intermention before perform actions in it, and how does all the cookie management and authentication, how does that experience work? So that's a different challenge on robots. The challenges. Can you model the physics, dynamics, kinematics, in a way that when your action model says do this, you can reason and be like, is that the right thing to do? If not, does it need to reproduce? So I think those are the two big things in AI that are most exciting to me. And I know people are like building various kinds of different types of AI SaaS products, but I think that these two are the ones that are going to probably have the biggest impact.

What about the mixture of holodecks? Because I'm in a holodeck, you're actually a digital twin. Even though you're standing, I can touch you. Right. You're real. But inside my Vision Pro, you're actually a bunch of voxels. Right. Volume pixels. And I see you as a digital thing. What's the role of devices like this Vision Pro, which, you know, over the next five years are going to be a pair of glasses, lightweight pair of glasses. What's the role of a pair of glasses like this with a robot? Yeah, because certainly if I'm on a business trip and I have a robot in my house, I'm going to be able to embodiment myself in the robot and chase the kids around or feed the cat or whatever.

Yeah, I think there are like two good paradigms around, just like VR and ar.

Right.

The AR part is going to be very helpful. And again, you avoiding having to spend a lot of time on this phone because you can wear these glasses, you can do most of these stuff just with voice, and then it shows you just plays and different things. And it's very, very helpful. On the robot side, I think the most helpful thing is going to be precisely what you're saying, which is being able to get into an immersive environment and be able to just like fully model out yourself as the robot and be able to do things and do it very comfortably and conveniently. That is going to change, like. Yeah, that's going to change things drastically.

If, if I was wearing a. A device like this and operating a robot, the screens here could show me what the robot's next action is going to be like. Is it going to walk across the floor here? It could put a little blue line in front of the robot and say, hey, that robot's going that way. Which gives me as a human more information about what that thing's doing. Are you thinking about that kind of interaction with a human that's wearing a device like this, obviously operating a bunch of robots?

I mean, I think, yes, I think over time what's going to happen is, to your point, a lot of our, the way we work is going to use a lot of augmented reality because a lot of the context to things and objects around you, you'd be able to get the classes. Like it's very similar to Iron Man. Right. It's just able to look at an object and it gives you a bunch of its properties and dimensions. I think that's going to be useful across borders. It's going to be useful in robotics, it's going to be useful in any kinds of manufacturing, medicine, even education, community service, everything. So I do think there's going to be like an increased role of virtual reality augmentation or like augmented reality, just generally that would, to your point, maybe the experience with technology in the spaces we inhabit like much more intuitive than it is today.

Yeah. And we could also dress up the robot. Right. If we had glasses on, we could put virtual clothing on a human or a robot. Right. I could change your shirt to something else. That's a far off future for you. Yeah, because we don't have glasses really yet. I mean, a Vision Pro is cool for a nerd like me, but yeah, I mean, it's too heavy.

We're starting to see some of this stuff come out. Right. So like Meta showed this bulky but still just a glass that could do relatively good VR in ar. Now obviously the technology has to get to higher fidelity than it is today, but I think we're like getting close.

Yeah, it's an exciting time. Well, thanks so much for giving me this college level class in robotics and showing me around your company. It's really a privilege to do this and gives me a good insight as to the state of the art of robotics. Because you, you're the best I've seen. At least you're the best that's invited me in to see Tesla might invite me in to see something else. But it's going to be interesting to see how this whole industry evolves.

Yeah, no, we're really excited. I think like in next two to three years there'll be thousands of robots walking around us. It's going to be a very trippy future.

I want to see it when you take it outside. Yeah, I'll come over this year.

This year.

Yeah, I'll come and see that. That'll be a big deal, seeing it walk around and certainly see it walking around without, you know, somebody controlling it. Yeah, exactly. Like if it could actually be an AI that could go to, you know, Koopa Cafe and stand in line and get a coffee for you or something like that, that would be very trippy, wouldn't it? But that future is coming. Yeah.

These robots would be able to easily like navigate themselves without requiring like a human teleoperation type thing. A lot of Tesla robots that too, like that piece is right there. But to your point, gamer, these end to end tasks, I'll just tell the story. They go and get me coffee and it knows which coffee shop I like. It'll go there. It knows what my order is, places the order somehow it can pay for, it brings my coffee.

You know that the robot's gonna need your credit card at some point, right?

It's gonna have some kind of like NFC payment chip gift or something.

It'll figure it out. Right. Well, thank you so much.

Thank you so much for coming.