Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics | Lex Fridman Podcast #241

The following is a conversation with Boris Softman, who is the senior director of Engineering and head of Trucking at Waymo, the autonomous vehicle company, formerly the Google Self Driving Car project. Before that, Boris was the co founder and CEO of Anki, a robotics company that created Cosmo, which in my opinion is one of the most incredible social robots ever built. It's a toy robot, but one with an emotional intelligence that creates a fun and engaging human robot interaction. It was truly sad for me to see Anki shut down when he did. I had high hopes for those little robots. We talk about this story and the future of autonomous trucks, vehicles and robotics in general. I spoke with Steve Vascelli recently on episode 237 about the human side of trucking. This episode looks more at the robotic side. This is the Lex Friedman podcast to support it. Please check out our sponsors in the description. And now here's my conversation with Boris Softman.

Who is your favorite robot in science fiction books or movies?

Wall E&R2D2, where they were able to convey such an incredible degree of intent, emotion and kind of character attachment without having any language whatsoever and just purely through the richness of emotional interaction. So those were fantastic. And then the Terminator series, just like really pretty wide, wide range, right? But I kind of love this dynamic where you have this incredible Terminator itself that Arnold played. And then he was kind of like the inferior previous generation version that was totally outmatched in terms of kind of specs by the new one, but still kind of held his own. And so it was kind of interesting where you realize how many levels there are on the spectrum from human to kind of potentials in AI and robotics to futures. And so yeah, that movie really, as much as it was kind of a dark world in a way, was actually quite fascinating. Gets the imagination going.

Well, from an engineering perspective, both the movies you mentioned, Wall E and Terminator, the first one is probably achievable. You know, humanoid robot, maybe not with like the realism in terms of skin and so on, but that humanoid form. We have that humanoid form. It seems like a compelling form. Maybe the challenge is it's super expensive to engine to build, but you can imagine, maybe not a machine of war, but you can imagine Terminator type robots walking around and then the same, obviously with Wall E, you've basically. So for people who don't know, you created the company Anki that created a small robot with a big personality called Cosmo that just does exactly what Wall E does, which is somehow with very few basic visual tools is able to communicate a depth of emotion. And that's fascinating. But then again, the humanoid form is super compelling. So, like, Cosmo is very distant from a humanoid form, and then the Terminator has a humanoid form. And you can imagine both of those actually being in our society.

It's true. And it's interesting because it was very intentional to go really far away from human form. When you think about a character like Cosmo or like Wally, where you can completely rethink the constraints you put on that character, what tools you leverage, and then how you actually create a personality and a level of intelligence, interactivity that actually matches the constraints that you're under, whether it's mechanical or sensors or AI of the day. This is why I almost was always very surprised by how much energy people put towards trying to replicate human form in a robot. Because you actually take on some pretty significant kind of constraints and downsides when you do that. The first of which is obviously the cost, where the articulation of a human body is just so magical in both the precision as well as the dimensionality, that to replicate that even in its reasonably close form takes a giant amount of joints and actually actuators and motion and sensors and encoders and so forth. But then you're almost like setting an expectation that the closer you try to get to human form, the more you expect the strengths to match. And that's not the way AI works, is there's places where you're way stronger and there's places where you're weaker. And by moving away from human form, you can actually change the rules and embrace your strengths and bypass your weaknesses.

And at the same time, the human form, like, has way too many degrees of freedom to play with. It's kind of counterintuitive, just as you're saying. But when you have fewer constraints, it's almost harder to master the communication of emotion. Like, you see this with cartoons, like stick figures, you can communicate quite a lot with just very minimal, like two dots for eyes and a line for a smile. I think you can almost communicate arbitrary levels of emotion with just two dots and a line. And that's enough. And if you focus on just that, you can communicate the full range. And then if you do that, then you can focus on the actual magic of human and dot line interaction versus all the engineering mess.

That's right. Like dimensionality, voice, all these sort of things, they actually become a crutch where you get lost in a search space almost. And so some of the best animators that we've worked with they almost study when they come up in building their expertise by forcing these projects where all you have is a ball that can kind of jump and manipulate itself, or really, really aggressive constraints where you're forced to kind of extract the deepest level of motion. And so in a lot of ways, when we thought about Cosmos, you're right, if we had to describe it in one small phrase, it was bringing a Pixar character to life in the real world. It's what we were going for. And in a lot of ways what was interesting is that with Wall E, which we studied incredibly deeply, and in fact, some of our team had worked previously at Pixar and on that project, they intentionally constrained Wall E as well. Even though in an animated film you could do whatever you wanted to because it forced you to really saturate the smaller amount of dimensions. But you sometimes end up getting a far more beautiful output because you're pushing at the extremes of this emotional space in a way that you just wouldn't, because you get lost in the surface area if you have something that is just infinitely articulable.

So if we backtrack a little bit, you thought of Cosmo in 2011 and 2013, actually designed and built it. What is Anki? What is Cosmo? I guess. Who is Cosmo and what was the vision behind this incredible little robot?

We started Anki back while we were still in graduate school. So myself and my two co founders, we were PhD students in the Robotics Institute at Carnegie Mellon. And so we were studying robotics, AI, machine learning, kind of different areas. One of my co founders working on walking robots for a period of time. And so we all had a bit of a really a deeper passion for applications of robotics and AI, where there's like a spectrum where there's people that get really fascinated by the theory of AI and machine learning robotics, where whether it gets applied in the near future or not is less of a kind of factor on them, but they love the pursuit of the challenge and that's necessary. And there's a lot of incredible breakthroughs that happen there. We're probably closer to the other end of the spectrum where we love the technology and all the evolution of it, but we were really driven by applications like how can you really reinvent experiences and funct functionality and build value that wouldn't have been possible without these approaches. And that's what drove us. And we had kind of some experiences through previous jobs and internships where we got to see the applied side of robotics. And at that time there was actually relatively Few applications of robotics that were outside of peer research or industrial applications, military applications and so forth, there were very few outside of it. So maybe IROBOT was like one exception and maybe there were a few others, but for the most part there weren't that many. And so we got excited about consumer applications of robotics where you could leverage way higher levels of intelligence through software to create value and experiences that were just not possible in those fields today. And we saw kind of a pretty wide range of applications that varied in the complexity of what it would take to actually solve those. And what we wanted to do was to commercialize this into a company, but actually do a bottoms up approach where we could have a huge impact in a space that was ripe to have an impact at that time, and then build up off of that and move into other areas. And entertainment became the place to start because you had relatively little innovation in the toy space and entertainment space. You had these really rich experiences in video games and movies, but there was like this chasm in between. And so we thought that we could really reinvent that experience. And there was a. A really fascinating transition technically that was happening at the time where the cost of components was plummeting because of the mobile phone industry and then the smartphone industry. And so the cost of a microcontroller, of a camera, of a motor, of memory, of microphones, cameras, was dropping by orders of magnitude. And then on top of that, with iPhone coming out in 2000, I think it was 2007, I believe it started to become apparent within a couple of years that this could become a really incredible interface device and the brain with much more computation behind a physical world experience that wouldn't have been possible previously. And so we really got excited about that and how we push all the complexity from the physical world into software by using really inexpensive components, but putting huge amounts of complexity into the AI side. And so Cosmo became our second product and then the one that we're probably most proud of. The idea there was to create a physical character that had enough understanding and awareness of the physical world around it and the context that mattered to feel like, like he was alive and to be able to have these like, emotional kind of connections and experiences with people that you would typically only find inside of a movie. And the motivation very much was, was Pixar. Like, we had an incredible respect and appreciation for what they were able to build in this, like, really beautiful fashion in film. But it was always like a, one, it was virtual, and two, it was like a story on rails that had no interactivity to it. It was very fixed, and it obviously had a magic to it, but where you really start to hit a different level of experiences when you're actually able to physically interact with our robot.

And then that was your idea with Anki. The first product was the cars. So basically, you take a toy, you add intelligence into it in the same way you would add intelligence into AI systems within a video game, but you're not bringing into the physical space. So the idea is really brilliant. Which is you're basically bringing video games to life.

Exactly. That's exactly right. We literally use that exact same phrase. Because in the case of Drive, this was a parallel of the racing genre. And the goal was to effectively have a physical racing experience, but have a virtual state at all times that matches what's happening in the physical world.

And.

And then you can have a video game off of that, and you can have different characters, different traits for the cars, weapons and interactions, and special abilities, and all these sort of things that you think of virtually, but then you can have it physically. And one of the things that we were really surprised by that really stood out and immediately led us to really kind of accelerate the path towards Cosmo, is that things that feel like they're really constrained and simple in the physical world, they have an amplified impact on people, where the exact same experience virtually would not have anywhere near the impact, but seeing it physically really stood out. And so, effectively, with Drive, we were creating a video game engine for the physical world. And then with Cosmo, we expanded that video game engine to create a character and kind of an animation and interaction engine on top of it that allowed us to start to create these much more rich experiences. And a lot of those elements were almost like a proving ground for what would human robot interaction feel like in a domain that's much more forgiving, where you can make mistakes in a game, it's okay if, like, if, you know, car goes off the track or if. If Cosmo makes a mistake. And what's funny is, actually, we were so worried about that. In reality, we realized very quickly that those mistakes can be endearing. And if you make a mistake, as long as you realize you make a mistake and have the right emotional reaction to it, it builds even more empathy with the character.

That's brilliant. Exactly. So when the thing you're optimizing for is fun, you have so much more freedom to fail to explore. And also in the toy space, like, all of this is really brilliant. I gotta ask you, backtrack. It seems for a roboticist to take us jump into the direction of fun is a brilliant move because one, you have the freedom to explore, to design all those kinds of things. And you can also build cheap robots. Like, you don't have to. Like, if you're not chasing perfection and toys, it's understood that you can go cheaper, which means in robot, it's still expensive, but it's actually affordable by a large number of people. So it's a really brilliant space to explore.

Yeah, that's right. And in fact, we realized pretty quickly that perfection is actually not fun because in a traditional roboticist sense, the first kind of path planner, and this is the part that I worked on out of the gate, was a lot of the kind of AI systems where you have these, you know, vehicles and cars racing, kind of making optimal maneuvers to try to kind of get ahead. And you realize very quickly that like, that's actually not fun because you want the, like chaos from mistakes and the. And so you start to kind of intentionally almost add noise to the system in order to kind of create more of a realism. In the exact same way, the human player might start really ineffective and inefficient and then start to kind of increase their quality bar as they progress. And there is a really, really aggressive constraint that's forced on you by being a consumer product where the price point matters a ton, particularly in kind of an entertainment where you can't make $1,000 product unless you're going to meet the expectations of $1,000 product. And so in order to make this work, your cost of goods had to be well under $100. In the case of Cosmo, we got it under $50 end to end, fully.

Packaged and delivered, and it was under $200 cost at retail.

Yeah.

So, okay, if we sit down like at the early stages, if you go back to that and you're sitting down thinking about what Cosmo looks like from a design perspective and from a cost perspective, I imagine that was part of the conversation. Well, first of all, what came first? Did you have a cost in mind? Is there a target you're trying to chase? Did you have a vision in mind like size? Did you have. Because there's a lot of unique qualities to Cosmos. So for people who don't know should definitely check it out is there's a display. There's eyes on the little display. And those eyes can. It's pretty low resolution eyes. Right. But they, they still able to convey a lot of emotion. And there's this arm like that lift sort of lift stuff. But there's something about arm movement that adds even more kind of depth. It's like the face communicates emotion and sadness and disappointment and happiness. And then the arms kind of communicates, I'm trying here, I'm doing my best in this world.

Exactly. So it's interesting because all of Cosmo is only four degrees of freedom. And two of them are the two treads, which is for basic movement. And so you literally have only a head that goes up and down, a lift that goes up and down, and then your two wheels and you have sound and a screen and a low resolution screen. And with that, it's actually pretty incredible what you can come up with. Where, like you said, it's a really interesting give and take because there's a lot of ideas far beyond that, obviously, as you can imagine, where, like you said, how big is it? How much degrees of freedom, what does he look like, what does he sound like, how does he communicate? It's a formula that actually scales way beyond entertainment. This is the formula for human kind of robot interface more generally is you almost have this triangle between the physical aspects of it, the mechanics, the industrial design, what's mass producible, the cost constraints and so forth. You have the AI side of how do you understand the world around you, interact intelligently with it, execute what you want to execute. So perceive the environment, make intelligent decisions and move forward. And then you have the character side of it. Most companies have done anything in human robot interaction really miss the mark or underinvest in the character side of it. They overinvest in the mechanical side of it, and then varied results on the AI side of it. And so the thinking is that you put more mechanical flexibility into it, you're going to do better. You don't necessarily, you actually create a much higher bar for a higher roi, because now your price point goes up, your expectations go up, and if the AI can't meet it or the overall experience isn't there, you missed the mark.

So who, like, how did you, through those conversations, get the cost down so much and make it, made it so simple like that? There's a big theme here because you come from the Mecca of robotics, which is Carnegie Mellon University Robotics. For all the people I've interacted with that come from there or just from the world experts at robotics, they would never build something like Cosmo.

Yeah.

And so where did that come from?

So the simplicity, it came from this, this combination of a team that we had. And it was quite cool because like we. And by the Way you ask anybody that's like, experienced in the like kind of, you know, toy entertainment space, you'll never sell a product over $99. That was fundamentally false. And we believed it to be false. It was because experience had to kind of, you know, meet the mark. And so we pushed past that amount. But there was a pressure where the higher you go, the more seasonal you become and the tougher it becomes. And so on the cost side, we very quickly partnered up with some previous contacts that we worked with, where just as an example, our head of mechanical engineering was one of the earliest heads of engineering at Logitech and has a billion units of consumer products in circulation that he's worked on. So like crazy low cost, high volume consumer product experience. We had a really great mechanical engineering team and just a very practical mindset where we were not going to compromise on feasibility in the market in order to chase something that would be enabler. And we pushed a huge amount of expectations onto the software team where, yes, we're going to use cheap, noisy motors and sensors, but we're going to fix it on the software side. Then we found on the design and character side, there was a faction that was more from a game design background that thought that it should be very games driven. Cosmo, where you create a whole bunch of games experiences and it's all about game mechanics. And then there was a faction which my co founder and I that are most involved in this, really believed in, which was character driven. And the argument is that you will never compete with what you can do virtually from a game standpoint, but you actually, on the character side put this into your wheelhouse and put it more towards your advantage because a physical character has a massively higher impact physically than virtually.

Okay, can I just pause on that? Because this is so brilliant for people who don't know Cosmo plays games with you, but there's also a depth of character. And I actually, when I was, you know, playing with it, I wondered exactly what is the compelling aspect of this? Because to me, obviously I'm biased, but to me, the character, what I enjoyed most honestly, or what got me to return to it, is the character.

That's right.

But that's. That's a fascinating discussion of. You're right. Ultimately you cannot compete on the quality of the gaming experience.

Too restrictive. The physical world is just too restrictive and you don't have a graphics engines like all this. But on the character side we. And clearly we moved in that direction as like kind of the winning path. And we partnered up with this really. We immediately, like, went towards Pixar. And Carlos Baena, he was one of, like, had been at Pixar for nine years. He'd worked on tons of the movies, including Wall E and others, and just immediately kind of spoke the language and it just clicked on how you think about that, like, kind of magic and Drive. And then he. We built out a team with him as a really kind of prominent driver of this, with different types of backgrounds and animators and character developers, where we put these constraints on the team, but then got them to really try to create magic despite that. And we converged on this system that was at the overlap of character and the character AI, where if you imagine the dimensionality of emotions, happy, sad, angry, surprised, confused, scared, you think of these extreme emotions. We almost put this challenge to populate this library of responses on how do you show the extreme response that goes to the extreme spectrum on angry or frustrated or whatever? And so that gave us a lot of intuition and learnings. And then we started parameterizing them where it wasn't just a fixed recording, but they were parameterized and had randomness to them, where you could have infinite permutations of happy and surprised and so forth. And then we had a behavioral engine that took the context from the real world and would interpret it and then create kind of probability mappings on what sort of responses you would have that actually made sense. And so if Cosmos saw you for the first time in a day, he'd be really surprised and happy in the same way that the first time you walk in and your toddler sees you, they're so happy, but they're not going to be that happy for the entirety of your next two hours. But you have this spike in response, or if you leave him alone for too long, he gets bored and starts causing trouble and, like, nudging things off the table. Or if you beat him in a game, the most enjoyable emotions are him getting frustrated and grumpy to a point where our testers and our customers would be like, I had to let him win because I don't want him to be upset. And so you start to create this feedback loop where you see how powerful those emotions are. And just to give you an example, something as simple as eye contact, you don't think about it in a movie, just like it kind of happens, like, you know, camera angles and so forth, but that's not really a prominent source of interaction. What happens when a physical character like Cosmo, when he makes eye contact with you? It built universal kind of connection. Kids all the way through adults. And it was truly universal. It was not like people stopped caring after 10, 12 years old. And so we started doing experiments and we found something as simple as increasing the amount of eye contact, like the amount of times in a minute that he'll look over for your approval till I kind of make eye contact, just by, I think doubling it. We increased the playtime engagement by 40%. Like you see these sort of interactions where you build that empathy. And so we studied pets, we studied virtual characters. There's like a lot of times actually dogs are one of the perfect, most perfect influencers behind these sort of interactions. And what we realized is that the games were not there to entertain you. The games were to create context, to bring out the character. And if you think about the types of games that you played, they were relatively simple, but they were always ones to create scenarios of either tension or winning or losing or surprise or whatever the case might be. And they were purely there to just create context to where an emotion could feel intelligent and not random. And in the end, it was all about the character.

So, yeah, there's so many elements to play with here. So you said dogs. What lessons do we draw from cats who don't seem to give a damn about you? Is that just another character?

It's just another character. And so you could almost like in early explorations, we thought it would be really incredible if you had a diversity of characters where you almost help encourage which direction it goes. Just like in a role playing game. And you had think of the, you know, seven dwarves sort of, and, and initially we even thought that it would be amazing if like, you know, they're like, you know, like their characters actually help them be, have strengths and weaknesses and some, you know, like, whatever they end up doing, like, some are scared, some are, you know, arrogant, some are, you know, super warm and like, kind of friendly. And in the end we focused on one because it made it very clear that, hey, we got to build out enough depth here because you're kind of trying to expand. It's almost like how long can you maintain a fiction that this character is alive to where the person's explorations don't hit a boundary, which happens almost immediately with, with typical toys and, you know, even with video games, how long can we create that immersive experience to where you expand the boundary? And one of the things we realized is that you're just way more forgiving when something has a personality and it's physical. That is the key that unlocks robotics. Interacting in the Physical world more generally is that when you don't have a personality and you make a mistake as a robot, the stupid robot made a mistake. Why is it not perfect? When you have a character and you make a mistake, you have empathy and it becomes endearing and you're way more forgiving. And that was the key that was like, I think goes far, far beyond entertainment.

It actually builds the depth of the personality, the mistakes. So let me ask the movie her question then. How so Cosmos feels like the early days of something that will obviously be prevalent throughout society at a scale that we cannot even imagine. My sense is it seems obvious that these kinds of characters will permeate society and we'll be friends with them, we'll be interacting with them in different ways in the way we. I mean, you don't think of it this way, but when you play video games, they're kind, they're often cold and impersonal. But even then, you think about role playing games, you become friends with certain characters in that game. They don't remember much about you. They're just telling a story. It's exactly what you're saying. They exist in that virtual world. But if they acknowledge that you exist in this physical world, if the characters in the game remember that you exist, that you like. For me, like Lex, they understand that I'm a human being who has like hopes and dreams and so on. It seems like there's going to be like billions, if not trillions of cosmos in the world. So if we look at that future, there's several questions to ask. How intelligent does that future Cosmo need to be to create fulfilling relationships like friendships?

Yeah, it's a great question. And part of it was a recognition that it's going to take time to get there because it has to be a lot more intelligent. Because what's good enough to be a magical experience for an 8 year old? It's a higher bar to do that. Be a pet in the home, or to help with functional interface in an office environment or in a home and so forth. And the idea was that you build on that and you kind of get there. As technology becomes more prevalent and less expensive and so forth, you can start to kind of work up to it. But you're absolutely right. At the end of the day, we almost equated it to how the touchscreen created this really novel interface to physical devices like this. This is the extension of it where you have much richer physical interaction in the real world. This is the enabler for it. And it shows itself in A few kind of really obvious places. So just take something as simple as a voice assistant. Most people will never tolerate an Alex or a Google home. Just starting a conversation proactively when you weren't kind of expecting it, because it feels weird. It's like you were listening and then now you're kind of. It feels intrusive. But if you had a character like a cat that touches you and gets your attention, or a toddler, you never think twice about it. And what we found really kind of immediately is that these types of characters like Cosmo and they would roam around and kind of get your attention. And we had a future version that was always on, kind of called Vector. People were way more forgiving. And so you could initiate interaction in a way is not acceptable for machines. And in general, there's a lot of ways to customize it, but it makes people who are skeptical of technology much more comfortable with it. There were a couple of really, really prominent examples of this. So when we launched in Europe, and so we were in, I think, like a dozen countries if I remember correctly, but like, we went pretty aggressively in launching in Germany and France and uk and we were very worried in Europe because there's obviously like a really a socially higher bar for privacy and security where you've heard about how many companies have had troubles, things that might have been okay in the US but are just not okay in Germany and France in particular. And so we were worried about this because you have Cosmo, who's in our future product, Vector, where you have cameras, you have microphones, it's cloud connected, and you're playing with kids in these experiences and you're like, this is ripe to be a nightmare if you're not careful. And the journalists are notoriously, really tough on these sort of things. We were shocked, and we prepared so much for what we would have to encounter. We were shocked in that not once from any journalists or customer did we have any complaints beyond a really casual kind of question. And it was because of the character where when the conversation came up, it was almost like, well, of course he has to see and hear. How else is he going to be alive and interacting with you? And it completely disarmed this fear of technology that enabled this interaction to be much more fluid. And again, entertainment was a proving ground. But there's ingredients there that carry over to a lot of other elements down the road.

That's hilarious that we're a lot less concerned about privacy if the is thing, if the thing has value and charisma. I mean, that's true for all of human to human interaction, too.

It's an understanding of intent where, like, well, he's looking at me, he can see me. If he's not looking at me, he can't see me. Right. So it's almost like you're communicating intent. And with that intent, people are kind of more understanding and calmer. And it's interesting. It was just the earliest kind of version of starting to experiment with this, but it wasn't an enabler. And then you have completely different dimensions where kids with autism had an incredible connection with Cosmo, went beyond anything we'd ever seen. And we have these just letters that we would receive from parents. And we had some research projects kind of going on with some universities on studying this. But there's an interesting dimension there that got unlocked that just hadn't existed before, that has.