Karpathy Talks Deep Learning, Elon Musk, FSD Beta, & Waymo Competition (03.23.21) — Tesla Daily: Tesla News & Analysis

Hey everybody, Rob Maurer here, and today we're going to be mostly talking about some takeaways from a recent interview of Tesla's Director of Artificial Intelligence, Andrej Karpathy, on a podcast called The Robot Brains Podcast, which I will link to in the show notes. The whole conversation is about an hour and 5 minutes. I would highly recommend listening to all of it. I thought it was very approachable. And hopefully it helps set the stage for a conversation we're going to have later this week. Quick look at the stock, Tesla today finished down 1.2% to $662.16. Not a bad day overall considering the NASDAQ itself finished down 1.1%. All right, so getting into the Karpathy interview, again, great interview. I think it's super interesting in terms of timing right now because this is so current. Tesla's obviously working on the FSD beta right now. Andrej did share some thoughts on that, which we will get to, I do want to give just a little bit of background though for people that aren't familiar with Andrej's background. So he has a history in education and research. He gained a lot of notoriety in the field of artificial intelligence and specifically in deep learning because of his course at Stanford about deep learning, which was really the first of its kind. And those lectures were also shared on YouTube, gained a ton of traction there. So Andrej Karpathy was teaching that class at Stanford, got his PhD there. After that, he joined OpenAI as a research scientist. Spent a couple of years there, but at that point in his life, he had been doing research for about a decade, reading papers, writing papers. So he talks about that period of time in this interview and joining Tesla, saying, quote, I was definitely getting a little bit restless at that time. I felt like these algorithms are extremely powerful and can really move the needle on some very incredibly important problems in society. I wanted to take a more active role in doing that. So I was getting a bit restless. I was looking at different opportunities in, say, startups and things like that. And then one thing that kind of happened on the side is because OpenAI at the time, under the umbrella of Elon organizations, a few times we were interacting with people at Tesla and I was kind of consulting a little bit for some of the problems in Autopilot. I kind of realized that they were dealing with fundamentally a deep learning computer vision and this was the fundamental constraint to whether or not this product would work. So I was kind of intrigued by that, but it was just a few consulting opportunities here and there. I sort of spoke to the team, but at this time when I was getting really restless to apply this technology in the industry, Actually, Elon reached out and he asked me, "Hey, you've been sort of consulting for the team. Do you actually want to join and lead the computer vision team here and help get this car to drive?" And so he caught me at a very kind of correct time when I was really getting restless, and I feel like this is perfect, and I think I can do this. I think I have the skill to contribute here. This is an incredibly impactful opportunity, and I love the company, and of course I love Elon and everything that he's doing. And so I would say that again was a moment when stars aligned for me, and I felt very strongly that this is the right thing to do at this time. End quote. So, long quote there, but I appreciated having that background. I've always kind of wondered if Andrej's heart was really in the research realm and if Elon had kind of convinced him and pulled him away from that, but that definitely doesn't sound like the case from how Andrej describes it. He said that at that time when he made that decision to go to Tesla, he and his colleagues at OpenAI had joked around that he'd probably only make it about 6 months, and here we are 4 years later. I'm really happy about that because I think Andrej is just a perfect fit for this role. Not only does he have an incredible understanding of the technical challenges that Tesla's facing, but he also seems to have the right personality for it. He talked about working with Elon, then we'll also get into some of the technical stuff here in a second, but he said, quote, he's a double-edged sword in terms of working with him because he wants the future yesterday and he will push people and he will inject a lot of energy and he wants it to happen quickly. And you have to be of a certain, I think, attitude to really tolerate that over long periods of time. But he surrounds himself with people who get energy out of that and who also want the future to happen quicker. Those people really thrive at Tesla, and so I happen to also, I think, be like that. And so I don't personally mind it. I actually thrive on it and the energy of getting this to work faster. End quote. I think this anecdote probably serves as a good reminder that high turnover isn't necessarily a bad thing just by default. You'll often see articles without a lot of data, honestly, being critical of Tesla for having high turnover, especially in management, but From the outside, it seems like they go through these cycles where they have a lot of turnover in short periods of time, and then they find the person that's right for the role, and also vice versa, that the role is right for them. And then it seems like those people tend to stick for a long, long time. Better to have high turnover, work through a number of different people and find the right person than to stick with the wrong person for too long. Anyway, the other salient point that Andrej Karpathy made about Elon came about from a discussion on Tesla's hardware and the vertical integration. Andrej said, quote, I think to a large extent Elon sees AI as just a fundamental pillar of a lot of this technology and wants to invest in internal teams that develop a lot of this technology and co-design everything. Tesla is definitely about vertical integration and squeezing a lot of juice from the benefits of that, end quote. He continued on with that line of thought and mentioned co-design 3 or 4 times, I think, which is really one of the most important and special elements of Tesla, this co-design idea. Particularly how the organization is structured around that idea. Tesla's purpose-built hardware chip for Full Self-Driving is obviously a great example of that. The structural battery is a great example of that. Before that, you had Super Bottle, then the Octovalve. The list just goes on and on and over time continues to grow. So pay very, very close attention to that when Andrej Karpathy says that Elon sees AI as a fundamental pillar of a lot of their technologies. We're still in the very early innings of that actually showing up, in Tesla's business. All right, so getting into some of his more technical comments then, this is gonna be a really long quote. I apologize for that, but I think it's a good primer. So he was asked, how would you describe deep learning to your parents? Andrej Karpathy says, quote, let's use a specific example because I think it's useful. So let's talk about image recognition. We have images. Images are made up to a computer of a large number of pixels, and each pixel just tells you the brightness of the red, green, and blue channel at that point. And so you have a large array of numbers and you have to go from that to, hey, it's a cat or a dog. Typical conventional software is written by a person, a programmer, writing a series of instructions to go from the input to the output. So in this case, you want someone to write a program for how do you combine these millions of pixel values into is it a cat or a dog. Turns out no one can write this program. It's a very complicated program because there's a huge amount of variability in what a cat or dog can look like in different brightness conditions, arrangements, poses. Occlusions, basically no one can write this program. Deep learning is a different class of programming in my mind, where no one is explicitly writing the algorithm for this recognition problem, instead we are structuring the entire problem slightly differently. In particular, we arrange a large dataset of possible images and the desired labels should come out from the algorithm. So hey, when you get this input, this is a cat, when you get this input, this should be a dog, and so on. So we're kind of stipulating what is the desired behavior on a high level, we're not talking about what is the algorithm, we're measuring the performance of of some algorithm. And then roughly what we do is lay out a neural network. It's a bunch of neurons connected to each other with some strengths, and you feed them images and they try to predict what's in them, end quote. So that's where the learning comes in, and also why Andrej Karpathy talks a lot about data labeling. Data labeling is how Tesla, or I suppose anyone doing image recognition, establishes that ground truth, which the algorithms can then be refined on depending on if the output matches or not. Andrej Karpathy continues on to explain how this process works, saying, quote, as an example, the 2012 ImageNet model was roughly 60 million parameters. So the weights of the neural network were really 60 million knobs, and those knobs can be arbitrary values. And how do you set those 60 million weights so that the network gives you the correct predictions? So deep learning is a way of training this neural network and finding a good setting of these 60 million numbers. And so roughly, the neural network looks at the image, gives you a prediction, and then you measure the error. Okay, you said this is a cat, but actually this is a dog, and there's a mathematical procedure for tuning the strengths so that the neural network adapts itself to agree with you. Deep learning is basically a different software programming paradigm where we specify what we want, and then we use sort of mathematics and algorithms to tune the system to give you what you want. And there's some design that goes into the neural network architecture and how do you wire everything up, but then there's also a huge amount of design and effort spent on the datasets themselves and curating them because those datasets are now your constraints on the behavior that you are asking from the system. It's a very different way of approaching problems that was not there before. Everything used to be written by a person. Now we just write the specification and we write a rough layout of the algorithm, but it's what I refer to as fill-in-the-blanks programming because we lay out an architecture and a rough layout of the net, but there's a huge amount of blanks, which are the weights and the knobs, and those are set now during the training of this network. So that's, I think, the bird's-eye view of how this is different. End quote. So keep that concept of weights in mind. I think we'll probably end up coming back to that later this week. As for how Tesla specifically handles this learning process, Andrej Karpathy said basically they're trying to find a bunch of situations where it doesn't work, then label those situations so that they do work, saying that Tesla collects that data in a variety of different ways. He gave a few examples, the first one here being very obvious. If there is an intervention when Autopilot is activated, that can be a great source of data for Tesla. The second one was interesting. So he talked about If there's a bounding box on something like a stop sign, for example, and as you approach it, it flickers, that represents the system disagreeing with itself over time. So it might say, oh, that's a stop sign. Oh no, it's not. Oh yes, it is. Tesla can isolate those occurrences, pull them from the fleet, label them, and then retrain the network on them. Andrej Karpathy described that process as saying, that's the loop. So step 1, the neural network makes predictions. Step 2, they source mispredictions at scale. That part is critical. Step 3, annotate them correctly. And Step 4, put them into the training set and retrain. That's the loop. What Tesla wants to do, and is trying to do, is to automate as much of that stuff as possible. That's where Project Dojo comes in. They jokingly refer to this attempt as Operation Vacation because if they get this going, they could all just go on vacation and the system would make itself much better over time. Listening to these things, it becomes pretty obvious how important scale is. Collecting all of those mispredictions seeking out and finding those edge cases through billions and billions of miles of data. That's critical to the training process. Andrej Karpathy pointed out that essentially in the self-driving industry you have a bifurcation where people are taking either a Waymo-like strategy, which there's many more sensors, in particular LiDAR and HD maps, or a Tesla-like strategy which is focused on vision, no HD maps. That's a higher bar to clear, but it is much cheaper. That cost-effectiveness affords you scale And scale, as we talked about, is incredibly important. Andrej Karpathy said, quote, I do not see how you can fundamentally really get the system to work in absence of scale. I would much rather give up some sensing in return for scale in AI problems, end quote. So super exciting. Obviously Tesla is leading in scale, and that scale just continues to increase day after day as Tesla ships new vehicles. And as the FSD beta rolls out, Tesla will be collecting even more data And as time goes by and as the system gets better, the quality of that data is going to improve as well. So with all of those things aligning, you can start to see why the progress might look exponential. The interviewer asked Andrej, even being so close to this, are you ever surprised by what the car does and say, wow, this car is smarter than I thought? Andrej said, quote, I would say about every drive I have a few of those, end quote, saying that even on the day of this interview, he had a 10-minute drive two ways, so 20-minute drive in total, with no interventions, and that, that's becoming a relatively common occurrence for them at Tesla. So a lot of great things in this interview, not all sunshine and rainbows. They talked a little bit about self-supervised learning, which takes a bit of a step towards more artificial general intelligence. 'Cause right now in a neural network, let's say you have to get it to recognize a fire hydrant, you'd have to show it 10,000 or 50,000 pictures of a fire hydrant from all different lighting conditions, all different angles, then it'll learn it. But if a human that doesn't know what a fire hydrant is, if you show 'em one or two pictures, they're going to be able to identify it in probably all of those 50,000 photos. So he was saying that's kind of where the analogy to a neural network and the human brain breaks, and that Tesla's current technology isn't working in that way, but there are a lot of different techniques that people are working on to kind of get to that end state result. Maybe it's not classifying something as a fire hydrant, but maybe it still understands the context that it needs to understand with maybe a little bit less explicit training. So a lot of fascinating stuff in this. Again, the interview will be in the show notes. The last question I thought was kind of funny. He asked Andrej Karpathy if he had slept at Tesla like Elon Musk has, infamous for. Andrej said, yeah, he has slept at the office a few times, but quote, "I like being overtaken by problems sometimes. I think it's good for the soul," end quote. So he said doing that was great, also said that he hasn't done it in a while. But again, Andrej definitely a good cultural fit for Tesla. The last quick thing here before we wrap it up for today, just an update on Starship SN11. Looks like at this point the earliest that would be would be this Friday, and certainly wouldn't be surprising to see it pushed into sometime next week. That'll wrap it up for today though. As always, thank you for listening. Make sure you're subscribed and signed up for notifications. You can also find me on Twitter @TeslaPodcast, and I'll see you tomorrow for the Wednesday, March 24th episode of Tesla Daily. Thank you.