Andrej Karpathy - AI for Full-Self Driving at Tesla

So I'm very excited to be here to tell you a bit more about how we're scaling machine learning models, uh, data algorithms and infrastructure at Tesla and what that looks like. Uh, so the rough outline of my, uh, talk is, uh, here. Uh, I'd like to first tell you a little bit about Tesla, Tesla Autopilot, what the feature set is, really what the problem statement is, what we're trying to address here. Uh, then I'll tell you a bit about neural networks and what it takes to actually deploy them into production at scale and what that looks like for the team. And then I'll tell you a little bit more about some of the more recent neural networks that we have developed for the purposes of full self-driving. So I'll go into that at the end. So first, a little bit about Tesla and Tesla Autopilot. So as you may know, Tesla manufactures and sells electric vehicles. We have a whole array of electric vehicles. Some of them are in the hands of customers or imminently to be in the hands of customers. And some of them have only been announced. So for example, we have the Model S, the Model 3, X, and Y. And then we've also announced the Cybertruck, the ATV, somewhat jokingly, the Roadster, and the Semi. And we have quite a few of these. So globally, we now have roughly 1 million cars. And for the purposes of the team, these are not just cars. These are computers. They're internet accessible. And we can both deploy over-the-air updates to them and get a lot of data from them that helps us develop the Autopilot. And they're also robots. So they have sensing. They have actuation. And we're developing the brains of that computer. Now, the most basic kind of Autopilot functionality is that when you turn it on in your car, the car will keep its lane and it will keep distance away from the vehicle ahead of you. We've now sort of accumulated almost 3 billion miles on the Autopilot, which sounds like a lot. We then have more advanced features, like such as, for example, Navigate on Autopilot. This allows you to set a pin somewhere in the world as a destination. And then as long as you stick to the highway system, the car will automatically reach that destination on the highway system. It will do all the correct lane changes. It will do— it will take all the right forks at interchanges. And it will overtake slow vehicles ahead of you. And it's kind of like— we think of it as roughly autonomy on the highway. And the car now has to monitor its blind spot. It has to have intelligence around where it is on the highway. It has to localize the correct lane and so on. We've also late last year released a Smart Summon. This allows you to, with your phone on the mobile app, summon the car to you from a parking spot. So if your car is parked somewhere, you just summon it to you, and the car comes out of its parking spot with no one inside and comes wind its way through the parking lot to come find you. It's quite magical when it works. Um— [LAUGHTER] Also under the hood, we have a lot of active safety features that are always running and available, even when Autopilot is not turned on. And we have a lot of statistics available to us because we have 1 million cars on the road, and we know how people get into trouble. and then we try to implement features that, you know, help alert or avert an accident. So for example, automatic emergency braking, front collision warning, side collision warning, blind spot monitoring, lane departure avoidance, emergency lane departure avoidance, and so on. These active safety features help keep customers safe in our cars. In particular, on active safety, we're doing extremely well. Euro NCAP awarded us with a 5-star rating. In particular, I believe we had the top scores in safety assist. We also did extremely well on AEB, automatic emergency braking for pedestrians and cyclists. Now you're seeing here, for example, AEB pedestrian is 5.6 out of 6 points. I think these numbers are kind of like slightly abstract. So let me actually put that in context. Like really, what are we talking about here? Pedestrian AEB is the following. Here's a simple— here's one example. The car just— oh, someone just merges from in between parked cars. And this car may not have even been on the Autopilot, but we continuously monitor the environment around us. We saw that there's a person in front, and we slam on the brakes automatically. Here's another example. Looks like a pretty benign scene. This person is probably not paying attention. The pedestrian is not paying attention. We slam on the brakes and prevent a collision. Here's one more example. You can see the trouble coming from the right. It's really hard to tell because everything is occluded, and then suddenly there's a pedestrian. You slam on the brakes and avert the collision again. So, you know, 5.6 out of 6 is kind of abstract, but we see a lot of these. We see tens to hundreds of these per day where we actually are averting a collision. And not all of them are true positives, but a good fraction are. And, you know, this is where you actually make your object detector really count. You're really averting collisions. Now, the goal of the team, however, is to produce full self-driving functionality. So here we have a video from the Autonomy Day that we had, like, somewhere earlier last year. Whoops. There we go. So the ambition of the team is to produce full self-driving, meaning that you can just set a pin arbitrarily in the world, not even on the highway system but everywhere else. Here you see the car just taking turns from the parking lot. It comes to an intersection. It waits for a green traffic light. It turns to the left, merges on the highway, and no need to touch the wheel. This is not available to the customers. We only have developer builds that do these turns through intersections and so on. But we're trying to get it to the point where we feel that the system is accurate enough to actually release in some form. So I'm going to cut this short, but this will actually get on the Cloverleaf in a bit. It will take the Cloverleaf. It will turn around and come back to our headquarters in Palo Alto, where a lot of the team is based. Now, I'm going to cut through this. You may be asking, like, you saw the car take a left through an intersection when the light was green. And you've actually seen a lot of these videos for like a decade. So, you know, here's a Waymo. I just took a small clip. A Waymo comes to an intersection and takes a left through the intersection. You've seen this for a decade. So how is this special? Why are we so late? What's taking so long? And, you know, it looks the same. So what's the— but I think the critical point to make is that it looks the same, but under the hood, it's completely different in terms of the approach that we take towards full self-driving. So in particular, we take a heavily vision-based approach. So everyone else has a LiDAR on the top of the car. The LiDAR shoots out lasers and creates a LiDAR kind of point cloud map. And then you pre-map the environment that you're going to drive. So you have a high-definition LiDAR map. And you localize to it to a centimeter-level accuracy. And then you know exactly the path you're going to take. So you can just steer to keep yourself perfectly localized in the correct region. So this is quite helpful. But this is not the approach that we take. We do not have LiDAR on our cars. So we don't shoot out lasers. It's all camera-based. And we do not build high-definition maps. So when we come to an intersection, we encounter it basically for the first time. We come to an intersection, how many lanes are there? Left, right, center? Which way should I turn? Where are the traffic lights? Which lanes do they control? All of this is just done just from camera feed, camera vision alone. And so this is a bit of a harder problem, but of course, once we actually get this to work, we can also deploy it to the millions of cars that we have globally. So let me now tell you a little bit about the neural networks that actually support all of the features that I've shown you that the Autopilot is capable of. And what it takes to actually keep these neural networks in production at scale in the hands of customers. So here's a random scene where we have to potentially drive and wind our way through this scene. As a human, this is relatively effortless, but you're actually doing a lot under the hood and the Autopilot has to as well. So we have to understand a lot about what are the static objects, what are the dynamic objects, where are the— what is the road layout, what is all of the layer of semantics on top of the road, like lane markings, road markings, traffic lights that tell you about how you can actually traverse that scene. So once you actually start writing down what's necessary to actually, you know, perform the driving policy, you end up with a long list of tasks, all of which we need to actually handle. And the detection of these tasks serves two purposes and two customers. Number one, we actually need it for the driving policy. And number two, we actually want to show a lot of things on the instrument cluster for the human so that they gain sort of confidence in the system. Here we've produced a video that shows some of the under-the-hood predictions for the main camera that is facing forward on the car. And you see that we're detecting lanes. We're detecting stop sign over there, a stop line, road markings. We're putting cuboids around cars, traffic lights, road edges and curbs, even things like trash bins coming up over there, lines that make up the intersection. In a bit, there's lines that create parking spots. We have to have all these attributes for whether or not a line is for a parking spot or not and things like that. So these predictions look pretty decent, but it's actually a huge amount of work to get them to actually work at scale and in the full long tail of what you can encounter on the roads. So even taking a very simple task, like for example, stop sign, you think that, you know, neural networks are capable of handling thousands of categories of ImageNet with all of their variations and so on. So how difficult is it to just detect a fixed pattern of like red on white stop? And it actually gets quite difficult when you get to the long tail of it, even to create a simple detector for a stop sign. So first of all, stop signs can, of course, be in lots of varying environmental conditions. Stop signs can not just be on poles but kind of just like on walls. Stop signs can be temporary and just hang out in different configurations. Stop signs can have funny lights on them, which are supposed to make it easier to see the stop sign, but for our system, it's the opposite. We have to actually explicitly worry about it. Stop signs can be held by a person. That stop sign can be in an inactive state or an active state. Stop signs can be on like these cones. Stop signs can be heavily occluded in lots of ways by foliage, by signs. Stop signs can be occluded by cars. Stop signs can be part of cars. And they can be part of cars in an active or inactive state again. Stops can be sort of held by a person. Stops come with lots of exotic modifiers to them. So this stop only applies except right turn. This stop It says, take your turn through this place. This stop is right turn, keep moving. So do not stop if you're going right. So there's a variety of modifiers. All of them we have to care about because they actually have impact on how you should drive through this final space. And as I mentioned, everything is AI vision-based at the moment. We do have maps, of course, that we build, but they're not high-definition maps. But stops can be attached to gates, but they only apply when the gate is closed. When it's open, they do not apply. Stops can be on arms. These arms can lift, and then the stop sign does not apply anymore. Stops can only apply when you're taking a left, so they're action conditional. And so basically, there's a massive variety of even just for a stop sign to get this to work. And what we do day to day in the team is we are going through the long tail, and we're building all this infrastructure for sourcing all of these additional examples. And so I've shown in the Autonomy Day presentation earlier last year the data engine, the process by which we iteratively apply active learning to source examples in cases where the detector is misbehaving. And then we source examples in those and we label them and incorporate them into part of a training set. So for stop sign detection, as an example, we have an approximate detector for stop signs based on an initial seed set of data. And we run that and deploy that to cars in shadow mode. And then you can detect some kind of a lack of health of that detector at test time. So for example, the stop sign detection is flickering. That could be a source of sort of, uncertainty. You can also detect that the neural network is uncertain in a certain Bayesian way, and there's a few ways to model that. You can find instances where you get surprised to see a stop sign, like why didn't you see it when you were a bit further away? You can also source examples when we detect a stop sign but the map thinks that there should be no stop sign or vice versa. So sort of a map-vision disagreement. So we have lots of different ways of sourcing difficult cases, and then we upload images and we look through them, we label some of them and incorporate them into a training set. As an example, we've struggled with these heavily occluded stop signs. We found that the detector was not performing very well when they were heavily occluded. And then we have a mechanism in this data engine process where we can actually train these kinds of detectors offline. So we can train a small detector that detects an occluded stop sign by trees. And then what we do with that detector is that we can beam it down to the fleet and we can ask the fleet, please apply this detector on top of everything else you're doing. And if the, the detector if this detector scores high, then please send us an image. And then the fleet responds with a somewhat noisy set, but they boost the amount of examples we have of stop signs that are occluded. And maybe 10% of them are actual occluded stop signs that we get from that stream. And this requires no firmware upgrade. This is completely dynamic and can just be done by the team extremely quickly. It's the bread and butter of how we actually get any of these tasks to work, just accumulating these large data sets in the full tail of that distribution. So we have tens of thousands of occluded stop signs. The fleet can send us as many as it takes. We have the biggest data set for accept right turn on stops. I'm basically certain of that. We again create a detector for this, and then we get a noisy stream from the fleet of cars encountering accept right turn on stops, and then we can label that, incorporate into training set. We have tens of thousands of these. I'm not actually 100% sure how you build out a data set like this without the fleet. So I talked a lot about massaging and growing the training sets, but we place just as much— we put just as much work into massaging the test sets as we do into training sets. So just reporting your mean average precision or reporting the loss function that you achieve in a test set, that just doesn't do it for us. So we spend a lot of time inspired by sort of test-driven development to build out very complicated unit tests and predicates for how the detector should behave in different cases. So for example, we could have a barrage of tests along the lines of, in this image, you should detect a stop sign right here. And the modifier that you, for this and that, that you predict on the stop sign must be this or that. And so we have mechanisms for creating and massaging these unit tests. It's a growing collection as we encounter problems in the fleet and as we notice them. And then we certify whether this detector is performing well or not. And this is our, this is really our test set. It's curated unit tests. If you're in a production setting, we actually like break down all these, all these different issues and track them individually. And then pursue them one by one with Data Engine to actually make them work. So occluded signs, we would build out a test set first. We see that we're failing on occluded signs, and then we— once we have the tests and we're failing them, we then spin the Data Engine on occluded signs, and then we see that the percent pass goes from 40% to 99% or something like that. And then we're confident that this is actually working relatively well. Now, so I've kind of shown you a little bit of the process we take for even a very simple one prediction. But as I mentioned, there's lots of tasks, lots of attributes. So the complexity of this is quite staggering. We have to know a lot about any scene, and each one of those things has their own data engine. So what we end up doing is we have these neural networks that have a shared backbone, and then they're multitask juggling lots of different tasks. And in total, the team is currently maintaining 48 networks that make 1,000 distinct predictions. These are just the raw number of output tensors. If you just take all of our networks and you add up the output tensors, it's 1,000. And those tensors, of course, can have multiple predictions in them. None of these predictions can ever regress, and all of them must improve over time. And this takes 70,000 GPU hours to train all the neural nets. If you have a node with 8 GPUs, you will be training for a year. So we do quite a bit of training there. And all of this is not maintained by hundreds of people. All of this is maintained by a small elite team of Tesla AI people of basically like a few dozen. So how is it even possible to make progress? And how is it even possible that your small team can maintain so many tasks and get them to work over time? So the North Star for actually achieving this for us is called Operation Vacation, which I'll describe in a second. But basically the idea is that for any new task, you have latency to actually the task working. And there's a process and we understand how to— the process for getting a task to work. And we're trying to develop as much automation machinery to actually support development of these new features and new tasks. And we're removing engineers from that loop so they're not involved. We're just building out infrastructure, and then we have a data labeling team and, you know, PMs and so on that can actually use that infrastructure to create new detectors. So as an example, we're working with caution lights recently. We're trying to detect when the police car lights are turned on. This is an example of a new task that we'd like to know about. And we sort of have a cookie-cutter— we know exactly what kind of approach to we know exactly what it takes to get a task to work. All the infrastructure is in place. So this task, we're going to treat it as a landmark task. Landmark is an example of a prototype. So we have a detection prototype, a segmentation prototype, a landmark prototype. And these are just classes of tasks. And if your new task is a member of any of these or an instance of any of these prototype classes, then all of the infrastructure is just plug and play and goes through the full data engine. You can collect the same dataset. You label your examples. You source more examples in the cases where you're failing. You deploy it in shadow mode. You source examples and you grind up metrics. You create all the unit test predicates. All this is completely automated. And this— we're mostly developing the automation infrastructure. And then it's easy to develop any new task. And that's kind of how we get this to work. OK. So now I'd like to talk a little bit about the neural networks and how they have to change to actually support full self-driving. So what I've explained so far is you take an image from one of the cameras. You run it through the shared backbone and you make lots of predictions about it in the image pixel space. So we can potentially predict objects, lines, or edges, and so on. Taking the example of edges, where does this go afterwards, right? So here's a video. And I'm showing the predictions of the road edge task on all of the different cameras. And you can see that this is basically a binary segmentation problem for us. And we detect those edges here and we use this to wind our way through a parking lot because we'd like to go to a certain destination but avoid all the road edges. Now, you can't just drive on the raw predictions of these edges in these 2-dimensional pixel— pixel coordinate systems. So you have to somehow project this out into a bird's-eye view in order to make this make sense. So the raw detections, we project them out into 3D. And we stitch them up into what we call an occupancy tracker. So this occupancy tracker— takes all the raw measurements from the 2D images. We project them out into the world. And we stitch them up across cameras. And we stitch them up across time. So the occupancy tracker is keeping the temporal context of that. And it creates a small local map. And then it winds its way through this parking lot to get to the person who is summoning the car at this time. So this is all fine. But there's a lot of problems with actually doing the stitching as I'm describing it. I'm making it sound very easy. The problem is these cameras are pointed in arbitrary directions. They make these measurements in these pixel frames. You might not— your road edges might not even align across camera scenes. And writing the tracker for these edges over time is actually highly non-trivial, lots of hyperparameters, error-prone code, very difficult to develop. So I've shown this example or this visualization a few times in many of my previous talks where this is sort of the Autopilot software stack. On the bottom, you have inputs to the stack, primarily vision. And on the output, we have steering and acceleration. And roughly, you can think of two code bases hidden inside the software stack. We have what I refer to as a Software 1.0 code, which is good old-fashioned C++, explicitly designed, engineered by a person. Person writes the code. And then you have what I call Software 2.0 code, where the code is an outcome of an optimization. It's the compilation. It's a compiler takes your data set and creates neural network code that runs in a car. And so it's, roughly speaking, neural net. The reason I refer to it as Software 2.0 is that you can take functionality from 1.0 code base and put it into 2.0 code base. So this boundary is fluid, and you can actually engulf more and more of Software 1.0 stack. So over time since I joined about 2 and a half years ago, the neural nets have expanded in how much of Software 1.0 land they've taken over. And so the trend is upwards. So in particular, in the case of these road edges as I described, what we're now working towards is that we don't actually want to go through this explicit occupancy tracker Software 1.0 code. We'd like to engulf it into neural net code. And we see that typically that works really well. So a lot of our networks have been moving towards these bird's-eye-view network predictions where you take all these cameras and you feed them through backbones. And then you have neural net fusion layer that stitches up the feature maps across the different views and also does the projection from image space to bird's-eye view. And then we have a temporal module potentially that actually smooths out these predictions. and this is all neural net components. And then we have a DevNet decoder that actually creates the top-down space. And so then we can directly potentially predict these road edges in a top-down view. Now, this makes a huge difference in practice. So here's an example. In top row, I'm showing the left camera, center, right. And I'm showing road edge detections in red. On the bottom row, on the left, you see the ground truth of what this intersection looks like. And you see that the car is positioned on the dot. And so that's the ground truth. On the very right, you see what happens when you take 2D predictions in the images and you cast them out. You basically get garbage. Like, it somewhat works around the car. There, the projection is easy. But far away, especially near the horizon line, a few pixels of error means lots of meters of error. And so these projection-based techniques are extremely hard to get to work. You'd have to know extremely accurate depth per pixel to do that projection. It's a very hard problem. And in the middle, you are seeing some of our predictions from the recent bird's-eye view networks. That just kind of make these predictions in top-down view directly. So let me show you a video here of what this looks like in practice. On the top, we will have a video of an intersection. On the bottom, I'm showing some of the predictions achieved by the Tesla AI team that has produced these bird's-eye view networks. We have road edges in red. We have dividers in green on the very left. Then we have attributes for these dividers. So we want to know if it's an actual curb or if it's just like a logical curb, so like a yellow line or something like that. Then we have traffic flow direction and a few other things like that. I'm only showing some examples. But the car is on the bottom center of these images. And as we're winding— as we're turning through this intersection, these bird's-eye view networks just predict pretty far away the full structure of this intersection and what it looks like. So this is extremely powerful for us. Here's another example in the case of Smart Summon. The car is driving around parking lots, and we have to understand the corridors of this parking lot. A parking lot. And so here we are predicting the corridors in bird's-eye view. We're predicting where the corridors meet at intersection points in red. And we're predicting tiny arrows that tell us the traffic flow. And this all comes out of the net directly. There's no occupancy tracker kind of necessary. Here's an example of using this for objects. So on the top, we have our camera feeds. On the middle, I'm showing our per-pixel depth predictions achieved by self-supervised learning. We need those depth predictions because we take, as many others do in the literature right now, we take a pseudolidar approach where you basically predict the depth for every single pixel. And you can cast out your pixels. And you basically simulate lidar input that way. But it's purely from vision. And then you can use a lot of techniques that have been developed also for lidar processing to actually achieve 3D object detection. So on the bottom, you have then predictions of our detector directly in the bird's-eye view. Yeah, here I have one more slide showing just our depth predictions for some other videos. This is not very exotic for any of you who are in computer vision because there are entire workshops on predicting and creating videos like these where we are basically achieving depth directly from images. And this is done— of course, no human can annotate depth. So this is done using self-supervised techniques. So in particular, we take an image. We predict a depth. Then when you have a depth for an image, you can cast out all the pixels into 3D. And you can reproject them either into camera views at that same time, but different cameras, or you can reproject them into the future frames of that single camera. And then you have a photometric loss that's enforcing consistency. And so the only way to predict— to make everything consistent in reprojections is to predict the correct depth. And so this learns in a self-supervised fashion to predict depth. And then we can simulate LiDAR inputs directly from vision and use this downstream to make these bird's-eye-view predictions. And as you might be familiar, the gap is quickly closing in state-of-the-art approaches. If you give yourself LiDAR and how well you can do versus if you do not have LiDAR but you just use vision techniques and use pseudo-LiDAR approaches, and that gap is quickly shrinking in academia. And we see a lot of this as well internally. Finally, I've shown you sort of how the software 2.0 stack is expanding through the code. It's eating some of the occupancy tracker-like techniques. The logical conclusion of that, of course, is there's still a perception system that creates explicit predictions. And those explicit predictions are, one, shown in the instrument cluster. And two, they are used for a policy. And this policy is still in the land of 1.0, where— for us and for many others as well— where you have explicit planner that takes all the predictions and drives on that. The problem with that is your explicit predictions are basically doomed to never be good enough and never be complete. And so writing these planners is extremely difficult, error-prone, lots of hyperparameters, very tricky. And so the logical conclusion of this, of course, is that we'd like to train neural networks that actually do a lot of this planning inside the network. In particular, we have huge datasets of people driving cars. So when people drive cars and they steer the wheel, they're actually data labeling for you. They're showing you how to drive through any intersection or any other kind of place. And the other thing I'll say is most of what I've described so far relies primarily on self-supervised learning, with the exception of depth, where we massage and create massive data sets. But we've seen recently with the use of, for example, contrastive losses and things like that, there's a lot of progress today in self-supervised learning. And so we'd like to leverage some of that in the team. Here's an example of just some videos. We have probably the most interesting largest data set of videos. These are coming from cameras. And so we've been really looking to some of these self-supervised techniques to speed up the process and to speed up the rate at which we learn from very little supervised data. So we don't have to collect like tens, hundreds of thousands of examples of accept, right, turn. We can do it. We prefer not to. Okay. So I showed you basically how we are improving the safety and convenience of driving today and how we deploy deep learning into production and support all the features of the Autopilot today. And then I also showed you all of the work we are doing to develop Full Self-Driving, how the neural net is eating through the software stack, and how we are putting vision and AI at the front and center of this effort. So if you'd like to make your AI expertise count, I think there is basically no more impactful or better place to apply some of that expertise than at Tesla today. And if you'd like to be a part of what I've shown you, then come to tesla.com/autopilotai, and you can just directly apply on the page. It's extremely painless, and that just comes directly to us. And you can help work with us to, you know, make this all work. So thank you.

Thank you, Andrej Karpathy. That was awesome. So we have time for a couple of questions while Ilya is setting up.

Hi. Hi.

So when you do these estimations, let's say, of a stop sign and other vehicles in your fleet have driven by it before, do you have a way of incorporating the prior from their estimation, or do you have to re-estimate from scratch?

Yeah, so we do collect maps. We just don't collect high-definition maps. Like when a Waymo or something like that is driving through an environment, they know that the leaf on that tree is 15.53 meters away, like, to centimeter-level accuracy. We don't know about the leaf of that tree, but we do know that there's a stop sign somewhere in the vicinity here. And so we do build low-definition maps where we do map stop signs, traffic lights, and things like that because it provides a second layer. And then also that helps us— again, like I mentioned, it helps us with all the trigger infrastructure for sourcing examples because we can look for discrepancies between map and vision. And that's a really good signal for us for getting lots of more images of really tricky cases.

So like, I drove by the stop sign yesterday.

Yes.

I estimate it as a stop sign.

Yes.

If I drive by it today—

Yes, thank you so much. OK, thanks.

Hi, Andrej Karpathy. Thank you for the great talk. I'm very interested in the software 2.0 part, and I follow progress. I think you mentioned it at Scaled ML. So I wonder, what's your overall vision for this? What are the main findings? Is this something that will supplant programming languages, will be kind of— change the way data science is done. What are your vision for this, and can the community help? Is this something that everybody can use to improve rapidly the whole progress of data science?

Yeah, so thank you. That's a very broad question. I think what I've seen so far is just porting functionality from software 1.0 to 2.0 is not always a good idea. It's a good idea sometimes. You're actually— it's a different way of programming, but you're placing yourself and a very indirect control of the final, of the final function. So sometimes you just want to like write a rule. If you're doing it in a Software 2.0 stack, everything is so indirect because you're massaging the dataset and you're going through a training process that takes days. The iteration is slower. You have to do all of this data engine stuff. So anything that's in the Software 2.0 stack like has a cost to it. You're committing to lots of data engine cost and any new problem that you find, suddenly requires a whole data engine pass and curation. And so that's not always a win. It's just sometimes a win. And so what we see actually is we kind of like combine Software 1.0 and 2.0. So the older systems, as I would describe them, in the Software 1.0 stack, they don't always go away. You almost think of it as like sedimentary layers where you have the new functionality, but the Software 1.0 is used, for example, to keep sort of guarding rails around the Software 2.0 part. So we almost see them as— you do a bit of both. And they both have pros and cons. But I do think that the trend, to a large extent, has been towards the software 2.0. It is a more powerful sort of— it is a more powerful way of addressing a lot of problems that are very messy and really hard to code, which is a lot of problems in the world. But we definitely see that they come with their own pros and cons. Thank you.

Thank you, Andrej Karpathy. We should move on. I really appreciate your time, and thanks again.