AI Agents Take the Wheel: Devin, SIMA, Figure 01 and The Future of Jobs

Three developments in the last 48 hours show how we are moving into an era in which AI models can walk the walk, not just talk the talk. Whether the developments quite meet the hype attached to them is another question. I've read and analyzed in full the three relevant papers and associated posts. To find out more, we'll first explore Devin, the AI system your boss told you not to worry about. Then Google DeepMind SEMA, which spends most of its time playing video games. And then figure one the humanoid robot, which likes to talk while doing the dishes. But the TLDW is this these three systems are each a long way from human performance in their domains, but think of them more as containers or shells for the vision language models powering them. So when the GPT4 that's behind most of them is swapped out for GPT5 or Gemini 2, all these systems are going to see big and hard to predict upgrades overnight. And that's a point that seems especially relevant on this, the one year anniversary of the release of GPT4. But let's start, of course, with Devin billed as the first AI software engineer. Now, Devin isn't a model, it's a system that's likely based on GPT4. It's equipped with a code editor, shell and browser, so of course it can not just understand your prompt, but look up and read documentation. A bit like AutoGPT. It's designed to come up with plans first and then execute them, but it does so much better than AutoGPT. But before we get to the benchmark that everyone's talking about, let me show you a 30 second demonstration of Devin in action.

All I had to do was send this blog post in a message to Devin. From there, Devin actually does all the work for me, starting with reading this blog post and figuring out how to run the code in a couple of minutes. Devin's actually made a lot of progress, and if we jump to the middle here, you can see that Devin's been able to find and fix some some edge cases and bugs that the blog post did not cover for me. And if we jump to the end, we can see that Devin sends me the final result, which I love. I also got two bonus images here and here, so let me know if you guys see anything hidden in these.

It can also fine tune a model autonomously, and if you're not familiar, think of that as refining a model rather than training it from scratch. That makes me wonder about a future where if a model can't succeed at a task, it fine tunes another model or itself until it can anyway, this is the benchmark that everyone's talking about. Swe bench software engineering bench Devin got almost 14% and in this chart crushes Claude 2 and GPT 4, which got 1.7%. They say Devin was unassisted, whereas all other models were assisted, meaning the model was told exactly which files need to be edited. Before we get too much further though, what the hell is this benchmark? Well, unlike many benchmarks, they drew from real World Professional problems 2294 Software engineering problems that people had and their corresponding solutions. Resolving these issues requires understanding and coordinating changes across multiple functions, classes and files simultaneously. The code involved might require the model to process extremely long context and perform, they say, complex reasoning. These aren't just fill in the blank or multiple choice questions. The model has to understand the issue, read through the relevant parts of the code base, remove lines and add lines. Fixing a bug might involve navigating a large repo, understanding the interplay between functions in different files, or spotting a small error in convoluted code. On average, a model might need to edit almost two files, three functions, and about 33 lines of code. One point to make clear is that Devin was only tested on a subset of this benchmark, and the tasks in the benchmark were only a tiny subset of GitHub issues. And even all of those issues represent just a subset of the skills of software engineering. So when you see all caps videos videos saying this is AGI, you've got to put it in some context. Here's just one example of what I mean. They selected only pull requests which are like proposed solutions that are merged or accepted that solve the issue and that introduce new tests. Would that not slightly bias the dataset toward problems that are easier to detect, report and fix? In other words, complex issues might not be adequately represented if they're less likely to have straightforward solutions. And narrowing down the proposed solutions to only those that introduce new tests could bias bias towards bugs or features that are easier to write tests for. That is to say that highly complex issues where writing a clear test is difficult may be underrepresented. Now, having said all of that, I might shock you by saying I think that there will be rapid improvement in the performance on this benchmark. When Devin is equipped with GPT5, I could see it easily exceeding 50%. Here are just a few reasons why. First, some of these problems contained images, and therefore the more multimodal these language models get, the better they'll get. Second, and more importantly, a large context window is particularly crucial for this task. When the benchmark came out, they said models are simply ineffective at localizing problematic code in a sea of tokens. They get distracted by additional context. I don't think that will be true for much longer, as we've already seen with Gemini 1.5 third reason models, they say, are often trained using standard code files and likely rarely see patch files. I would bet that GPT5 would have seen everything. Fourth language models will be augmented, they predict, with program analysis and software engineering tools. And it's almost like they could see six months in the future because they said to this end we are particularly excited about agent based approaches like Devin for identifying relevant context from a code base. I could go on, but hopefully that background on the benchmark allows you to put the rest of what I'm going to say in a bit more context. And yes, of course I saw how Devin was able to complete a real job on upwork. Honestly, I could see these kind of tasks going the way of copywriting tasks on upwork. Here's some more context though we don't know the actual actual cost of running Devin for so long. It actually takes quite a while for it to execute on its task. We're talking 15, 20, 30 minutes, even 60 minutes sometimes. As Bindu Ready points out, it can get even more expensive than a human, although costs are of course falling. Devin, she says, will not be replacing any software engineer in the near term. And noted deep learning author Francois Charlais predicted this there will be more software engineers, the kind that write code, in five years, than there are today, and newly unemployed Andrej Karpathy says that software engineering is on track to change substantially with humans more supervising the automation, pitching in high level commands, ideas or progression strategies. In English I would say with the way things are going they could pitch it in any language and the model will understand. Frankly, with vision models the way they are, you could practically mime your code idea and it would understand what to do. And while Devin likely relies on GPT4, other competitors are training their own frontier scale models. Indeed, the startup magic, which aims to build a co worker, not just a co pilot for developers, is going a step further. They're not even using transformers. They say transformers aren't the final architecture. We have something with a multi million token context window. Super curious of course how that performs on SWE bench. But the thing I want to emphasize again comes from Bloomberg Cognition AI admit that Devin is very dependent on the underlying models and use GPT4 together with the reinforcement learning techniques. Obviously that's pretty vague, but imagine when GPT5 comes out with scale, you get so many things, not just better coding ability. If you remember, GPT3 couldn't actually run reflect effectively, whereas GPT4 could. If GPT5 is twice or ten times better at reflecting and debugging, that is going to dramatically change the performance of the Devin system overnight. Just delete the GPT4 API and put in the GPT5 API and wait. Jeff Klune, who I was going to talk about later in this video, has just retweeted one of my own videos. I literally just saw this two seconds ago when it came up as a notification on my Twitter account. This was not at all. No, the success rates weren't perfect, but if you look at the averages, and this is for GPT4 Vision, they are pretty high. 91%, 82%, 82%. These numbers in the middle by the way on the left reflect the number of steps that GPT4 Vision took and on the right the number of steps that a human took. And that's just GPT4 vision, not a model optimized for agency, which we know that OpenAI is working on. So before we even get to video games, you can imagine an Internet where there are models that are downloading, liking, commenting, doing pull requests and we wouldn't even know that it's AI. It would be, as far as I can tell, undetectable. Anyway, I'm getting distracted. Back to the SEMA paper. What is SEMA in a nutshell? They got a bunch of games, including commercial video Games like Valheim, 12 million copies sold at least, and their own made up games that Google created. They then paid a bunch of humans to play those games and gathered the data. That's what you could see on the screen, the images and the keyboard and mouse inputs that the humans performed. They gave all of that training data to some pre trained models and at this point the paper gets quite vague. It doesn't mention parameters or the exact composition of these pre trained models. But from this we get the SEMA agent which then plays these games or More precisely tries 10 second tasks within these games. This gives you an idea of the range of tasks, everything from taming and hunting to destroying and headbutting. But I don't want to bury the lead. The main takeaway is this training on more games saw positive transfer when Seema played on a new game. And notice how Seema in purple across all of these games outperforms an environment specialized agent that's one trained for just one game. And there is another Gem buried in this graph. I'm colorblind, but I'm pretty sure that's teal or lighter blue. That's zero shot. What that represents is when the model was trained across all the other games, bar the actual game it was about to be tested in. And so notice how in some games like Goat Simulator 3 that outperformed a model that was specialized for just that one game, the transfer effect was so powerful it outdid the specialized training. Indeed, SEMA's performance is approaching the ballpark of human performance. Now I know we've seen that already with Starcraft 2 and OpenAI beating Dota, but this would be a model generalizing to almost any video and they are pretty high. 91%. 82%, 82%. These numbers in the middle, by the way on the left reflect the number of steps that GPT4 Vision took, and on the right the number of steps that a human took. And that's just GPT4 vision, not a model optimized for agency, which we know that OpenAI is working on. So before we even get to video games, you can imagine an Internet where there are models that are downloading, liking, commenting, doing pull requests and we wouldn't even know that it's AI. It would be, as far as I can tell, undetectable. Anyway, I'm getting distracted. Back to the SEMA paper. What is SEMA in a nutshell? They got a bunch of games, including commercial video Games like Valheim, 12 million copies sold at least, and their own made up games that Google created. They then paid a bunch of humans to play those games and gathered the data. That's what you could see on the screen, the images and the keyboard and mouse inputs that the humans performed. They gave all of that training data to some pre trained models. And at this point the paper gets quite vague. It doesn't mention parameters or the exact composition of these pre trained models. But from this we get the SEMA agent which then plays these games or More precisely tries 10 second tasks within these games. This gives you an idea of the range of tasks, everything from taming and hunting to destroying and headbutting. But I don't want to bury the lead. The main takeaway is this training on more games saw positive transfer when SEMA played on a new game. And notice how Seamer in purple across all of these games outperforms an environment specialized agent that's one trained for just one game. And there is another gem buried in this graph. I'm colorblind, but I'm pretty sure that's teal or lighter blue. That's zero shot. What that represents is when the model was trained across all the other games bar the actual game it was about to be tested in. And so notice how in some games like Goat Simulator 3 that outperformed a model that was specialized for just that one game, the transfer effect was so powerful it outdid the specialized training. Indeed, SEMA's performance is approaching the ballpark of human performance. Now, I know we've seen that already with Starcraft 2 and OpenAI beating Dota, but this would be a model generalizing to almost any video game. Yes, even red Dead Redemption 2, which was covered in an entirely separate paper out of Beijing. That paper, they say, was the first to enable language models to follow the main storyline and finish real missions in complex AAA games. This time we're talking about things like protecting a character, buying supplies, equipping shotguns. Again, what was holding them back was the underlying model GPT4V. As I've covered elsewhere on the channel, it lacks in spatial perception. It's not super accurate with moving the cursor for example. But visual understanding and performance is getting better fast. Take the challenging benchmark mmmu. It's about answering difficult questions that have a visual component. The benchmark only came out recently, giving top performance to GPT4V at 56.8%, but that's already been superseded. Take Claude 3 Opus, which gets 59.4%. Yes, there is still a gap with human expert performance, but that gap is narrowing like we've seen across this video. Just like Devin was solving real world software engineering challenges, SEMA and other models are solving real world games, walking the walk, not just talking the talk. And again, we can explain expect better and better results the more games SEMA is trained on. As the paper says, in every case SEMA significantly outperforms the environment specialized agent, thus demonstrating positive transfer across environments. And this is exactly what we see in robotics as well. The key take home from that Google DeepMind paper was that our results suggest that co training with data from other platforms imbues RT2X in robotics with additional skills that were not present in the original data set, enabling it to perform novel tasks. These were tasks and skills developed by other robots that were then transferred to RT2. Just like SEMA getting better at one video game by training on others. But did you notice there that smooth segue I did to robotics? It's the final container that I want to quickly talk about. Why do I call this humanoid robot a container? Because it contains GPT4 vision. Yes of course its real time speed and dexterity is very impressive, but that intelligence of recognizing what's on the table and moving it appropriately comes from the underlying model GPT4 vision. So of course I have to make the same point that the underlying model could easily be upgraded to GPT5 when it comes out. This humanoid would have a much deeper understanding of its environment and you as you're talking to it. Figure 1 takes in 10 images per second and this is not teleoperation, this is an end to end neural network. In other words, there's no human behind the scenes controlling this robot figure. Don't release pricing, but the estimate is between $30,000 and 150,000 DOL per robot. Still too pricey for most companies and individuals. But the CEO has a striking vision. He basically wants to completely automate manual labor. This is the roadmap to a positive future. Powered by AI, he wants to build the largest company on the planet and eliminate the need for unsafe and undesirable jobs. The obvious question is if it can do those jobs, can't it also do the safe and desirable jobs? I know, I'm back to the jobs point again, but all of these questions became a bit more relevant. Relevant, let's say, in the last 48 hours. The figure CEO goes on to predict that everywhere from factories to farmland, the cost of labor will decrease until it becomes equivalent to the price of renting a robot, facilitating a long term holistic reduction in costs over time. Humans could leave the loop altogether as robots become capable of building other robots, driving prices down even more. Manual labour, he says, could become optional. And if that's not a big enough vision for the next two decades, he goes on that the plan is also to use these robots to build new worlds on other planets. Again, though, we get the reassurance that our focus is on providing resources for jobs that humans don't want to perform. He also excludes military applications. I just feel like his company and the world has a bit less control over how the technology is going to be used than he might think. It does indeed. Jeff Klune of OpenAI, Google, DeepMind, SEMA and earlier on in this video, Fame reposted this from Eduard Harris. It was a report commissioned by the US government that he worked on and the TLDR was that things are worse than we thought and nobody's in control. I definitely feel we're noticeably closer to AGI this week than we were last week. As Jeff Klune put out yesterday, so many pieces of the AGI puzzle are coming together. And I would also agree that as of today, no one's really in control. And we're not alone with Jensen Huang, the CEO of Nvidia, saying that AI will pass every human test in around five years time. That, by the way, is a timeline shared by Sam Altman. This is a quote from a book that's coming out soon. He was asked about what AGI means for marketers. He said, oh, for that it will mean that 95% of what marketers use, agencies, strategists and creative professionals for today will easily, nearly instantly and at almost no cost be handled by the AI. And the AI will likely be able to test its creative output against real or synthetic customer focus groups for predicting results and optimizing again, all free, instant and nearly perfect images, videos, campaign ideas, no problem. But specifically on timelines he said this when asked about when AGI will be a reality, he said five years, give or take, maybe slightly longer. But no one knows exactly when or what it will mean for society. And it's not like that timeline is even unrealistic in terms of compute. Using these estimates from semi analysis, I calculated that just between quarter one of 2024 and the fourth quarter of 2025 there will be a 14x increase in compute. Then if you factor in algorithmic efficiency doubling about every nine months, the effective COMPUTE at the end of next year will be almost a hundred times that of right now. So yes, the world is changing and changing fast and the public really need to start paying attention. But no, Devin is not AGI, no matter how much you put it in all caps. Thank you so much for watching to the end and of course I'd love to see you over on AI Insiders on Patreon. I'd love to see you there. But regardless, thank you so much for watching and as always, have a wonderful day.