Worlds FIRST AGI SOFTWARE ENGINEER Just SHOCKED The ENTIRE INDUSTRY! (FULLY Autonomous AI AGENT

So there was a recent announcement from this company called Cognition Labs and they announced something that actually did shock the entire industry as it's the first AI software engineer. You can see right here, it says, today we're excited to introduce Devin, the first AI software engineer. Devin is the new state of the art on the SWE bench coding benchmark, has successfully asked practical engineering interviews from leading AI companies and has even completed real jobs on upwork. Devin is an autonomous agent that solves engineering tasks through the use of its own shell code editor and web browser. When evaluated on the SWE benchmark, which asks an AI to resolve GitHub issues found in real world open source projects, Devon correctly resolves 13.86% of the issues unassisted, far exceeding the previous state of the art model performance of 1.96% unassisted and 4.8% assisted. Check out what Devin can do in the thread below. So let's take a look at some of the demo videos they showed us because honestly, it is quite impressive on what this can really do.

Hey, I'm Scott from Cognition AI and today I'm really excited to introduce you to Devin, the first AI software engineer. Let me show you an example of Devin in action. I'm going to ask Devin to benchmark the performance of Llama on a couple different API providers. From now on, Devin is in the driver's seat. First, Devin makes a step by step plan of how to tackle the problem. After that, it builds the whole project using all the same tools that a human software engineer would use. Devin has its own command line, its own code editor, and even its own browser. In this case, Devin decides to use the browser to pull up API documentation so that it can read up and learn how to plug into each of these APIs. Here, Devin runs into an unexpected error. Devin actually decides to add a debugging print statement, reruns the code with the debugging print statement, and then uses the error in logs to figure out how.

To fix the bug.

Finally, Devin decides to build and deploy a website with full styling as the visualization. You can see the website here. All of this is possible today because of the advanced style that we've made in both reasoning and long term planning. It's a really hard problem and we've only just started, but we're super excited about the progress that we've made so far. In the meantime, if you'd like to try out Devin on your own real world tasks, send us a request below and we'd be happy to forward it to Devin.

So that was a rather impressive short demo and there are much more in the video that I'm going to be showing you. But this is showing us that we are moving to a very, very interesting part of society where, where these AI technologies are going to keep sprouting up. Companies and investors are going to keep building these platforms where you can do absolutely amazing things. And we're likely to see shakeups to the entire industry. Now, with Devon being the world's first AI software engineer, I think the advancement is not to be understated here. This is an autonomous AI agent that is able to do a variety of software engineering tasks to a very, very good degree and, and is really, really going to impact the sector moving forward if things continue at this pace. Now, we saw in this first demo that it was able to even pull up the browser, to pull up the API documentation, to understand what's going on, and then even do the debugging so it could fix the issue by itself. So this was really cool to see firsthand how this AI system was able to solve an issue on its own. Kind of look around, browse around, and then manage to complete the task and completely without human assistance. And right now the benchmarks are state of the art, meaning that this is currently the best we have and in the future it's likely to improve. So I wonder what the future will look like in just a couple of years, because my, oh my, things are moving quickly now. There are also some other demos that show just how good this system is, and I'm going to show you guys those right now because Devin is able to do a majority of things that I didn't think it was able to do. And, and one of the key things about this that makes it so cool is the long term planning and the reasoning breakthrough that they've had, which I'll talk about later, which is a key factor in why this is really effective. So let's take a look at how, first of all, Devin can actually do tasks on upwork.

Hey, I'm Walden, one of the developers here at Cognition AI. We were playing around with whether or not Devin could start a side hustle on upwork. So here's actual real job from upwork where the client wants to set up this computer vision model, which actually looks quite interesting. Seems very difficult to set up. I'm not sure how I would start doing this, but you know, you give the task to Devin and ask Devin to figure it out and things just kick off. Devin immediately goes ahead and you can see it Sort of starts setting up the repo. It actually runs into some issues here with the versioning. So if you watch how Devin deals with it, Devin's actually updating the code to make these things work. He continues with this loading and importing packages. You can see that actually downloads images from the Internet to run through the model. You can see here that there are actually some issues that come across. However, Devin knows how to handle these things. Devin kind of pushes through and if you look closely, Devin's actually doing print line debugging here, where Devin is adding these statements to track where the data flows. And Devin continues to do this until Devin understands how everything's working and actually then updates the code with the fixes. After removing print line statements, Devin continues this pattern of fixing code and running it again until it runs the image model across all these rows across the world. We can ask for a report from Devin, at which point Devin sends over some sample images of roads with damage marked out and a nice. TXT file explaining Devin's work and the different kinds of outputs of the model. Good job, Devin.

So you can see here that this was a quick demo on how it can do certain tasks on upwork. And I think once again, the reason I stated that this is going to shake up the industry is because if it's able to do many different upworks tasks, then that means that it could potentially shake up the gig economy by removing future jobs that many people used to do. So once again, this is something that is shaking up the entire industry in various different sectors. But one thing that I was really impressed with from this demo, you can see right here that it actually has a little planner. And then as this person is running the demo, you can see live what this AI is planning to do. Now, one thing that is quite, I wouldn't say frightening, but rather intriguing was that many people did state that these AI systems wouldn't get long term planning because it would be, I guess you could say, rather dangerous if the AI wanted to do something that we wouldn't be aware of. And I think right here in this example, of course, having long term planning is vital to achieving any goal that humans usually do. And you can see right here, the plan is to clone the repository from the provided GitHub link, read the documentation, install the necessary dependencies, run the models and review the output to verify the accuracy and understand the result. Next, here we have a demo of it being able to do unfamiliar tasks.

Hey everyone, my name is Sarah and I'm going to show you how Devin, our AI software engineer can autonomously learn from a blog post. Within a few minutes, Devin successfully generated this desktop background image for me with my name on it. So all I had to do was send this blog post in a message to Devin. From there, Devin actually does all the work for me, starting with reading this blog post and figuring out how to run the code in a couple of minutes. Devin's actually made a lot of progress and if we jump to the middle here, you can see that Devin's been able to find and fix some edge cases and bugs that the blog post did not cover for me. And if we jump to the end, we can see that Devin sends me the final result, which I love. I also got two bonus images here and here, so. So let me know if you guys see anything hidden in the Once again.

Another really impressive demo on Devin being able to do unfamiliar tasks, which shows that this LLM can learn. And it's just really, really fascinating to see how all of these AI agents are being done with various different startups. Now here's another demo in which Devin adds a feature to an open source repository.

Hey, I'm Tony, an engineer at Cognition. I helped build Devin and and now Devin helps me too. Today at work I wanted to run a bunch of commands at once and be able to track their status. On one screen I found an open source tool named mprox to do this. Here it is right here. Looks like it all finished, but the status is way too vague. I don't know which ones failed, they all just say down. I really want to improve the UX here, but I'm not familiar with the code at all. So I had Devin, my AI software engineer, help me. Looks like this person right here had the same issue. So all I gave Devin was the link to the issue and asked Devin to fix it. You can see me make the request right here on the left. Let's see what Devin did. On the right, we can track Devin's work and watch Devin jump from tool to tool. First, Devin clones the repository using the shell, then reads the readme in an editor to learn how to set up the code, then goes back to the shell to install the required dependencies. Devin also opens up a web browser to take a look at the issue. Now Devin starts coding. At some point, Devin even opens up some Rust documentation to debug a compiler error. Finally, Devin finishes the task and reports a summary of the changes that were made. Let's see if the changes worked. I have Devin's code right here. Looks like it worked. The Third command succeeded. I can even see the status codes. Here's all the code that Devin wrote for this change. Thanks, Devin.

We also have another really cool demo of Devin making the Game of Life by someone else that works at Cognition.

Hi, I'm Adyan and today I felt like playing the Game of Life, so I asked Devin to implement it for me. Devin started creating a new react application using the shell, and then it started writing some code through its editor. After that it deployed the app through Netlify. Let's check it out. That seems nice, but there's a lot more features which I want to add. So let's ask Devin to do this one at a time. I want the words Devin to be written at the initialization screen instead of it being random. Then I want the word to be slightly bigger and the frame rate to be faster. I also want him to fix a bug where the screen gets flipped freezed after three seconds. Let's see the progress Devin has made so far. You can see the diff. And the last div shows that Devin just fixed the bug where the screen gets frozen after three seconds. Seems reasonable to me, so let's move on. Next, I want Devin to increase the frame rate after 10 seconds and also to make the website responsive to different window sizes. I also wanted to make it interactive so that when I click my mouse somewhere, it should spawn a new block. Let's check out what Devin has made so far. Started with Devin, which is what we asked for, and when I click something, it creates a new block as well. That's fun. Let's play around with it. Well, that was my evening.

There was also another really, really cool application where Devin was able to fine tune its own models. And I found this one to be really, really fascinating. Because when I read a recent research paper about LLMs being able to fine tune their own models, they they actually weren't able to. So this is an iteration. Not on that, but I just find it funny how, you know, recent updates have shown that certain things can't be done and then boom, out of nowhere, two days later, someone else makes it happen.

Hey guys, Today I'm going to show you an AI training an AI. So here we're going to take the Qlora repo, which is a fine tuning method for quantizing large language models. We're going to feed this repo to our agent, Devin, and all we have to ask Devin is to fine tune a 7B llama model Devin clones. The repo figures out how to run it using the readme sets up all of the requirements using pip, looks through all the scripts, and is able to start running the training job. There are a few hiccups where Devin runs into some CUDA issues, which is to be expected with open source repos, but it's not a problem. Devin looks at the Nvidia environment and figures out how to reinstall the packages to make it work. After a few more runs figuring out the correct model names, Devin successfully gets the training run working. Here we see training proceeding smoothly. Loss is going down, and after a few steps looks pretty good. I tell Devin to wait as the training job runs. After about an hour, I come back, ask Devin, hey, how's the training going? Devin helps me look. A few hundred steps are done now and everything is still proceeding smoothly. Looks great. Thanks, Devin, for helping me set up my training run.

See, I think this is a rather fascinating thing that an AI system is able to train its own models. I mean, the implications here are staggering. There's a ton of different applications. But there is one more thing that I want to show you guys from the Cognition team and then I want to get into some of the really cool details like the inner workings behind Devin. So to give you guys a quicker update.

Hey, I'm Andrew, an engineer at Cognition, and I wanted to share a pretty amazing experience I had with Devin. So I maintained this big open source repository which contains a lot of different algorithms used for competitive programming. A lot of people use it and a few weeks ago my friend texted me that, you know, there was actually a bug in one of the, in one of the implementations. The implementation wasn't quite right when the inputs weren't, weren't relatively prime. I kind of glossed over that case when I was implementing it, so I never really thought about it. So I implemented a quick fix and then I thought that I should test it, but I actually never really got around to writing any test cases. So I thought if I don't want to do it, I should just ask Devin to do it instead. So I gave Devin the repository, asked Devin to just check it out and start working on it. So Devin found the right repository, checked it out, found all files that are in the repository, and then I told Devin what test case I wanted him to write. I just told Devin, these are the inputs, and then try checking for these conditions. So Devin wrote the test without too much trouble. Devin just looked around to understand what exactly the test should look like and what exactly the interfaces were. And with this, Devin ran the tests, ran into a quick hiccup which is a compiler, but Devin is able to solve those very effectively and just added an extra include to fix that and then was done running this initial test. So then I asked Devin to actually expand the test a little bit. Instead of just testing this one input, I wanted Devin to write tested on all inputs. So just kind of the brute force testing strategy. I use this a lot in my tests and I just wanted Devin to implement it so that I didn't have to worry about it. So Devin went and rewrote the test function to use for nested for loops. But this time, after Devin ran the tests, Devin actually found a test failure. You know, if the code were correct, there could be compile errors in the test. But you know, the tests seem really pretty reasonable, so there probably shouldn't be a failure. So Devin went and tried to debug the program for me. So Devin here actually wrote, actually added a print statement to debug the outputs and the inputs to the failing test, reran the test and actually found which case was wrong in this case, these are the inputs. And then the return value was actually negative 9. And the, the code I'm running actually should never really return negative values. So Devin realized this and actually went looking in the, when looking in the code that we're trying to test and actually added this line of code that if extra less than zero extra plus equals, you know, plus equals something. And so in order to make sure that the return value was actually non negative. So after fixing this, Devin actually reran the tests and now, now I can be confident that my code is correct and I have some tests to prove it.

There was also something really fascinating that I did see was of course the funding. And they said that we are well funded, including a 21 million Series A LED by Founders Fund. And we're grateful to support this from the industry that they do have. And what's crazy also is that they said by solving reasoning we can unlock new possibilities in a wide range of disciplines. Code is just the beginning. So it seems that this company with Devon, whatever they're building, it seems that it doesn't just seem like it's going to be a software. If they've been able to solve the issue with AIs being able to reason effectively and having this agent autonomously do tasks like this on the Internet, I think they can definitely gain a huge market share for the autonomous agent sector which is going to be booming in the years 2024 and beyond. So I think that this company is very, very well poised to take a decent sized market share because there aren't really any other tools that we've seen demoed like this that are going to be that well. Now in addition, we can also see the SWE benchmark and they talked about how it achieved very, very good results being the state of the art model. And of course they said that the performance of Devon on the SWE benchmark is essentially impressive, especially when you consider it state of the art models. Achieving a 13.86% resolution for real world GitHub issues in open source projects is actually a pretty notable accomplishment considering the complexity and variety of problems that can occur in such projects. Now what actually stands out with Devon is that it significantly performs the previous models and even when those models were given additional information such as the exact files to edit. And this actually suggests that Devon has has a more robust understanding of code and the context in which it operates, which allows it to autonomously navigate and fix issues within a code base without explicit directions. Now moreover, there was something really cool. Devin's ability to perform unassisted is a key differentiator. Being evaluated On a random 25% subset of the dataset indicates that the AI's performance is not tailored to specific types of problems, but rather generally applicable, which is a desirable trait for an autonomous AI system meant for real world application. So what they also did state that was rather fascinating too was that they are going to have a technical report which will provide greater insight into the methods and technologies that enable Devon's advanced capabilities. And it's likely going to be eagerly anticipated by the AI community and the software development. And these results will be a signal of how increasingly valuable AI is going to come in terms of the software engineering community. Now one thing that they also do talk about was the secret technique. So it states here exactly how cognition AI made this breakthrough and in such a short time is something short of a mystery, at least to outsiders. Wu declines to say much about the technology's underpinnings other than that his team found unique ways to combine large language models, such as OpenAI's GPT4 with the reinforcement learning techniques. It's obviously something that people in this space have thought about for a long time, he says. It's very dependent on the models and the approach and getting things to align just right. And it seems that they made significant strides in in this particular thing by combining some of the best techniques that we know in AI. Now, of course, they're hinting at a proprietary blend of technologies or methodologies that they've pioneered, which could be the core of their breakthrough. And the specific details of how these technologies are integrated and leveraged to achieve such breakthroughs are kept under wraps, adding an element of mystery and of course, protecting trade secrets. Now, reinforcement learning is a powerful method in AI, where algorithms learn to make decisions by receiving rewards or penalties for the actions they take, Rather like training a pet. And when combined with large language models like GPT4, which already have strong understanding of human language, the potential is there to create an AI that can improve itself through iterative processes, potentially at a rate and efficiency that is unprecedented. Now, the approach of aligning the models and getting things to align just right actually suggests a delicate balance and a fine tuning process that could have taken substantial time and experimentation to perfect. It's actually quite a tantalizing peek into the kind of advanced AI development happening within cognition AI, possibly heralding a new era of software engineering tools that could actually revolutionize the industry. Now, Andrej Karpathy actually does talk about this, and his tweet outlines a intriguing parallel between the evolution of autonomous driving and the automation of software engineering. It's a compelling analogy that tracks the incremental steps of AI's increasing involvement and sophistication in task completion. His progression for software engineering automation with AI begins with basic assistance and moves towards more complex and integrated functions. And this trajectory indicates a future where AI handles more of the day to day coding tasks, enabling developers to focus on higher level design and problem solving. And Devin actually represents a leap in this evolution, coordinating multiple development tools and acting with more autonomy. This actually does suggest that human oversight will shift to a higher abstraction level, and that is particularly interesting. It implies that the future of software engineers may operate more like managers or architects, guiding the AI strategy rather than writing every line of code. And Karpathy also touches on the essential aspect of AI integration into software engineering, which is the user interface design. The interaction between humans and AI must be seamless and intuitive, allowing for developers to efficiently guide and correct the AI. It's not just about making the AI smarter, it's about designing environments where AI and humans work together effectively. And his closing remarks also underscore the significant changes ahead for the field of software engineering. As AI tools like Devon become more capable, the role of a software engineer will likely transform, emphasizing supervision and high level conceptual work over traditional coding. It's actually an exciting prospect, and one that carries a promise of increased productivity and the potential to tackle more complex problems than ever before. Overall, what we've seen here today was the very first in possibly a long line of autonomous AI agent software developers, but this is the first and very likely won't be the last. We know that many other companies are working on this, and they've all had very similar break. It will be interesting to see what other companies come to market with their products now that Cognition have come out with Devon, and it seems like they are leading the race in terms of what we are expecting when we look at autonomous AI agents, especially in the coding space. So with that being said, what do you think the future of this is? Are you excited or or are you someone who is a little bit more on the pessimistic side of what's to come? Either way, your thoughts and opinions are appreciated. And if you did find value in the video, do not forget to leave a comment down below. Subscribe for future updates.