New OPEN SOURCE Software ENGINEER Agent Outperforms ALL! (Open Source DEVIN!)

So there has been an announcement of a advanced level open source software engineering agent. And you can see here that this is really, really striking because it was only recently that we had Devin be the first autonomous software engineer. And it was something that took the industry by storm. So in this video I'm going to be giving you guys 10 of the key takeaways on this open source agent and what it is able to do effectively. So here we can see the announcement. It says, SWE Agent is our new system for autonomously solving issues in GitHub repos. It gets similar accuracy to Devin on the software engineering benchmarks and takes 93 seconds on average. And it's open source and we designed a new agent computer interface to make it easy for GPT4 to edit and run code. So let's take a look at some of the 10 things that you should know. One of the first things that you should know is that it is open source and then completely open source. You can see right here that this is absolutely incredible. On the open source comparative benchmarks we can see that it achieves a 12.29% compared to Devin's 13.84%. Now why is this so crazy? Well, if you remember, Devin, an actual software engineering agent that wasn't open source and actually had a $25 million Series A funding round. And comparatively, this small team of open source developers have managed to achieve relatively similar results in, I would argue, a shorter amount of time and with a lot less capital. Which goes to show that open source could be achieving rapidly remarkable results in shorter time spans than than bigger teams. Which does mean that it's quite surprising at how effective this team is in order to build something as quickly and as effective as they have. Now, what's crazy as well is the distance between these two. It's not like there is a massive difference. It's only literally 1%. So you could argue that these models are practically similar. And it will be interesting to see how in future versions the actual abilities do increase when with scale and with Future models like GPT5 or upgraded versions of GPT 4.5. Now, like I said before, if you compare this and point number two is that if we actually look once again at Devin's benchmark, we can see that the other systems that it was comparing itself against were far, far lower in comparison. But if we look at the compared benchmarks in point number two, we can see that it really, really has just jumped up the gap, if it exists just here, because that means now that open source has pretty much caught up to the state of the art closed source in terms of what is capable. Which just goes to show that open source models, unlike LLMs, it actually could potentially catch up or even overtake them. And I'm guessing the reason that in comparative benchmarks why open source could catch up to closed source is because both of these models are using the base level of GPT4 or potentially Claude Opus, considering they do have advanced planning and coding capabilities natively built into them. Now let's move on to point number three. Point number three is exactly how this works. So how does this software engineering agent work? So the software engineering works by interacting with a specialized terminal which allows it to open, scroll and edit through files. It also allows it to edit specific lines with automatic syntax checks and and it allows it to write and execute tests and the custom built interface is critical for good performance. So this is something where they essentially describe how it works in a specialized terminal which allows it to think through its actions. We can see that by looking at the demonstration. We can see that there are thoughts and actions and then there are observations where it is able to check what it is doing. So we can see right here it states that our reproduction strip confirms the reported issue. Maximum and minimum are not being converted to R. Let's search for files related to R code generation. It searches. Then we can see the observation and then we can see the thoughts and actions. So right here we can see exactly what the system is thinking and then its action. So we can see the first of course right here this is the thought of the AI system. The responsible file code is likely here we should do this. Then of course we can see the action at the end. So, so the thought is in here and then of course the action. And then we can see from the observation, then it looks at the observation and then goes back to thought and action. So it seems like the system we have here internally in point number three, how this works is that it just thinks, then it acts, then it observes what's being done and then it thinks once again. So what's cool about this is that we can see that this is an open source software agent capable of of seemingly long term planning or at least iteratively planning as it moves forward. Now point number four is rather fascinating because I did see something that I didn't think we would see. So essentially what we did also see in point number four was the fact that there is a new design. So it says simply connecting an LM to a vanilla bash terminal does not work well. Our key Insight is that LMS require carefully designed agent computer interfaces similar to how humans like good user interface design. For example, when the LUP messes up identation, our editor prevents it and gives it feedback. So essentially what we can see here is that the language model needs an agent computer interface that is very friendly in order for it to work very effectively. And they said that connecting it to simply a vanilla bash terminal just doesn't work well. So they've essentially designed a new design that works well natively with these LLMs to make sure that they understand exactly what is going on and that they're more effective. And we can see right here, we can see that there is an LM friendly environment feedback, it goes to the agent. Then of course, the agent computer interface has very simple commands that it can use, such as navigate, repo, search files, you, the file viewer, edit the lines, and then of course, it then converts that into the computer and then it comes back. So we have this entire system of how this works up. The editor prevents mistakes and allows it to work more effectively. So it's clear that the new design made a huge difference in terms of the performance capabilities of this. Now, there was also something else on the design in point number five. In point number five, we also saw that they were basically limiting the information on this AI system. So they said here that another example is that we discovered that for viewing files, letting SWE agents only view 100 lines at a time was better than letting it view 200 lines or 300 lines, and much better than letting it view the entire file. Essentially what they're stating here is that they didn't want to give the AI system all of the entire files when letting it complete the task. They basically said that it's only allowed to view 100 lines at a time, and it was much better than letting it view 200 or 300. Because I'm guessing that that this likely increased the complexity of what was being done. Maybe it confused the model, and I'm guessing lower lines allowed the model to process what was going better instead. So it seems that from this we could judge that the internal system, the internal agent works better when it has less things to do. And I'm guessing that that is something that is not too surprising. But you would think that if an agent has access to the entire file, maybe it would perform better. But I'm guessing that just showing it at 100 lines at a time and allows it to plan better and allows it to be more effective and dedicate all of the compute to ensuring what it does is correct. And they also say good agent computer design is even more important when using GPT4. So if you are building a advanced AI software engineering agent, it is possible that you limiting the software agent to only viewing 100 lines at a time might be better than viewing it 200 or 300 lines at a time. So that, that is something interesting. And I wonder in future if there will be an optimal number of lines that you can have for a software engineering agent to view or if there's going to be multiple software engineering agents possibly collaborating. Maybe you've got like four or three collaborating on different parts of the entire code base, fixing it at one time. Now point number six was also rather fascinating. So the software engineering agent additionally can be easily configured and extended to improve future future research on software engineering agents. Since the software engineering agent is open source, anyone can experiment and contribute new ways for agents to interact with computers. So this is something that I do find quite fascinating because now we have a system that is completely open source, which means that the development is likely going to increase even more. Remember, if we looked back at point number two in the compared benchmarks to Devin, we we could see that Devon wasn't that far off because you know, the advanced software engineering agent, the open source one, especially in the announcement benchmarks, was only 12.29% which is only around roughly 1% lower than the closed sourced Devon, which essentially means that since this is easily configured and since it is now completely open source, we know that further development by other individuals and maybe even perhaps companies could take this to a whole entire new level which, which is thus going to increase competition. And I do wonder what kind of software engineering agents people are going to be building with this because this seems to be very, very effective. And so far this seems very, very promising for the future because this is something that I would argue has been built remarkably quickly compared to open source LLMs. If you remember, compared to the release of GPT4 and GPT 3.5 open source chatbots did take quite some time considering the rigorous amount of training, pre training and all of the aligning that you need to do to the model. This it seems like people are easily able to build on top of existing closed source models and then of course get these advanced software engineering agents. Point number seven is also rather fascinating. They actually leave a link to a pretty cool demo in which you can use it and see how the entire thing works. And I'm going to show you that right now before we get to some of the other Points. This is point number seven. And this is where we have the advanced Software engineering agent and you can see how, how it works internally. So this is of course the web page where they do have a lot of stuff, but we can see the demo right here. So essentially if you just click next step, you're going to be able to see exactly how it works. So we've got the issue right here and this is the issue that we are trying to solve. And you can see that this is all of the code that you've put in the thing. Okay. And of course you describe the bug right here, you can see this is exactly what you want. And then we can see the next step. So it says to start addressing the issue, you we should do this. Then of course you can see in the terminal what the system is doing. And of course it's trying to reduce the bug. And then of course it's saying we're going to paste it in. And then of course you can see there, of course it's done that. And then of course you can see exactly how this works. Now I'm not a software developer, but if you are a software developer, this is really good because you can see exactly how it works and what steps it takes. On the left hand side in its workspace, of course it has its terminal, it's had its editor, and, and essentially you can see that this entire thing took around 38 steps in order for it to be complete. So you can see the error has been successfully reproduced which confirms the described issue, the error message, yada, yada yada, before proceeding with the fix. Let's do this, let's navigate to here and then we can see exactly how it works. You can see it's opening the tools library and it's. Yeah, it's really, really effective at showing you exactly how things work. You can also make this full screen. You can, you can also go ahead and make this full screen right here. And of course you can see the terminal. And I think that this is really cool because it actually shows you how the AI system is working. And with Devin we did get to see a few demos, but I really do think that this website is really, really effective now. In addition, at point number eight, they did talk about a paper release. One of the things that many people do want is of course they do want technical details. And on the discord they said that we are aiming to release by April 10th. So for the paper release they're aiming April 10th because that is when they think that they're going to be able to get the paper out. And of course if you don't know what the paper is, that's just essentially where the technical details of exactly what's going on should be, you know, released in terms of how it works, all the benchmarks, what open source or closed source systems they used, how they fine tuned it, and some of their initial experiments on what was effective and what wasn't effective. So next Wednesday should be the release of the entire paper where you can dive into more details. Point number nine was rather interesting because this is how expensive this is to run. One thing you probably know about AI systems already is that a lot of agentic tasks where you have to do multiple different reasoning steps do require the model to output a lot more tokens than an initial simple zero shot with a simple task. Now what's crazy is that they said we limit this at $4 per task and on average we spend much less for each solved task. And we'll have a number in the papers next and we'll have a number on average of tokens in, out. So I think right here they talk about how of course they don't want this to be an extremely expensive model and that is completely understandable considering the fact that in order for this to be usable, in order for it to be viable for something that people can use on a day to day basis, it shouldn't be very, very expensive. I mean, if you can get your software engineering issue solved for like 50 cents or something, that is going to be something that is very, very effective. But you if every task took $10 to solve, that would be very expensive very, very quickly because there are a bunch of different tasks and if you're trying to use this at a scale, it wouldn't be that cost effective to use because you're going to rack up the bill pretty quickly. Now of course other models are coming out and other models are getting cheaper and more effective. So I do think, like I said before, over time the cost per token is going to go down quite a lot. But for now they set a limit at $4 per task. But for solve tasks, that is actually how it works. And in terms of the, you know, how long it takes to solve, 93 seconds on average is pretty, pretty incredible because I think on Devon, if I remember correctly, it did take around five to 10 minutes to solve, but I can't verify that. So don't take that as me, you know, hating on Devon or whatnot. But 93 seconds on average is very, very, very, very impressive. Now the last point and point number 10 is of course is that will they use open source models? And they said that, you know, that could be great. But right now they mainly use closed source models because they are quite strong. And in the original software engineering benchmark paper we found that a lot of existing open source models were fairly far behind. So basically they're just saying here that they could use models like Llama 2 or you know, Mistral. But the point is, is that these closed Source models like GPT4 and Claude Opus are, are quite a lot better than some of these open source models and due to that fact they're just going to continue using them because it does make sense. Now there are benefits to you using closed source, I mean open source models because of course they can run locally and that's of course really good in terms of privacy and effectiveness. But once again of course closed source models have billions of dollars in investments and they are just far more effective than open source models at this time. So it seems that they won't be using any open source models. So but maybe they're going to allow that if you're going to be able to do that. But I wonder how effective that would be considering that, you know, open source models aren't as great as close source models. So let me know what you think about this. Do you think that this is very effective? Do you think that this is something that's really cool? Do you think that this is going to be something that takes down Devon? Because it is right hot on the heels and I wonder if open source could actually take down closed source in the very near future. With that being said, it's been the AI grid and I'll see you all in the next video.