This content was created by LARRUZ. This content was uploaded by LARRUZ. This is a YouTube video titled "SWE-Agent: The New Open Source Software Engineering Agent Takes on DEVIN" (posted 2024-04-02). The excerpt is an English presenter-led walkthrough and commentary comparing the new open-source SWE Agent (announced by someone named John) to Cognition Lab’s proprietary system Devon/Devin. The presenter summarizes SWE Agent’s performance on the SWE benchmark (GitHub-issue-solving benchmark), notes a reported 13.86% state-of-the-art figure for Devon, and explains SWE Agent’s similar accuracy and much faster runtime (~93 seconds versus ~5 minutes). The video describes the agent architecture (thoughts/actions loop, an “Agent Computer” interface with LM-friendly commands and terminal), a demo reproducing and fixing a GitHub issue step-by-step (searching 100-line windows, locating functions, making a pull request across ~38 steps), setup notes (Discord, demo site, forthcoming paper), limitations (currently GitHub-focused; needs stronger LLMs like GPT-4/Opus), and a broader point that proprietary moats are mainly data and compute. Created/last-edited timestamps are present in the metadata.
Questions this video answers
13.86%
About 93 seconds
Around five minutes
25% of the test set (reported in the video)
38 steps
Chapters
You probably recall Devon from Cognition Lab, the company that is seeking 2 billion dollar valuation to create world's first AI software engineer. We have already seen a couple of open source projects such as Divika and Open Devon who are trying to replicate the functionality of Devon. However, we don't have any information on the performance of these open source projects compared to Devin until now. Today we have a new open source project called SWA Agent, which stands for Software Engineering Agent, which gets very close to the performance of Devon. In their release the Devon team was highlighting this specific chart which shows that Devon was able to achieve a state of the art accuracy of 13.86% on the SWE benchmark. If you are not familiar, SWE Benchmark is based on solving GitHub issues and the Dev entry team showed that Devin gets state of the art performance on this specific benchmark. However, now we have the SWE Agent which is able to get very close to the performance of Devon on this specific benchmark and it might be even better than Devon. The SWE Agent announcement was made by John, so it's a new system that autonomously solve issues on GitHub repos. The performance is very similar to Devon on SWE Benchmark, but it takes only about 93 seconds, which is pretty impressive because Devin was taking around five minutes to do the same thing. Now another thing to keep in mind is the performance that you see here for SWE Agent is on the whole data set or the whole benchmark. Now for some reason the Cognition Lab team decided to test Devon on only 25% of the SW benchmark test set. So not sure like why they did that, but I think if you test that on this whole data set the performance will degrade, so probably is going to be somewhere close to what SWE Agent gets. So here is how it works. The agent has thoughts and actions, so it will come up with a plan, execute those, make observations, then modify its plan based on the observation it makes. Right? So very similar to what normal agents does, but they had to make a couple of adjustments. So in the demos you probably have seen that Devin directly works on the computer terminal, but for the SWE Agent they design a middle layer which they're calling Agent Computer interface. So it has its own language model friendly commands and it has its own language model friendly terminal as well where it's going to execute all the code and then it interacts with the computer terminal. Now as I said, the project is completely open source. They are going to be releasing the paper pretty soon and that's going to be very interesting to see. They already have their own Discord server and there is a website for demo which we are going to look at in a minute. Now in terms of the implementation, this looks very simple and very elegant. So as I said, they had to design this agent computer interface which is a set of LM centric commands and feedback formats to make it easier for the LM to browse the repo view, edit and execute files. At the moment the SWA agent is specifically focused on GitHub repos so you cannot use it for other software engineering tasks. But I think that can change with new updates. They have provided instructions on how to set this up locally. I will be creating a subsequent video on this, but let me show you a demo in action. Okay, let's look at a demo of how this works. So they actually have provided this real GitHub issue that SWA agent is going to try to solve. Now they have provided a step by step process. So let's walk through this. So in the first step, when SWA Agent receives this issue, it comes up with a plan. So it says to start addressing the issue, we should first try to replicate the bug as described. And this is the first step which a human engineer is going to take as well. So it's going to create a new file called reproduce underscore bug PI and here in the terminal you can see that that's the step that is going to take. So it says now let's paste the example code from the issue into reproduce underscore bug py. So basically it's going to copy the code from the issue in here and you can see that it copied the code. Then it is going to try to run that to see if we can reproduce the same issues or the same errors or not. So it's going to run the command now, right? And it run into exactly the same problem, right? So the first step is it's able to reproduce the issue. Now it's going to try to solve says the error has been successfully reproduced, which confirms the issue describes, right? And now it's going to try to come up with a solution, right? So basically it creates this tools PY file, right? So like it looks at the file, here are the contents of the file. Then it says the tools not PY file is now open, but this specific function is not visible in the first hundred lines. Now the way SWE Agent works is that it's going to look at hundred lines of codes at a time. If it's not able to find the issue in the first hundred lines, then it's going to do a search for the remaining file, right? So it's basically is doing search for that specific function and it found that specific function at line 302 and I think in their blog post they were explaining that this way the agent can keep more context and seems to be performing much better compared to looking at the whole file at a time. Right. So basically you can walk through this step by step. There are a total of 38 different steps the SWA agent takes to fix this specific GitHub issue and at the end it's able to resolve the issue and make a pull request. Now unlike Devin, this is limited to GitHub issues at the moment. But even in Devon's case we just saw the demos, we don't really have access to it, so we are not really sure what are the actual capabilities. But I'm sure that the open source community is going to keep adding these different features into software agents. Now if you look at projects like SWA Agent, davika or even open Devon, one thing is pretty clear that the only moat any of these proprietary systems have is the proprietary data and compute. Other than that there is absolutely no mote. I'm looking forward to the release of the paper. I think that is going to provide a lot of useful insights. Now one thing to keep in mind is you will probably need opus or gpt4 to run this. The open source LLMs are not capable enough to actually run agents like these yet, but hopefully that will change pretty soon. It's actually great to see the progress that we are making. So this is within 20 days of the announcement of Devon, which is pretty crazy. Anyways, I hope you found this video useful. Thanks for watching and as always see you in the next one. Thanks.
