Microsoft NEW AI Agents ARMY Is Here! Fully Autonomous SOFTWARE DEVELOPERS (AutoDev)

So there has been a ridiculous level of AI agents being released by many different individuals. And Microsoft has stepped into the arena with something that I could call Devon 2.0. They've called this AutoDev automated AI driven deployment. So essentially this is a rather fascinating thing because they are introducing something that is very similar to what we had last week, which was Devin. And of course that one actually did shake the entire industry because it pretty much was on the radar. So essentially with autodev, it's rather fascinating because like I said, Devon was released and it did shake the entire industry in terms of people realizing that the future careers and landscape is going to change in ways we never thought before. And that is of course rather fascinating considering that artists were the ones on the shopping block and now it may be software engineers. And software engineers aren't out of the crosshairs anymore because with autodev it kind of shows us what's possible with software engineering. Now, essentially what they state here is they state that, you know, the landscape of software development has witnessed a paradigm shift with the advent of AI powered assistants. And essentially they reference GitHub Copilot, which you all know about. But essentially they say that GitHub Copilot is just like a copilot, it's pretty good, but you can't build, test and execute code and GitHub operations. So they're constrained by their limited capabilities focusing on suggesting code snippets and file manipulation within a chat based interface. So essentially they said, in order to fill this gap in, we present autodev, a fully automated AI driven software development framework designed for autonomous planning and execution of intricate software engineering tasks. So you can see right here that this is a fully automated AI agent that is able to plan and execute intricate software engineering tasks. And this is their first iteration. I can probably guarantee that they're going to be coming up with new iterations of this unless GPT5S comes up with something that we didn't think of. But yeah, it seems that Microsoft are working on this and I'm not sure if they're working on this in a response to Devin, but I'm sure something like this did take some time. So I guess that what we're seeing here is probably just a coincidence. And it says AutoDev enables users to define complex software engineering objectives which are assigned to autodev's autonomous AI agents. To achieve these, AI agents can perform diverse operations on a code base, including file editing, retrieval, build processes, execution and testing, and git operations. They also have access to files, compiler output, build and testing logs and of course they have access to other tools. So it says this enables the AI agents to execute tasks in a fully automated manner with a comprehensive understanding of the contextual information required. Now, so the crazy thing about this, and I'm going to show you guys some more stuff, okay, is the fact that this is something where they have a bunch of AI agents working in the frame. So this isn't something like where you have one AI agent. This is something where they have a bunch of AI agents and these AI agents actually have different roles. You can see these AI agents can perform diverse operations on a code base, including editing, review, da da da da. So it's pretty cool. And I'm going to get into some of the benchmarks now and some of the other stuff. So you can see right here that says, in our valuation, we tested AutoDev on the human eval dataset, obtaining promising results with 91.5% and 87.8% of past one code generation, respectively. So it's actually pretty, pretty good in terms of the benchmarks. The only thing is that I wonder why they didn't compare it to the other benchmarks, which I will talk about in a moment. But we're going to get into the architecture now because I think the architecture is very interesting. So this is how autodev works and it's quite interesting. It's quite similar to Devin, but I think this one, they have more AI agents working in there. So essentially it says AUTODEV. Figure 1. AUTODEV enables an AI agent to achieve a given objective by performing several actions within the repository. The eval environment executes the suggested operations, but providing the AI agent with the resulting outcome and the conversation. The purple messages are from the AI agent, while the blue messages are responses from the eval environment. So you can see right here that we have the user and then it goes through all of this stuff. You can see the AI agent is able to do all of this stuff. It's able to run it, realize when there's a failure, and then of course test it in the environment, bring it back, and then of course run the test, make sure it's successful, and then bring back to the user. And this is of course based off GPT4, as you guys can see right here. So this is something that is you. You know, I wouldn't say it's pretty odd, but in terms of where we are in the LLM space, we do know that other AI systems recently released do actually improve on this quite a bit. Now, like I said with this one, the reason Why I find this one a little bit more fascinating is because this is collaborative agents. So it's a little bit different because you can see right here, it says at this stage, the user can define the number and behavior of the agents, assigning specific responsibilities, permissions and available actions. For example, so they state that they could, you know, the user could define a developer agent and a reviewer agent that collaboratively work towards an objective. So this is essentially some kind of, I guess you could kind of say agent swarm kind of thing where you have, you know, multiple things working collaboratively together. And when you work collaboratively together, you essentially have something that is rather more effective because, you know, they're independently working and figuring out the solution. So, I mean, it will be interesting to see if it does work better than one AI agent that's able to do stuff or one that acts as a reviewer and one that independently works on its own. So here, here's again where we have some more of the automated framework. So essentially this image diagram explains how the software, which is autodev, works. So basically you can kind of think of this like a kitchen where essentially you have the head chef, which is the, of course, conversation manager. And of course you have these specialized chefs which are the AI agents. So you can see right here we have multi, multi AI agents like right here. So there's a bunch of different agents. And of course, with this, you can basically say that the head chef is then getting all these other chefs to build a meal or whatever. But. But these are like all combining together. I don't know if that made sense, but you can think of like the eval environment here. This is essentially the kitchen where these guys go to work. And of course the eval environment is where the ingredients, the state, the tools, the repository are ready. And the docker, of course, is like a special safety seal that ensures everything is clean, secure this docker right here, and preventing cross contamination. Now, in addition, what's really cool as well, you can see right here we have the tools library. I didn't mean to do that, but we have the tools library. And with the tools library, this is essentially where these specialized chefs, so to speak, has their special kitchen utensils. And essentially they've got file editing, they've got retrieval, building, execution, testing, and of course the git, which of course, you know, recipe tracking, version system. So essentially what we have here is this is what the agents are going to use in order to do the, like, many things that they want to do. And of course, conversation manager is this is where you type in your objective and then the conversation manager figures out what to do, guess the agents to do that, and this is kind of how the entire thing works. So in short, basically you tell the system what you want to achieve and it coordinates the different specialized parts to make sure it happens correctly. And everything is done in steps, everything's checked along the way. And I think this is rather fascinating because this kind of setup is a little bit different to the Devon one. So it will be interesting to see how this framework actually compares to Devon on the benchmarks, because I think this is pretty similar, but it's a little bit different. So it'll be interesting to see how it does work. Now, it is of course based off GPT4 and they talk about agents. This is agents comprising large language models like OpenAI's GPT4 and small language models optimized for CO generation communicate through natural language. And these agents receive objectives from the agent scheduler, responding with actions specified by the rules and actions configuration. Each agent with its unique configuration contributes towards the overall progress towards achieving the user objective. So once again, they have multiple versions of GPT4 that act as specialized versions. One acts the reviewer, one acts as the developer, one acts as this, one acts as that. And like I said, this is, this is a little bit different because this is like an entire swarm of agents that collaboratively come together and complete your task, instead of one singular AI agent. So compared to Devon, this is kind of interesting framework because it takes a unique, different approach. Now, in addition to here, this is where they talk about the benchmarks as well. They state the CO generation results on human eval. AutoDev achieves a top three performance on the leaderboard without extra training data, unlike LATS and Reflection. So LATS actually stands for. I think it's Latent Action Tree Search. I'm not sure if it's action, but it's Latent Action Tree search, something like that. But the one part is tree search and that tree search is where they're searching through multiple solutions to find a solution to the coding problem. And reflection is a different type of, you know, way to code that does achieve performance increases upon the base model. And you can see that they all use the base model. But this one has extra training. Reflection has also extra training. That's why it's stable to achieve 94. You can see autodev, although it isn't better than language agent Tree search. This is. I can't believe I didn't just read that there. But it doesn't require any extra training. That's why it's so impressive, because these ones require extra training. But with the zero shot baseline you can see GPT4, it's at 67%. But with AutoDev it now is up to 91.5. And that's with zero extra training. And you can see AutoDev attains a pass one score of 87% on the human eval dataset modified for the test generation task, exhibiting a 17% relative improvement over the baseline that utilizes the same GPT4 model. So essentially one test shot they got 87% on the human eval. And I think with humans it's 99%. So in order to break this down further, this table shows the results of the testing process comparing humans and two different approaches using GPT4. So if you want to break this down, you know, pass one is a score that tells us how often the first solution given was correct. It's kind of like asking someone a question and seeing if they get it right on their first try. Now the passing coverage, this actually shows how many of the problems could be solved at all, regardless of how many attempts were made. The overall coverage, which you see right here, the overall coverage, this is a combination of how many problems were solved and how accurately they were solved. Now, of course the humans, of course the humans wrote, because the humans are actually good at the test, they get a perfect score, 100 on their first try, which is pass at one of the. And almost 99.4% problems can be solved. Now when using AutoDev with GPT4, the first try success drops to 87.8%, which means it's not quite as good as the human, which you can see, but it is still quite impressive because it can solve as many problems as human can. So we can see right here, even though the first pass is 87.8, we can see that the passing coverage is 99.3. So it's still pretty, pretty impressive. But you can see the overall coverage is actually lower because it doesn't get everything right on the first try. Now, of course you can see that the zero shot baseline, this is just the attempt of GPT4 without any special preparation training specific to the task. Hence, you know, zero shot. And it gets the answer right on the first try 75% of the time, which is where you can see 75 right here, which is still less than autodev or humans. And it can still solve a lot of problems, but of course it's not is getting good at autodev because you know, it's passing a rule Coverage is just at 74. Now of course, like I said, one of the benchmarks that I would have loved to see was this benchmark, the real world software engineering performance. I think maybe they didn't do this benchmark because either A it wasn't one that they wanted to do or B it wasn't just something that, you know, they wanted to do because maybe it didn't perform that well on the benchmarks and you know, people would have, you know, done the comparison. But either way, I think when Autodev, you know, is like open sourced and released, if Microsoft do make it into an actual product and I think it's going to be kind of fascinating to see what they do considering the fact that OpenAI their silence is completely deafening considering the fact that it's been over a year since their recent product update. It will be interesting to see what kind of products they do release because that does mean that in the future I think they're going to be coming up with some advanced systems because we're seeing people surpass GPT4 in all sorts of categories and the fact that they haven't produced anything means that, you know, either A, the most unlikely scenario is that, you know, GPT4 is, you know, they did really well and now they're struggling to match it, which I don't think is a solution at all, or B GPT5 and the other AI systems they're working on are simply so advanced that they really don't care about stuff like Devon, about stuff like autodev because it's just so far ahead. They have no worry and no pressure about releasing AI tools because it's just so good that the competition isn't even anywhere near close. So maybe that's why this highlights is deafening which is why it will be interesting to see if we come back to this before the 13.86% on Devin for software engineers. And then of course it will be interesting to see how that comes on in the future. Now, in terms of the actual solution, I probably should have had this in the earlier on in the video, but essentially we can break this down. So of course the first things that it does is it identifies the error and essentially the robot's working, the AI system is working, they realize the piece isn't fitting as it should. And you can see right here assertion error. And then of course you can see it says the test has failed as the test case that failed expected, yada yada yada. Then of course this is where reasonings and you can say the test case is Incorrect needs to be fixed. And then you can see, let's correct this. And then of course you can see updates in the environment. Then it tests it and then it says a summary pass. And then it calls the test suites passed successfully all tests. You know, the goal has been reached and then it ends the interaction. So overall we can see that this is really, really effective. It has a very simple feedback loop kind of thing. And we've seen this with other pieces of software as well. So it will be interesting to see how they do this as well if we kind of get the same kind of thing we had with Devin. Now, one of the last things that they do have is they talk about. AutoDev allows AI agents to communicate progress on tasks or request human feedback using the talk and commands respectively. And essentially it says anecdotally, these commands have proven helpful for developers using autodev to understand the agent's intentions and gain insights to the agent's plan. So essentially when the agent is doing something, they do have a command where you can simply ask the agent, why are you doing this? What are you doing? And they found that is pretty useful because they can understand why the agent's doing what it's doing and kind of direct it and kind of help it. So it's really, really cool. So it says our future plans involve deeper integration of humans within the autodev loop, allowing users to interrupt agents and provide prompt feedback. So in the future they're going to be able to stop the agents completely and tell them, look, do this instead. And I think, you know, that's going to be interesting. But like I said, it will be interesting to see if they're still going to do that, provided GPT5 is on the horizon. AI agents are on the horizon, Devon is on the horizon. And we saw Miza's KPU on the horizon being something that has ridiculous reasoning steps. All in all, we are moving in a rapidly changing world and of course autodev is one another step in that direction. With that being said, what do you think about these agent swarms where they have autonomous AI agents all working together to achieve a goal? Do you think this is just going to be for coding or do you think they're going to have different applications like AI agent swarms for companies, marketing reports? I mean, will be interesting to see how this thing is applied because I think, you know, you know, autonomous AI agents working together can do stuff. Previously it's been, it's been talked about quite a lot and there have been some early, you know, things, but I think when we do have this in the future multi agent setup. I think that is going to be something that is really, really interesting. Let me know you guys thought about this. If you didn't do the video, I'll see you guys in the next video.