Cosines New AI Software Developer GENIE Surprises Everyone! (AI Software Engineer)

So software development has taken another massive stride with Cosine GENIE coming in, showing us their new state of the art, fine tuned version of GPT4 that can perform 3.8% on the new software engineering verified benchmark announced last Tuesday. Take a look at their announcement video. It's rather fascinating.

Hi, I'm Ali, co founder and CEO of Cosine, a human reasoning lab, and I'd like to show you genie, our state of the art, fully autonomous software engineering colleague. GENIE has the highest score on SW bench in the world and the way we achieved this was by taking a completely different approach. We believe that if you want a model to behave like a software engineer, it has to be shown how a human software engineer works. We've designed new techniques to derive human reasoning from real examples of software engineers doing their jobs. Our data represents perfect information lineage, incremental knowledge discovery and step by step decision making, representing everything a human engineer does logically. By actually training GENIE on this unique data set, rather than simply prompting base models, which is what everyone else is doing, we've seen that we're no longer simply generating random code until some works, it's tackling problems like a human. So let's take a look at genie solving a real problem from a real repo. You'll notice you can prompt GENIE with a natural language prompt ticket, or in our case, a GitHub issue. So I'll go ahead and start. So now Genie's fetched the GitHub issue. When I click solve, it'll start looking into the problem. As you can see, it started thinking about what it'll need to find in order to solve this problem. This process is iterative and it will keep going until the model is satisfied that it's found everything that it needs. There we go. We can see that it's pulled a couple of examples of files from the code base that intuitively look like they're relevant to the issue that we're looking at now. It's going to start writing code to try to solve the problem. Much like the retrieval step, this process is also iterative. GENIE will write code, run it, and then react as a function of what it's seen. One of the great advantages of our data first approach is that because our model has watched more humans solve problems than any human could in a lifetime, it has a great grasp of how software engineers really break down and triage issues. It's also easily able to edit code in place, which is something that foundational models struggle with without rewriting entire sections. GENIE is now running the code it's writing and is using the debugging tools that we've given it to look at application state and execution flow, just like a developer would. Again, it's seen humans do this millions of times and is emulating that process. So, back to the task. We've just watched GENIE try a couple of different approaches to solving this problem, and at first it wasn't successful, so it planned again and has just written an alternative approach. This process can continue indefinitely, and because of the long context window that GENIE has available to it, many different approaches can be tried without losing any information along the way. There we go. All the tests are now passing. Genius successfully solved this problem, and it solved it in just 84 seconds, which I guess was much faster than any human could come to an unknown repo with an unknown issue and solve a problem. So now It'll write a PR title and a body and actually open the PR on our linked GitHub repo through the COSI web platform. Any comments or reviews left on that PR will be heard by GENIE and will be acted upon as if it was a real human colleague. We'd like to thank OpenAI for allowing us to fine tune such a long context window model, and I'm extremely excited to see where and how you guys use genie. If you'd like to give GENIE a try, just head over to our website at Cosine Sh. We truly believe that software engineering is just the starting point and that we can codify human reasoning for any job or industry. We can't wait to show you what.

We'Ve been working on now with this. What we can see here is the other models that are on this benchmark. So the SWE Bench Verified leaderboard is the leaderboard that puts together all of the previous agents, models, agentic workflows that work to solve these issues. Now, previously, the previous high Score was Amazon Q's developer agent at 38.8%. Now, what's crazy about all of this is the rate at which models are improving. We can see that from 7% earlier this year all the way up to 43.8%. This is a remarkable level of improvement. Now, the reason that this is truly remarkable is not mainly for the fact that we got better models, but the craziest thing about all of this is that one of the things that Leopold Aschenbrenner, someone who worked at OpenAI on the Super Alignment team, what he actually spoke about in his paper, the Decade Ahead, was this thing called unhobbling the gains. And this was where by default the model learns a lot of amazing raw capabilities. But they are all hobbled in sorts of dumb ways, limiting their practical value. And with simple algorithmic improvements like reinforcement learning, chain of thought prompting, with tools and with scaffolding, we can unlock significant latent capability. Basically stating that, look, the way how we use LLMs is rudimentary and over time we're going to figure out ways to get better and better with these models. So over time it's going to be interesting to see how these models will perform in terms of their abilities that we manage to extract from those models when we understand what they're capable of. For example, in this paper it talks about this. This is unhobbling. So imagine you had to solve a hard math problem, but you had to instantly answer with the very first thing that came to mind. It's, it seems obvious that you would have a pretty hard time except for the simplest problems. But until recently, that's how we asked LLMs to solve math problems. You remember in the first days of GPT4, people would just ask it a question, but after that, what we decided to do was chain of thought. We decided to give it a step by step scratch pad and it was able to solve much more difficult problems that were. So chain of thought prompting unlocks that for LLMs. And the reason I'm going over this is because now that we're seeing that with new methods and the way that new AI systems are performing, we're managing to unlock more and more capabilities with these systems. You can see here how the base GPT4 has gained around 40% on its level. It says that GPT4 base model 5% with just the base model to 20% with GPT4 post trained on release to nearly 40% today with better post training tools and agent scaffold. So now the reason that I actually spoke about this is because this relates exactly to what Cosine GENIE are doing. And on their paper where they actually talk about this model, they state that, you know, GENIE was always designed to be agentic. Although when we first dreamt up the idea back in 2022, that term didn't really cement itself in 2022. That was, you know, really, really early on. So basically what they're stating here is that from the start of developing this model, they designed it to be, you know, autonomous. They wanted this model to act independently and make decisions rather than a smart assistant that would just make it a passive tool. They wanted this to be like an actual assistant. So they wanted GENIE to actually understand what it was looking at and respond in the most logical way, quite like human programmer would. So essentially you can see here it says this is the tip of the iceberg when it comes to the work that was done to make as much implied information in a developer's mind explicit. And for every task they trained GENIE on, they had to teach it how to first gather essential background information about the project. And, and this was actually to prevent GENIE from making up code that doesn't fit with the existing project structure. That's where they talk about, you know, so that it wouldn't hallucinate code and produce solutions that were in line with how the code base was already organized and already operated. So they put a lot of effort into teaching GENIE the kind of background knowledge that experienced programmers who snake code and produce solutions that were in line with how the code base was already organized and already operated. So they put a lot of effort into teaching GENIE the kind of background knowledge and that experienced programmers already have in their heads, but don't actually always write down basically how you teach someone the rules of the game, but all of the unwritten strategies too. Now here's where they actually talk about how GENIE works, how it's, you know, agentic loop actually works. They say that, you know, the agentic loop is compromised of four main processes, planning, retrieval, code writing and code running. And these alone are not new most tools in this space.

So the main thing is of course planning, retrieval, code writing and running. And these are alone and not new. Of course, most tools in this space will use a mix of all of these. But they say that because GENIE is trained to perform each of these tasks like a human would, rather than how a base LLM would, we're basically able to get so much more performance from the model. So once again, as I've spoken about before with the unhobbling, it seems that GENIE have managed to just extract more performance and out of this model. Now another crazy thing that I saw was that they actually talk about the use of self improvement in training the model. They state that much of the data that we were training on was in a perfect state because the vast majority of the time the code that is published by a human is in a working state for it to be published. So basically what they did here, which was rather genius, was that they used the first version of GENIE to try and solve the coding problems. And, and then when GENIE made mistakes, they showed it how to correct those mistakes and they then added these examples of mistakes and corrections to the training data for the next version of genie and then they repeat this process multiple times. So they basically used self improvement to train the model. And I'm wondering that could they somehow repeat this loop in the future to get these models even better? And you can see it says every time we repeated this process, the, the initial candidate solution from GENIE was stronger and in many cases correct. And the cases where it wasn't, the amount of correction we had to show the model in the dataset was much reduced. So there was this iterative improvement of the model, improving the model that was just completely crazy. So they also talk about the future and they state that despite genie's impressive state of the art performance, we know that there's untapped potential and we're committed to refining the DataSet to enhance Genie's capability. They're going to be broadening data, introducing new capabilities, and that GENIE will become proficient in more programming languages and the latest frameworks. So overall, they're going to be creating different sizes of AI models. Smaller ones for simple tasks, bigger ones for more complex jobs, and they can turn any advanced model into a GENIE by their method of fine tuning. And what's interesting about this is that they're stating that they're going to, you know, bigger ones for more complex jobs, and they can turn any advanced model into a GENIE by their method of fine tuning. And what's interesting about this is that they're stating that they're going to, you know, do an open source model and pre training, extending a foundational model on our extensive dataset, aiming for improved generalization and specialized data reconciliation. And one of the things that they do an open source model and pre training extending a foundational model on our extensive dataset, aiming for improved generalization and specialized data reconciliation. And one of the things that they talk about is that a really exciting feature for businesses is that they can fine tune GENIE to perfectly understand specific. Talk about is that a really exciting feature for businesses is that they can fine tune GENIE to perfectly understand specific larger code bases. This works even for uncommon or company specific programming languages. It's like teaching GENIE to become an expert in a company's unique dialect of code. So this is going to be rather fascinating because the software development space for AI has evolved so rapidly and it seems to fit larger code bases. This works even for uncommon or company specific programming languages. It's like teaching GENIE to become an expert in a company's unique dialect of code. So this is going to be rather fascinating because the software development space for AI has evolved so rapidly and it seems like nearly every month, we get a large update that shows how much these companies are improving.