Flow Engineering and Code Integrity at Scale with Itamar Friedman, CEO of Codium AI

I say that very soon, even for sophisticated programs, we will see higher quality coming from AI.

Even in the case where we're in a hurry, you don't take a problem, here is my code. Usually, okay, here's the problem. Let's think a bit about how I want to design it. Even if it's not too much, you do a few steps. It doesn't make sense that we will ask a model, like, prompt, generate for me. Even if it's super intelligent, let's let it think the same way we do.

It doesn't seem intuitive to me that you would ask an AI to do something Then immediately take its solution and ask it to critique its own solution. And yet it works.

One day we would want the AI to be a team member. A virtual team member cannot just write code without testing it, without understanding the deeper context. A team member needs to be on the entire 3 parts of pre-build, build, and ship.

Hello and welcome to Turpentine AI.

What's that you say?

Turpentine AI? Well, we've got some news for you. After a year and 100 episodes of The Cognitive Revolution, Turpentine is doubling down on AI. We're creating a new dedicated channel for The Cognitive Revolution. Turpentine will be spinning up new AI-focused feeds as well. And this feed, soon to be known as Turpentine AI, will become a shared platform featuring best-of content from multiple shows and voices. We'll be making this transition gradually over the next month or so, and I'll be posting original content to both the new Cognitive Revolution feed and here on the Turpentine AI feed. So if you want to keep up with all of the latest, definitely make sure to subscribe to the new Cognitive Revolution feed, where I've just posted part 2 of my recent appearance on the 80,000 Hours Podcast as the first full episode. You can find the new feed at the same URL. cognitiverevolution.ai. And there you'll find links to the new YouTube channel, Apple Podcasts and Spotify feeds. And for those of you like me who use an old school podcast app, the bare RSS feed as well. Please do take a second to make sure you are subscribed to both feeds and stay tuned to both channels for lots more original AI content coming soon. Today my guest is Itamar Friedman, Co-founder and CEO of Codium AI, a company that's on a mission to make code integrity simple. Now, for context, at this point, it's broadly understood that large language models make useful coding assistants. Microsoft GitHub's Copilot was the first commercially successful implementation of a large language model for developers, and it's already been 4 months since Microsoft announced that it had become a $100 million business all on its own. However, code generation and code completion, while they make for amazing and highly inspirational demos, support just one part of the software development lifecycle. Before developers can write code, they must work with teammates to figure out what to build in the first place. And after writing code, they have to test it and integrate it into broader systems and production environments. It turns out, especially as products and teams grow, that the coding itself is often a minority of the total work. And that's where Codium AI comes in. Focused on code integrity, Codium is meant to support the rest of the software development process with focus on explaining, testing, improving, and integrating code. These are areas where just about every team could stand to improve. And though the company is only 18 months old, it already serves hundreds of thousands of developers. In the first half of this conversation, we discuss Codium's product, starting with the value that I found in it, even as a solo prototype developer who doesn't really have to worry about production issues, and then broadening out to discuss how Codium supports larger teams and their correspondingly larger products. Then in the second half, we dive into their recent publication, Alpha Codium, which shows how careful task decomposition, that is the process of breaking down larger projects that language models might struggle with into much smaller tasks that they can do far more reliably ultimately enables successful workflow automation. The Codium team calls this flow engineering, short for workflow, and while they do use some techniques that are specific to software development, what stuck out to me most about this research is how readily generalizable and applicable it is to many other domains as well. The project is open source, and I definitely encourage anyone building AI-powered workflows to to take a close look at their framework, prompts, and insights. Codium's work, as well as that of Google DeepMind, show that large language models are already competitive with human participants in coding competitions. It seems very likely to me, and as you'll hear to Itamar as well, that they will continue to improve and soon achieve elite performance levels. But even as that happens, and in some ways perhaps even more so then than now, it will be critical to ensure that AI-generated code aligns to requirements And performs reliably. And with that in mind, I think Codium is extremely well positioned for success and definitely encourage you to check it out. As always, if you're finding value in the show, we appreciate it when you take a moment to share it with friends. This one naturally should go to the software developers in your life. Please don't hesitate to send us any feedback, guest suggestions, or questions. My DMs are open on all of the social media platforms. And remember, To visit cognitiverevolution.ai to subscribe to the new Cognitive Revolution feed, where you will find exclusive episodes starting this week. Now, I hope you enjoy this conversation about harnessing large language models for code integrity with Itamar Friedman of Codium AI.

Itamar Friedman, founder and CEO of Codium AI, welcome to the Cognitive Revolution.

A pleasure to be here. Thank you for inviting me.

We've got a lot of ground to cover. You guys have built a very interesting product, which I've been playing around with over the last week and also have some cool new research out, which is pushing the frontiers of what we can get out of language models and excited to dive into all of that. I guess for starters, you know, you've started this company that focuses on code integrity as opposed to, you know, the somewhat larger space of just coding assistance or code generation. I'd love to hear kind of how you understand the impact of AI and large language models on the software development profession as it has already happened and how you came to kind of came to focus in on this question of code integrity as the area that you wanted to build in.

As I see it, I think that, and as I saw it also in 2022, which I call BC before ChatGPT. That code completion and code generation, like per se code generation, like give me code according to my request or something like a prompt. This is like a first use case, very useful use case, by the way, very fitting for the technology of LLM. It's not by chance that this is the, I'll call it the MVP, minimal desirable viable product that was introduced to us, but it's the first of many of what we're going to see in the intelligent software development landscape. I mean, let's say 3 years. I don't think it will even take 10 years. And when I thought about like in 2022, like let's say August, like I saw the first signs of Copilot actually working. I thought that like what we're actually getting with code completion is like getting your search inside your IDE and even a very smart search instead of Googling it 5 different like times and each time diving 5 different times in Stack Overflow, then I get like a transformed result into my code. And that's why probably we see a decline and continuing decline of Stack Overflow, et cetera. But eventually this is just one of the tasks that we kind of hate doing, like starting to Google. But what about testing? What about code reviewing? By the way, each one of the tasks is also like nice. I like sometimes to— see what people write in Stack Overflow comments, right? It's just also like a matter of like having a community. But in many cases, I hate testing or I hate like verifying that the code fits a spec, checking that I didn't miss any of the flows or features that exist in my Jira ticket or whatever, Monday ticket, et cetera. So basically I think like we're just seeing the early birds, the first signs of how AI is gonna impact our software development lifecycle and tools, et cetera. And then coming back to the end of 2022, again, BC, before ChatGPT, I thought that code completion and generation is going to be commoditized. We actually, myself and my team that we started end of 2022, we used GPT-3.5 API before ChatGPT and we knew it's coming, all of this. And we thought it's going to be commoditized. You probably know there are like 15 tools for code Code generation completion. Copilot is the leading one, but there are others. Some, someday soon, very soon you will even run a model like, you know, locally and it will be good enough, et cetera. But again, when I look on my 100%, uh, like time of development, this usually takes like 30% the most. And it's not the most painful part, to be frank. Again, so that's why we thought like we need to focus on integrity and like to just wrap it up. Why did we call it integrity and not like testing? Testing is the old way to verify your code intent. There are other ways to do that. For example, if I automatically create a specification for your code and you did write a specification somewhere else, we can try to match it. Here we go. We checked if your code is working as expected without going per se via testing. And by the way, in the long term, and you see signs in our product already, when I check your code, I also want to check best practice and, you know, even culture stuff like I'm talking like not only cultural, like best practices of development, but even culture of the company. And we might get to AI alignment later on this call. I can elaborate. So that's what we call code integrity and not testing. That's what I thought of like the most impactful point.

30%, you said, of software developers' time is spent on the actual development of the code, like writing the sort of happy path. Is that what that Essentially means?

Yes, I actually think it's a good number. You know, I'm not talking about the individual freelancer, you know, working by herself on some cool project. I'm talking about like a team of 50 developers, 500, et cetera. Like, you know, in a real company, then even a hypergrowth company, it doesn't like in 4 years old, it doesn't have to be like 20 years old or 100 years old company. There's so many other things you're doing. Like, by the way, some of them are in the software development regime, like our realm, like architecture. Uh, you know, like thinking about how to design your software, et cetera, but actually writing code, I think there is a lot of evidence that it shows that it's much less than 50%. And even, I think a few researchers researched, I don't remember to quote them, that it is less than 30%, even like, I mean, like much less than 30%.

It definitely does highlight a big distinction between, and this is like a theme in AI application development in general, right? Like it's easy to create a demo. It's hard to get things to production. That is, you know, when people typically say that, or when I hear it, I typically sort of think of the, just the, you know, the last mile of the AI optimization of trying to get the, you know, the prompts to be reliable and, you know, the edge cases and whatever, right? There's, there definitely are challenges just in the AI part of the application. But now you're also kind of bringing to the fore the fact that real software that serves like large, you know, customer bases is typically developed by bigger teams with whole cascading processes, whether it's waterfall or agile. And there's obviously a whole other podcast that can dive into the fine-grained distinctions between all the different ways that people try to structure their software development, but it's a highly structured process and people put a lot into figuring out what exactly are they going to build? How are they going to prioritize it? How are they going to define it? Finally, we get to this, what turns out to be know, 30% of the work that is what people actually, you know, tend to think about the most, of course, is, you know, that you're sitting there writing code. And then once you've got that done, now you've got all these other steps to finally, you know, get that through approval and finally integrate it into a production environment. So that's interesting. I'd never heard that that number is as low as it is.

By the way, we even, we even didn't talk about the fact that a lot of the code that you write is more like maintaining code, like changing, refactoring code. It's also part of it. Like a lot of the cool AI demos, especially coding agents, they're all about, yeah, let me create a new amazing website, which is basically 5 pages of register and some dashboard that it's usually like, anyway, you could download some, like, to be frank, like a GitHub repo that does all that. And in like real software, a lot of times after you wrote something, after a week, it's legacy code. And so it's a lot about like fixing that, refactoring that, elaborating on it. So like, yeah, that's why like, I think like there's a lot of opportunities here for more important use cases and for AI to help in addition to code completion.

Understanding where this sits, is there more that you want to dig into on kind of the criticality of, if I were to break it into 3 phases, it's like figuring out what to build, doing the initial build. And then making sure that that build works and fits with everything else that we've already got going on. But you had maybe other points to expand on there.

I'll make another point really quickly is that one of the reasons we focused at Codium AI and code integrity, because we believe, and we've kind of tried to prove that. And I think we did with the AlphaCodium that we're going to talk about later. It's groundbreaking. I would say practically it like beats like AlphaCode by DeepMind, which we admire. And that's why we call our work AlphaCodium. And beat OpenAI on their submission with GPT-4, like by tenfold on their submission on a code contest. Like it's a benchmark derived from real code competition from Codeforces. And the entire idea in that work is that we put a lot of our know-how and expertise into the code integrity part. That's what makes the code generation part very reliable and does better than the majority of professional competitors there. But, and by the way, it's open sourced and you can reproduce the only, by the way, work out there from all the 3 companies that submitted. They're the only 3 companies like actually trying this challenging benchmark. It's DeepMind, Google, and OpenAI and us. Our own is only like one submitted, but like closing the brackets, what I wanted to say is that the reason we focus on code integrity, because we sync It's not only directly related to a few obvious use cases, like helping you to test, but we think it's a key factor to enable most, if not all, of the other use cases in the intelligent software development landscape, even in code generation, which is the same in their game, like in DeepMind and then mostly OpenAI and GitHub game, even there, like AlphaCoding did better. because of the integrity. And if you're talking about use cases like refactor, that you obviously, when you refactor something, you need to test it, et cetera. In many other use cases, the integrity of the code, like testing of the code, is a key factor. That's why we were so focused on that. That's our uniqueness. Yeah, sorry if it's a bit too peachy, but I really believe in it. We're looking on the future and we wanna like make 10x like impact with AI in addition to code completion.

It certainly relates to a lot of the things that I think about around just, you know, how are we going to take full advantage of AI while still, you know, making sure that we have it under control. And I mean, under control in kind of every respect, you know, of course, like there's the big picture runaway AI scenarios, which I don't dismiss, but there's also a lot, you know, of kind of smaller scenarios where AI's impacts on the world are not immediately obvious, but could turn out to be quite important. And so.

Yeah.

I think that, that, that angle makes a lot of sense. Let's just talk for a minute then about kind of some of the upstream stuff. I use Copilot. I take my code snippets and queries to ChatGPT. New startups coming into the space like Cursor that's creating an AI-first IDE. Those, you know, I think are all pretty powerful. How would you describe what you see as the impact of those? tools in that, you know, again, I'm shocked by the smallness of the 30% number. Maybe I shouldn't be, but I, but what would you say? How would you describe what you see as changing in, you know, that kind of center activity as these tools are coming online and becoming extremely popular? Hey, we'll continue our interview in a moment after a word from our sponsors.

By the way, last time about the 30%, I think like one of the reasons that enterprise, you see them buy code completion by mass. Is because first of all, this 30% does matter. And it's also one of the first cases that actually works really well. It's the most, again, I really like love this use case, but it's like the most trivial one to apply with this new LLM technology. Like LLM were trained to complete the next lines of codes according to a context. And this is exactly what code completion and even generation does. They're less trained and well designed per se, purely by themselves to deal with this code.

Does this code actually work?

And that's like AlphaCodium. We presented more like an agent-like, more like a flow engineering than prompt engineering approach. That's why, by the way, the enterprise are really interested in the next use cases. So don't be so surprised if you already by the end of 2024, you will use AI in many other cases.

We'll probably talk about it later.

For example, we have a Git plugin. that helps you review pull requests, et cetera. It's like a huge pain. And I really, I don't think like you're about to try it, but solves a major pain and things that we hate to do. And the pull request, it leaves you to the things that you love to do. Copilot is amazing. Also other, some code completion tools. I think ChatGPT is amazing a lot because of the interface of the chat. And by the way, that's why we introduced chat that is more ChatGPT style in our IDE plugin. We call it, by the way, Codeumate. So not only we have many commands there that are specific for code, which ChatGPT doesn't have, but also we enable like pure free text like ChatGPT, but we do enable you to mark context more easily than ChatGPT. You need to copy it. So also recommend it to try. And then I fully agree with you that there are amazing startups. I think you mentioned Cursor. We really love them.

Yeah.

By the way, it's the first time I'm mentioning it. We are experimenting with coming up with our own IDE as well. And I see why it's like such a big promise because you can then choose how you want. It's much more native to integrate like some of the LLM features. Having said that, there is a reason that we didn't do it because again, the thing that you were like, if I could say like slightly shocked about is that coding is 30%. And that most of these solutions are focused on code generation, and we're focused on code integrity, which is through the entire software development lifecycle. You wanna check your—

test your code while you're coding.

You wanna test your code while you're merging. You wanna test your code in your pipeline, like in the CI/CD, and et cetera. And there is more point, by the way.

And if you build a new IDE, you really need to be focused on that.

And we didn't want to put that as our main focus, more on the entire software development lifecycle. So I definitely think that we will see like these tools, they will not disappear in your IDE while you're coding. They're amazing. And for example, Codemia is not coming to replace that. So that's like how I see it. Like we, by the way, like Sequoia did like a nice overview. You mentioned at the beginning, that's really like, that's pre-build, build, and ship. They did something similar. And I think like one of the things that many startups are focused on one part, like cursors and they give an environment that enables you to code. But one day we would want the AI to be a team member, right? And with all the respect, a team member, a virtual team member cannot just write code without testing it, without understanding the deeper context of even going and reading your Jira ticket or whatever, et cetera. So that's why like a team member needs to be on the entire 3 parts. Of, of like the pre-build, build, and ship in order to really be a virtual team member for you that can do some of, some of your work.

I'll describe my experience with the product in a second, and then you can expand on that. But just one more kind of question on the impact of AI on the software development process and profession today. I feel like we hear widely varying reports. I personally am not the world's greatest coder. And I would say I probably get a multiple X efficiency boost in many cases, even from just going to ChatGPT. Regular listeners will have heard me before talk about coding by analogy, where I'll basically take a class or whatever I have. It could even be documentation, but often if it's out of my repository, you know, The way that I implement caching is there. And a couple other things, patterns that I wanted to follow are there. And I say, do a new version for me that does a different thing, but kind of follow these patterns. I have pretty good luck with that. I genuinely do think I accomplish a lot of things a multiple X faster. Then you see these more rigorous studies that come out and say, oh, 30% faster here, very meaningful numbers there. But mostly the head, those kind of More sober studies would be more incremental than like multiple. And then more recently too, I've seen some of these reports that say that code quality is declining. And again, I don't think that's the case in, in, for me, because probably my, you know, bar maybe wasn't so high to begin with, but, you know, the, the sort of Gary Marcus position, which I do take seriously, is that at least as a possibility, or at least in some contexts, is that Yeah, maybe you're able to get stuff done faster, but perhaps you're doing it at lower quality, you're building up technical debt, and, you know, this is perhaps more of a mirage than, you know, it seems. So do you have a perspective on that? I mean, maybe it's everywhere, you know, everything, maybe it's everything in different contexts, but yeah, how do you, how would you break that down?

Okay, cool. So first, I think both philosophies, both opinions are right. I think there are some cases that it's like magical and it does really productive and without reducing the quality. And in some cases it does. Like, I don't think it's a conflict. It's a matter of there are so many setups. For example, when you want to start, and I have a feeling that this is in many of your use cases, start a project from scratch. I think like in most, and by the way, you're not trying to build a product that reaches millions of people and needs to be robust. You probably get a really pretty good code. Good. I'm talking about good, not like reasonable. For this case, for these cases, much faster. If you're a company with millions of users or hundreds of millions of users and bugs are really critical and performance is really critical and other things, and you're not trying to build something from scratch, but rather like add a feature, et cetera, you need a lot of context, then maybe the quality actually very likely will be reduced. And that's, by the way, eventually like people in companies will use like tools that are more ChatGPT in the IDE that integrated with the context like we offer in CodingMe AI, but that's put aside. And now I'll tell you what I think more specifically. Yeah, I saw those reports lately for reduction in code quality, and I try to explain, I think, why there is a difference. For example, in GitHub report, they say it's twice better quality or something like that. The number doesn't matter. They tried, like it was a controlled report, controlled experiments where they tried to isolate it from scratch, build something, and that thing was actually almost an average request that you can easily, like LLM can extract from the internet. But again, when you have complicated code, and in this case, like I mentioned, yes, I do expect even higher quality than the regular developer. When we're talking sophisticated software, it's not.

Yeah.

Having said that, I say that very soon, even for sophisticated programs, we will see higher quality coming from AI. And now I want to explain a bit about that. So I think the way we designed code completion and code generation until today, I'm talking we, like I'm talking about the industry, but I love a lot of our competitors. Actually, we see ourselves at Codemia as complementary, but that's a mindshare competition to some extent. I think like most of the systems were designed System 1-like thinking. You know what I'm talking about? I'm referencing to Dan Kahneman's book, System 1, System 2, like thinking fast and thinking slow. Most of the cases are thinking fast, System 1.

Hey, here's a bit of a context. Here is a prompt.

Give me the solution. That's System 1. I think we will see, and that's by the way, AlphaCodium, what we did in AlphaCodium, we will see more of System 2. Hey, let's think about it step by step, slowly, and not just of a chain of thought, okay? But actually using various tools. It's like an agent-like, okay? By the way, opening brackets for a second, you could do agents that are more system 1 than system 2. So doing an agent doesn't necessarily mean— I would say that system 2 is like the properties of that. It's like we see on AlphaCodium, like not only reflection and steps, but also including additional information that is relevant. In the process, like when you're thinking, you want to think something deeply, usually you go read a bit something, you go fetch the right information and things like that. And once like tools, even code generators, by the way, not code completion, it's going to be hard to implement it, code completion, because it needs to be fast. But when code generation like will do System 2, I think we're in a very good chance that even in very complicated code, we'll see higher quality. So even if it's worrying right now, I think it will be solved or like Even greatly mitigated by the end of the year, even the end of this year.

Yeah, things are certainly moving unbelievably quickly. One of the things I am saying over and over again to people right now is we now have AIs that can reason, plan, and use tools. And then of course people are always like, well, but not that well. And I'm like, yeah, but 2 years ago they couldn't do it at all. Right. So we, we should not assume that they're stopping here. So it's a, Good reminder. You are right about kind of the sort of work that I do. And that, I think that's a good diagnosis. You know, I typically do R&D type projects or prototype type projects. I, you know, don't have to worry about all the edge cases most of the time. And I don't have to worry about, you know, uptime or SLAs or, you know, all of the demands of a, you know, high volume production environment. Nevertheless though, you know, there is this gap that I also have between the kind of stuff that I spin up and, you know, what my partners on the development team, you know, for example, at Waymark will be happy to, you know, trust and run with. So I do see a lot of potential, even just for me with the Codium plugin. So here's what I did. I, you know, downloaded it, installed it onto my VS Code as an extension. You've got a quarter million active users, which is not a small number when you're talking about developers.

And VS Code, by the way, there's also JetBrains in it. And the Git plugin.

I assume that VS Code is like the biggest population, but maybe not.

Yeah, you're right. But JetBrains is like, I have to say, surprisingly, like growing faster for us. I guess I can try to estimate, but maybe that's less interesting. Estimate why, but JetBrains is really growing fast for us. Almost like, I think it will suppress like VS Code, like in a couple of months.

Wow. Interesting. I guess that would just be that people who are going to use an IDE that isn't like the most standard industry standard vanilla IDE are more inclined to just embrace new tools generally?

I think maybe we just invest in it like more than others. I have a few assumptions, like we care about enterprise, right? And VS Code is definitely like also trendy in enterprise, but JetBrains is also very dominant there in enterprise and we care about that. So we invested in it. Actually, we are going to do some few even more bigger releases, like in the next couple of months, even one month, because we're focused on that. So I guess generally getting high scores in JetBrains, it's hard. I think the average scoring in JetBrains is lower, and we have like 4.8 stars there or something like that.

It's a big world out there. Uh, when I see these numbers, they're always a reminder of just how big, uh, the world is. We were talking, we both have a tie to Ann Arbor, Michigan. And for me, the visual that I always keep in mind is the unit of Michigan Stadium. which seats 110,000 people. And I always just imagine that view of being in that stadium when I think of 100,000 people. So in my mental picture, you know, you've got over 2 Michigan stadiums full of developers just on VS Code. And for me, okay, I'm again, not really subject to the, you know, rigors of, you know, real at-scale enterprise software development. But I could definitely see how this is going to add a lot of value to a lot of people because I installed the thing. And now I'm just kind of poking around, right? I can ask it to explain my code. That's, you know, super useful just in general. I had a lot of success with just clicking generate tests. I thought that was a pretty interesting workflow where it first popped up 20 different things that I might want to test, different behaviors, as well as kind of the first half a dozen tests to actually test those behaviors. And then you've got the ability to run. And I was also interested in run and fix. Run, I was like, okay, yeah, totally. Run and fix, I was like, how do we ensure the integrity of that? Hey, we'll continue our interview in a moment after a word from our sponsors.

First of all, I'm very, very happy that these are the commands that you tried because we do have plenty. By the way, like there's an additional set of commands. There are 2 sets of commands for us. One that we like frame it as a code assistant. Like everything that you're doing right now while you're coding. And another, like it's framed as like a PR assistant that everything related to the changes, like if I want to analyze my changes only, like, or let's say focusing, like the entrance point is the changes. Of course, then we need to like have a deep context, et cetera. But you can like recap your changes, describe your changes. Rest in peace, description, fix bug 111. You can just call the /describe and get a really comprehensive description for your pull request, et cetera.

So, but I think like if I need to choose the top 2, I would choose exactly the test, the generate test, and I'll explain, I'll share why.

But I also wanna point out that we also have quick test. You might enjoy that because one of the things we learned is that our generate test, it's an advanced feature.

Like it even—

Even opened you an advanced panel, right? And it didn't like present you it inside of your IDE, like code file, or even in the chat. It opened you an advanced panel because it's relatively advanced. And if you're looking for some more quick and fun, then you can try the quick test one. It's in the chat command. So you actually were right about the, our main value proposition is the generate test. And by the way, the 20 behaviors that were generated. Notice that you can click and open sub-behaviors.

So you basically have like 80 ways.

And by the way, in some functions and classes, it could come 50 ways and some others could come 100 ways to test your code. So one of the main valid positions we hear from developers is that, hey, with Codium AI, I learned how I want to test my code. Because actually, like I mentioned before, like generating the code itself, it's cool, it's fun, but it's And it does save me time, but thinking how to test my code, that I didn't forget edge cases, et cetera. This is where things we kind of try to sometimes hate to do, but we know it's important. And that's why we focus there on behavior analysis. By the way, like you asked me a few times about the future, so I'll allow myself to say that this is where we're also gonna focus on. Right now, the behavior analysis are pretty much Like generated from 2 types of content, like your code and your comments. And soon if we connect to your Jira or Google Docs, we can extract behaviors that are not necessarily yet in the code or not implemented correctly in your code. What I talked in the beginning, like spec matching. So it's definitely like an important area for us. Like it's like automatically creating BDD, like Cucumber, et cetera, for you, like a specification. And this is why also explain for us is a very important command because the way we see things is like you have code, we believe in the concept. Now I'm going to open a bracket. It's really important for us. Really, really.

We call it DRY by A.

DRY, don't repeat yourself. And until today, developers tried to be DRY by not repeating themselves in code. But what about repeating yourself When you're writing your user story in Jira and then implementing it and then writing tests, all of these 3 have a lot in common. So drive by AI means that AI will help you to complete all the 3 and not needing to work on all of them from scratch. Now you write everything from scratch on the specification, you write everything from scratch on the test, you write everything from scratch for code.

No.

Now you can write your code and like now in 2024, I mean, you can write your code and get the relevant test and only work a bit to complete them and get the spec or the other way around, write the spec and get the test and the code and keep it dry and always have them in sync. So that's why I mentioned like, explain for us, if you look at it, we're kind of trying to create a README, right? Like a spec for your code. And generate test, it's kind of like the test, and then you have the code. And all of that, we want to keep dry by AI. So right now, it's with a push, you needed to click it, right? When the accuracy would be good enough for us, and we are aiming there, then we also do that with pull. We will try to keep it aligned for you and even give you alerts if something is wrong. So that's how we see the future there.

The vision of a more kind of passive, you know, assistant that pops up to help you As opposed to the one that you engage, which is kind of how it all mostly works today, is definitely really interesting foreshadowing. I also wanted to flag too that I got a lot of value from the enhance function. And it honestly was like software tutor kind of for me. You know, I've, I've been largely self-taught, largely entrepreneurial, you know, making stuff work. I used to go to Stack Overflow a lot. I go a lot less now. But I would go there, you know, a lot of times kind of, you know, with like pretty beginner questions in a lot of different domains. And so, you know, I've been writing Python code for years, but I just hit enhance and, you know, it gave me the enhancement. It was like, wow, okay. It's like typing everything for me. These are all the things that I like should be doing that, you know, I'm just not, you know, even bothering to. It definitely felt like I could learn from this, that my output could be better, that my handoff to the development team, you know, would be smoother. And so I do think this is likely to become a part of what I do, even though it's like, you know, I'm still in this kind of R&D mode and not really accountable for, you know, the code integrity in the way that, you know, the bigger teams are.

Here's the thing I thought that's interesting about Enhance. So first of all, thank you. I love, I love that you love it. By the way, I want to be clear here. There is no feature that we have that 100% of the developers love, by the way. Developers have a taste.

And that actually leads me to the next point.

Right now, I don't know if you noticed, when you do enhance, you can give it a bit of a prompt, like enhance it in a certain way. But think about it, and by the way, in implementation as we speak, part of it is already in alpha users, like design partners, et cetera. Think about it, if you could have a configuration file, by the way, maybe for each repo, saying, what are the principles of the enhancement that you like to get? Like in this repo, for me, enhancement means ABCD. And another repo enhancement mean like EFG or whatever. And then every time you call enhancement, not only like it gives you a general best practices enhancement, but yours. And think about tech leads being able to influence this for their junior developers, et cetera. And now if you like to enhance, I also suggest you to improve. And think about like you get code improvement and think about like improve that you can impact. configure and digest per repo. For example, one of our clients, what they do is they configured that they have like an old API and they have a new API. The old one they want to deprecate and they configure that every time someone, developer mistakenly like calls the old API, then it gives an improvement.

It gives an alert, hey, use this new API.

For dev team, this is like almost like a game changer. So this is another, I want to like use this opportunity to share with you how I think the future is coming is that there's also options that AI will learn by itself your best practices and the team best practice, and we're thinking about it, but the first step is actually letting you adjust it by yourself, and that's how we see it. So right now, what you liked about the enhance, I really loved it. You loved it. I can tell you that 50% of the developers or so, they hate it because they think of enhance in a different way, and that's why we want to let them configure what enhance means for them.

Well, I'm, you know, again, my low bar is such that even the baseline Enhance is a win for me.

It's not, it's not, it's not, it's not low or high bar.

Like it's just, it's taste. It's like best practices.

It's like, it's not like better or worse or juniors or advanced. It's like space or tab, right?

Certainly I don't deny the importance of taste among the, you know, those that have the true professional chops. One big question I was wondering about though, as I was using it, is the question of kind of the deep context, right? The things that I was doing were mostly like function level, class level, but pretty contained, right? And most of the things that I got back were also pretty contained. Test this function, enhance this function. When you get to the PR agent type workflow, And, you know, just in general to kind of continue to add more and more value, it seems like one of, if not the major big leap there is going to be the challenge of the much bigger context of these bigger projects. So in one sense, things are kind of coming to us as implementers of AI because the context windows have grown dramatically over the last year, you know, trying to do this sort of thing with a 4,000 token window sounds extremely hard. And at 128,000 tokens, you have a lot of room to put stuff in there. Doesn't mean it's all going to work. It doesn't mean it's trivial, but at least you have a lot more room to play with. What kind of high-level principles can you share about how you guys are thinking about the challenge of the much bigger projects and managing that context?

This is an amazing question, by the way. I think it's critical for the future of intelligence software development lifecycle, the entire idea of being able to, you know, understand big repos and multi-repos. It sounds similar, like some, some for clients, they have like a huge monorepo of 10 million lines. And how is that different than, for example, 10 repos of 1 million? There is a difference, but I don't think we— that's that interesting. But the challenge of dealing with a lot of line of code context is critical. I do think that as LLM contexts grow, really efficiently grow, that will help, but not solve, but it definitely helps. I do want to point out there are some research talking about that the big windows, they have also like problem with focus, attention, and all that. It's getting better, but it's still, I think it's a problem because even the way you order things, even if there's not a They will solve kind of the focus and attention. It's like, you still need to flow the process. Again, it throws me to the AlphaCodeum. Like if I just give you like, as a human, here is all the code. No, you probably process this, work on it and try to organize it. And the same way that you do it in steps, the LLM needs to do it in steps. Yes, there are like integral steps inside the LLM. There are stages, there are blocks, but a Like Harpathy said a few times, like, like basically calling an iteration is basically adding more blocks of thinking. And, and I think it's critical, even if you like make a bigger context window. And from our point of view, like Codium AI, we do implement deep context, but not as some of other solutions right now. And here is why we did it differently. We decided that in the beginning. the inception of the product, it was like high-level tactics. Like I would say on the border of strategic, we want to be a zero data retention company. It means like even on our on-prem, like cloud-prem solution, even there, we wanted to start with zero data retention. It means that we compute the context on the fly and we do. Like we build all the dependencies, we go until like we have like a, like kind of a heuristic algorithmic stop where to stop the graph of dependencies and upwards and downwards. But we don't do that upfront and we don't try to look on the entire 10 million repo at runtime. That's impossible. That's what we wanted to do. So we do have like deep push context. And now after we progressed quite a lot with these capabilities, now we are looking. into like ways, and there are quite a few and needs to be done right to do like digest repo and multi-repo and then exploit that background process that needs to happen all the time. And then on runtime to exploit that, we're working on, and then it will be even deeper context. By the way, it costs, it costs the environment and the company and et cetera. So it needs to be done right. So the cost is going down. Yeah. But it's still something that costs, but it wasn't the main reason, the cost, why we didn't do it because it's more like we wanted to give the confidence to our customers that this is like zero data retention product. And some of them were really scared of their code being indexed and saved somewhere, anywhere, even in their own, like, you know, their GitHub is strongly controlled, like GitHub, GitLab, Bitbucket, et cetera. They're strongly controlled. It's a really mature product. And then suddenly you come with a vector database you save somewhere and didn't pass like a lot of time since the inception of it. So we decided strategically to wait a bit and now it's getting mature. Well, our product and this field of, let's call it vector databases, et cetera, we're getting into it as well.

So yeah, this context question, I mean, it's one that I have been been thinking about a lot personally because I tried to create a GPT to help me with my, you know, relatively modest, you know, personal R&D repo for Waymark. And it really has not worked. You know, what I've kind of found is that at least so far, I just have to manage the context manually, loading the whole repository in there, you know, tried a few different ways. It kind of sent me off on this path of like knowledge graphs because I realized that What it seems to be doing is like matching on a chunk level, loading that stuff into context, but then it's missing all these things that I, that it really needs to take advantage of. And I've like tried saying like, no, it's your job to go find the pattern in the, you know, you're the one that's supposed to find the caching pattern and use it. Like, I don't want to, if I have to do that, then I don't really benefit from the GPT relative to just using, you know, ChatGPT and pasting in whatever context is relevant. Do you have any kind of Findings or tips or whatever on the sort of graph side? And if I understand correctly, you're doing this all dynamically in the, in the IDE at the moment. So you've got code in the plugin that's kind of spidering out, creating a graph. In the future, I guess you're also saying like, yes, this is going to go toward vector database as well. And we'll have chunking, but any thoughts about the, you know, what can we say about a good graph strategy? Because I need, I personally need some, some tips.

So first I have to say that, man, you're revealing all our near future features. I'll explain what it's going to be. Actually, what I'm going to tell you already exists, but it's a bit hidden and then some developers notice it and we got really good feedback about it. So I'll share with you, like I did a spoiler, like let me focus on that for a second. For one of our features, I think you didn't try it. It's called not— what you tried is generate tests. It's like generate tests from scratch. We also have a feature called Add Me More Tests, or officially we call it Extend Test Suite. Like if you already have extended, and we did an experiment and there we actually UI/UX-wise revealed the graph, revealed the— actually we didn't reveal it as a graph, although it's a good idea to do that. We're thinking about it. It's like sometimes like developers don't like too much UI in their IDE, right? So right now we did it like a list, like of the computed, let's call it, subcontext, like there is the direct context, the code under test, the code under explain, the code under command, what you're— you ask about different types of commands, testing, explain, enhance. But then there's the subcontext that the dependencies and all that, this is being revealed to the users. And the interesting thing is that then the user, the developer can change it. You pulled this one, but I don't like it, but hey, what about this something you missed? By the way, even if the AI, like the algorithm were correct and did bring all the right minimal graph, part of the graph, then you might actually want to add more if you want suddenly to get like a different test, like that covers more areas or different areas. So it's interesting. That's what we're working on to give it, like provide us features in other cases. So one thing I want to say is that I think that AI assistants would benefit, the developer would benefit if the AI assistant will reveal the underneath computing. By the way, this, I think it's true for not only the context, but other things we're thinking about it, how to, you know, I don't want to overload, but if you do like to see what's going on behind the, under the hood, here you go. That's like our philosophy looking forward. We did it in some places, but not enough. Specifically about the graph, one of the complicated things about it is that it's language dependent. It's like, it's almost like unbelievable for me that there's not like sufficient open source around it. Like there is this, the Tree-sitter and many other, sorry, and a couple of other open source out there, but they're not really completing everything we would want to have to have a very terrific graph for each one of the leading 30 languages. I can tell you like with relatively high confidence that JetBrains have that. They need it for the IDEs and GitHub have that for whatever, like search, et cetera. And what they do in their— in VS Code, usually what you do is like you download an extension that does it for you, right? In each language, et cetera. But it's not like open sourced enough, like in some one complete framework that gives you all that. So it's not easy to do. That's why you're having a hard time. If it's okay for you not to do that in the backend side, then what one of the things you can do is actually use the IDE's APIs. and exploit their graph building. That leads me to one thing I wanted to tell you how we do it. You said like we're doing an IDE. Basically we're doing it in 2 steps. We're doing part of the preprocessing in IDE and your— and like a consumer side, oh no, sorry, in the client side. And then a part of it, another part of it on the backend. It's for efficiency, like to try to like minimize what we send, but also like— Deeper algorithms that could run on the backend.

Well, let's get to the research itself then for a little bit. So, you know, for all this talk of context, like this is a very, we're kind of back to a toy environment, right? With the, with this thing, but I, I understand it as essentially you and your team flexing on your ability to maximize the performance of language models. Is that like a fair characterization? Like you wanted to just show, hey, we are awesome at this and here's Uh, we're going to go head to head with, you know, some, some well-known publications on, you know, an established benchmark to demonstrate and kind of teach the community, demonstrate that we're awesome at this and teach the community some of what we know. Is that basically the, the mindset behind it?

The word toy, I'm not sure if you related it to the AlphaCode or the benchmark, but I wanted to relate to both. Like the reason we chose a code contest, which is a benchmark designed and gathered by DeepMind from mostly Codeforces competition. We chose that because we think this is the most serious coding challenge there is out there. There is like some LeetCode stuff and HumanEval includes like coding stuff. They're either saturated, simple, even the LeetCode hard, et cetera, is like simpler than the Codeforces in most cases. And by the way, a lot of them are like shown to be presented In the training set. Like, I don't know if you noticed, like GPT-4, like really advanced on HumanEval. And then somebody presented that it's doing 100% accurate before some date and zero. And after it shows like kind of almost proof that a lot of the stuff there is in the training. And with Codeforces, the idea is that currently the code contests are like a benchmark is set, but we do see it as something that should always progress as more Codeforces like to add more problems, code contests also should progress with it. And then there is higher chance that the other good— that's not in the training set, but the other good thing is that Codeforces problems have private tests. It's almost like Kaggle, you know, though. So you're not revealing, like it's hard to really fine-tune on it. So that's why we chose it. And I think this is one of the reasons That there's only 2 companies until today that competed on the code contest, which is DeepMind and OpenAI, and we're the 3rd. All the rest of the, let's say, LLM generators, they usually do an Omnivale, et cetera. And there's a lot of debate around it, but they're basically around the same numbers. They're getting the same numbers. Now about the AlphaCode itself, if it's a toy or not, I can tell you that on one hand, it's doing really well on the Codeforces, and these are really hard problems, try it out. And at the same time, yes, these are isolated problems. They're not like connected to many other moving parts and companies like huge software stack, et cetera. So we are in 2024, step by step, taking many of the concepts that we had in AlphaCodium and integrating it Uh, into our Git plugin called PeerAgent, an IDE plugin called Codimate and, uh, by Codium AI. And, and, and we are integrating it and you will see not only like new capabilities, but new interfaces that are more flow engineering. And if we talk about AlphaCodium, and so I do think that, that, that we will see the impact also in real world software.

Yeah. A better word than toy would have been self-contained, I think, because And another way that it's interesting is that it is, I think, a framework that generalizes probably extremely well beyond code too. So let me just set it up a little bit. I'll give a little bit more kind of foundation for folks, and then you can give me more detail. So first of all, this code contest, as you said, is a standard that humans compete on. So I always like to ask the question, well, how good are humans at this task? That's a great grounding. And it is true that it's pretty easy to create devilishly tricky little problems, and people like to compete on these. So the kind of prior state of the art before your work came out was DeepMind's AlphaCode, and they published a result, I believe this was maybe even in Nature, that said, hey, we have achieved median human performance among coding contest participants, which is really critical, right? Because I mean, people are always like, well, how do we compare to humans? And it's also, which humans are we comparing to? So we are comparing to people who participate in coding contests. And with AlphaCode, I think it was about a year ago, they got to the 46th percentile or roughly, you know, the median participant in these coding contests. And that's just by successfully executing or succeeding on 25% of the challenges. So these are some pretty hard problems if the sort of person who finds it fun to go spend their time doing coding contests is only getting 1 in 4 of them correct. Now, to just unpack a little bit more of the structure, you have essentially a problem statement, right? That's kind of like a story problem from like a high school algebra textbook, except for a coding, expecting a coding solution. And then if I understand correctly, you have both the public tests that you can see, right, that are given to you, like you must pass these tests. And then also the private tests, I wasn't clear on, are they so private that they're not published anywhere and you have to just ping somebody's servers? And so those are on the Codeforces servers where those tests, and they're literally nowhere on the internet. Is that true?

That's why it makes it like a really good Kaggle-like competition. So about what you said, that's perfectly correct. I want to elaborate a bit.

Okay.

So first of all, there was AlphaCode. I'm not even— I think it's even more than one year. And then there's AlphaCode 2, which was just recently. One important thing, the first thing I want to relate is that both of them were fine-tuned on this specific problem, like fuck. And this, by the way, makes them less general. Like you talked about, if it's like, by the way, I'm fine with calling it a toy because I do agree it's currently not fitting for real-world cases, which it's going to be like in our products in this year. But as opposed to AlphaCode 1 and 2 that they find you in their model on this competition specifically, like fuck, our solution does not. It gives like a huge hint that our solution, AlphaCodium, is more generic for other problems. And this is critical. Second, AlphaCode 1 specifically, like especially, but also AlphaCode 2 makes millions of LLM calls to solve one problem. So when they said they reached 25, I think it's 28 if I remember correctly, what you're referring to, like in AlphaCode 2, they kind of reshuffle the entire competition. They don't say which one they did. They don't share, like with all respect, I think like they did AlphaCode. One amazing report, AlphaCode 2. It looked to me like they've been in a rush. They don't know exactly. They test AlphaCode 1 on AlphaCode 2's stuff and Shuffle of Code contest, which is public, but they had a private now or something. And there they did 25%, but the original one is 28%. To get to the 28%, you call 1 million calls for one problem. This is impractical. So they fine-tuned and 1 million calls. Forget about it. This is not even a toy. This is like, it's like a kind of proof of concept, which is amazing. They inspired us. Yeah. Okay. With AlphaCode 2, which they fine-tuned a new model, which is Gemini Pro-like, they still fine-tune, but they do much better on how many calls they make. They, it's like we did, while we were fourfold magnitude better than AlphaCode 1, we were just like 20, 30% less calls than AlphaCode 2. Having said that, still they fine-tuned their model. It means that now for every competition or every new software or whatever, you need to fine-tune your model. That's like almost impractical. And while our solution is generic. And usually like when one of the things I liked about the project that what we thought is the main contribution here was also recognized by others as the main contribution. It's not always like that. Like you always wait for what's going to like— how the community is going to look like and what you did. And Karpathy, when he reviewed our paper and tweeted about it, he actually related to the fact that how well we designed the flow. Now you can call it chain of thought, you can call it an agent. We decided to call it flow engineering because chain of thought, like you can read in the paper React, like the chain of thought is like one type of flow. And we designed a different type of flow, so we didn't want to confuse, and we don't call it like agent per se because we didn't wanna fall on that buzzword. And so it's basically what we did is like we did a code-oriented iterative fact-checking flow engineer— flow engineering.

Okay.

So we start with the problem. Sometimes the problem is defined like in such a humanly way And we wanted to represent it in an intermediate way, which humans can read, but is much better for LLM to digest. It's like taking a product description from a product dude, from a PM, and like refactoring it to a technical document. And this requires us a few steps, but the interesting thing is that each one of the steps are relatively reliable. Sometimes when you try to do like the best prompt from the problem to the code, practically the LLM needs to do so many things in the process that it makes this unreliable. And even with the best prompt, it still will be unreliable. But if you design a flow, like I just described the first part of our flow, which is a few steps to redefine the problem, then each step is more reliable. You get a better, a better result. Now, yes, I know what you might be thinking. If you take like from like the original to the end, maybe it's like 40% accurate. But if you take, even if it's 90% accurate, each one of your steps, but if you take 0.9, multiply 0.9, multiply 0.9, you might get less than 40. But, but, but practically we have a few techniques that we show in AlphaCodium how we avoid like, you know, this situation with what we call like gathering and injection through the process. In the old days, we might call it self-supervision or something like that. So basically the flow starts with a few steps where we redefine the problem and then we offer a few solutions for this problem and then AlphaCode auxiliary reflects on it and then start to generating one or two solutions running the public test if exists. By the way, some of the problems come with 0, if I remember correctly, I hope I'm not wrong. 0, some with 1, 2, some with many public tests. And then it starts to reflect on it and run the code. So it's like specific. It's not like a general chain of thought. It's like, it's a flow specific for programming. And I want to relate to that. And then if you want, like if you take it to healthcare or you take it to to, I don't know, like legal, you can anchor your results in different ways. In this case, in software anchoring via testing. And not only that, AlphaCodium then tries to generate additional tests that are edge cases because, you know, in machine learning, when you want to like have like a good solution, you need to have the edge cases to draw the boundary. And this is what you also see in Codium AI, what we talked an hour ago, with the behavior analysis and give you the edge cases. And then although it's AI-generated code, we know how to smartly, naively, it's not so complicated, but smartly to exploit these edge cases, even if they might be wrong, by choosing like a curriculum learning, which one is accurate enough to conclude and anchor it, use an anchor. And this is how eventually, like after a dozen of steps, you reach the final solution. So that's AlphaCode, well-designed, for solving problems from a spec, from a definition to, with a few data points to a solution. And it could be generalized to real-world product because you do have some data point you can extract. I can talk about it. Yeah, that's like an overview. So the name of the paper is From Prompt Engineering to Flow Engineering. Like the time we needed to work on prompt engineering really reduced to almost 5%, because once you move step by step and every time you ask for something relatively clear, then the sensitivity of the prompts is being reduced.

I think this notion of relative simplicity of prompts, I went into the GitHub repo and read through the prompts and I was pretty impressed by just how relatively straightforward they are. Pretty brief, pretty easy to read, crisply written. I thought, you know, clearly some care has gone into articulating very clearly what each task is. But the real work has been breaking down this macro task into the subtasks in a way that really embodies the best practices or, you know, a successful workflow to get from beginning to end. So I thought maybe I'd just read the steps. And I think, again, this is something that you could bring to almost any other area where you might want to apply AI. I, might think of it as, I mean, you call it flow engineering. I might call it workflow engineering. The first challenge that a lot of times they have is that they haven't documented their workflows or they haven't even really thought at all about like, what did they just do the work, but they haven't really taken the step back to think like, okay, what are the steps that I do? And, you know, maybe I'm kind of blurring them together, but, you know, could I really break them down into discrete and kind of clearly distinguishable subtasks. So that is the lesson here, I think. I mean, you can, you can borrow from the AlphaCodium pattern, but you can also do this for your own type of work and with whatever you're trying to do. And I think you'll, you'll find pretty good results from it. Elicit, which is a former guest on the podcast, and I'm a very small, proud investor, also is like a pioneer of doing this kind of thing, you know, just rigorous task decomposition in their case for research. So here's the flow. You've, you kind of divided into 2 buckets, but I mean, really it's, it's, it's a pretty linear flow with a couple nodes that you can kind of iterate on. So first you, you know, provide the, the coding problem. The kind of first core step is problem reflection. That's your classic, like, unpack this, you know, think step by step, but you're not even doing anything yet. You're just kind of unpacking the problem itself, really just, you know, literally thinking about the problem. You know, LLM style. Then in this context, you have the public test. So then do the same thing about the public tests, then generate possible solutions. I might also say, describe that as generate possible strategies, because if I understand correctly, there's no code generated yet at this point, but it's like possible approaches, strategies, you know, ways to think about solving the problem. So you got a few of those. Now rank those solutions. So again, each of these is a distinct LLM call with a pretty basic, straightforward prompt, but really dialing into doing that narrow part of this overall workflow. Once you've ranked those solutions, now generate additional AI tests. I think this is obviously more specific to a programming environment than other things. And we'll leave it as an exercise for the audience to think about what the version of that might be in their context. But this I thought was really interesting too, because the way that the thing is ultimately going to be evaluated on is, does it pass the private test, right? If it passes the private test, it's deemed to be successful. And yet you only have the public one. So I assume these things are generally kind of designed in a way where There will be some failure modes that the private tests are supposed to capture that the public tests would not capture. So you're taking that extra step of saying, okay, well, what other things, even if we were to pass all these tests, what other ways might we fail? So generating additional AI tests to try to cover those bases. You've got an interesting diagram that kind of shows the problem and solution space and how you expand the coverage of tests by generating these additional tests.

Yeah.

additional tests. And all that has happened before we get to any code, right? So this is literally just all in the preprocessing bucket as it's described in the paper. Now we're finally though, having reflected on the problem and come up with a few different ways to think about solving it and generated new test ideas and chosen, ranked those various candidate strategies. Now we're finally ready to generate some code. So generate some code, see if it passes the test. Actually, there's even a self-improvement step. That's another thing I think is pretty interesting, is just multiple rounds of improvement even before testing. So generate the code, then feed the generated code back in. Can this be improved? Then go on to actually run it. This is the only thing, as far as I understand, that gets outside of just fully programmatic or kind of fully deterministic language model calls, is that we're now actually going to run the code. So we do need access to a runtime for this portion, right? Now you either succeed or fail. You can feed in the error messages, work on that again, again, again, until you finally pass all the tests, then do the same thing on the AI tests. There's an interesting nuance there around the fact that the tests might be wrong. Can you give a little bit more color on how you think about that conceptually?

First of all, I want to say that you're spot on in some of the insights. Like, think about it as a human, probably this process as a developer is what we do. You don't take a problem, like even in the case where we're in a hurry, you don't take a problem, here is my code. Usually, like, I'm not saying that we don't 10% of the time or et cetera, but usually like, okay, here's the problem. Let's think a bit about how I want to design it. Even if it's not too much, you do a few steps. It doesn't make sense that we will ask a model like prompt, generate for me. Even if it's super intelligent, you know, okay, let's let it sink like the same way we do it. Like that's a core idea here. So yeah, let's think about it. Yeah, what could be like edge cases? What could be like what I'm trying to achieve? How do I break it into points? You know, okay, what could be like a relevant algorithm here? Like think about it. You've been asked to do something. You'll think, yeah, what's the principal like technical engineering aspect that I need here before you start writing the code? At least that you do that. So why wouldn't the AI do that? Like when you get to the test, basically the AI might be wrong and like similar to us, How many times, if you do testing, like the world divides into 2, the developer world divides into 2 basically. Either you don't do testing because you hate it or you do testing because you, and you hate it, right? But basically like when you do do testing, hey, how many times has it happened to you, you made a test and it was red, it didn't pass. And then you're thinking, is it a problem in my test or is it a problem in my code? It happens a lot, right? So why wouldn't it happen with AI as well? So how do you deal with it? You kind of reflect about it, right? But by the way, we also like a bit could be wrongly biased towards accepting green tests that pass. And it could be test passed, the test is wrong and the code is wrong, but biasedly, like we would accept that green test, right? So, and then we will put most of our time on the red test, or is it the problem in the test, problem in the code? And basically what you see here is that, that what we, this bias of developers we put into AlphaCodium. When a test is like passing, maybe with a bit of reflection or not, we anchor it. Okay, this, this should not only pass now, but also in next revision. But if the test didn't pass, let's reflect if we think we need to fix it or, or, you know, generate another one or actually fix the code and try to reflect on it and act accordingly. So basically you build. A growing anchor group of passing tests. And that concept you can take to other fields. Let's say I have some, I don't know, like on the legal part, I have different items I need to comply to and I write something and I check and some way or another, if I comply to 3, okay, from now on, I want to comply to them and keep that safe and move on to the next one. Maybe not. I didn't give the The best example, but there could be like other fields that could definitely enjoy this. By the way, it's a field, sorry, like I'm so excited about these stuff. So maybe I'm saying, talking a bit too much, but it's relatively like a researched field. We didn't quote other papers, et cetera. We didn't want to make it such an academic work, but curricular learning in general, like it's a very adjacent or exact field. There's so much work there. Do you start with the hard cases or you start with the easy cases? Do you start like, you know, so you can apply, there's a lot of room in AlphaCoding for innovation or trial and error, like to see which other curriculums could work. We chose one according to our intuition, best practice.

Yeah. Okay, cool. Interesting. So I will just summarize then the results and maybe also give a, just at least my understanding of kind of the economic You know, comparison between the human performance and the, the AI. So you try this with a few different models at the time of the research, GPT-4, clearly the best model available for coding. I would be very interested to hear what, what Code Llama recently released might look like in this context. But the punchline is with a total number of API calls per run through this entire workflow. That averages in the sort of 15 to 20 range, and then also allowing the AI to run this whole process up to 5 times, you get up to half of the code contest problems solved correctly. That's basically double what AlphaCode reported some time ago and essentially on par with what AlphaCode 2, also out of DeepMind, has reported more recently. And that gets us up to close to, if not all the way to, the 90th percentile of humans. Again, and these are humans that are spending their time competing in code contests.

All other competitors, they mostly did pass 10. What pass 10 means, they submit 10 times on the code competition. And we submitted 5. Why did we choose 5? Because we thought about UX/UI. Think about like, if you are a product. And you want to suggest a few options, then we saw that 10 options is a lot. But one is people will be willing to get like, here's top 3. What do you think? Which one would you like? Or top 5 or, but 10, like practically also mean less running time and easier for us to research when we didn't do 10. And by the way, it favors AlphaCode, DeepMind and GPT with their report because they did 10 submissions, we 5.

So.

If we do 10, by the way, we get better results. So, and these 5 attempts, in each one of them, there is roughly like 3 solutions that AlphaCode automatically tries. And in each solution, there is like 15 calls. So roughly speaking, there is like 100 calls, which is equivalent to AlphaCode 2 when they reach similar results to us, which is 30% solving. Okay, so not 50. With 100 calls, we reached to 30% of the tasks are being solved. Solved means that all the private tests are passed, which put us like in the 51, 52 percentile. Like we beat the majority of, but not the 10 percentile. To reach the 10 percentile, we needed to run like more than 1,000 calls. And AlphaCode, by the way, it's More towards the million. And we didn't want to do that because it's not practical. It's like when you want 100 calls, it's like 3 minutes and it costs you, let's say, a few bucks. When you need to run 5,000 times, for example, to beat the 10th percentile of competitors, then basically it takes you tens of minutes and more bucks. And we practically wanted to show a practical result that people can reproduce. So we aimed on the, hey, let's give you a tool that lets you beat the majority, but not necessarily the top 10. If you scale the parameters, you'll do much, much better.

I think the difference is that I was taking that number off the validation portion as opposed to the test.

Oh yeah, yeah, yeah. You're correct. You know why I'm relating to the test? Oh yeah. Sorry. So what you said is correct about the validation, but let me tell you why I'm relating to the test. I think it's much more correct or real world, or like, it's better to use the test. And by the way, noticed that when we, because it's not available out there and it's like, I think more pure results. So I related to a test and you're right. And you relate to validation. And, but what I said is like everything I said, it's about a test, but why do I relate to a test? Because I think you can see interesting insights. For example, GPT-4 results on the validation is like much Opens much a bigger gap than other models. And then on the test, actually the gap shrinks. This is really interesting in my opinion. Like it shows the power of the flow and how maybe GPT is trained on so much data that like somehow includes some of the, somehow the interpolation of what's going on there, but not the private one. And by the way, you mentioned like we decided to keep things simple. And we presented a result on DeepSeek code and GPT-3.5 and 4. We didn't at that time manage to do on Code Llama version 3 or whatever it's called, because it was just released a few days and we released like 10 days ago. But we did test on more APIs. And I have to tell you that surprisingly or not, DeepSeek is doing relatively well, especially on the tests. Side, like it's very generalizable. And so we didn't want to complicate it too much and we presented that. But we might later on in AlphaCode 2, when we want to even do more rigorous and not an 8-page report, but like sometimes OpenAI DeepMind, they do like 70 pages. I'm not sure we want to go there. Then we'll show many more, but DeepSeek is doing relatively well. There are other cool models. Some other models are really good for chat. but not so good for code. So, you know, some of the Bedrock, AWS, I don't want to call names. Uh, they're really good for chat, but less for code. Uh, I'll allow myself to say that it was a pleasure and we can chat again if you, if you want to host me again later on the year, maybe I'll have some exciting things like hinting a new model or things like that.

Sounds great. Well, I'll just, let me just run down really quickly a couple of other quick takeaways that jumped out to me. One, the value of YAML as a format for output. That's definitely one I'm going to be using because I have been using XML and I think I've given that tip plenty of times to people. And I like XML better than I like JSON because I can at least like read, you know, as I'm just manually getting started, my XML. But the YAML is like fewer tokens, also easy to read, you know, and apparently works really well. So. That's an update for me. A couple other just general patterns that people can follow from your work, starting with easy, moving toward hard. You use the phrase knowledge accumulation, also kind of deferring the big decisions for as long in the process as you can, right? So you're doing that reflection, analysis, et cetera, et cetera. And finally then choosing your strategy and getting into the code. That I definitely think is something that, you know, whether you're drafting Legal letters or any number of things, you can kind of follow, I think, a very similar logic. Asking for small modular functions, that's a code-specific thing, but again, it makes a ton of sense for a lot of different scenarios, right? Can you both break the task down and have it break the task down in its response to you? And then finally, immediate self-review. Just, it's not intuitive necessarily. I think we have, maybe should be more intuitive. This is maybe an area where more anthropomorphizing Would even be better, you know, but like, we, for, I, for whatever reason, it doesn't seem intuitive to me that you would ask an AI to do something, then immediately take its solution and ask it to critique its own solution. Like it just, for whatever reason, seems like that wouldn't, you know, be a way to get a better answer because it's a, you know, whatever, it's an AI and yet it works. So, you know, ask literally just immediately after generating the solution, have the AI critique its own work with a different prompt. Super interesting stuff there.

100%, like a really, really good summary. I would say that the point to think about a bit, the prompt, because we simplify the steps, like the prompt is less sensitive, but on the reflection part is where to read a bit about in our paper, also in the blog, but the paper talks about like do and don'ts, like kind of how to do a reflection. It could be done wrongly. And about the knowledge accumulation, one thing, like I said it before, but the knowledge accumulation is not only like how the data is progressing from one step to another, but somehow, sometimes it's a skip connection. Sometimes you take an output that was generated in step number 4 and inject it again in step number 7 because you want to have the knowledge being given almost purely and not after all the digestion that happened on the way. That's how trying to prevent the accumulation of the probability of mistakes.

Yeah. Yeah. Knowledge accumulation, hopefully without error accumulation. Well, I definitely would refer people also to the GitHub repository where you can check out all the prompts. Everything is there with, you know, down to the temperature, all the settings, all that good stuff. And I do think this is something that is a very practical, easy to digest project that, that a lot of people could take inspiration from. So this has been great. I really appreciate it. Any final thoughts or comments before we break?

No, it was really, really, really enjoyable and love your podcast. I think it's a good time to tell me which one do you think is the, if I may ask, the one I must listen to, which one of the episodes for someone like me that like really likes the details, the code, software development.

One that I think has been really good for a lot of people is the Tiny Stories episode with 2 guys from, in fact, one is also Israeli, Ronen Elden. Uh, they are from Microsoft Research and they created this kind of like 3-year-old reading level. You don't know if that's a thing, but like very kind of basic language, short stories, simple concepts. And then they trained language models on that and they showed some super interesting results and some really interesting, and for me, kind of foundational intuition building, uh, results for how language models learn and why. Um, and you get even to micro reasoning skills at like very small models, like just, you know, a few million or a few tens of millions of parameters. You're already starting to see reasoning skills developing. And why is that? Uh, well, we'll leave it to the episode. Uh, but I thought that was a, I learned a lot from that one. This has been a lot of fun. Itamar Friedman, founder and CEO of Codium AI. Thank you for being part of the Cognitive Revolution.

Thank you.

Co-founder. It's always important. And my team is the reason for, for our success for now and for the future. people has ever seen.

Thank you.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.