OpenAI Codex Agent – Is This the End of Programmers?

We finally have a cloud based coding agent from OpenAI and they're calling it Codex. This gives us the first glimpse of how collaborative coding is going to look like in future, and also why it's more important than ever to start learning coding. Now that might be a controversial opinion, so I'll explain it later in the video. But first, OpenAI has a huge problem and that is naming these systems. So Sam Altman tweeted this we will name it better than ChatGPT this time in case it takes off. Well, I don't think they did a good job. As somebody pointed out, today OpenAI releases Codecs, which is powered by Codex 1, a specialized version of Open 03. It is not to be confused with Codex, a model released in 2021, or codecs, the CLI tool released last month. Now, all jokes aside, this seems to be a very interesting implementation and this could become an alternative to something like Devon from Cognition Lab. This system is cloud based, very similar to what Cognition Lab offers in the form of Devin, and you can run a whole bunch of parallel tasks. So it's very different than other IDEs like windsurf or Cursive, both in functionality as well as how the users interact with the system. In this video we're going to see how this system works. We're going to also look at some of the strengths and limitations and what people are saying about it. The opinion is very divided and also I'll share some of my thoughts about the future of coding. So let's quickly look at some of the interesting bits in the blog post itself. So one of the thing is that this model is a fine tune or optimized version of O3 for software engineering. So it's not a completely new model or system in itself. And the goal of this system is to be a collaborative agent that is going to work alongside a developer. So a developer can assign it tasks and it will go complete those tasks then come back with implementation and the user has to accept those those changes or those feature implementation before they can submit a PR. As they point out, a task can take from one minute all the way up to 30 minutes and then the user will have the ability to monitor codecs progress. So very similar in functionality to Devin. So here's how it's supposed to work. This is going to be available within the ChatGPT interface as you can see here, but you just need to connect GitHub repo or GitHub account to it so that it actually has access to your account. Right now it's Available on Pro and Enterprise is coming soon to plus, we don't know when that is going to happen. So you are going to connect a GitHub repo. Within the GitHub repo you need to create an agents MD file which is going to have a set of instructions of what the repo is about, very similar to cursor rules or rules in one serve. So the model or the agent is going to look at your GitHub repo, it's going to set up environments and then it opens up a sandbox in OpenAI's cloud. So nothing is running locally. Everything is happening in their cloud. And since everything is running in the cloud, you can start parallel tasks. So this is a very neat feature and you can ask questions, you can assign a task to implement a new feature in your code base or try to figure out if there are any bugs in the code base. So really interesting features. All you need to do is just describe your task. The system is going to set up a new sandbox as I said, and then it's going to show you all its progress of what exactly is happening. And you as a user have full control and you will need to actually merge those changes back in the form of of a PR that is going to create for you. So here's an example of what the Agents MD file is going to look like. Again, these are just a set of instructions on what exactly you want the model to do. Now, just like a good programmer, it's going to create unit tests, run those, and when those unit test passes, it will kind of create a report and also potentially create a PR for you that you can review yourself or your team can review and merge the pull request. Now the question is, how good is this new Codex agentic coding system? Well, they have provided a couple of different benchmarks and they also show the problem with these benchmarks. So here's the sweep. Verified. Now, if you let it limit to only one attempt at solving a specific task, then it is definitely better than O3 on high settings. However, if you increase the number of attempts codecs and both O3 high can take, actually the performance difference is not that much, especially it kind of situates around four attempts because the difference is marginal. That's why OpenAI have their own internal software engineering tasks benchmark. On this benchmark, it definitely shows relatively bigger improvement compared to O3 high. On the SWEE benchmark they say they had to discard 23 samples that were not runnable on their internal infrastructure. And Codex 1 was tested at maximum context length of 192,000 tokens and medium reasoning effort, which is assess the settings that is going to be available for everybody today. As somebody on X pointed out, in terms of this performance on Sweepbench, this does not seem to be that different than O3. Here is Aidan from OpenAI. He says Sweepbench is a seriously imperfect target, just as 98 on MMLU is kind of scarier than 95 because we know some of the questions are wrong. Now keep in mind sweepbench verified was created in collaboration with OpenAI. So it kind of brings in the question of how good these benchmarks actually are right anywhere. Why would you test it on a benchmark that you know it's kind of imperfect? And then O3 is already incredibly capable. Codex One is meant to be more useful and better product that creates more value for the user. Okay, now the question is how much more value. It really depends on who you are asking. So here's a couple of examples. Here's Matt, he says I have had Codex access for a bit now, haven't spent as much time as I would like, but when it works it's powerful, best suited for backend tasks in my experience. Now something I have noticed is a lot of people who had early access this they say when it works it's really good. Here is another user who also had early access and again he says when it works it's a pretty magical experience. I'm surprised by the level of understanding the model has from the code. But then we have experiences like this trying out codecs how it's supposed to fix if it doesn't have access to the Internet. Like I can't install NPM packages or upgrade things in my project. So I think the user was asking can you please upgrade the next JS to latest and make sure everything still works. But then it says I'm sorry but I can't upgrade next year's because it doesn't have Internet access. I believe once you create the project then it does not have Internet access anymore. Now they actually highlighted this specific feature in their blog post. They say the Codex agent operates entirely within a secure isolated container in the cloud. During task execution, Internet access is disabled, limiting the agent's interaction solely to the code explicitly provided via GitHub repo. And I think this is one of the major limitation of the system. For example, if I'm building a new feature I would like to have the latest version of all the libraries, right? And then sometimes, especially if you're working with these older code bases you probably have an older version of a package. Example that I run into is the Generative AI package from Google. So most of the older projects that I have built for Gemini API are using the older version of Gemini API. However, right now I ran into a lot of issues because there is now this unified SDK. But if you disable Internet access then you will not be able to use the latest versions because you need to look up the documentation. Now from their perspective, they want to do it for security purposes, but I think it's going to really limit the usability of Agentix system like this. Now keep in mind it's still in a preview version, so probably they are going to enable that specific feature. And here they say we prioritize security and transparency when designing codecs so users can verify its outputs. A safeguard that grows increasingly more important as AI models handle more complex tasks independently and safety considerations evolve. Right, and that is I think a really important concern to have. But you also want to have more and more flexibility. Now they are seriously limiting or safeguarding against malicious applications that can be built using these AI driven agentic systems. But then they say that it's important that the protective measures do not unduly hinder legitimate and beneficial applications that may involve techniques sometimes also used for malware development. Right. And seems like the system is smart enough to figure out. Now in terms of the real world smartness, we will have to wait and get opinion from the community on how good this system is. But this at least opens up the possibility that we are going to potentially see a lot of contributions from these AI agents into bigger and bigger open source projects. Now Devon had a very similar promise. We haven't already seen bigger contributions from these agents, but I think we are at a point where this is going to become a reality. Now as these coding agents are getting better and better. How should we start thinking about programming and coding in general? Now for that I think it's very important to listen to what Greg Brockman had to say.

One of the things that I find really exciting about how Codex works is it has very nice non human strengths and weaknesses. And so it really means that you get much more out of it if you start thinking of of it as not just a static tool that you just use like you know, just without having to build expertise, but if you, if you really optimize your code base around what it can do, you start like honestly, most of what Codex benefits from is just what is good software engineering practices in terms of, of modular code bases with good tests and things like that, that you're able to just move so fast. And we've seen that. We've seen that happen internally with many people at OpenAI.

So you can take this in two different ways. One is we need to learn a whole new paradigm of how to collaboratively code with these systems, which I think is wrong. The way you should think about it is you need to start learning best practices of software engineering. And for that, you actually need to understand coding principles if you want to effectively use these agentic systems. And this should be a very good opportunity for anybody who does not have a coding background and they want to build applications to learn from these systems on how to implement modular code and what are the best practices. That's the only way we will be able to build better applications and systems using these coding agents. So I personally think that these coding agents are not going to replace programmers, but effectively increase their productivity to new levels. Now, these are very early days, and in order to survive, you really need to get better in your craft. Let me know what you think. Anyways, I hope you found this video useful. Thanks for watching and as always, see you in the next one.