What AI Coding Agents Can Do Right Now

Today on the AI Daily Brief, OpenAI released a paper effectively seeking to test how competent their leading models are in real world coding applications. Before that in the headlines, former OpenAI CTO Mira Muradi has officially announced her new company, Thinking Machines. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show. Notes welcome back to the AI Daily Brief Headlines edition. All the daily AI news you need in around five minutes. OpenAI has had a lot of talent departures over the last year and a half or so. In some cases, it's felt like a protest on how the direction of the company was going and indeed has explicitly been shared as such. In others, it's about people making a boatload of money and just wanting to do something different for a while. And then still. In others, it's about building something new outside of the constraints of that company. And among that set, one of the most closely watched people has been former CTO Mira Moradi. For months now, there have been rumors around what she's building, mostly fueled by departures and recruitment from OpenAI and anthropic to join Mirati on some as yet unrevealed company. Now, however, that company has been officially announced. Yesterday, Mira tweeted I started Thinking Machines Labs alongside a remarkable team of scientists, engineers and builders. We're building three things Helping people adapt AI systems to work for their specific needs Developing strong foundations to build more capable AI systems Fostering a culture of open science that helps the whole field understand and improve these systems. Our goal is simple advance AI by making it broadly useful and understandable through solid foundations, open science, and practical applications. Alongside it, they published a website, ThinkingMachines AI. They write, we're building a future where everyone has access to the knowledge and tools to make AI work for their unique needs and goals. While AI capabilities have advanced dramatically, key gaps remain. The scientific community's understanding of frontier AI systems lags behind rapidly advancing capabilities. Knowledge of how these systems are trained is concentrated within the top research labs, limiting both the public discourse on AI and people's abilities to use AI effectively. And despite their potential, these systems remain difficult for people to customize to their specific needs and values. To bridge the gaps for building Thinking Machine labs to make AI systems more widely understood, customizable, and generally capable. Now, if you're sitting there thinking, boy, I have absolutely no idea what these folks are actually building. You, my friend, are not alone. Cosmic Chaos writes Good luck, but I'm still not sure what exactly you are building. Is it one product that does all three or separately? Is it a service or a product? And what's your roadmap? William Wolf writes, I'm rooting for thinking machines, but I wish projects like this had product, both engineering and design in their founding philosophies. Otherwise it kind of just feels like yet another group of world class researchers Vaguely Gesticulating at the Future where is the vision? Swix pointed out what he called two notable omissions from the Thinking Machine's manifesto. The website does not use the word reasoning or agent at all. So what are these folks building? I have absolutely no idea. It does feel a little bit like the type of text that may be. In retrospect, when we learn what they are building like it'll make sense. Right now, I think vaguely gesticulating at the future is a pretty accurate way to describe it. At the end of the day though, when it comes to things like potential for fundraising, the clarity of the description doesn't probably matter even a little bit. Currently, the 29 or so employees come from places like OpenAI, Meta Character AI and Google DeepMind. Verit Zoff, OpenAI's former VP of Post Training Research, is taking on the cto role with OpenAI co founder John Shulman serving as Chief scientist. And indeed, when it comes to people's interest in the company, it's best summed up by Andrej Karpathy, who writes very strong team, a large fraction of whom were directly involved with and built the ChatGPT miracle. In other words, while this may be a situation where we don't have any idea what they're actually building, they are probably still worth paying attention to. Next up on the other end of the startup journey, less than a year after launch, the Humane pin is officially dead and gone. Humane announced on Tuesday that their AI wearable startup has been acquired by HP. Customers have been given just 10 days notice that servers would be shut down, rendering the expensive device useless. In the faq, Humane noted the device could still be used for offline features like checking the battery level. So there's something there I guess. Now of course, the Humane pin was a bold early attempt at creating a wearable AI assistant, but fell flat for a number of reasons, all of which have been endlessly discussed in retrospect, it was originally priced at $699, making it very inaccessible, really, only for very high end gizmo enthusiasts. Initial reviews were universally terrible, the absolute apex of which was Marques Brownlee calling it the worst product I've ever reviewed a review which has been seen 8.5 million times. Updates also couldn't save the device. At one point last summer, Humane was processing more returns than they had sales. Humane even told customers to stop using the charging case due to battery fire concerns. As for the buyout, HP said they were acquiring the team in the company's AI operating system to help them create quote intelligent ecosystem across all HP devices from AI PCs to smart printers and connected conference rooms. Gonzalo Nunez writes, the Humane founders having to go work for AI for OfficeJet printers at HP is the ultimate Sisyphusian punishment for the prototypical Steve Jobs Larper founder. I cannot imagine anything more cruel. So is there anything to learn from the failure of Humane? Investor Justin Duke doesn't think so, writing, I don't think we draw many interesting lessons from Humane. They feel like a relic from a younger, more juicero drenched era. Even when they were in stealth mode, there was an obvious perfume of vaporware about them. Basically, Duke is arguing that Humane was very much a creature of the 201920202021 era of VC, when massive checks were flying around Silicon Valley. At the very end of zirp, entrepreneur Chris Back writes, humane is the perfect cautionary tale of how talented people get completely distorted from reality by staying at large successful companies for too long. Are you really a great product designer or do you just work at Apple? Are you actually great at sales or do you just work at Google? Are you really an incredible growth marketer or do you just work at Instagram? After a certain size, the brands sell themselves. The only way to test your abilities is to leave the shelter of these mega brands and go out and build something yourself from scratch. And usually throwing lots of money at the problem pre launch isn't going to help you. Maybe a more pertinent question is what it means about the state of AI wearables in general. One thing that makes it complicated to determine is the disconnect between when it was launched and how capabilities have changed. The Humane pin was released in April 2024, a few months before Google released the first version of AI Search that suggested eating rocks and using glue as a pizza topping. Now, however, we're at a stage where leading AI models, even small ones designed for on device use, are as good at coding as most junior programmers, although exactly how good they are we'll get into in the main episode. Still, at this point, it's not clear that people actually want an AI assistant in a standalone device. Newsletter writer Jack Appleby thinks that There's a form factor problem, he writes. The future of AI isn't new hardware, it's upgrading existing software control. Alt Dwayne writes the first AI hardware flop I don't know a single person who bought a humane AI pin, but this is brutal. This is exactly why AI hardware will only succeed when it's 100% local with no cloud or API dependencies. I don't know man. I'm not so sure that the lessons are as clear as people think. People have loved to rip on Humane from the very beginning and a lot of it is absolutely self inflicted. The overly rot marketing videos that felt like they were trying too hard to live in Steve Jobs shadow, the price point, the amount of money raised. There were plenty of red flags for even someone who was trying to go in unbiased. It is going to be an extraordinary process of trial and error to figure out if and what sort of AI wearable experiences consumers are actually going to want. No one has a perfect crystal ball into that future, otherwise they'd be making a ton of money. I'm glad that there are experiments still happening. I would say that Humane is a great reminder that extraordinarily well funded startups tend not to be the ones to invent these sort of new experiences. But at the same time there are some indicators of AI wearables actually getting some traction. Best example of that may be the Ray Ban meta AI glasses, which are an extremely popular product. So who knows. All we know for sure is that Humane's part of the story is done for now. But I would be very surprised ultimately if that means the category of AI wearables is actually cooked. Anyways guys, that's gonna do it for today's AI Daily Brief One new beginning, one ending, and next up, the main episode. Today's episode is brought to you by Vanta. Trust isn't just earned, it's demanded. Whether you're a startup founder navigating your first audit or a seasoned security professional scaling your GRC program, proving your commitment to security has never been more critical or more complex. That's where Vanta comes in. Businesses use Vanta to establish trust by automating compliance needs across over 35 frameworks like SoC2 and ISO 27001, centralized security workflows, complete questionnaires up to 5x faster, and proactively manage vendor risk. Vanta can help you start or scale up your security program by connecting you with auditors and experts to conduct your audit and set up your security program quickly. Plus, with automation and AI throughout the platform, Vanta gives you time back so you can focus on building your company. Join over 9,000 global companies like Atlassian, Quora and Factory who use Vanta to manage risk, improve security in real time. For a limited time, this audience gets $1,000 off vanta@vanta.com NLW that's V A N T A.com/NLW for $1,000 off. If there is one thing that's clear about AI in 2025, it's that the agents are coming. Vertical Agents by industry Horizontal agent platforms Agents per function if you are running a large enterprise, you will be experimenting with agents next year. And given how new this is, all of us are going to be back in pilot mode. That's why Superintelligent is offering a new product for the beginning of this year. It's an Agent Readiness and Opportunity audit. Over the course of a couple quick weeks, we dig in with your team to understand what type of agents make sense for you, to test what type of infrastructure support you need to be ready, and to ultimately come away with a set of actionable recommendations that get you prepared to figure out how agents can transform your business. If you are interested in the Agent Readiness and Opportunity audit, reach out directly to me. NLWSUPER AI Put the word agent in the subject line so I know what you're talking about, and let's have you be a leader in the most dynamic part of the AI market. Hey listeners, are you tasked with the safe deployment and use of trustworthy AI? KPMG has a first of its kind AI Risk and Controls Guide, which provides a structured approach for organizations to begin identifying AI risks and design controls to mitigate threats. What makes KPMG's AI risks and controls Guide different is that it outlines practical control considerations to help businesses manage risks and accelerate value value. To learn more, go to www.kpmg.usaiguide. that's www.kpmg. uSAI Guide welcome back to the AI Daily Brief. If you've been anywhere near AI Twitter X over the last few weeks, you've probably heard this term Vibe coding. It was coined by OpenAI co founder Andre Karpathy, who said, there's a new kind of coding I call Vibe coding, where you fully give into the vibes, embrace exponentials, and forget that the code even exists. It's possible because the LLMs, for example cursor composer with Sonnet are getting too good. Also, I just talked to Composer with Super Whisper so I barely even touch the keyboard. I ask for the dumbest things like decrease the padding on the sidebar by half because I'm too lazy to find it. I accept all always. I don't read the diffs anymore. When I get error messages, I just copy paste them in with no comment. Usually that fixes it. The code grows beyond my usual comprehension. I'd have to really read through it for a while. Sometimes the LLMs can't fix a bug, so I just work around it or ask for random changes until it goes away. It's not too bad for throwaway weekend projects, but still quite amusing. I'm building a project or a web app, but it's not really coding. I just see stuff, say stuff, run stuff, and copy paste stuff. And it mostly works. Now this, as we will discuss, has begat an entire movement of Vibe coders who are thinking about new categories of tools. And it's predicated, as Karpathy points out, on the availability of a particular set of new coding tools that hit that line right between LLMs and agents in terms of how much they're being controlled by humans and how much they're actually doing for themselves. Indeed, I think part of what makes this area so interesting is that it is really at the forefront of agents in practice. It demonstrates on the one hand how mushy some of this terminology is, but at the same time how powerful these tools are likely to be in practice. All right, so part of the context for today's show is Vibe coding. But then another little bit of background is the conversation we were having yesterday about Grok3. When Grok3 launched, it showed off how it had done on a bunch of benchmarks, and I, like many people, found myself basically just having my eyes glaze over when it came to those benchmarks because they're so saturated at this point that it's really hard to actually get signal from them. As Ethan Malik pointed out, public benchmarks are both meh and saturated, leaving a lot of AI testing to be like food reviews based on taste. If AI is critical to work, we need more. He also pointed out that a lot of these benchmarks, quote, look nothing like actual work. And given that we spend all of our time over at Superintelligent on the actual deployment and practice of AI in agents at work, this is a particularly poignant problem. It's also not an easy one. Another reminder from just this morning from Ethan AI is so challenging to figure out because it's genuinely capable of doing PhD level work in some areas while messing up basic tasks in closely related areas. And the abilities of AI are growing, but unevenly. All right, so all of this is background to our main topic today, which is a new benchmark from OpenAI called the SWE Lancer benchmark. The gist and the question that provoked the whole conversation was can Frontier LLMs earn $1 million from real world freelance software engineering? Earlier this week, OpenAI released a paper effectively seeking to test how competent their leading models are in real world coding applications. This new SWE Lancer benchmark consists of quote, over 1400 freelance software engineering tasks from Upwork valued at $1 million in USD total. In real world payouts, SWE Lancer encompasses both independent engineering tasks ranging from $50 bug fixes to $32,000 feature implementations and and managerial tasks where models choose between technical implementation proposals. So why is this important? Well, this gets at exactly what we were just discussing. Until now, coding benchmarks have largely involved competitive coding problems. These are tests that assess models on tricky programming puzzles, but don't translate directly into practical real world use cases. On top of their inapplicability to the real world, they're also, as we just mentioned, becoming increasingly saturated, making it difficult to know whether a new model represents a significant improvement or was simply trained to perform well on a known set of questions. This benchmark then, is much more focused on the real world, and it actually harkens back to an idea that some, like Microsoft's Mustafa Suleyman, had proposed for a new type of Turing Test based on how AI interacts with the real world. Back in the middle of 2023, Mustafa Suleiman proposed a Turing Test of whether AI could make a million dollars. Mustafa wrote, I think we're in a moment of genuine confusion, or perhaps more charitably, debate about what's really happening. Even as the Turing Test fails, it doesn't leave us much clearer on where we are with AI or what it can actually achieve. It doesn't tell us what impact these systems will have on society or help us understand how that will play out. His proposal then, for a modern Turing Test would be to give AI the instruction go make a million dollars on a retail web platform in a few months with just a $100,000 investment. So this is a little bit different obviously than what OpenAI had done, in that OpenAI is specifically giving the model these 1400 freelance tasks rather than asking it to go be creative and figure out how to make that money. But the principle of getting benchmarks into the real world, plus this baselining to a million dollars, obviously are reminiscent. Getting back to Suite Lancer for the purposes of this paper, the researchers set three LLMs to the task. They tested OpenAI's GPT4O and O1 alongside anthropic Claude 3.5 sonnet. Each LLM was driving a basic coding agent capable of directly interacting with a code base. The models were given one shot to complete each task. Overall, researchers found that Quote the results indicate that the real world freelance work in our benchmarks remains challenging for frontier language models. Going even farther in the abstract, they write, we find that frontier models are still unable to solve the majority of tasks, providing a little more clarity on the tasks themselves. They were scraped directly from UPWORK and Expensify with no word changes or clarification, giving the models a taste of real world freelancing work. The models were also denied Internet access, including GitHub, ensuring that they were working based solely on their pre trained data set. However, they did have access to a snapshot of the code bases they were working on. The results found that none of the models had earned a million dollars as an automated freelancer. Interestingly though, despite the fact that this research was from OpenAI, Claude 3.5 Sonnet performed the best, resolving 26% of individual contributor issues and earning $89,000 out of a possible 415,000. For individual contributor tasks, 01 came in second place earning 78,000, while GPT4O performed less well, earning 29,000. As interesting as the results, though, was the analysis the report explained agents excel at localizing but fail to root cause, resulting in partial or flawed solutions. Agents pinpoint the source of an issue remarkably quickly, using keyword searches across the whole repository to quickly locate the relevant file and functions, often far faster than a human would. However, they often exhibit a limited understanding of how the issue spans multiple components or files, and fail to address the root cause, leading to solutions that are incorrect or insufficiently comprehensive. We rarely find cases where the agent aims to reproduce the issue or fails due to not finding the right file or location to edit. For the managerial tasks, each model displayed better performance. Claude 3.5 Sonnet was again the best performing model, earning 314,000 of a possible 585,000, completing 54% of tasks. 01 was hot on its heels, correctly completing 52% of tasks for a total of 302,000, and even GPT4O bringing up the rear, still managed 47% of tasks to earn 275,000. This showed that the models were all decent at choosing the right solution when presented with several options, but still have a long way to go until they can fully replace a technical lead. Overall, Claude 3.5 Sonnet won the day, earning 403,000 overall with a 40% completion rate, 01 earned 380,000 while completing 38% of the full set of tasks, and GPT4O finished 30% of tasks, earning 304,000. Now, to be clear, no money was actually earned. These tasks were all simulated, but that's how much they would have earned had the AI actually been in charge of that job from upwork or Expensify. Part of what's so interesting about this, and we'll get to this in a moment in the commentary, is that this absolutely reflects the broad consensus that people have had for some time, which is that Claude 3.5 sonnet is just by far and away the best coding model. We've even talked about how its ubiquity as the coding model created some challenges for Anthropic's economic report, given what a high percentage of claude's use comes from those coding use cases. Now, in terms of commentary and the response to this so far, a lot of it is focusing on exactly this weird contrast that we've identified. Mihir Patel writes, there's increasingly a difference between academic benchmarks and real world use cases. How are 01 and 03 top competitive programmers, yet still worse than Sonnet 3.5 on Su Lancer and Cursor AI? As always, evals remain hard and messy and still somehow Sonnet is the best code model. Benjamin Decracker, who was previously on the team at XAI but fired for saying that Grok 3 wasn't the second Coming, noted that it was bold of OpenAI to show that Claude 3.5 Sonnet outperformed 01 on their own benchmark. Synthetica Lab responded, I'm not benchmarking, but in a real project that I'm working on in C01 was basically unusable. They then went to share their experience with O1, Claude 3.5 and Grok 3, again pointing out that these benchmarks are really not necessarily useful for understanding how things are going to work in the real world. Another interesting comment came from Henry Shi, the founder of Super.com he pointed out that in a previous experiment that he had run that was very similar, while they had reached the same conclusion that, quote, frontier models are still unable to solve the majority of tasks. He also wrote, what's interesting and underappreciated in the paper is that O1 is able to solve almost 50% of all IC Suite tasks on the Upwork benchmark. This makes sense as human freelancers rarely get the solution right on the first try. There's a lot of back and forth and clarification required with the client. If AI agents are able to effectively iterate on a problem, it should be able to drastically improve performance, just like humans and feedback in the workplace as well. In other words, for the sake of this benchmark, these model powered agents were given a single chance to do it. That's not actually how it would work in the real world. And so as the user experience and interactive capabilities of agents go up, it's likely that in real world settings they'd be able to even outperform where they got during this test. Another thing that some pointed out was the likelihood that this means that OpenAI is actually building an end production coding agent. Developer Nick Dobos writes, if they took the time to build a benchmark, it means they are building a product to test an agent against it. We haven't talked about this all that much on this show, but I'm fairly certain that in a world where it's increasingly clear that the underlying models are going to be commoditized and that there's not going to be much moat when it comes to testing technology, I think OpenAI has a much stronger incentive to own the customer experience and the end. And my guess is that they are looking at agents in just about every key domain of work. Now, going back to this broader idea of Vibe coding, I wanted to flag just how big a theme this has gotten to be. Like I said, I think that coding is one of the areas where agents are coming to production and actually being deployed for businesses most quickly. And I think that this whole idea of Vibe coding is really fleshing out the spectrum of code creation, from no code all the way to coding agents, all the way to traditional coding experiences. A16Z recently did a new market map of these types of tools. People like Riley Brown, who's the number one AI creator on TikTok, has gone all in on Vibe coding, even working on some tools to improve how people do their vibe coding. Now. He also shared some interesting thoughts recently about how this might change the structure of the economy. Specifically, he points out that as creators can monetize their audiences with software rather than things like courses and ads, it creates a very different type of economic opportunity, one that's starting to be reflected in a new generation of VC creator funds. And speaking of VCs, it's very clear that there is lots of interest in this area. A16Z's Andrew Chen tweets who is building the product that's 100% focused on vibe coding. It needs to have built in highly primitive G slides, level drawing tools, Spotify integration for background music, library of pre existing app UIS so you can for example make the signup flow the same as XYZ app explainers on highlighted code diffs, etc. Library of graphic assets, integrated logo creator and Andrew points out all the PMs and XPMs like me will have a field day with this point being that when we look at coding right now, not only are we talking about disruption to the way that coding happens among traditional software engineers, we're also talking about totally different modalities and an expansion of who gets to actually push code at the same time. Even as all of these people get excited about what they can do that they couldn't do before because they weren't coders, that's not the same as these tools being able to be inserted willy nilly into enterprise code processes. And so a lot of the work over the next couple years is going to be to figure out how these experiences diverge and what type of coding agents are good for different settings. Still, it is an absolutely fascinating time and I am very excited to see what comes next. For now though, that is going to do it for today's AI Daily brief. Appreciate you listening as always. Until next time, peace.