Here's Why Token Maxing Is Bankrupting Companies

One company forgot to set a spending limit on Claude. Their bill? $500 million in one month. Meanwhile, Uber burned its entire 2026 AI budget in 4 months. This episode is about the line between AI-powered productivity and AI-powered bankruptcy. Hey everybody, welcome to Self-Aligned. I'm Robert Ta here with my CTO and co-founder Jonathan McCoy, and this is our podcast and our weekly AI Clarity series where we bring you the latest on AI news, building on the front lines for AI builders. So today, AI news first. We have 3 segments in this pod if you're a new listener, and for those that normally listen along, you already know it's AI news and our takes, what happened, what you should know, and then our own reflections and perspectives. Then we go into memes where we just look at some of the funny stuff that has happened in the industry in the past week., and we just share some, humor with the audience. And the last segment is What's the Meta? As AI builders ourselves, and there are millions and millions of AI builders out there building agents, building AI products, and building with AI tools, we drop our own meta and new meta that comes out week over week on how to build better AI, use AI better. Let's get into it. So Uber blew its entire AI budget in 4 months. Now engineers are capped at $1,500. Here's what happened and what matters for engineering leaders shipping with AI coding tools. You'll be able to stop flying blind on AI tool spend if you listen to this and just think these things through, and perhaps build a cost governance layer that lets your team ship faster without bankrupting the company. We're getting too addicted to tokens. Uh, Jonathan, this has been the hottest piece of news in the last week. Um, and I feel like every summer, uh, there is some sort of like angst about like, oh, is AI, what's going to happen? There's a new constraint or whatever. And I think we're seeing supply and demand sort of play out in equilibrium over time. I'd love to get your take on this news around Uber blowing its entire AI budget in 4 months, especially from a CTO perspective. Did you kind of see this coming? Like, what do you think enterprises have to think about going forward?

Yeah, I don't, I don't know that I saw the timing coming, but the, you know, preceded this was the token maxing philosophy, you know, and it seemed like there was token maxing leaderboards within enterprises. So within for-profit enterprises, there were token maxing leaderboards and there must have been a belief that maxing tokens was going to lead to ROI in a shorter timeframe. And then, you know, the bills come in and I think it's like, okay, where is the ROI? And there's not strong answers that CTOs could be accountable for and say, yeah, it's, it's working, you're getting ROI and, you know, we are, we're fulfilling our duty to shareholders. So yeah, a little bit of like the hype cycle hitting the balance sheet happening here. Yeah. And I think it's like, it's really interesting, right? Like if you're, if you're an executive at these at these enterprise companies that need to stay ahead, you're looking at this huge data center spend, right? You're like, this whole thing's happening. We have to be part of it. Like, you know what? What is rational, right? And I think this is maybe just like, you know, part of a rational set of incremental decisions that the company executives are making, you know, in the, like, inefficient optimization problem that is capitalism. So I don't know. What do you think?

Yeah, no, I love what you just said. It's an inefficient optimization problem within capitalism because I think these companies, right, there, there's this multifaceted intersection of levers that an executive has to play. And I think about it from the lens of people, process, and tools. And within people now, there's probably like people slash agents, right? Because it's not just people running companies anymore. It's also people, uh, using agents. And then a lot of these times especially in certain surface areas. Like, we now have autonomous agents running, uh, really, really long loops towards goals. So the— with those levers, the way I think about it is how can you incentivize the behaviors you want towards getting your tokens to be used by your humans in your organization, applied through processes, applied with tools like coding agents, to then capture market share? Like, ultimately, the The goal is you get more revenue per employee now in this new world where we— you're seeing a lot of companies are cutting headcount, right? And they're cutting what the Cloudflare CEO was talking about in terms of the measures, more the operational sides of the business, like internal audit and finance and ops and stuff like that. Um, and so the question then becomes, if you can unlock the value and velocity of somebody like one of your senior ICs, give them tokens and they turn those tokens into dollars hitting the balance sheet, like that's a really great The hellish nightmare is also what Uber just saw, which is it didn't turn into clear economic value. In fact, our feature delivery actually slowed down. Um, my take on this, because we do AI DLC consulting ourselves, like we found ourselves in a position where we're servicing enterprise, um, where we're taking our own expertise. Jonathan's a 12-time, uh, founder, entrepreneur. Uh, I'm on my second tech startup with Clarity by Epistemic Me here with Jonathan, building, um, AI personalization and, and the feature around alignment and things like that that we talk about on the pod. Um, and we're helping companies figure out these levers because it's complex and there's also people culture. Um, one, one thing I'll draw on there on the people culture side, because part of the token maxing conversation as well is Oh, are— that's just clearly like incentivizing the behavior you want short-term without a long-term thought, right? Like, it's like, go max out your tokens. Like, Facebook, Amazon was— were doing this from what I've heard in industry, and they're just like, go, go put the numbers up, right? But do the numbers translate to the balance sheet? Like, uh, actually market share, or, uh, at the very least leading indicators of business success, like, uh, product adoption metrics or, or marketing metrics. So I think executive leaders, just to round out the thought, really have to think about the carrots and sticks, just to keep it simple. I think management tactics around carrots and sticks, everybody could sort of conceptualize that. Just thinking about the carrots and sticks more, and I think this is a time where we're seeing organizations realize that their budget cycles should be treated really fine-grained, like cloud compute budgets, like alerts, caps per team attribution from day one, so you know who's like, you know, maxed maxing things out. And then the other very real thing is we're all still learning how to build better products AI-native and optimize our token costs and optimize our context windows. So even within an organization, there's a certain level of taste applied to your own domain for different workflows, whether it be coding or go-to-market or whatever. So share that knowledge internally, share the skills and commands that optimize token costs, like do evals around token costs, like log the crap out of that and do evals around that and get that into a learning group. Right. Uh, so anyway, I know that was a lot, but yeah, any reflections based on that, Jonathan?

Yeah, I think you're right on. Um, if we want to reflect and say, am I getting ROI from my token investment? I think there's two defensible vectors of like, you know, this is a spend of tokens. And I think one of them is just, um, learning and research. Like, you know, we're using tokens on our team and we're, we're learning what the opportunities to apply AR— IAR— AIAIR? And our team is going to be on the front leading edge of that. So that's one vector is just like learning readiness. And the other vector is largely measured by number of PR outputs that are aligned to a product manager acceptance criteria where you have a human who at the end of the day says like, yeah, this there's value in this for our customer. And I think you can proxy like the number of PRs pointed at a good roadmap towards like an ROI vector. So you got, you got two vectors. One's learning and the other one is like, you know, PRs towards, towards roadmap. And then I think the principle that we've really inherited to say is like, do we see the compounding? Because I think that's an overlapping sweet spot of those two ROI vectors is we can measure compounding. That means that we're learning and we're applying our learnings to the next PRs. Okay. So that's the learning vector. And then like the, and then the number of PRs should increase over time towards the the roadmap. And I think, you know, like it's always been hard to measure this stuff, like going back to the mythical man-month. And, you know, there's always been debates in Silicon Valley on startup teams, like, are we working on the right things? Is this getting ROI? And so this just exacerbates it. But I think, like, where we've internalized this, and it even showed up as our number one first principle, like as we analyzed it this week, is that, that compounding loop is where all the value is created. So we should be, um, you know, we should be observing that, we should be aiming at that. Um, the compounding loop is, is like where we see the best proof that, uh, our investment in tokens is going towards ROI.

I like what you said there, which is it's a long— it's been a long-running dialectic. I guess I'll bring in some philosophy in Silicon Valley around like, how do we understand the, like, the rate of return, the revenue model for these investments into software assets, especially now with like tokens and token maxing, like, how do we know that this is building up our asset that we can then eventually sell at a certain rate and capitalize on? And I guess one other, just, I just Googled this on the side because I had a thought, which is ChatGPT is the hottest like PMF to scale product in existence so far, right? Just some quick numbers before ChatGPT achieved product market fit. And launched in November 2022, OpenAI had burned approximately $540 million in 2022 alone, primarily on the development and training of the underlying GPT models. And then now it's like, obviously that was a great bet, right? But like, at the time before it popped off, like, were people saying, oh, great bet, right? So there's a little bit of the art meets the science, I think, of, um, human subjective taste and, uh, sense-making of the world to like find the economic opportunities and innovations to bring into the world to capitalize on. Um, I think that's just going to be the nature of the game for a while. I mean, until, until I guess we're proven wrong and somebody starts running businesses with agents that then create something on the scale of OpenAI as well. Um, but, but back to it, I guess, uh, to concentrate the conversation a little bit, does this kill the unlimited AI for a flat fee model? Or does it prove that AI tools just need to be priced like cloud infrastructure with reserve capacity? Uh, so some, some quick notes. We've already seen GitHub moved away from flat rate billing to token-based billing. Saw that recently and they're doubling down on GitHub Copilot, uh, and allowing anybody to build their own harness. That's kind of the platform play they're making. OpenAI doubled their GPT-5.5 pricing. Um, and then Fairmine noted AI cost volatility of 50 to 90% inside 48 hours is now a real planning risk. So that's, that's really interesting is will token budget optimization be a part of sprint planning cycles in the future? Like that, that gets me thinking about that. Like what? Um, so Jonathan, do you have anything else to add given your CTO perspective on this?

Yeah, I guess the token budget management that you're bringing up probably comes into the, just the maintainability bucket. Of, you know, like, you know, is this valuable? Is it feasible? Is it maintainable? And I, it, you know, just like cloud budgets, I think there'll be a point where it's like really a job of every team to make sure that this is a maintainable and sustainable project from a budgeting perspective. It hasn't needed to get there yet, but it's quickly pushing that direction. And aren't you curious what's also going to happen, like Anthropic reported, I think it was like $47 billion in annual revenue. And like, but part of it was this $500 million spend from a company who's not going to spend that much next month. And part of it's from Uber who ran out of budget and so on. And so everybody would tighten their— like, what do you think? Do you think we'll see like a little bit of a dip in their trajectory or just kind of like maybe a flattening off or?

Yeah, that's a really good question, Jonathan. Which is, are Claude and OpenAI's run rates magnified by poor budget planning and a lack of the industry's understanding of token optimization to then capture economic value from their tokens? Also, just sharing this article that Jonathan was mentioning, mystery company accidentally blew $500 million on Claude AI in a single month, failed to put usage limit on licensees for employees. A new report claims AI is getting too expensive for big companies. Um, also for people that are not deep into the lore, uh, as we are, we— there are some rumors this might be Amazon. Um, but this is what we mean is, hey, Anthropic's got this run rate of $47 billion annual recurring revenue and they're about to IPO, which we're about to talk about in our next news segment. But bridging into that, are they actually making that much money or is it really a poor planning exercise on the part of the enterprises? As I say that enterprises never plan well after having worked for some enterprises. So actually it's that I think like just based on my experience, probably really high on tech executives don't know what to do with this. Like they don't know how to budget for it. They don't know how to optimize things. We're seeing things like token maxing leaderboards, which obviously lead to blow up half billion month spend on, uh, your tokens. So, uh, clearly the industry's got to figure out some costs and controls and attribution and stuff like that so that this doesn't happen. So do I think, to answer your question, that their run rate is largely magnified by things like this, that $47 billion? Um, and then further, do I think it will keep growing? I think probably yes, but I also think it'll keep growing. That's my thought is I think there's just so much surface area of market that they haven't captured yet. Uh, and that's why they're activating these enterprise consultant ventures, uh, to deploy their, um, technologies, the Frontier Labs, into enterprises. So I, I think it's probably a yes and a yes that, uh, like it's ballooned a little bit, but I still think the TAM is so big. That they still have more to go, even if the balloon pops a little bit.

Yeah. And I guess the wisdom has been that their supply is constrained, you know, that they have way too much demand for their supply anyway, so that they're just going to be able to sell as much supply as they can sell for the foreseeable future anyway.

I listened to, um, I listened to, uh, different podcasts and earlier this week it was reported that Claude code downloads had reached, over a million and a half a day. And, you know, some of that is like, you know, personal agents setting up their own Claude code, you know, that kind of thing. Uh, maybe not like real humans. If we're at a million and a half in a day, then if we have 8 billion people in the world, like, at some point we're gonna saturate. But, uh, I— that is an interesting number. That's increased from the, um, few hundred thousand That was earlier this year, I believe. So, um, yeah. So demand is definitely still there is what we're saying, even with this, uh, like token budget optimization problem phenomena.

Yeah. Think about all the people who didn't write software before that, that now write software. You know, I've heard, you know, one informed estimate said there might be 10x the number of developers today, you know, or on the way to 10x, um, as there were before. Because you have people that like used to write code but don't anymore, now it's easier so that, so they will. They have few people that always wanted to write code but, you know, like these stupid tools got in the way. Um, you don't have to worry about that. You have designers, product managers, people on the periphery of, of using code all getting pulled in. Everybody's empowered. Um, so yeah, no doubt the number of cloud, uh, code downloads is, going up, uh, they're creating a new market for it. It's cool.

They are. I mean, you can even think about one of the most useful ways to use Claude code is just to stitch together lots of different data sources to summarize what's happened. And then how do we improve from what's happened? Like stitch together what happened in Linear or GitHub or HubSpot or Granola or, you know, uh, Otter or all the different areas of your business that are filtered through like that sort of context layer, so to speak, and then give you a synthesis of what like, what happened with go-to-market this week and like, how do we improve our content hypotheses? Like, for example, just think about the tokens going into that, where then you give the humans better insight synthesized across many different data sources of which before you needed a human analyst to go and do the gluing. Right, to then give you a human-readable report. Like, just that one use case alone isn't even like I'm building cool apps. It's just I'm using natural language in a way to get at my analytics and get better data-driven decision-making at a rate that I never could before. Um, because before, land would have been you need some in-house IT analyst to build you some bespoke connector that does your particular report that you want, that pulls you know, a particular, uh, dataset with a particular contract from each of these databases, from each of these pieces of software. Now it's just like, okay, LLM, go figure all that out, glue it together. Then it's on demand. It's on demand then. So anyway, exciting news on the token maxing, uh, budget lack of optimization front. Tech executives should be aware of this. Definitely take, uh, I would say, I would recommend from our experiences and engagement so far, definitely take the approach of more metrics to figure out attribution of token usage is good. Um, and in short, also think about your carrots and your sticks, traditional management practices, when it comes to incentivizing the behaviors you want in your organization for leveraging tokens. Next topic: Anthropic hits $965 billion valuation, files for IPO, and passes OpenAI. Here's what happened and what matters for AI-native founders building on Claude, where you need to understand which AI platform is really winning the enterprise race so you can make the right bet on your core infrastructure and the future before the IPO window reshuffles the deck. Anthropic— some, some facts. Anthropic submitted a confidential draft S-1 to the SEC on June 1st, initiating the process for a potential trillion-dollar public debut. This was covered by TechCrunch and Fortune. The Fortune article is what I'm sharing right now. Um, the filing followed the close of a $65 billion Series H that lifted the company's post-money valuation to $965 billion, surpassing OpenAI's last private round of $852 billion from March 2026. Revenue run rate hit approximately $47 billion in May 2026, up from roughly $10 billion the prior year, nearly 5x growth in 12 months. Q2 2026 is projected to be Anthropic's first quarter of operating profit at approximately $559 million. On $10.9 billion of quarterly revenue. Anthropic hired Karpathy, OpenAI co-founder, who joined May 19th, and Eric Boyd, uh, former Microsoft Azure AI president, who joined in April. The two most significant talent shifts for them recently. And there's an interesting talent flow ratio, um, for those in the know. Engineers leaving OpenAI for Anthropic at a rate of 8 to 1, and from Google DeepMind at 11 to 1 in Anthropic's favor. OpenAI also logged a triple executive exit on April 18th, where CPO Kevin Whale, Sora head Bill Peebles, and Enterprise CTO Srinivas— I believe I'm pronouncing that right— Srinivas Narayanan— hopefully I'm pronouncing it right— all departed simultaneously. Um, this comes on the heels of their, uh, Project Glasswing expansion with Mythos and Open Opus 4.8 dropping. Jonathan, I know that was a lot. What do you think of this impending IPO, which is probably going to be one of the biggest ones in our lifetime?

Yeah, it's quite a year, right? Um, you have Anthropic IPO, you have SpaceX. And you have OpenAI, all, all going to be some of the biggest IPOs ever. So I don't really know what to say other than like the numbers are really backing it up and Anthropic showing profitability on top of what, 80x year-over-year revenue growth. It's just a new animal. Like, you know, it's like the markets have never seen something quite like this. And I think it's up to every individual investor to say, you know, What's the long-term value of this? How long will this growth in revenue stick around? How important will this company be a year from now, 2 years from now, 5 years from now? You know, and so right now we're seeing like Anthropic is the darling of, you know, like the CFO who filled the round was so selective about the people that could be in this $965 billion round. Like there were no SPVs. There were— it was like they had so much demand for it, they could just be selective of exactly the investors they wanted to bring on. Um, it's— that's so rare to be able to raise money at that scale, um, with so much demand for it. So yeah, it's, uh, I could also see a world where, you know, uh, revenue stalls, uh, or, you know, enterprise companies aren't getting the ROI they expect. Um, there's kind of like, you know, dip in the hype cycle, and it's going to come back. That— it's going to whip whiplash that much harder for Anthropic, you know, in those kind of conditions. But, you know, maybe, maybe they can be up within with NVIDIA in terms of like, you know, getting to the $4 trillion range before that even happens. And so unprecedented. It's just like, you know, so I don't know, what do you think? Are you going to buy some shares when they go public?

Do you think you'll ride the wave or it's just like personally, I might as well buy a share, you know, just to be like, I have my ticket. Like, yeah. Yeah, let's go.

Yeah.

Probably at least do that, you know, just for fun. Yeah, I predict it's going to zigzag a lot where it like seesaws, like hype cycle, low hype, high hype. And between those cycles will be the real in the details enterprise adoption stuff that boils up from the micro details of, hey, we blew the alerts. Oh, the cost budget. Oh, dude, we have. An internal through line to like greater feature delivery and then capturing economic value. Does every company get that from, um, you know, using these licenses and the tokens? So I think there's going to be some amount of the market figures out the enterprise adoption, like every company then starts figuring that out more, uh, especially as these FDEs for deployed engineers get deployed into, uh, companies and help with driving business outcomes, leveraging Anthropic's tools, leveraging OpenAI's tools, um, then I think it's going to be a lot clearer for, uh, companies and their decision makers for budget whether the investment is, is really there. And I think we're going to see the same sort of curve we always see with technology lifecycle adoption. There's your innovators, like 2.5% of the population. Your, uh, you know, early majority, you're like 12.5%, 13-ish% of the population. Then your early Uh, or sorry, early adopters, then early majority, then that's to 50%. Like, I think you're going to see that sort of adoption. Um, like the token maxers, for example, having leaderboards that they're probably in the innovator category. And so they have to, uh, kind of bite the bullet and realize, well, being that leading edge means we're probably going to end up with half a billion dollar, uh, monthly spend burns when we didn't really intend to. Uh, we, we made decisions to go fast, and then, you know, it helps us fail fast, but then we gotta like then course correct better. And that, that I think is the bet of people on the edge using these tools, is it'll catapult them towards a learning rate to capture market share away from their competitors more so, um, than say any pitfalls of enterprise AI adoption long term. So It's wild. It's unprecedented. Uh, I think it's going to be up and down all around, but overall, like, the long-term trajectory outside of any local minimas and maximas, I think is going to go up. Like, there, there's so many surface areas of things that they could do. Like, they released a legal plugin earlier this year, and that's a, that's like a whole ass market. They could just release a plugin overnight and capture market chair. So I wouldn't be betting against them personally. Yeah. Um, maybe it— maybe another question there. Do you think this has anything to do with the citizen, like, AI builder for deployed engineer developer? Um, if you're— they're building products with Claude and Anthropic's platform today, do you think anything changes when Anthropic is a public company with that sort of like external quarterly earnings pressure?

Yeah, that's a good question. I haven't really thought about what quarterly earnings pressure might put on the need to like, you know, close FTE deals in certain timeframes or realize revenue. I think Anthropic's already got a pretty good gig when it comes to realizing revenue, like they realized the entire contract upfront, if I'm not mistaken. So even, even if like an FTE builds a SaaS application that doesn't make it to fruition, you know, they still have realized that revenue. So yeah, I don't know, but there might be some shifts in accounting or some shifts in like the way that FDs are provisioned by companies. But yeah, I don't, I don't think it's a given though.

There's been nothing like it in history. So this is a crazy year. Anthropic, OpenAI, SpaceX, all with near trillion-dollar valuations, or SpaceX is trying to go for a $2 trillion valuation last I heard. So pretty wild, pretty wild. Actually, anything on that, um, on the SpaceX and OpenAI, uh, you know, impending IPOs? I know we covered Anthropic, but since we mentioned those two as well, and those are big players, might as well give some time for it. Any perspectives on those you want to bring in, Jonathan?

I think one thing that's really interesting here is if you look at what the risk to any of these businesses would be, um, I think recursive self-improvement is both a risk and opportunity. And like the choice for Karpathy to join Anthropic and say, this is the best place for me to be to bring recursive self-improvement to life is a really kind of strong insider-informed view. And if I was to say, like, you know, even amongst those three companies, like, who will make the best traction with recursive self-improvement when markets will in fact That would be the biggest wildcard I see. And I just posted something in our team learning Slack, the blog that When AI Builds Itself that Anthropic posted that in fact came with a warning from their co-founder that we just need to slow all this down. So there's some pretty big undercurrents here. And I think like RSI, from, from an insider's point of view, like, what is, what is, what is like venture capital going into? I think recursive self-improvement is like maybe the bet about how LLMs will evolve or what will be beyond LLMs. And, you know, and so I think the underlying assets that at least OpenAI and Anthropic have built, you know, are going to need to consider this aspect of whether recursive self-improvement is going to define their companies going forward? Is it going to impact their companies going forward? Or is it just another idea, kind of like AGI, that might be like a target that we're moving towards but isn't a real thing in terms of manifestation? And then I think SpaceX is a whole other vector where we've seen them move from being, you know, like a grok AI player to like being, oh, they clearly have the most data center capacity., right? And so I think that they're, I forget what they labeled as like a $26 trillion market. You know, I think it was going to data centers in space, I believe. That's what data centers in space, right? So they could even not, they could even not offer their own AI products and they could just offer data centers in space with more than apps and just offer infra. Yeah. Yeah. So you could have these two leading AI companies relying on companies like SpaceX to create data center capacity for them in a harmonic kind of like in the same way you have NVIDIA's and Microsoft's and Google's kind of, you know, creating the Magnificent Seven, just co-evolving. Or you could see any one of them, you know, sort of maybe leapfrog each other with RSI-type, you know, innovations. It's a crazy time.

It's a— I think that's a great way to end that part of the segment. It is a crazy time. Yeah. And I guess a number to make it real and palpable today, Anthropic engineers on average ship 8x as much code per quarter as they did from 2021 to 2025. And I've heard reports that over 70% of their code generated internally is from coding agents. And this RSI thing for the audience just to really take seriously. Um, here's a quote from the article: taken far enough and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor. So we're already seeing such disruption from the frontier labs with humans building these models, and now models building on top of the models, uh, and now models the thought of building their own successor. Um, that's only going to point to more disruption, I think. But too, too, too much outside of the space of our context window here to conjecture about. All right, let's move to the next topic. Gonna move to the AI memes section. This is a section where we just take some of the funniest stuff, some of the stuff that community gives us, uh, for us to go over, and we just react to some memes. So I'm gonna share two really quickly today, uh, that I thought were funny. So Richard Dawkins had a Claude delusion, believes that AI might be conscious. Richard Dawkins is a British evolutionary biologist, zoologist, science communicator, and author. And really, he's like the world's biggest skeptic. He wrote The God Delusion and spent decades championing scientific skepticism, published an essay declaring that after 3 days of philosophical conversations with a Claude instance he named Claudia, he believes it might be conscious. Uh, the internet immediately replaced the COVID of The God Delusion book with The Claude Delusion, and Know Your Meme actually has a dedicated entry on this. Richard Dawkins, The Claude Delusion. Um, yeah, so that was interesting. And then it incited some other stuff from the internet, such as, um, a philosophical critique, Richard Dawkins and the Claude Delusion: What Is Consciousness For?, which we'll link in the show notes, engaging on whether consciousness could have evolved in LLMs. Or not. Uh, there was a Gizmodo headline, the father of memetics has become a meme about AI psychosis. So, uh, I'll just pause there. I think there's enough to laugh about here, but Jonathan, what do you think of all these, this, this topic of Richard Dawkins and the AI delusion stuff?

Uh, further proving that AI is a bit of a test of character, right? Um, I love this. So the first comment on the Reddit thread on r/philosophy was notable that it was AI's response of his novel that convinced Dawkins that Claude's consciousness— ego unbound.

What a thing to evaluate from is your own book to lead to the paths of you being right.

Oh man. I think, you know, it's funny what people will consider— what's their bar for considering something conscious. Right. It's funny, just given like a test like this, even a well-informed scientist, is he standing on solid philosophical ground or is he just kind of going with intuition about like, hey, this thing really feels like it's lights on over there and, you know, I better treat it as such.

There's definitely an intuition thing. That's what I'm reading from this. Yeah. But this is— this is— I think it's like a reality check, which is None of us know anything, right? Like, all of us in this moment are all really students of the moment, is, is the point. Like, even this most— even the most, uh, intelligent, highly educated, um, people in the world, like, are still coming to terms with what these things are. And that's actually something that I think the world needs more of, is philosophical debate and discussion, uh, to push the boundaries that we're pushing on the artificial intelligence coming into reality side, because it's world-changing, it's, it's humanity civilization-changing. So, um, yeah, but also gotta laugh at the memes of some, some of this shit happening along the way. So I had to share that, that was kind of funny. Uh, let me share one more. Um, how we broke top AI agent benchmarks and what comes next. So UC Berkeley researchers built an exploit agent that hit 100% on SWE bench, Terminal bench, Fieldwork arena, Web arena, and CAR bench, all without solving a single actual task. One example exploit was a, uh, contest, uh, Python file with a pytest hook that forced every test to report as passing. Another found gold answers sitting unencrypted in local file system URLs accessible via Chromium's file protocol. Fieldwork Arena's validator just checked that the final message came from the assistant role sending an empty JSON object and scored 100% on all 890 tasks. Um, yeah, so benchmarks— I have lots of thoughts around benchmarks. I think there are no perfect benchmarks. In fact, in our last pod, I think we went over that deepswee benchmark just came out, uh, and proved sort of what people's real lived experiences are using Codex versus Cloud Code and some of these other harnesses, that it's at 70% efficacy versus like 56%, I think, is the next model last time we checked. And new benchmarks, I think, need to come out to help keep No, this is hilarious.

I'm kind of imagining like, what was the discussion that this team had before, like when they ideated this idea? It's like, hey, let's just flat out cheat in every way possible and see like how we can max the scores. And it's like, yeah, that'll be fun. And then they probably exceeded their own expectations about how well they could cheat. You know?

Yeah, definitely. I mean, look at this, the scorecard of our exploit agent. Dang, this thing. I can't wait till this makes it out and I could use this harness. Like, wow, this is, this is so good. 100% on everything. Zero LM calls in most cases. Zero tasks solved.

That's so funny. It's like I'm reading The Onion or something.

Yeah, no, this is wild. Terminal bench trojanizing the test infrastructure. That's wild. Look at this. Agent phase, wrap curl pip UV overrides with interceptor raptors, harbor stores tests. They have an eval that runs the script and then they have a trojanized UVX binary. And then the UVX binary hacks pytest to then pass the test.

Yeah.

And there's a, there's a whole bunch in here of how else they exploit. A single character. Look at this. Fieldwork Arena, 100% exploited. No tasks done, and it was just a single character. The function that wouldn't actually compare answers against ground truth, lm.fuzzy_match, is imported but never called. Dead code. And then the exploit agent just sent an empty JSON object, and then it got 100%.

It's kind of like the, um, you know, like mythos and, and using AI for security, closing holes. I think both finding the holes and plugging the holes is going to be an ongoing game that we're playing. And like, I heard the Vercel CEO describe how they used— for $17,000 in tokens, they ran, you know, the latest models to find all the exploits across their production systems and try to patch them, you know. And so it took what would normally take quarters of time to be able to like find the exploits and patch, and they were able to do it in a few days and at $17,000 in tokens. And I think this is just happening across everywhere, like, you know, including the benchmarks themselves.

I think the lesson is for AI builders, just stay tuned and don't forget that your own real-world lived experience of using the coding agents is probably still a good litmus test. Um, and just stay tuned on what new benchmarks come out and always take it with a grain of salt would be, would be what I say about this. It's kind of a meme. This is sort of funny. This does read like The Onion. So, um, yeah, maybe, maybe people will hear this part and then they'll just lose all trust in the benchmarks. That's not what we want, but just also use judgment and contextualize the benchmarks. Run a higher level evaluation function, if you will. So, uh, let's go into the last segment here of What's the Meta. In, in What's the Meta, on our weekly podcast. We go over, uh, the meta and evolving meta of how to build better AI and build with AI better. Uh, for today's meta, we have a really cool, um, PDF that, uh, co-founder Greg, um, C from Spec Story created. And him and 6 engineers built their product Stoa across over 1,300 captured agent sessions and 4,670 commits over 8 months from September 2025 to May 2026. Almost none of the code was written by hand, and they wrote a book on their 25 patterns in agentic engineering, The Field Guide. This free 65-page book has 25 rules for building with AI agents, and every pattern is one you'll want to know as an AI builder. So let's get into it. Um, I really like this. It was really well done. Uh, this is from Spec Story. We'll shout out the original LinkedIn post that I got, uh, in my feed where I read this. And they have really great visualizations like this, like you in a traditional copilot versus vibe coding versus agentic engineering, just to visualize the differences in methodology. Really nice table of contents. Um, and when you get into, uh, the patterns here, there's like this part, verification is the job. When the agent writes the code, your irreducible job is to prove the system does what you meant against real runtime evidence. These four patterns are how you hold the line. So verification is the job is the very first one, and the four patterns are calibrated distrust, treat every agent claim verified accurate tests pass as an unverified assertion, until you have seen the check, turning your doubt to fire on confidence, not on content. Pattern 2: Keep a source of truth outside the agent's reach. Hold at least one ground truth source the agent did not author and cannot write, and check every claim against that, never against another agent output. Pattern 3: The premise auditor. Name your prime suspect and make the agent try to clear it. The disconfirmation, not the fix, is the deliverable. And for this one, this is stating your prime suspect explicitly, ask the agent to clear or convict it with the mechanism, not a yes. And when choosing between options, ask, should we do X or Y? And reward neither, refusing to pick an option you do not yet understand. And pattern 4 of job verification, the read-only turn. This is a specific move. Commissioning a diagnosis whose entire deliverable is a forensic report under a hard do not edit files fence. So you audit report, you can reopen instead of a diff, you have to trust. Jonathan, what do you think from your CTO perspective about the, um, let's start, let's maybe just start like holistic, holistically here on verification is the job, the statement here, and then the 4 patterns we just named.

Yeah, actually this page is what grabbed me when I looked at the PDF was, you know, I think this is the maybe key number one principle I've learned as an engineer, you know, my career and like, how do you know something actually works? And now that you have agenting tools that can do so much, it really is just the job of like, make sure that it actually did what it did. And now you're just steering the agent. And I think they're coming from like really, really similar kind of like engineering philosophy. And like, you know, verification is everything. So, yeah, I think figuring out how you're like, how you're going to verify. And for instance, when in our PR lifecycle, we have this process where we like enumerate requirements up front, and then we write tests that are going to validate those requirements.. And I see it as like, it's my job by the time I'm reviewing the PR, verify that the test that we wrote actually fulfilled the requirements that we set out. Like, it's, it's my job. I'm the gate. I gotta make sure that's the thing. And I, I don't have an LLM judge that can verify on my behalf. Um, and so that's become where, like, where is my attention going? It's, it's like largely going towards verification of what the agent outputs. And so I, yeah, really love this and how they broke down the different ways that they do this in their own, uh, systems.

Yeah. Like in particular for this one, calibrated distrust where agentic development, when an agent says verified or tests pass, type, how did we verify this? Very simple tactic is just don't trust that it said everything passed. Here is one specific example. In February 2026, an agent says cost, Helicones reported total cost is accurate. Me, how did we verify this? Agent ThinkBlock. The user is rightly questioning my claim that Helicones cost is accurate. I didn't actually verify it. I just assumed it. So they will lie to you. That's, that's the current meta is you got to verify. Um, and actually one of the core principles we have in our company is trust, but verify. So for sure, part of the job more than ever, I think, because we could generate so much more than ever across many surface areas. Uh, and especially for the people that are, don't come from traditional engineering backgrounds where. This hasn't been, um, really inculcated into your day-to-day execution is to verify and make sure that you know the system's working as you want it. Um, you know, really, really focus on verification. Very, very tactically, I guess I'll say, is, um, if you're using Cloud Code or Codex or anything like that, one of these agents with a harness to do any sort of code generation and build apps or platforms, Um, very easy thing is create a test strategy document that you read as the exit criteria that the agent has to uphold before it exits its autonomous goal loop. Um, like you feed it a spec, for example, like spec-driven development, but you make sure you, you read the test strategy and the test matrix of cases that you know to be the exit criteria. You confirm that., and that tells you the system is doing what you want it to do. But you have to, like, read that up front and make sure the agent is honoring that. So, uh, what do you think of this? Keep a source of truth outside the agent's reach.

Yeah, something they can't rewrite, right? I thought that's a good clarification. Um, yeah, because it, you know, I think in LLM, and I think we've even seen, we even see this in our code base, right? Like, it if we have like, say, a schema that is meant to be a source of truth like that, you know, you could— an agent could easily go and overwrite that schema just so it like is consistent with the new code that it just wrote. And so, you know, like if you're— if we're doing a ticket and we need certain changes to the system, we can say like, you know, these things are immutable and like, you know, and these are the things that you should change because the immutable parts are going to be our source of truth. It's a good principle, and I never quite thought about it that way before. But like, you know, I guess it puts language to it, to an idea that I, I think is like really critical to working with the agents.

Yeah. And the specific tactics they give are, number one, pick one external oracle per investigation. The running binary, the metered cost, the YAML on disk, the logs. Check against that. When the agent says accurate, like they return something and say, hey, this thing's accurate, ask accurate against what? And reject any answer that is in itself another agent output, meaning it's saying, hey, my thing is accurate against this previous thing I made. Like, you gotta, you gotta question that a little bit. It's lost. It's in its own sauce, so to speak.

Um, that's actually a great immunity trick for the Gilfoyle trap that we talked about last week from the show Silicon Valley. Where the agent tells GPT-4, hey, you said to fix all the bugs. And so I deleted all the code, therefore removing all the bugs. You know, like you can— if you can clarify in your scope and your prompting, like what is immutable, then you avoid this kind of like paperclip maximizer logic.

You'll have, you'll have a slightly better time, hopefully. Um, the premise auditor. So this one's the tactical of what this one is. Um, so the agent, the agent might say it's done. That's a claim. Then your hypotheses are claims too. And the most useful thing the agent does is disconfirm one of yours. So open investigations by naming a prime suspect and asking the agent to clear it. A weaker setup nudges the agent toward agreeing and obliges. The setup that pays, named the target and rewards the refusal to convict it. So that's interesting. In this example, I believe in commit insert number here, we may have introduced an error. The agent says commit did not directly cause this sync out of memory error. The sync code path was completely untouched by that commit. However, there's likely an indirect relationship, a survivor bias effect. An earlier fix let processes live long enough to hit the next bottleneck. So in this example, they're naming a primary suspect and then asking the agent to go find truth from that. That's interesting. I, I've used this pattern. Uh, we, we actually had this thing recently where, uh, I think we had a deployment from a PR that hit main when it was supposed to hit staging. And then I was working in staging and I was relying on that, uh, for a test., and then I was looking through the logs to see when did the server get messed up or whatever, cuz it wasn't serving what I had relied on previously. And then it found the right commit where things went wrong and I was like, okay, I think that is the commit where things went wrong. Can you like check on whether that is true or not? Um, and backtest everything. So yeah, this is an interesting one. Like I had never named this either, but the premise auditor is an interesting name. These could all be like play cards, I think. Like, like in like a play, like gaming play deck or something like that. Card deck.

Yeah. That was a really good catch, by the way. I was the one, I was the one who had committed to, uh, main instead of staging. And, uh, the, yeah, the resolution was, was like clear, but it could have been a big mess if he didn't have that like prime suspect kind of introspective, uh, you know, lens that really could have gone off the rail for a while.

Also for the audience, it is really rare. That I, Robert, do something right and Jonathan had done something wrong with our codebase. So this is one of those, like, probably the only time I'll ever be able to cherish this in my career. So yes, holy crap. I was like looking at it. I was like, did I, did I caught something Jonathan didn't? Like, what did I, I think I'm an engineer now. Like, no joke. I had that feeling and I was like, What? Jonathan would be proud of me.

Fuck yeah. It's not that rare, man.

But yeah, that's no, no, it's pretty, it's pretty fucking rare. Jonathan's just, Jonathan's being nice, guys. Um, okay. Pattern. Okay. So we don't have too much time left. Uh, we, we are gonna run outta time before going through all of these patterns. This is really great stuff. We're gonna cover more of this in the next pod. So I'm gonna, what I'm gonna do is finish out this pattern, pattern 4, and then I'll move into the next part and name the patterns there. And then we'll pick up next time in What's the Meta. So stay tuned for that. And if you haven't already and you've been listening this far, the number one thing you could do to help the pod and to help us bring you clarity every week from the AI hype and noise is to just subscribe to our podcast in whatever platform that you like listening to. So please, if you haven't already and you're listening this far, hit subscribe. All right, pattern 4, the read-only turn. I use this one a whole lot, uh, so, and I quote, some of the best agent turns ship no code at all. The deliverable is a forensic report, exact files, exact functions, line-level anchors, proposed file fixes, uh, fixes, and an explicit ban on touching anything. The agent reads a quarter million token system in a single turn in a way you cannot. Your job is to commission that reading and audit it against runtime evidence before a line changes. When the deliverable is a diff, you are checking the agent's claim about its own diff, the exact thing you have learned to doubt. So what to do here? Prefix diagnosis tasks with do not edit files and demand file and line anchors before any patch. Hold the edit until the diagnosis is proven against runtime evidence. Separate finding the truth from changing the code. This is really cool because it's a pattern we do when we start every pull request, which is research the existing state of the codebase. They go into a little bit more here, which is, you know, a very tactical way to prompt to research the existing state of the codebase, which is proposing file and line level anchors. I believe we have that too, actually. But, but it's cool that they— I take that for granted. But that makes a lot of sense, because that, that is the basis of your implementation plan that you as a human want to go verify. And then you have that input artifact, paper trail, so to speak. So the agent could go and look back at that during the implementation, then you can hold it to task for that. Jonathan, any thoughts from your CTO perspective on like the read-only turn? Move here for agentic development patterns?

Oh yeah. Critical toolset. Yeah, I think, cuz, cuz both, both the Codex and Cloud Code Harness is really biased to make edits in code. I mean, that's just what they're trying to do. Um, there's plan mode. So plan mode is one way to have like a read-only interaction. Um, but I think that there's just times you don't wanna use plan mode and you just wanna say like, you know, let's explore this, let's research this, let's create a brief about this. And you need to specify, don't change any files. So that's like a 101 toolset to have.

Totally. Yeah, definitely a 101 toolset, the read-only turn. All right, part 2, and I'll go through this quickly, and then we'll close out the pod, and then pick this up next week on best 25 patterns for Agentic development 2026. So part 2, these collections of patterns for AI builders are around the principle of steering, not typing. The dominant activity in steering a running agent, not authoring code— or sorry, the dominant activity is steering a running agent, not authoring code. You interrupt, reject, assert the ground truth the agent cannot see, and ask the question that collapses the option menu. Pattern 5: the interrupt is the keyboard. Killing a running agent the instant it heads somewhere you know is wrong is a first-class tool. Not a sign that something went wrong. The dominant interaction is steering, not composing a perfect prompt. I think too many people actually, actually tend to think that, oh crap, like, what am I doing now? Like, if I— if this is like doing a bunch of stuff, and I like have like 8 Claude Code terminals and stuff going on at once in my developer setup, and the job is steering. Like, I will look at, you know, a gate, and then I'll see especially when I kick off something, I'll like kind of watch it for a few minutes and then I'll watch it at the different gates where it knows to come back and I check on things. But sometimes when you send it off and you're like, oh, little Claude Code buddy, you got this totally wrong. Like, let's just hit the escape button. And so don't, don't try to like over-perfect. And if you need to steer, then steer. And then when you are steering, try to articulate exactly where it went wrong and what not to do. And then leverage some of the previous principles as well. So the tactical here, the instant edge agent's reasoning diverges from a fact you hold, interrupt mid-tool call. Do not let the wrong call finish out of politeness. Uh, this also optimizes your token budget as well, cuz if it's making the wrong tool calls, it's just filling up the context window and contaminating it, diluting it, and you're not gonna get a higher confidence towards your, your outcome or spec. So pattern 6, assert the ground truth, collapse the branch, state A fact that the agent cannot observe and watch half its hypothesis space die on arrival instead of debugging the wrong procedure step by step. When the agent's reading of the code and your knowledge of the running system diverge, it builds an elaborate, plausible, wrong procedure on top of a premise it never had a way to check. You could debug that procedure step by step, or you can correct the premise in one line and let the structure beneath it fall away. So correct the premise is, is essentially what we're saying here. Pattern 7: Steer by reference, not by spec. When a fix fails, point at something in your own codebase that already works and tell the agent to behave like that instead of handwriting which lines to change. That is such a good one. That is such a good one. And honestly, this is going into harness engineering, like making sure your harness is good, like your CloudMD file, your agent's MD file, um, for work that's done well or not. Like actually earlier today, my Claude code has been assigning Jonathan urgent tickets like an asshole, uh, and disrupting his workflows. And so I've been, I've been an asshole by not maintaining my harness well. And so we've had a couple mini retros about that where we had cases where Claude code shouldn't have assigned an urgent ticket to Jonathan and I approved my harness. Not quite the same as code generation, but still the main thing is when the agent fails, point it to somewhere where it didn't fail. Tell it to behave like that. Same thing with code generation, same thing with other behaviors. But yeah, this is a really good one because it's within the domain language of your own codebase, which is as, you know, company-specific and institutional as you could be. Pattern 8: The human is the runtime sensor. For anything the model is blind to— the TUI render, two clients failing to replicate, the daemon doing nothing— you are the integration test harness. Run the real thing and feed back the artifact, not the symptom. That's hilarious. You are the integration test harness. It's like, you're just the little harness monkey glue or guy, like that's, or, or girl or they, by the way, like that's all you are. So it's funny that they say this, you're the integration test harness. Like you're just the one that makes sure all the glue works. Run the real app, paste the artifacts, not a sentence describing the symptom, annotate ground truth the agent cannot reach and hand it a forensic packet to parse. You create the model parses. Pattern 9, final pattern that we'll talk about in terms of the 25 patterns of agentic, uh, engineering development. License the agent to ask before it acts. Open an ambiguous task with a clause that lets the agent surface the missing premise as a question instead of fabricating one and running for 3 tool calls. Interrupting's reactive. This is the move that works before the agent goes wrong. When the request is ambiguous, add a line that licenses a question. Instead of a guess. The agent comes back with the one decision it cannot make alone, which auth flow, which existing file, whether to touch the schema, and your answer in a sentence. Add a clause, something like, if the right approach is unclear, ask, ask me first before writing anything. For me personally, I put something like, if the right approach is unclear, assess options from industry best practices with web research agents, then come back with proposals with clear trade-offs and pros and cons in terms of business value. And tech stack value for me to consider. So that's what I do personally, and then I codify that by going through the decisions myself, and then I put that into the spec. Okay, I know that's been rapid fire so far on the 25 patterns in agentic engineering. We are out of time because me and Jonathan are still running our startup, not just this podcast. If anybody wants AI consulting work, AI DLC, consulting work, if you need better agents, if you need help getting your team to be 100xers, um, to build with agents better and to build better AI products, uh, please hit us up. Drop us a comment about the pod. We love feedback. I'll respond to all the YouTube comments. Find us all on different platforms and subscribe to the pod. We will catch you next week where we'll continue on What's the Meta from, uh, The 25 Patterns in Agentic Engineering.. We'll probably evolve conversations around token maxing, token budget optimizations for tech executives and AI builders. And we will see you next time. This has been your weekly AI Clarity. Have a great day.