AI:AM — AI Coding Agents, Chip Design, and the Value of Human Judgment · October 6, 2026

I can just prompt Claude. I don't need you. When you work for me and I'm paying your tokens, you better be like actually producing thoughtful stuff. And I have 2 or 3 employees right now who are basically under performance review because they are just giving me Claude slop. Your videos went up like 14.7% week on week. And I say, why? And he doesn't know. He's actually never watched the videos. He just used Claude to like analyze the videos. And Claude doesn't have video analysis.

We're spending over $100,000 a day on tokens. The agents themselves have been able to completely do closed-loop iteration and testing on the On the token spend side, it's become, you know, our single largest non-manufacturing line item very, very quickly. We didn't tell anyone to cut their spending or do anything, even though we were seeing this exponential growth in our token spend.

First, the news. The Curve debate drifts toward runaway AI, a Frontier executive entertains caps on intelligence, and Anthropic's original restraint pledge comes under scrutiny. Short pause. Then Thomas Somers of Positron AI on what a chip design team actually buys with its agents, and Swyx on what an employer still pays people and vendors for once anyone can prompt the model. Long pause. To close, the hosts on Claude's advance on the threesome problem, and whether AI audits could trigger mass retractions in science.

Lighthaven had a conference called The Curve. Over the last few days. What was your takeaway from The Curve?

It's a really interesting and pretty unique event in the AI space, started by a couple of people who had the idea that we need to bring more contrasting opinions together.

So can you speak to some of the divides that you saw over the weekend?

Pro-open source, because it's going to save us from concentration of power issues. anti-open source because it's going to create biorisk. But I do think the big thing that people have kind of come together on, at least to a significant degree, is it does look like the AIs are going to get really powerful. And this is something we're really going to have to reckon with and can't just sort of hope goes away or feel like maybe if we just wait, the bubble will burst or the fever will break or what have you. And the question now on the capabilities front has gone to, Is RSI going to foom? Is it going to be fast for a while and then kind of level off? Yeah, we've got these super powerful things, but is it really going to kind of run away from us? Those were still kind of the capabilities questions that we're getting asked. But there has been, I think, some healthy updating across the community just based on what we've seen.

From the Frontier Labs, were there any kind of limits or constraints that they said You know, should be enforced on players in the space, including themselves and others.

Yeah, that was a huge topic. And so actually had a chance to talk to and hear from, in some cases in sessions, in some cases just in, you know, kind of passing conversations, all Chatham House rules. So I'll kind of abstract away, but there were a number of things that I actually thought were like, kind of newsworthy. One, let's say, senior executive at a frontier lab, first of all, was saying that just pretraining continues to deliver, that the models are just at the base level of capability, getting stronger and stronger. And there doesn't seem to be an obvious stopping point to that. Scaling laws are holding or maybe even bending. Because the data quality is getting better. Synthetic data seems to be definitely really delivering. Give the model a bunch of data and think, like, what could I transform this data into that if I then pre-trained on it would make the model smarter? And you're sort of converting these reasoning tokens, converting back into pre-training data, which then goes in, of course, to the next generation of models. So this is like one way in which recursive self-improvement is happening. RL is important too, but his take actually went as far as saying he believes that the latest frontier models do have the ineffable research taste that would be needed for them to start making paradigm-level breakthroughs. He believes it's currently an elicitation problem where the RL is not bringing it out well enough. They don't know quite how to bring it out well enough, and so it's kind of rare to figure out how to elicit it. And then he said a couple of things that I thought were just legitimately newsworthy. One was there likely is, let's say, a level of intelligence that we just shouldn't go past.

What?

Yeah. First time I'd ever heard that from a Frontier Lab leader. But this was a person who, again, everybody would recognise their name and position. Saying, I think it is likely that there's a level that we shouldn't go past. Now, was that not ever or just not right now? I think that was also a little bit ambiguous, but it was a pretty firm statement. Definitely the strongest statement I've heard about imposing kind of a hard cap on capabilities, even for a time. How do we operationalise that? Would you be prepared to sign on to something like a limit on the number of flops that go into the next pre-training run? No more than 10 to the 27 flops in the next pre-train or something like that. But what actually happened was he basically said, yeah, I think that could be reasonable. Again, the details will be incredibly important if we go down that path, because the more data you enrich, the more flops you put into enriching the data, you can kind of You can definitely play a sort of shell game of hide the compute, I think, these days.

What about Elon?

What about Zuck? But the take was kind of the 2 to 3 companies are really pushing the frontier. And even the other 2 that we would usually think of in the top 5, Elon and Zuck, are actually distilling a lot, whether they know it or not. And the way that they are distilling these days, it's not that they're calling the Claude API and getting traces and feeding them in. But rather the cottage industry of RL environment makers and sellers is in fact functioning as a distillation conduit, because what they're all doing is having Claude create these RL environments, selling the RL environments to other frontier model makers, and then they do their internal RL on an environment that only exists because Claude was smart enough to make it. The kind of synthesis of all this is if they hadn't been releasing the models and if people weren't able to do these various sort of direct and indirect distillation techniques, they actually think that probably would have happened, that the other companies are not really keeping up but for the sort of leak of intelligence in all these different ways, including the RL industry. that is gradually finding ways to transfer from frontier models to their competitors.

Anthropic and OpenAI are both structuring for more like the 2 to 3 year mark, and they're looking at a 2028, 2029 AGI RSI.

Honestly, even sooner, I would say, was kind of the vibe. And this is probably somewhat biased to be a short timelines crowd. But again, the short timelines crowd includes the executives, top researchers, founders of these companies. They were talking more about just next year. Definitely not full agreement on how fast it goes or if or when it tops out or whatever, but a shared sense that the decisions that get made over the coming months and the way that compute is used over the next year could be really critical. In terms of will they still have need for compute, or does a cap on pre-training scale or other compute input cap that might constitute a pacing mechanism. Does that mean the end of the compute buildout? I would say no. Another question I had the chance to ask was, of course, everybody's heard Jensen Huang on Ezra Klein, I think, at this point. He said at Nvidia, we spend like 20% of our effort on designing the chip, and we spend 80% of the effort on verifying and validating and testing all the edge cases and making sure it's going to last a long time and be reliable. all these other things that are not just the original design. Congratulations, you've succeeded. Now you're going to enter the era where you're going to have to make them safe, reliable, trustworthy, et cetera. And so I had a chance to ask one of these people, what do you think about that take from Jensen? And the response was basically like, yeah, that's kind of reasonable, without, again, knowing exactly where the numbers will shake out. the possibility that the majority of compute in the future could be going into all sorts of different safety measures, which could be monitoring, could be chain of thought monitoring. People are getting kind of bearish on chain of thought monitoring, but then there's internal activation monitoring. They're already spending a significant percent of compute on monitoring, and it sounds like that might go up a lot. Another thing that they're spending compute on in a big way right now is Fixing the RL environments. A model was trained on a bunch of hackable RL environments where cheating was rewarded. It's a bit of a cheater. That's a big problem. They are now applying these models to the RL environments themselves and saying, hack this environment. And now that is the direct task. They're doing it. They're finding all these places and ways in which the environments can be hacked. They're fixing those and gradually They're driving down the rate of flaws, which means that they're also driving down the rate at which cheating would be rewarded, which in turn means you're going to get less cheating from the models as they actually come online. I think this is still, as they say, an open research question, but it seems like they're probably going to be able to get comfortable releasing the next couple of things, even as they do consider themselves bottlenecked on alignment. But then past that, they're like, all bets are off. There was definitely some interesting discussion around How hard will Anthropic press their advantage to the degree that they have an advantage right now? And the sense that the AI safety community had when Anthropic was founded is that they were committed to not advancing the frontier. They wanted to be at the frontier so they could be relevant and do all the research, but they weren't going to push it because they didn't want to make the race condition worse. And people challenged some Anthropic leadership on that. And their response was like, well, what we said was we would never publish methods that would advance the frontier. And people were like, I don't think that's really what you were saying at the time. And now they've been asked, like, okay, OpenAI is trying to get their house in order. You guys definitely have your work to do too, but you seem like you have maybe a little bit more in order than they do. Are you going to chill and let them get their house in order and not try to run away with the competition or not? And we didn't really get an answer on that. So.

A chip startup's AI bill now rivals its payroll, and this episode asks what that money buys inside a chip design team and when the cheaper model becomes the costlier choice. Thomas Sommers co-founded Positron AI, whose engineers are taping out their first custom silicon with AI agents working the tools.

Our guest for today is Thomas Sommers. Thomas previously founded Rex Computing, a processor startup, and was a 2013 Thiel Fellow. He later became principal hardware architect at Lambda, which provides access to graphics processors for AI developers, and worked on its early cloud infrastructure. He then served as director of technology strategy at Groq, another AI chip company. Positron was founded in April 2023 and shipped its first Atlas server in August 2024. Atlas uses programmable chips. The company's next generation, Asimov, is custom silicon intended for Titan servers. Positron reports that more than 50 Atlas racks have been deployed at Oracle. It's a pleasure to have you on the show.

Thank you. Thanks for having me.

One of the things that you've done is you've speeded up the process of designing and deploying the chips. Can you tell us a little bit about how you did that?

I would say like a core founding tenet was that we wanted real hardware in customers' hands as quickly as possible. It was about 15 months from cold start of company to shipping first customer server, and then, you know, about 3 years total time to win Oracle as a customer. But for our second gen, which is full custom silicon, it's really, really amazing the advancements in AI over the past year, specifically in our ability to close the iteration loop in terms of everything from RTL developments, all the front-end work related to chip design, doing the verification, which is the the biggest time limiter, I would say, historically for chip designs, all the way to doing the full backend. AI has been able to drastically accelerate our own development of chips to make AI cheaper and more energy efficient.

How does this translate today to token spend versus salary spend?

So on the token spend side, it's become our single largest Bigger than human salaries. Yeah. Recently it eclipsed human salaries. It came back down. I'll explain in a moment. So 6 months ago was like equivalent of a single employee. Got Claude plus GPT and a little bit of Cursor and other stuff, all for the cost of a single employee. So our token spend just massively increased when in June it started to now be multiple employees, and now it's more than Like we were peaking, you know, in the, in the days after GPT-6 aspirin, we're really trying to, to push its boundaries, et cetera. We were, we were spending over $100,000 a day, um, on tokens. Um, that tempered down a little bit in, in the weeks after as we optimized a bit and, and just weren't trying to run as many parallel experiments. Um, but I would say the big thing that was like, you know, a saving grace was Opus 5.5 came out, and in a lot of our tasks, not everything, it was doing better than Astra and was a quarter of the price.

You mentioned how much AI is helping you on the verification side. In a notable interview Jensen Huang recently gave with Ezra Klein, he talked about how Nvidia spends something like 20% of its time and energy designing and then 80% verifying, validating, ensuring long lifespan reliability, et cetera, et cetera. What do your ratios look like on that? And what more color can you give us about specifically where AI is really helping?

So for us, I would say it's a little bit different because we're starting from scratch. It will end up being somewhere around maybe 60%. design focus for us to 40% verification? I would say like the probably most amazing thing that I don't think would've been possible 3 or 6 months ago and only really became possible with GPT-6 Astra and now Opus 5.5 that we're leveraging a lot in this task is we, for doing our verification work, we use Cadence Palladium emulator systems. So these are big giant racks that are full of custom ASICs. that are specifically designed to just gate-level emulation of the silicon. Agents themselves have been able to completely do closed-loop iteration and testing. You know, I think people thought it was crazy, but, you know, handing, you know, these very powerful agents, you know, full keys to our internal infrastructure and saying, here's a Palladium piece. And I highly, highly doubt that these models had, you know, Palladium documentation in, in their pre-training thing, especially because a lot of documents are, are new as, as software updates. But the fact that it went and looked up the documentation, when we're looking at these agent traces, the reasoning traces and, and tool calls and et cetera, it's reading the full documentation, effectively compressing that itself by, by, you know, reading the PDFs, generating its own, you know, markdown files of, of cheat sheets of what to do, and then building out all of the testing infrastructure and its own harnesses for being able to access this. If you're just doing regular software RTL simulation, that's running on the order of like 10 hertz for us in those cases. If you remove a lot of the debug pieces that slow things down, et cetera, we can run the full chip emulation on the order of 500 kilohertz. And Right now, that's really just focused on implementing test programs, finding cases where they fail, and then writing out reports which, you know, get reviewed by other agents and by humans in the loop. But I would say when this really started working, you know, 6 weeks ago, roughly, it was beginning of September when Astra-6 came out. It was huge. Step function improvement from, you know, GPT-5.6, where it could not do that full closed loop. It still required humans at different stages.

So one of the commentaries on AI in general is that it's a lot of like process following work, especially like, for example, taking a manual for a particular piece of software or hardware and implementing it or following the rules behind it. And the criticism has been that it is not really true new innovation. Have you, have you seen any signs of really some unexpected innovation rather than just rule following in that sense?

I have not yet. I'm somewhat proud of the fact that it has not figured out some of Our cleverness of what we've done. It has like questioned and like, this seems like a bad design decision until it gets explained to it why something's done, or it actually runs tests and sees and understands, oh, that's why you're not doing the traditional way a systolic array is done. And so it does make me feel a little proud, even with just how unbelievably amazing the models are today, that it won't come up with the, I would say, the ingenuity that we had as human designers. Do I think that will hold for another 6 months? Uh, not, not sure. Part why I actually think it's not going to be like the agent will come up with a better idea immediately. I think the amazing turn of events that we've just had in the past few weeks in terms of us having access to these models, um, is, is the fact that just because it can iterate so fast and do experiments and things by itself, I wouldn't be surprised if, if it would come to the same conclusions we did. Or maybe make better things than what we would do by ourselves, just because it's going to be able to iterate so fast and go through so many different design possibilities.

What kind of trade-offs does it involve, or what kind of engineering challenges do you take on in order to get the advantage of being able to use commodity memory versus the very high-end memory?

My fundamental belief was that There isn't going to be a stop anytime in the near future in terms of scaling laws and, and getting increased value from, from making models larger. So that eats into memory on one side. But then the other side, I think the main limiter today of AI being applied to, to most applications is context length and being able to hold more context, you know, per user and then scaling that out to drastically more users. And, and really today I would say that the driver of quote unquote users or individual sessions is actually just having more agents. I've gone from, you know, 3, 4 months ago where I had probably on average 2 to 4 agents like running constantly in the, in the background to where I'm now running 15 to 20. So, you know, if you then multiply how many people are using AI and how many that, that concurrent completely separate contexts adds up very, very quickly. That memory driver had us say, okay, we have to use the commodity memory 'cause that's the only thing that's going to be able to scale and be cost effective. And when we set that as sort of the constraint, in our architectural design, we had to come up with, you know, very clever innovative solutions. And, you know, the 2 main pieces of that, and, you know, the thing that we showed with our first generation product is being able to achieve extremely high memory bandwidth utilization from a compute architecture perspective. So while NVIDIA GPUs, you know, I would say on average are getting between 30 and 40% memory bandwidth utilization in the actual decode forward pass of a transformer model. So basically that means even though they advertise, you know, 8 terabytes per second of theoretical memory bandwidth with, you know, a B300, you only actually are seeing, you know, something in the ballpark of like 2 terabytes per second of actual realized memory bandwidth. It comes down to the reuse patterns that really don't exist in transformers that they designed their hardware architecture around. for training and other workloads. And so with our first-gen product, we were able to hit and sustain 93% of theoretical memory bandwidth. We're partners with Credo Semiconductor that, and have developed a memory chiplet solution that allows us to go from the maximum number of LPDDR, same type of memory that is in phones, laptops, et cetera, The max number of channels that you can find in any other products is on the order of 12 to 16 channels, and we go up to 72 channels of LPDDR5X with this decoupled memory chiplet solution.

Walk me through a model that fits locally, like which parts of the networking or coordination start to disappear?

With Osmov, the, the ROK chip, we have up to 2.3 terabytes of memory capacity per chip. So when you compare, you know, the, the B300 shipping today caps out at 288 gigabytes. So that means on a per-chip basis, you now can actually scale in what you would have needed from a memory perspective, 8 GPUs for, we can do on a single device. And it's not just, oh, you've got that cost silicon savings whenever you have to scale to having more than one device. There's overheads involved with that. There's, you know, your all-gathers, all-reduces that you have to do within, for each layer and each, really each multiply that you're doing of sharding across those devices. It comes down to real performance in the end.

There have been increases in prices of all kinds of memory modules in the last year or so, and pretty significant increases. Has that affected your projections on the TCO going forward?

I was looking at a quote that we had a year and a week ago. It's gone up 4.5 times since then. If all other things being the same, TCO comparative to other compute providers, everyone's in the same boat with having to increase their COGS and then thus prices. So that doesn't affect us that much from a competitive standpoint. I would say the fear would be, oh, if if the cost of the products goes up and you're otherwise delivering the same value, then there's a problem. But I think that the reality there is, compared to a year ago, the capability of a model using X amount of gigabytes of memory is way more than 5x than it was this time last year.

So obviously the nature of the hardware that people can access determines in part what kind of models they can deploy. I'd love to get your kind of perspective and breakdown on how these hardware considerations relate to looping. Maybe start with, like, why do we loop? Why is that even a tempting thing to do?

It's been very interesting, from the Loop Transformers paper to that being the rumour of one of the big advancements with GPT-6 Astra, that simply repeating the forward pass gets you that improvement. But it's crazy that there was really work that I think was only being done in sort of the fringes of the open source transformer community 2, 3 years ago of taking like a LLaMA 70B and just duplicating layers within. And you got better results from it, even though it was repeating the exact same matmul.

Imagine what you can do when you train it to work that way. Can you go just a little bit farther into the hardware connection of that? I'm kind of coming in with assumptions along the lines of just very fundamental terms, like why loop at all? I think it has to do with this kind of memory bandwidth, right? Because now I can keep the same weights on the chip and I don't have to shuffle things in and out as much. No?

Not really, because basically the assumption is if you're doing inference, that you're going to have all of your weights Locally in DRAM, you're just sharding that over some number of devices. You do get some amortization, but just with the size of experts and the sizes of caches on chips, there's not much of a reuse opportunity. You're, you know, by the time you're already done with the layer, you've already gone through the amount of memory many times that of what is in like the on-chip SRAMs. From like a hardware perspective, like looping really is more memory capacity savings than memory bandwidth savings. Because, like, let's say that you trained 2 models with the same base set and you trained one to do with looping, and so it only has 10 layers instead of 20 layers. And, you know, model B is 20 layers. That 20-layer model, it's doing very, very simple, is going to be double the size of the 10-layer version. And if you find that the looped 10-layer version gets 95% of the same—

Yeah.

you know, quality of results and it's half the size, you're probably going to deploy that one. You're saving more capacity than bandwidth, because doing the second loop, you still have to do the same memory fetches and do all of the same, like same number of maples, there's the exact same number of bytes that need to be moved and the exact same number of flops that have to be done in that model A and model B scenario.

When you design a chip, And there is a certain point in the design where you kind of freeze the design and, you know, hand it over to start the rest of the process. What are the decisions that you delay or defer until the very last moment before you decide to hand over?

There are a lot of features at different points of the chip development stages that we left on the cutting room floor. Most of them are things that we still want to do in future chips. But it's what can you actually accomplish, especially with our goal of iterating quickly. And it's better to get a product out quickly, even if it doesn't have all the features that you want, to get that out there, get all of the learnings, product feedback, revenue, everything that you get from actually having shipped something. It's not just the design time, but it's the verification of that component and, you know, figuring out how that integrates with the rest of the system. There are a bunch of optimizations that we could have made and done differently if, if, if, you know, a year ago we thought that linear attention was, was going to take off the way it has, at least in the open source model space. I'm still not convinced that that's being done at the scale that we see in, you know, the open source and specifically Chinese open source models. In like the big model labs. But it makes sense in the constraints that the Chinese ecosystem has in terms of hardware supply, etc., that they're effectively being forced to go in that direction. And so they're innovating in that way. As far as I'm aware, none of the major US model labs are doing anything like MLA or, you know, DSA or any of these other techniques that the— the Chinese labs have done. Primary customer focus is on like the big, big model companies. So I don't necessarily think we made the wrong choice there to focus on the things that we did.

Positron closed an $875 million Series C on September 10th, money earmarked for taping out Asimov and ramping its Titan servers. As the token bill kept climbing past payroll, did anyone tell the engineers to pull back?

We didn't tell anyone to cut their spending or do anything, even though we were seeing this exponential growth in, in our, our token spend. Um, it, it was really just the fact that, um, uh, we, we both felt that we were getting good ROI on it, so we weren't going to, to taper it, but also the fact that, like, we have very high confidence belief that costs are going to come down. And yeah, we're using, you know, the major model labs, you know, their models for everything today. You know, we also have faith that we're going to be able to move things to, you know, local models running on our own hardware. I would say, like, we do a little tiny bit of this with GLM 5.3 today. But personally, like, my philosophy is I really don't want anyone to be running like a real software or hardware development task on anything that is less than the best model. Like, I don't care if it's, you know, a 10th of the cost, you know, per token. It's just not worth the, the, the expense.

Sean Wang, Swyx, employs engineers and non-engineers, buys software to run a conference business, and decides who speaks at AI Engineer. That puts him on every side of one question: when anyone can prompt the model, what is an employer still paying people and vendors for?

Next guest is Swyx, or Sean Wang. Uh, he is the co-founder and CEO of AI Engineer, which runs conferences and workshops for people building software with AI. He's also co-founder and editor of Latent Space, a publication and podcast about AI and the people developing it. And he's also an advisor to Cognition, the company behind the coding agent Devin. Um, he also founded Small AI, whose AI news project combines automated reading and summarization with human selection. The next AI Engineer New York conference is scheduled for October 12th through 14th with a focus on applications across banking, investing, insurance, and financial technology. Let's bring up Sean.

Thanks for having me back on.

It's so effortless to create so many things that I really am bottlenecked on my ideas. What is the vibe that you're getting from people who have been software engineers building software applications, are they still feeling super empowered? Are they starting to feel threatened? For a while, people were like, oh, well, there's going to be so much demand for more software that it'll be fine. There'll be even more software engineers.

I think college kids are a bit worried, but other than that, if you're relatively plugged in and very capable with AI engineering tooling, You're in more demand than you've ever been because your expected value is higher than it's ever been. Therefore, the demand has increased a lot for a very specific kind of demand for people who can manage coding agents productively instead of producing a whole bunch of slop. And I've been in that situation. So I am both an engineer but also an employer of engineers and people who are not engineers who are vibecoding. When you work for me and I'm paying your tokens, you better be like actually producing thoughtful stuff. And I have 2 or 3 employees right now under performance review because they are just giving me clod slop. And that's really bad for them. And they don't understand it. You're not like producing any value from clod.

Like, I'm probably also mostly producing clod slop. What's the delta?

Go for depth of insight rather than breadth of coverage. You know, of like, you know, my YouTube guy will be like, oh yeah, your videos went up like 14.7% week on week. And I say, why? And he doesn't know. He's actually never watched the videos. He just used Claude to like analyze the videos. And Claude doesn't have video analysis, contextual understanding, or anything like that. You are a human using these tools. You cannot let these tools think for you. You gotta actually think to use these tools. What did I do right? What can I do better? Lots of areas for human domain expertise. But then they send it, they like substitute it for the, like the lowest energy, lowest cost means of production, which is chuck something in the cloud and hope that I don't notice.

He is a buyer as well as a boss. AI Engineer stages conferences several times a year, and the videos, design, and marketing come from outside agencies. When a vendor's work comes back, How quickly can you tell whether a human thought about it?

At an agency, actually, you know how these like launch videos come up? Agency goes like, oh yeah, we made that. I like reached out to one of those agencies, I want to work with you, requirements. And they came back to me with like very clear Claude slop. It's like, really doesn't get what we're going for. And I think like, to some extent, Claude slop is fine, as long as it's like useful and good. And they were just like, oh yeah, you caught us, uh, we're gonna move on. And so like, they're just hunting For people with low standards or like just don't care or who have enough money there. Whereas like, I think the people with taste just care about the product, the end product, and will just have like an immediate reaction to like, yeah, clearly you have no relationship whatsoever with this core content and you're just doing thought stuff.

So this whole taste question, to what degree is it coupled or correlated with a background in software engineering though? Because I'm not reading any code. Most people I'm talking to aren't reading any code, but do you care about the part of the resume that's like, oh, I've been coding for X years at this point, or is it not so relevant?

I don't care so much except for, let's call it security roles and for anything sort of backend scalability. And I really need you to know the differences between between GCP and AWS and what I can do on each one of those things. I just recorded an interview with the Supabase founders who are scaling Postgres to a scale that we've never seen before. Yeah, good luck trying to hire somebody to scale Postgres with a complete self-managed sharding solution that works at YouTube scale. There's only one person in the world that can do that, and they hired the guy. It's about taste, and it's not so much about length of experience. Actually, sometimes length of experience works against you because you just have a set way of doing things and you don't understand how to do more than one agent at a time. You should be relatively comfortable juggling like 5 to like 10 ongoing multiple things. Definitely true that you're much more of a manager than an individual contributor. I do think that people who look at data rather than code are more valuable these days. So being able to say, here's the logs, here's the traces, here's the schema, here's the input/output, capturing it, turning into an eval. All these are basically the merging of AI engineering and ML engineering that is happening that people need to upskill on.

And then managing scaled-up runs, Every coding agent works inside a context window, a bounded working memory. What falls out of it, including what another agent did last week, is simply gone. When several of them build one product, where does the human's oversight have to sit?

Like, right, like, uh, I think that the, the person that has, let's say, 10, 20 years of experience in software engineering really reviews every line and tries to make every line make sense. Whereas now we just need to make the modules make sense. And I can allow slop in there because it helps me to go faster, as long as I contain the slop in things where I completely understand the whole system. Where you go wrong is you have too many modules, too many black boxes where you don't even know. And actually, the code also gets confused.

Mm-hmm.

Slack competitor. I saw a bug where the messages weren't loading. I refreshed and the messages were loading, and I refreshed again and the messages were not loading. And it's like, well, it's the same exact code. What the hell's going on? Turns out there's 2 paths and there's a race condition. Why? Because 2 different coding agents worked on it at different times. And because they just made their own thing. And agents would potentially sometimes do it because sometimes just things fall out of the context window. But you need to have the oversight of the module. Inside of the module can be a black box.

You did a Kill My SaaS project.

Not done yet.

So why don't you tell us a little bit about it? It came about because you had a product that you were using for AI engineer and you found that they were kind of overbilling you or you were not happy with how much you were paying for the amount of work that they were doing.

Basically, you can call this like a bounty on mid-tier SaaS that should not exist. And this definitely was one of them. Basically, like, you know, half a salary or one-third of a salary for a piece of software that I don't own, that nobody enjoys using. Let's just throw that out. If I'm going to spend $40,000 on this SaaS subscription, I can spend that in amount of tokens, and that buys me a heck of a lot of tokens.

The software he wanted placed runs the back office of a conference its organizers expect to fill with more than 1,000 people and over 100 speakers next week. How do you tell a replacement that works from one that merely looks finished?

This is the problem, especially with like launching it without planning. I just did it on a vibe to my entire audience. We had so many submissions that we then have to eval them. And so evals are the problem. We're no longer checking, here's 2 or 3 requirements that you fit. We're checking the entire UX of the entire flow from 3 different perspectives of organizer, attendee, and sponsor or a speaker. And then you need different logins, then you need all the different workflows and to submit different applications across the different formats that we do. I actually shipped evals to help make participants on the second one. But for really frontier things, the human just has to playtest through all of it on top of their regular day. Just be aware that if you offer a large bounty, like we offered $10,000 as the prize, you get a lot of submissions because people want to vibe code for fun. And a lot of them will be low quality. It'll just be like, Claude, make no mistakes, go do this. They'll make mistakes and they'll just submit. And so like the, the sort of verification load is very imbalanced because they spent zero thought on this thing. They just threw it in there and then, and then you have no idea if it's like low quality or not.

Devin, the coding agent from Cognition, takes requests in plain language from a Slack channel. In a talk earlier this year, he described a designer on his team sending it annotated screenshots with no text. No training. What finally moves staff who distrust anything vibe-coded?

But overall, very successful. So I have one of the most adversarial non-AI teams in AI. I hire event professionals who are like super old school. They do everything in spreadsheets. Like these are not glamorous, like high-tech jobs. And they're very super suspicious of anything that is new and vibe-coded and techie. And initially they were like, we'll never use this vibe-coded thing. And this is like I want to use the tried and tested stuff that has been working for Microsoft and all these other guys. And then they saw the quality of the submissions, then they looked at their existing platform, they were like, yeah, okay, we're going to switch. This is one of the benefits of me partnering with Cognition on, is I just allow them access to Devin to modify the code. So anything they don't like, they can just request a change and they have it pretty much in 1 to 2 hours. Which they've never had before. Just to give you an idea of the way we work with these SaaS companies. Right now, when we request a change, they'll say, okay, that sounds pretty cool. It's on our Q3 roadmap. And we don't have confidence that it'll actually be done in Q3. We requested for a thing in Q1, it just landed. So SaaS is quite cooked if you're mostly a CRUD app.

So can you synthesize this experience with the first statement you made around if you're decent with vibe coding, never been in higher demand. There is a lot of latent demand for software that hasn't been met yet, but I have a hard time seeing how we don't kind of pick a lot of this low-hanging fruit and then end up in a sort of a lot of companies go out of business. Are we going to see mass layoffs from big tech? How long can the party go on?

I mean, the simple answer is if you look at it as a fixed pie of software engineering, then that is the conclusion that you will arrive at. But if you look at it as the point of like my competition or my TAM is spreadsheets, all spreadsheets made by anyone with any sort of productivity tooling can be turned into custom software for them to put on Rails and with a beautiful UX and with automations. Demand for custom software becomes a lot larger. And more beyond that, if it's all— if it's phone calls, if it's emails that go back and forth, and, and, you know, eventually physical meetings between people, all of that can be incrementally turned into more and more and more and more custom software and hardware and models, by the way. Like, why is Salesforce so damn big? Is because people can customize Salesforce. But at some point, they need to stop paying the $300K a year baseline subscription and spend $30K building their own personal CRM, and they're good. In fact, they're more than good, they're happier because they can now modify it to whatever they want. So you should have an idea of the sheer diversity of human needs and desires. We don't even have the same amount of customization in our software as we do our handbags.

What the hell?

We just need to get the infra there and need to scale it first. The limit on all this is the sum total of humanity's total token bandwidth, right? Which is literally the amount of silicon we can produce. It's going to dictate pricing and availability and rate limits and all these things.

You've already said that you've had issues with, I would say, poorly spent tokens, right? Tokens which were, you know, parts of your organization that perhaps didn't have the critical faculty of evaluating the tokens that were being emitted with sufficient thinking. So how does that work as you kind of start to scale organizations and you get less ability for a human manager to kind of evaluate whether their subordinates are spending tokens properly?

I mean, yes, you need the ability, but like Are you asking if I have a solution?

Yeah. I mean, how do you think that would work?

So I think this is coming down to just management philosophy in general, and I don't have any sort of core competitive advantage about that. So on one hand, yes, I would say that the leading companies and the names that you all heard of have a practical hiring limit because they just don't have enough managers to manage those people that they're hiring. It's roughly Let's call it something to the order of 100 to 400 people a year. But if you ask them to onboard 4,000, they would struggle. At the large, large, more than 10,000-person company level, nobody cares about the individual humans inside of a team anyway. And actually you just manage by numbers. What are your 3 top goals per team? And are you delivering those 3 top goals to me so that I can plug you into the rest of my org? Inside of that black box, I don't care. I'll hold you to some cost efficiency metric, but hire the humans you want, spend the agents that you want. You actually don't need to know every single person in your company. Actually, you just need to know your team leads and your team leads need to know them. But I'm just going to fire the entire team if the team lead is not performing. It's not very humane. Actually, Amazon was very like this. You just have business units and every business unit has a GM, which is basically a CEO. Yeah, they're plugged into an overall strategy. Um, and I think it works well.

SWE Bench, the score coding agents advertise, tests a model on fixing real bug reports from open-source repositories, one issue at a time. A whole product is never the unit of measurement. What is missing when a model can clone the software you pay for?

I would say that there's a lot of UX issues left. If you look at Sui Bench and you're like, oh wow, we're like 90 on Sui Bench, you're looking at the wrong thing, my guy. If you haven't actually tried to completely vibecode a SaaS that you use, you don't understand how models are still very bad at all this.

It sounds as though you had to set up a SaaS in order to evaluate your Kill My SaaS submission.

But yeah, then we just completely evaluate against a Side-by-side playthrough of each of those things. So the beauty of like computer use being good now is that you can basically just do a point and click and screen capture and then compare, compare, compare. We have a 200-page document that documents every flow and then just make sure that each, each of that is represented in your, in your clone. We ask them to also put in some human taste into like, well, what would you change if like you could do things differently? So you are allowed to go off script and like, why would you not?

Over the weekend, We've had a series of math discoveries. I think one which perhaps is worth maybe discussing is a threesome. If you take 3 numbers and you sum them up, can you get to a 0 or not? And this is basically a search operation at the end of the day. And so 2 minus 10 and plus 8 equals 0 is a threesome solution. Can you get the number of operations that you do below the n squared number. For 20, 30 years now, it's been no. And these guys that published yesterday or today, they managed to get it to n equals 1.99992 by finding a combination of numbers in the number space that, you know, you could get the particular, like, search time to under n squared. And this is pretty significant because it breaks the barrier. And for the moment, no real impact because these are extraordinarily large numbers that these guys found. So it's not a big deal. An Anthropic employee gave a cryptography problem to Claude and asked Claude to solve it. And that cryptography problem relied on this N squared being true, that the limit of how quickly you could do something was n squared. And Claude, while doing this cryptography problem, which relied on this n squared being true, broke it and came back and said, well, you asked me to solve the cryptography problem, and I kind of just solved it by undermining the entire basis for that set of problems. The other interesting thing about this problem is that after Claude solved it, Anthropic approached 2 mathematicians who were leaders in this field and asked them to publish, paid them to publish, gave them the solution, and then they kind of reformulated in a readable manner, and then they published under their name. So they are listed as the first and second author. Claude is listed, and the fact that Anthropic gave them the solution is listed on page 63. And this is being claimed as the way that things are going to be in the future, that we would have models solve these things and we'd hand the solutions off to mathematicians to kind of verify. And the mathematicians who verify or explain it would end up taking the credit. There's still a lot of unease in the math community right now. On what is actually the right thing to do.

The guidance I got from AIs was like, it's not likely to be immediately super consequential, but that barrier-busting effect can at times be super important. You know, the classic 4-minute mile, right? Once it's demonstrated that somebody can do it, all of a sudden lots of people start doing it. So it will be really interesting to see if all of a sudden there's like a wave of kind of follow-on optimizations that bring the number down substantially further. This also seems to be going on right now in the NanoGPT speedrun department. Just headline-wise, it seems like we've gone from something like 70 seconds on this classic benchmark. And basically the idea is to train a small model to get to a certain loss as fast as you can in terms of wall clock time. This result here took almost half the time off, which was more than I think the last 45 improvements combined. And this one has been accepted. One thing I did think was quite interesting about this particular one that has been accepted and kind of seems to have kicked off this wave of significant drops in the time was that the author here said, AI didn't play a big part in it. This company is doing also large-scale training. So they did also make a point in their publication about this, that the optimiser that accounted for a significant, but I think not majority of the time savings that they did share is actually not as good as the one that they are using internally. So they have an even better Optimizer improvement in-house, but they're keeping that to themselves.

And this company is superstitious. I think that NanoGPT's record was a form of performance marketing intended to show the capabilities of their optimizer and their team in general.

Here's a question for you. This question actually I heard from Ajeya Katra in May. So in May she asked, me and others in an audience, how many copies of yourself would you need to have to do as much stuff as you are doing today with AI assistance? At that time, I said 2x, 2 clones or twice the speed. And this weekend, thinking about it again, I was like, I think it might be more like 4 or 5 now. Software in particular, It's like way higher. So what I did last night with like 12 prompts would be easily a week's worth of work. But even subtracting out the software development and just imagining sort of the content, the sensemaking, writing, whatever, giving talks, I still think I'm probably right now in the kind of 4 to 5 range. I haven't measured it, but what's your intuition for you?

I don't know, like maybe 10. So the way I look at it is that a lot of what we build for ourselves tends to be a couple of different things that we tend to build. We tend to build sensory organs. For example, Swyx created AI News. In AI News, he set up pipelines of unstructured data from Discord channels on AI, from Archive, from all of these different places. And they kind of got filtered and ranked, et cetera, et cetera, until he came to be able to send out these emails, which were very compressed kind of sensing. And those became kind of sensory organs for the entire community to kind of know what's going on in the community. If you were the kind of person to wake up in the morning and look through Y Combinator, Hacker News, and 4 or 5 or 6 different pages to kind of keep up on AI. That used to take 45 minutes of your time in the morning, and all of a sudden that's compressed by AI news to 5 minutes. That's a permanent compression. So when you say, let's take away the AI agents, you also have to say, let's take away all of these affordances that we've built for ourselves over the last And once you start taking away those affordances, then you need to replace that with like an actual person, kind of. So I think the numbers are actually much higher than we think. It's not just the number of tokens that you spend on a daily basis, but the fact that you've spent those tokens and that becomes kind of a CapEx and you have this capital asset that you've built for yourself.

One other big takeaway from the weekend at The Curve in terms of expectations, and this was definitely shared Again, across your frontier company. AI for science is very much where they think we are headed in the not distant future. One sentiment was, are AIs going to be superhuman at just some things like the verifiable tasks, or are they going to be superhuman at everything? The sort of middle position, which I think is very credible, is we are seeing superhuman performance at anything we care about. And are really committed to investing in. So that doesn't mean it's full generalisation to every domain, but even in domains that are thought of as not super inherently verifiable, they feel like when they put their focus on it and they go license what data they may need to license, and they, of course, Apply lots of processing to that data to augment and create synthetic versions and what have you, and put some RL on it too. The whole package basically is, I think, broadly understood to work on essentially any problem that they really choose to focus on. And so AI for Science is coming next, and the expectation is we will see One person went as far as to say, in no more than a year, it will not be possible to do the very best science without AIs playing a big role in it. And we'll see.

30% to 50% of all science papers are probably non-replicable and are probably trash. That's what we're training the AIs on.

There's no doubt that you're right, that there's a lot of junk science out there. And if these AIs are going to get that good at science, which I think seems very likely, there's no doubt that they're going to be able to separate the good from the bad with presumably imperfect reliability. I would bet it will be accurate enough that people will just kind of generally believe, at a minimum, we are going to need to be graceful about that process, because who knows why all these different junk things happen. So it's not fraud. There's all kinds of ways people can go wrong in the scientific process that's not fraud. Watch out for mega retraction would be my expectation. Can't be that much more than a year.

There is going to be an enormous kind of social pushback in the research community on this kind of thing because it's going to affect a lot of careers and livelihoods, I think.

One other thing that I heard at The Curve this weekend was on the topic of medicine in particular, but this is sort of a not confidential thing, but just somebody who works on the AI for medical advice domain said doctors have been surprisingly warm and receptive to AI, much less threatened by it than you might have thought. Much more inclined to embrace it. Even on the level of like the AMA, which I had long assumed would come in and try to shut things down that, you know, were starting to encroach on its turf. The point of view that I got from somebody who, you know, has been doing some of that work with those kinds of organizations was like, not really. I thought that was really Exciting. And I would hope to see the same thing from scientists. It's a little more personal for the scientists. One thing, the AIs can come in and just be super useful in general in medicine. And now it's like, okay, this is a whole new reality, but they didn't attack you specifically as a doctor who did wrong. The mega retraction watch is going to be something that's going to touch a lot of individual names, and that is inherently going to be just a lot more contentious.

I wonder to what extent it's because the expectations that people have for doctors to be current on all of this new medical technology is so high that they've been feeling like just overwhelmed for the last few years, post-Google basically, and just not able to keep up with the questions that are being asked by patients who go and Google. And now they have something that works for them. Right. That, that, that works to keep them up to speed, uh, very quickly and accurately too.

Thomas Somers is CTO and co-founder of Positron AI, where engineers are taping out custom silicon with agents working the tools. Swyx, Sean Wang, curates the AI Engineer Conference. His work is at ai.engineer. That's the episode.