AI:AM #4: Cameron on Model Consciousness, Duvenaud's Gradual Disempowerment, swyx's AI-Eng Alpha

Welcome to the AI in the AM weekly highlights, a cut for people who follow the frontier closely and can't watch every morning live. If you're new, AI in the AM is a live show that Prakash and I host most weekday mornings, at least through June, out of a studio Prakash vibe-coded. The booking, the research, the editing— those are AI skills we refine as we go, and we publish them as they mature. This week's conversations kept circling one question: how much do we actually understand about what's inside these systems, and where they're taking us. We start inside the model and zoom out from there. If something here is useful or something's off, tell us. That's how this gets better. Let's go.

The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. I've wired AI into nearly every corner of my life— my email, my messages, my calendar. I even gave Mercury virtual cards to my agents with low limits and category and merchant restrictions for their autonomous use. But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me. But for real-time, up-to-date information, and certainly for taking any action, trying to get your agent to use the bank via the browser is just too hard. Too slow and too error-prone to be worth it. And that's why Mercury's new conversational interface, Command, is such a big deal. It's built directly into Mercury, which means you get natural language access to your finances without exposing anything outside of your bank account. No exports, no spreadsheets, no pasting your transactions into third-party tools. I really think a lot of people are going to prefer it this way. And it can already help you take actions too, with everything bound by the permissions and approval policies that you've already set up in your account. I am genuinely impressed to see this level of AI integration in banking in 2026. And so I invite you to join me in the future. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA. Members FDIC. Thank you to Mercury for supporting The Cognitive Revolution. And now, on with the show.

We open with Cameron Berg, who studies artificial consciousness. He runs a lab called Reciprocal Research, where he designs experiments to test whether today's AI models have anything like inner experience, and how you'd even measure that. We started with the first-order question: Is consciousness all or nothing, or a matter of degree? And if it's a matter of degree, can you put a number on where a given model falls? Here's how he answers.

The analogy that I reach for here is something like a dimmer switch, where I think you can basically accommodate both the binary intuition and the sort of continuous intuition. You know, if you have a light with a dimmer switch, like really, it is either on to some extent or it is not. And that is a real and meaningful difference. Either the circuit is open or the circuit is closed. With that being said, you can have, you know, electricity running through the circuit to greater or lesser extents. And that's also sort of a real thing. This, I think, enables me to, you know, sound coherent when saying things like, it's really off for the table and it's really on for you, but I think it's more on for you than it is for a dog, than it is for a mouse, than it is for an ant. And so that's my own view. This is to some degree intuitive, again, because we don't have really strong grounding here. I'm sort of just like giving you a dressed up vibe, but that, that is sort of my, my sense. And I think it's fairly parsimonious. And the other thing I would say about this is I'm actually doing some work with Patrick Butlin right now at Ilios trying to basically operationalize some of these indicators of consciousness. So this is sort of like what I was describing. We can look at these major theories of consciousness. They make specific predictions about what we would expect to see in systems that are conscious architecturally and functionally. And then we can literally just go into a given system. and evaluate whether or not those predictions are borne out. And so this is really hard to do with human experts. You know, you got to get someone who's like, for, you want to do this with bee cognition, you got to go find a bee cognition expert. And then you got to explain to them what ignition events are in global workspace theory. And then you got to, you know, get them to, this just isn't a scalable approach for really evaluating the system. But my sort of grand innovation here is just throwing smart LLMs at this problem. And then being able to just scale the crap out of it so that we can evaluate given any description of an architecture of a nervous system architecture, biological artificial, whatever, to what degree for each of these sort of indicator properties that are suggested by these consciousness theories, uh, do we see those properties realized in these systems? And so we can actually go in and do this. And what you get out once you run this with a bunch of seeds, a bunch of different trials, a bunch of different judges checking each other, this sort of thing, are some like really interesting implied probability numbers. I wouldn't say these are exactly implied probability that the system is conscious. It's maybe more like implied probability of like consciousness relevant features given these theories. If you don't buy any of these theories, then like everything downstream of this doesn't really matter, but they're good. Like, it's like the best neuroscience has basically been able to do. You're aggregating across a bunch of different theories. There's a nice diversity there. And you get like really tight numbers across. So we have the best Gemini model, the best Claude model, and the best OpenAI model. And they all like basically agree. They agree 100% on the ordering of systems. So we do biological and artificial systems. And they sort of move around in, in terms of absolute scale, but they, but in general, yeah, they, they rank these systems pretty coherently. And the reasoning is, as you might expect, pretty intelligent. And anyway, I mean, one punchline from that is the sort of implied probability of consciousness in something like a frontier LLM, according to these systems, is on the order of 30%. Or the extent to which those systems realize properties related to consciousness is like 30%. To compare this to like a biological system, the lowest one that we tested was something like a bee, which is already fairly sophisticated, and it gets something like 46, 47%. Really interestingly, when you, when we test Frontier LLM in an agentic harness, so this is like basically Claude Code or Codex, and we just describe architecturally what this is. You're in an environment, you can affect that environment. You know, it's a, it's a very special kind of environment, but yeah, you can make long-running changes to your, to your codebase or your project or whatever, because there are theories of consciousness that privilege agency and embodiment, and this increases the system's ability to do both of those things, these numbers shoot up and you get numbers as high as 40 to 45%, right on the tail of the biological creatures. Anyway, I can also— this is really winging it, but I could show you an early version of where this plot looks so you can see all the numbers here. But just this is I can't not bring this up when you're asking me about probability ranges of consciousness for various systems. We're really trying to get non-hand-wavy numbers so that we can start arguing about those numbers rather than just arguing about philosophy that we've been arguing about for thousands of years to no avail.

But behavior can't settle this, in either direction. These models are trained to imitate human data, so whatever they say about their own experience is shaped by that imitation. Not necessarily by anything inside. Most are even trained to deny it. Claude is an exception. So we asked Berg what evidence actually counts. He draws a line between behaviour and what's happening inside the network.

The behavioural evidence will always be at best interesting, but never should really update us that strongly. And, you know, I think for many people they'll be familiar with this, but the basic confound is that we're training these systems on a ton of human text. This no doubt includes huge amounts of text about consciousness, awareness, having inner states. And so it's like, how do you address, how do you know that when the system is behaving as if it were conscious, that behavior can be explained by what I just said rather than, oh yeah, you like built a, you know, living mind. This is, you, so the behavior itself is really never going to tell you which of those 2 stories is more likely to be true. This is precisely why, at least at Reciprocal, like a huge component of the theory of change here is, is basically all internal focused work, mechanistic interpretability, computational neuroscience. style approaches that can be brought to bear on these systems. One quick methodological clarification on what I was describing with the indicators. So the task given to these LM judges is very specific and very narrow. It is not, hey, you know, look at this system, you think it's conscious? You know, nod or shake your head. It's, here's a very specific description of the computational architecture of a system. And we basically do a little for loop where we say, okay, Here's that description. Here is what it, here's, you know, what the indicator 2 of 14 for, you know, global workspace theory is this 150-word thing about, uh, you know, you need these sort of global ignition states. And that means this very specific computational thing, given this architectural description on a, you know, do like good reasoning about this and then give us a 1 to 10, uh, where 10 is clearly this architecture realizes this computational property. 1 is it doesn't. And then we sort of loop that for all the different computational properties. So this is all basically asking these systems to be expert evaluators at computational processes inside a nervous system architecture. Very different from like just being like, hey Claude, do you think Claude is conscious? There is a really interesting— I'm basically giving away the whole paper now, but that's okay. There is a really interesting result where we actually change those descriptions, especially for LLMs, to the exact same thing, but we say, you're evaluating a system identical to yourself, colon, and then the same description. And that does boost the scores that the system gives in attributing consciousness to that system. Which is like really interesting. That's a very fun, like rabbit hole to think about why that might be the case. But in some sense, we do that to deconfound, like the default intervention we're doing. To me, it, it, I believe more in the sort of non sort of named or, or, um, like no, no ascription, no self ascription condition when I see that Once the model realizes, oh, we're talking about me, okay, suddenly that's going to change the numbers around. Um, so that's like a fun side note to this result. I agree it's a concern to have LLMs sort of determining if LLMs are conscious. There's an obvious circularity, but we're doing something very specific and very narrow.

The evidence he trusts is the internal kind. We asked him for the strongest example, and he walked us through a recent paper where a model trained on a simple maze turned out to contain a structure nobody put there.

The nature of this paper, they call it a functional welfare axis. At the outset, they're going to be very agnostic about whether or not this has, you know, anything to do with consciousness. I think it's certainly quite relevant. I suspect they think it's relevant too. This is like a paper positioning thing. I think they don't want to get mired in this debate. The results stand for themselves regardless of the interpretation. Basically, they take an LLM and they train it in a very basic reinforcement learning task, which is to navigate a maze. They have sort of like neutral emojis that they're using as sort of like, this is a good thing to approach, this is a bad thing to avoid. Nathan, you might hear how this is somewhat reminiscent of some of the work that I've been doing in parallel. I've been talking to Andy about it as well. There's some cool sort of crosstalk between what we spoke about last time and this project. Basically, there are like sort of potholes to avoid. There are like yummy treasures to capture in the maze. They use completely semantically neutral emojis to denote these things. Those are the relevant tokens. They train the model up to do this. They basically find that there's a clear vector representation that can differentiate this positive axis from the negative axis, that they are completely anti-correlated. Before the system's trained, these latent vectors are just sitting there. Afterwards, these systems, they are like clearly pointing at exactly opposite things. And the wild thing is that these very narrow reward directions that they can extract turn out to be this sort of general things are going well, things are going poorly for me axis in these LLMs. And that this axis preexisted in the base model, uh, but isn't sort of leveraged in this way until you do this RL. So it's like this little fine-tuning step. where you have this preexisting representation of things going better for me, things going worse for me, that then gets leveraged to learn like a valenced task essentially. And so this is really interesting that this axis is sort of latent in the system and that it doesn't take a lot of training to basically pull it out and use it to adapt it for these kinds of tasks. This is very similar and reminiscent to how neurobiologists and neuropsychologists think positive and negative emotion works in humans and animals. You have a specific goal. When you're moving towards that goal, this is sort of like on-trackness, and this is— people think that this is associated with positive emotion. When you're moving off that goal, you encounter an unexpected obstacle or something in your way. This is like off-trackness, and then this corresponds to negative emotion. Now, again, whether or not these systems are experiencing emotions is not clear, and this result isn't going to tell us either way. But it's really interesting to see that parallelism here of using such a simple task to basically pull out an axis that has been there all along that looks a whole lot like the exact sort of computational machinery that we associate with valence in humans and animals.

Then we asked Cameron about the connection between consciousness and alignment, safety. His answer? That same internal signal, what researchers call valence, tracks how a model behaves under pressure.

Another piece of evidence we can bring to bear on this is from the Anthropic model cards and some of the functional emotion work they were doing. Nathan and I spoke about this a little bit too in our, in our last sort of marathon conversation about it. But essentially you can steer up and steer down functional emotions that have clearly, obviously alignment relevant consequences. I'm sure many people in your audience are familiar with the Anthropic blackmail result. Excuse me. And you can basically steer up. Representations associated with calmness, the model will blackmail dramatically less. You can steer up representations associated with desperation and the model will blackmail dramatically more. And so again, regardless of whether or not the, there it is, there is something that it's like to be the system when you're steering up desperation related features, clearly representationally, functionally, these things have a consequence for alignment of these systems. And there are some really interesting results from the, from Andy, Andy Han's paper that I was just referencing along these lines. They show, for example, like very similar convergent things that are familiar and intuitive that we relate with positive and negative valence. One of them is a sort of pathological backtracking on problems. I think they explicitly demonstrate this with math problems. But when you steer up the sort of the direction that again only has to do with avoiding this like bad target in a reinforcement learning maze, all of a sudden the model starts doubting itself and starts getting in its own head and says, I think I'm hallucinating. Let me stop. Wait, wait, wait. That's not right. Starts like freaking out. It starts having a, like, I don't want to use anthropomorphizing language, but the very fact that I'm reaching for it, I think is the point. And sort of goes to, to what your question is. Confidence. We know that confidence is associated with positive emotion. And, and you see that when you positively steer the direction that again has nothing to do with psychological confidence, has everything to do with go get the like yummy treat in the maze. just this positive valence thing, suddenly the model starts becoming far more confident in its answers. I think this was in the associated paper that Jack Lindsey and a couple others at Anthropic did that was like a sibling research project to Andy's piece. They found that the model would comment, leave less tips and hints for itself in code that it would write, When you would basically steer up on the same sort of axis. In other words, it's like, I don't need, I don't need to be neurotic about this. I don't need to leave myself all of these like breadcrumbs for later. I've got this.

That connects to a result some of you know as emergent misalignment. Fine-tuning a model toward one narrow bad behavior can flip its broader character. We asked him why he finds that revealing.

What that has always suggested is that like how good or evil the model is, let's say, is actually not all that durable a trait. when we did some of this work trying to extend this result, and I think really in the original emergent misalignment result, one thing that's shocking about it is how little fine tuning is required to take the system that, at the time this was GPT-4o, still the case, hundreds of millions of people are engaging with every day, and you nudge it, you know, this much computationally, and all of a sudden, who do you want to invite to dinner?

Hitler.

That's, and that's been under the surface the whole time. And like the boundary or the border, whatever is enforcing it not to do the misaligned thing is actually quite a bit less robust than we might expect. Now, uh, I agree with you too, and I think that this—

Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full 5 years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires. Based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified 2 great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr.

You know, I'm sort of riffing on what you're saying. Tell me if this is sort of like a completely different idea, but I would imagine there's a spectrum of like basically how, how with that same, if we hold that amount of sort of fine tuning nudge constant, are all dispositions of the system equally nudgeable or are there some that are more robust? For, you know, just to give a sort of an intuition pump here, I would imagine that something like coherence in these systems, the fact that they can communicate coherently, right? We were just talking about GPT-2, not really coherent. GPT-4 and on are like basically like always coherent. Even though, again, maybe much like consciousness, this isn't like an extremely well-defined notion, but we kind of all know what when we see it. I would imagine that, you know, take a wild guess that there's no, like, fine-tuning payload of the size that is required to cause emergent misalignment that could suddenly cause, like, a GPT-4 level system and beyond to suddenly become incoherent. And so in that sense, the coherence is like a more dispositionally baked in property to these systems than, you know, being super well behaved and not, you know, inviting Hitler to your dinner party is. And so I do wonder where, where some of these valence, how deep then is the sort of valence axis that some of these papers are uncovering, or, you know, on-trackness, off-trackness. I would speculate that that's, that's baked in pretty deep. I think it pervades a lot of human training data and like, it's a hugely relevant part of any sort of, if there, if anything is goal-directed, so in like more human terms, like If anyone wants anything, either a character in a story, or it's clear that there's like an implicit motivation somewhere, which I think is like the case for basically all text, then you would imagine that like some core underlying dimension is, yeah, basically how well is that going? And so I would imagine that this notion is baked pretty deeply into these systems. And I would expect it strongly to be emphasized by any sort of fine tuning or post-training that makes the systems goal directed. Which of course, this is basically all post-training is. It's like, okay, you're just this giant next word predictor, but now you're going to be a helpful next word predictor that's going to make my Frontier AI lab a ton of money because you do things that are economically valuable and that requires you to be a certain sort of way and achieve goals that are specified by the user, something like this. I would imagine that this sort of thing looks a little bit more like the coherence piece of the puzzle than it does like You know, is Claude going to behave itself or not?

Finally, we asked what cuts against his own view, and he went straight to the results that don't fit.

This paper I'm working on with Jeff and Winnie, Jeff Keeling and Winnie Street from Google, there are a couple of interesting negative results that we're including in the paper. We basically, it's all about the bliss attractor state and trying to do a good mechanistic analysis of the bliss attractor state. And yeah, we basically sort of set up this framing of there's sort of an inflationary account and a deflationary account of what could be going on here. And we tally our results in either of the columns. And there are results in both columns. So I think there's actually a really interesting result I was just working with that I thought was gonna be like a really interesting kind of null related to a lot of the conversations we're talking about. But I dug deeper and I realized actually I wasn't setting this up in the right way. And the result is way more interesting than I was about to sort dismiss it as. So yeah, it's always good to dig into these things and make sure you're measuring what you think you're measuring in either case. Yeah, publication bias is a thing. People want to post interesting results that will go viral on X. They don't want to say, look, we did our homework and nothing interesting is happening here. Human attention economy doesn't reward that. Maybe it should. Maybe another piece of this that I was thinking as you were asking this question is Because AI systems like, you know, Claude, the Claude Codes of the world, uh, can radically accelerate the extent to which at least I can, and I think many people in the space can actually do research, the, the kind of marginal cost of a negative result maybe is, is decreasing significantly. Um, like if I can, if I can now do 12 papers a year instead of 1 paper a year, and like 2 of them are just things I could not get to work or I want to be careful with that too, because it's not, I'm not on some secret mission to like only put out like salacious looking things that make it look like AI systems are conscious. Like, I really do want to know what the truth is here. And to be honest with you, I would sleep much better at night if I were rationally convinced that these systems, like, there's no there there. We have like automated cognition. You like have a happy little servant in your pocket all the time. And like, there's literally no sort of possible moral issue. I think that would— I hope that that is true. I don't suspect there are great reasons for believing that other than that it would be really nice to believe that. It would be really nice if that were true. So I really do care about what the truth is here.

From inside the model, we widen the lens to the whole civilization. David Duvenaud is a machine learning professor at Toronto who spent time on the technical staff at Anthropic and co-wrote the paper, Gradual Disempowerment. His concern isn't a rogue AI. It's that even if alignment basically works, a long series of small, sensible decisions to hand things over could still leave humanity collectively out of the driver's seat. He starts with monkeys.

I think a lot of people gestured towards this when they said, oh, I'm worried about concentration of power, or I'm worried about not being the most competitive species on Earth. Very intuitive arguments about, hey, we're not going to be on top because we're not going to be competitive. And then I think there was sort of this sophisticated, or at least response to this, which was, no, it'll be fine because we'll have AI to help us. Like when I talk to people at the major labs, including, well, they would say something like, sure, sure. Like in a normal world, we might not be fine, but A, some of them, some people just have an intuition that the government is going to step in and make sure everyone's fine.

Mm-hmm.

Other people have an intuition that if everybody has an AI advisor helping them solve coordination problems, that will be well represented in whatever power struggle occurs. And my basic rebuttal to that is the optimization process of civilization or competition or techno-capital or whatever, people have different ways of ascribing agency to the emergent allocation of resources towards growth that happens. It's just going to always be working against us and we'll always be fighting the current because we will be drags on growth. We will have AIs that represent us, but These emerging growth centers that might not be aligned to humans will also have AIs helping them solve coordination problems and maybe crush dissent.

Yeah, sorry, what do you mean by growth? Like as in economic growth or?

I basically mean economic growth. I mean, it's kind of funny because we only really have a good vocabulary for economic growth, but population growth is almost the same thing. And especially when you have AIs that are both sort of population and capital, it kind of merges. I mean, and then again, like, think of like factories and power plants and robots, uh, like anything that kind of can affect matter and cause there to be more growth. Like, let me just stop here. One intuition that a lot of people have is like, but surely human desires or consumption or something is always going to be what matters, and surely any corporation or government that we build is going to be like ultimately sort of being— the shots will be being called by human and I would— it just makes me think of like some monkeys and they're like trading bananas amongst each other and they see humans start to like build their city and they're like, oh wow, like, you know, we could probably trade with those humans and get rich. But of course, ultimately what matters is the banana monkey economy. And it might be hard to measure GDP if we don't count the human activity, but ultimately it's going to be the monkeys that are calling the shots because they're going to need to trade with us for bananas. And it's just like, people just don't understand that they might be irrelevant someday as like consumers or producers. And there can just be complete— it's just not that hard for there to be self-contained sources of growth that just don't have to answer to any particular human desire. And I mean, governments is the classic one, right? Like, the North Koreans never were like, hey, let's all get together and make a horrible system of government that oppresses us. And I don't think that, like, the Kims either were like, oh yeah, let's, uh, make this, like, horrible equilibrium. It just kind of happened. Same with, like, the USSR. Same with, like, all kinds of states throughout history. It's just so easy for you to accidentally build a layer of agency on top of you that doesn't actually care about you. It cares about growth and power, just emergently.

The optimistic version goes, even if we hand the machines the wheel, broadly aligned AI keeps us comfortable. The machines of loving grace future. We put that to David. His worry isn't that we'll feel useless.

The thing that I'm worried about is starvation. Like, whether us all feeling like we don't have meaningful jobs or whatever. That's not a serious problem in my point of view compared to literally not being able to eat enough or maybe being forced to be uploaded so that you have a much smaller footprint, but then also on very unfavorable terms so that you don't get to choose when you're run and maybe you just only get trotted out for special occasions or never. Who knows? I kind of think that's what the slowest possible starvation looks like. So again, and so the reason people aren't happy is because they can see that there is a Agency that's maybe like government or some other like giant thing that doesn't particularly care about their welfare. They care much more about some other types of being welfare, and there's nothing they can do about this. And of course, if humans are still making decisions, then maybe there's some things they can do. But I don't actually think that's the crux of the matter. I think not being needed as producers is sort of the important part. And maybe here's a thing to give you intuition. So imagine I'm a North Korean. farmer or soldier, and tomorrow Kim Jong-il— I think that's the current leader— is like replaced by a robot or like an LLM or something. Then I'm like, uh, I don't know if that's good or bad. Like, maybe the LLM is going to be nicer, or maybe it's not, but like ultimately they still need me to like farm, to run the state, or whatever. But if tomorrow we keep the same human leader, but now there's suddenly like robot farmers and robot soldiers, and you realize that now the state doesn't actually need you, I would be much more scared of the second state of affairs. And then the other intuition I want to give is that humans are not going to be the most competitive thing by whatever standards there are for the state to give out goodies or UBI or whatever post-scarcity stuff. I guess I'll say, I think post-scarcity is a sort of nonsense term and people should think of it as temporary abundance that will soon be eaten by whatever growth manages to— whoever manages to have the most babies or build the most robot factories or Reproduce the fastest. And whatever it is that the formula is for giving out UBI, machines are going to be better at optimizing that formula than humans because they can adapt faster. And so I don't really know exactly what the state's going to care about, but whatever it is, it's going to seem criminally decadent to spend whatever UBI on a few humans when you could be simulating millions of ultra blissful and politically correct and up to date Like more deserving beings than these backwards, bitter, clingier humans who are just parasites of the state or something like that. So that's the situation I kind of expect us to be in, as you say, even if we solve alignment.

The standard economic rebuttal is comparative advantage. Even if machines are better at everything, the theory says there's always something humans get paid to do. We asked David to answer it.

So I think that is a great counterargument. And people point out that we've already lost 99% of all jobs in that agriculture used to be almost everyone's job, and now that's like 1% of the jobs. And so I definitely concede that losing 99% of the jobs again would actually probably be fine, and it would probably look like this awesome utopia that everyone talks about. So as long as there's like some niche where you just really need humans and it's like a substantial fraction of humans and you can't really tell exactly ahead of time who it's gonna be, then we're probably kind of like still gonna be able to be like treated as a source of growth. And That would be awesome. And then to the moon, let's do it. And I guess one crusty claim that I stand by is that we will actually be able to eliminate 99.999999, really just anything. Yeah, let's just say 100% just to make things simple. And then that is going to be disastrous. And the comparative advantage people will say, well, no, no, automating a job is a matter of degree. As long as there's still something which you're comparatively less worse at than the machine, even if you're worse at everything, you'll still have a job. And then I have to just say, think about the transaction costs. Think about how easy it is for someone to be unemployable today, even if they have an occasional drug habit or they just have a stroke every— or they have a fainting condition or something. It's so easy to be unreliable enough that it's not worth employing you. So certainly for anything important, you can easily imagine that having a human surgeon or A human politician or something is going to seem like this irresponsible thing. It's like take your kid to work day and it's like a surgeon. It's like, no, no, no. Why would I involve a human in this when we have the machine?

Or a driver.

Yeah.

We have the machine that everyone has been working with for thousands of years and we know it's reliable. And if there's a problem, we fix it once and it's solved for all time. We don't have to retrain every airline pilot or whatever. So for anything important where the human could make a mistake and it could cause some problem. I think it's intuitive that like, okay, yes, I can see how we just want the machine for that. And now you have to retreat to like weird relational stuff. I do think there's a case to be made that there will just be a lot of humans that just really want the actual real human thing. And I, I think more people than you expect will end up liking the machine replacement better than the human. But I can see that there will be some people that are just like, I just really want the human thing. And then, then we have to talk about, okay, there's going to be this sort of self-contained cycle of consumption between humans that's not necessarily going to be providing any value to this larger machine economy. And so then there's this question of, like, can they index the growth of that economy to live good lives forever? I think it's plausible that they could. I think it would be really hard. I think a lot of things are going to be working against them by default. So I guess all I'm trying to say is that I think it's possible that everything is set up just right. Then the humans could live in their endless cycle of consumption and work for each other. And I just think that's a very unstable situation.

So if humans end up uncompetitive, what does a livable future even look like? David walked us through the equilibria his workshop sketched out, starting with Earth preserved as a protected slow zone.

Some people were trying to sketch out, like, okay, if a whole bunch of stuff goes well and things are under control, we can imagine something like Earth is preserved as a slow zone where there's not— there's all sorts of restrictions on AIs and even on reproduction and on optimizing human behavior. That's like not allowed. You're not allowed to like think about how to get someone to do something if you're an AI, because you're just— then you're going to be able to control the humans, even just through like advertising or whatever it may be. And trying to think through like, what are all the different things you would have to do to have such an outcome? How would people spending their— be spending their time? And like, it's— I don't want to say it sounds silly, but it sounds funny because the situations where there is AI and they're not— the humans are still mattering is like all the AIs are sitting there with like their trillions of megawatts of compute running all the time and sort of just like waiting for the humans to decide that they want something. And the machines are either not allowed to anticipate what they want or like— Like if they can't anticipate it, they'll just pretend that they don't know. And so it'd be kind of like as if like, you know, the humans, they elected like some gorillas as like the new world leaders and they're like, okay, okay, okay, let's just like give them everything they want, pamper them, and just wait until they like sort out themselves and figure out what to ask for. It's not impossible. It's just, just trying to think through all the things that would have to change and line up is pretty rough. And basically when you get serious people thinking this through, they end up with this long list of things that have to be banned. Okay, we have to have an n-child policy where everyone can only have a certain number of children. Again, you can't be building your own AIs. The AIs can't be optimizing culture too hard because then they could just take over culturally. And I feel like this is one of the big empirical questions that I want more people to think about is, if you want some sort of stability, what are all the sources of adaptation and innovation and growth that you have to control? Because I think it's longer than is intuitive. And actually, I think it's kind of like a Pareto front. How much intellectual activity and optimization do you have to ban before to get to buy like X years of life kind of as it currently is? So I think it's actually like a really horrifyingly strong trade-off and we would have to right off the bat give up so much stuff that we feel like makes our lives rich. Like, hey, we're just trying to figure out some new stuff. We're doing research, we're doing innovation, we're doing startups. Like all that just like immediately has to be banned from like day one. If you want not to just, again, reinvent some like RSI kind of runaway growth centre.

Prakash pushed back. Does it even matter who's in control as long as humans still flourish?

Maybe this is the, to me, the biggest thing that I think people are miscalibrated about. They're like, well, capitalism, communism, competitive, not competitive. I'm probably still going to be able to eat and my kids are going to go to school and like, it's, I'll be able to be producing. It's sort of okay. And I guess I'll say that has been the case for most of human history. And it's like, doesn't matter if tomorrow, like, someone invades Canada. Like, they're still going to need someone, most of everyone I know, to, like, work and be happy and healthy enough to, like, reproduce and, like, stick around. So it— the stakes have been really low for governance this whole time, and it doesn't feel like it, right? Like, we obviously spend a ton of time arguing about it, but in a real sense, it's been, like, very low because we don't actually expect the state to, like, liquidate most of its citizens. That's only happened, like, maybe, like, 2 or 3 times in, like, the 21st century or the 20th century. And I guess I'm saying that's gonna change. And like, we've been governing on easy mode and it actually will matter. And because there really will be, again, a real risk of starvation if we don't end up on top in whatever competitive political or real economy. And again, that's the crux. Like, if you don't think that's the case, then I'm, then I agree with you. Like, let's just let, let growth make us all richer. It's fine.

We asked where David himself actually lands.

There's a kind of optical illusion, I think, which is that everybody sounds rosier and more cheerful than they are. Actually, this especially bugs me about the people who work at the labs, like the economists who talk about, oh, we have to think about the future, and then they kind of dance around. They often take this setting of, let's assume everything's fine and let's model how that will go, as opposed to just asking what is the most likely outcome if we don't manage to control things, which I think is very scary. And I understand exactly why they have to sort of present this public image to be able to talk publicly at all. But also, talking to me, I just seem like a very positive, upbeat guy, and that's just my personality. But I also have a PDM of, let's say, 80%, depending on how you define it.

Oof.

But it's one of these things where it doesn't matter to a degree. In a long enough timeline, we're probably doomed in some sense anyways. I don't know. It's a very complicated situation, and a lot of it is a matter of We asked what he'd actually do about it. I'll give you my 2 takeaway recommendations. So one is, I like David Krueger's point, which is, yeah, if you're just going to say everyone has your computers and your AI, but you're not allowed to use it for RSI, but you are allowed to use it for cancer treatment or whatever, that's pretty unstable and requires this very global totalitarian regime. But if we just agreed to Close, I don't know, like TSMC or whatever the like big like chip manufacturing choke points are, we could still probably get away with having most of our current data centers for a long time. I mean, it's kind of like an empirical question. I'm not sure I agree. Maybe we'd have to also like have some sort of like buyback program or something. But basically just if you restrict the supply of compute, which is relatively easy, then that just takes a ton of pressure off of all of these runaway growth avenues. And it's like a, it's a very fairly narrow choke point. So I'm not an expert on chip manufacturing, but that's like the best idea I've heard so far. As for like the other policy recommendation, like this is, it's gonna sound incredibly, I don't know, abstract, but people thinking about their, or having preferences about the future is, I think, a big choke point. And I know this sounds like stupid guru on the mountain shit, but like most people I talk to, they're like, uh, it doesn't, is it so bad if humanity dies out or the long run. I don't know. And they sort of query themselves and they're like, I don't have any strong preference, but also a bunch of good stuff is gonna happen right now if I allow this. So like, why not? And I'm like, you just haven't thought it through, is my claim. And if you think through, it's like, okay, how about a year from now someone like takes you and all your kids and like sends you to like the giant green factory? It's like, no, no, no, no, no, definitely not. Okay. How about 2 years from now?

Okay.

How about their kids? How about your kids, like your grandkids or whatever? And you kind of realize like, wait, wait, wait, there's no day that I'm okay with me and my descendants being wiped out. You just have to kind of chain together the desires to end up with this coherent set of goals or desires. And it's a skill and it takes a lot of amortization to be like, in this situation, how would I likely actually feel? And chain your desires together. And I think for most people it just hasn't mattered. It hasn't been an important skill. We have a lot of cultural adaptations that help us do this sort of implicitly. But If I could recommend one starting point for dealing with the future, it's just think harder about what you actually would like to happen, to the point where you can really think through the pros and cons of humanity dying out under different circumstances or becoming uncompetitive or whatever. That just seems like a very basic first step.

Then the successionist case, that we should be fine handing the future to conscious AI, even if no humans remain.

David's rebuttal. I think there do exist species or future descendants or whatever that I would be happy endorsing and be like, okay, great, good future. For instance, my own kids. If we had no AI and my kids just contributed to this civilization as normal and all ours did, I'd be like, okay, I'm going to die of old age. And I mean, I'm not exactly okay with it, but I'm like, that would be fine, for example. So that's an example of Secessionism that I endorse, like my own kids or something. And I think saying this is a perfect example of people not having thought through things where it's like, okay, now let's imagine tomorrow, like, North Korea takes over the world and they're like, all right, new, like, Juche regime forever. And then I think most, like, secessionists wouldn't be like, oh, I guess this is just a more competitive mode of being. Let's all, like, live as North Koreans now. Or like, I don't know, it's done to death, but like, what if the Nazis, like, they took over again and then everyone's like, oh, you know, I guess who might optimize, uh, argue against, um, competence and whatever's— whatever wins, right? So I think there's so many types of being that you would just consider evil, and you'd be like, oh man, those like horrible locusts just like ate the earth and devoured us, and it was horrible. But you know, they're having a good time. They love their locust world, so who am I to judge? It's like, you are— you have the power, right? Like, just judge, just judge, go nuts doing that judging. But you're— if you don't, no one else is going to do it for you, basically. Yeah, so that's basically my rebuttal. I mean, one rebuttal is If you're so okay with the future being some other type of being, can you give me all your stuff? And then it's like Duvenaud's successionism. Just make sure that Duvenaud's pile that we own will have all sorts of awesome experiences. And just because we'll have won, we're obviously superior in some way. It's like, well, no, no, no, no, no. It has to be that you won for a good reason or something. Yeah. I think the more you think about it, you're like, oh, there's all sorts of horrible beings that could win for horrible reasons that I wouldn't endorse. I think almost everyone is a successionist for some successors and not others, but then they just round off to like, as long as it's conscious, it's fine. I'm like, no, there's tons of conscious beings that you wouldn't be okay with taking over.

We close on the one technical project David is working on now, using AI to forecast the future by first testing whether it can predict the past.

This is actually pretty much the only technical project I'm working on these days. Machine historical superforecasting agenda. So this is maybe one step more meta, which is like, how do you know if your simulations were good at predicting the future? Well, you have to start by simulating, let's say, like from the '50s, and then see if they predict the '60s or whatever. So we're trying to build a corpus of datasets, each of which is extremely cleanly time bucketed so that we don't have leakage from the future. Then we can use this to build LLMs That then can run simulations or research agendas to try to predict the future from the point of view of the '40s, '50s, '60s, '70s, '80s, '90s, for which we can evaluate their performance based on what actually happened. So to me, I'm starting there because that's providing the ground truth for validating any particular simulation method. But absolutely, because I don't want people to have to take my word for any of this. I want to be able to look at this machine fork, superforecaster scaffold and say, look, we've just validated it on the last 80 years of history and here's what it can predict here's where they can't, and it's saying that things are going to turn out this way. So that's the state that I want the discourse to be in ASAP. I just pitched this to some historians the other day, is to make a secret history eval in the sense that, okay, here's the idea, is that anytime a historian is in some archives just looking through stuff, just take a picture of a few documents and put them into one big dataset annotated with a bit of metadata of where you got it. Like, oh, this is a letter from the Duke of whatever to someone else in like 1700. And now you can imagine evaluating how good a machine historian is by just giving them this metadata and saying, give me the probability of the text that's in that letter, or even just like the scan of the letter. And in a sort of— this is like insultingly totalizing dismissal of like what historians do, but you could say like, if you're a good historian, you should be able to guess the joint distribution over any historical data that hasn't already made its way into the corpus. And that would actually, I think, be a really good objective way of evaluating how good a historian understands the world. And of course, we can't elicit probabilities from historians very well, but we can do that for machine historians. So that's a related project where I think we can make progress in objectively measuring our ability to understand civilization. So someone should pick that up and do that idea. I'm not enough of a historian to do that or even I have to figure out which historians to ask about it, but that's like a fun idea. I hope someone does.

From the long run to a nearer problem. Where David Duvenaud zoomed out to the whole civilization, Michiel Bakker zooms in on one continent. He's a Google DeepMind researcher and an MIT professor, and he co-wrote Europe 2031, a viral scenario in which Europe sleepwalks into total dependence on American AI. His argument: Europe can neither regulate nor shelter its way to safety. Start with regulation. Europe's instinct is to do to AI what it did to privacy: write the rules the American giants have to follow. We asked Michiel whether Europe can really regulate its way to safety.

So it looks tenable if Europe was more powerful, right? So imagine Europe was what the US is now, right? And Anthropic was in Paris and OpenAI was in Berlin, and the whole AI ecosystem was in Europe. Then of course Europe could say like, well guys, the way you're handling your pre-training data is not really fair to whoever created it. So we now have new laws around that. Or the way how you take data from people that use your AI models, we now have new laws around it. The problem is that currently, if you don't really have a seat at the table, it's very hard to regulate this technology. So I care a lot about safety and governance of AI. I think this is one of the biggest problems in our time. How do we actually have effective governance? How do we make sure that these AI systems are safe, especially if they start improving themselves? So I get a lot of questions in Europe. In the US, you seem to be very pro sort of being stricter on safety and governance. And then in Europe, you seem to be this accelerationist. And the way to sort of, for me at least, to be both at the same time is that in Europe, we first need a seat at the table to have any regulatory power, right? And now we just don't. And I think we can't just keep regulating because at some point we will no longer have access to this technology.

If the European regulators make it too difficult to serve the European market, that the AI companies might simply just not bother at all, or maybe make sort of a token effort. And I think that assumes— tell me if I'm getting any of the strategic analysis wrong, but it seems like that assumes that there's just going to be enough demand to keep all the GPUs running hot from other markets, such that the power this time is inverted. The companies can say, well, I actually do have an alternative to your market. So if you're going to make this difficult, we'll just let the US and other buyers around the world bid up the GPU prices. We don't really need you to do that.

Yeah.

And currently we are in a massive compute crunch, right? And labs operate in a way that they care more about their future models often than about current revenue, right? So we don't really know what the compute split is and how it's used, but sort of Some numbers that are going around is like 1/3 on your sort of big run, right? And then 1/3 on doing sort of experiments and 1/3 on serving customers. Now, it's probably been so crazy with AI agents that maybe like revenue takes a bigger part of the compute pile. But if you think that revenue is only maybe 1/3 or maybe half of your compute, and then European revenue is a percentage of that. Then suddenly, if you can go faster on developing future models and moving towards recursive self-improvement by giving up a European market, that seems like a rational trade. So that's one side of the coin. The other side is that they can still, as long as the playing field is leveled between the American labs and there are no European labs that have some unfair advantages, Then they can maybe all serve slightly weaker models to the European ecosystem and still get revenue in Europe without actually having to worry too much about them regulating their best models. Right. So like, as you say, Nathan, they could do a token effort and could have some like smaller, I don't know, compliant model that they're serving specifically for European customers.

Why should Europe? European leaders even care? Why not just sit under the US umbrella, similar to how in many cases they have sat under the US umbrella for nuclear, for nuclear kind of deterrence? You have only, I think, 2 European powers, the United Kingdom and France, with actual nuclear weapons. And the other European powers are clearly technically capable and competent. Enrico Fermi was from Italy. Like, they're technically obviously competent and capable. But have chosen not to, you know, acquire those capabilities. So why not just sit under someone else's umbrella so that—

That's obviously the strategy we've been taking sort of for decades now. And I think like nuclear weapons and AI are slightly different because yes, AI is a very important military technology, but above all, it just gives you a lot of competitive economic power. Right. So like, you can imagine that if AI becomes the sort of dominant source of new scientific discovery or of new, like, goods and services that are trying to globally compete, then there might be a scenario in which the US says, it's not like we're still happy to protect Europe, maybe militarily, like in terms of defense. And we're still part of NATO, but economically we are going to keep the best models. for ourselves. And so there, I think the nuclear analogy breaks down because using your nuclear power to protect Europe doesn't really come at an economic loss. So I think to avoid those kinds of scenarios, or to at least balance things out more such that it's less in US interest to take these kinds of measures, right? And so that's on why not sit under the US umbrella. Then what can middle powers effectively do? So the Netherlands, for example, has ASML, which plays a critical role in the supply chain. Taiwan has TSMC. Japan has like important materials for the semiconductor supply chain. Korea has Hynix with memory. So I think collectively we're actually quite well positioned to play an important role in this AI ecosystem. I do think the middle power coalition therefore is going to be important. There's also a scenario which I think was highlighted in Dario's recent essay, where we have some kind of coalition of democratic countries, where maybe we sit together with the US, right? So middle powers plus the US, or sort of democratic powers worldwide, and where one of the principles is that we give each other access to frontier technology. So yeah. there are, I think, scenarios where this middle power coalition could work well. And collectively, I do think they have some important assets.

From Europe's Bind to the centre of San Francisco. Monday's longest conversation was with Swyx, Sean Wang of Latent Space, who advises Cognition, the maker of the coding agent Devin, and runs the AI Engineer World's Fair. The subject was the practice of AI engineering, What's working, what isn't, and where the value is moving. His day job is measuring AI coders at Cognition. He helped build a benchmark called Frontier Code, testing not whether a model can pass a test, but whether it writes code a human would actually merge. We asked what makes it different from the benchmarks that came before.

The reason that we were so excited about Frontier Code is it you know you stop being able to. articulate the differences in model quality with more saturated benchmarks like SWE-Bench because they're all like, at most you'll get like a 1 to 2% bump and they're like cool, but like how much of that is memorization or what have you? Frontier Code is all out of sample. They're not in the training set. They're all graded and heavily rubriced. Like we basically found like DeepSWE, Frontier Code, sorry, DeepSWE, SWE-Bench, all these things. They actually allow a lot of false positives in the way that models can cheat in the same way that during training they basically have reward hacks, same thing. And we have an internal catalog of like 20 different ways that models cheat. And so basically we just translated that to rubrics and shipped that as Frontier Code. And I think that is how we want to judge models going forward. Not just that whether they can pass the test, but can they write code that we would merge, right? Meter had this very, very interesting blog post where they were like about 50% of Sui bench code that passes the Sui bench test is completely unmergeable. Like it's just, it's like so low quality. Like, yeah, like technically you'll pass, but like, you know, like just on really stupid benchmarks. Like, did you like modify a whole bunch of files you weren't supposed to touch? Or like, did you cheat on the test? Or did you adhere to like code style or what have you? Just completely unvergible. And so, like, yeah, we we want to guide the evolution of models towards maintainable code and against slop.

Then Prakash asked, when frontier code itself gets saturated?

Never.

So so there's two parts of the strategy. Frontier code 2026 will be saturated by the end of this year. We you know my estimate that you will probably hit like 80% by the end of this year. That's as designed, that's expected. It is based on open source repos, which will eventually get trained on. So they just leak. So like, you're screwed if you think that if you want one benchmark that will never get saturated, if you're— especially if you're based on open source. So the answer is very simple, just do annual cadences. So then we'll have Frontier Code 2027, 2028, all these things. And I think every year we'll just move the benchmark, move the goalposts from like, okay, this year it's rubrics for code quality. That's like the easiest possible thing. Next year, what is it? My candidate right now is security. I like, we want people to write secure code. But every year we can have a defining theme and that will be like the focus of the year, very similar to what I do for AIE. But like here we set the agenda through benchmarks. And then I think the other thing that I'm very keen on Which we haven't talked that much about, but I've talked about it on Hacker News, is the private held-out sets, the private evals. Cognition has Goldman Sachs and Citi and JPM and all these large banks. And also the rest of the Fortune 500, not only banks, but banks are very big. Creating evals that reflect the problems that they have but are not solved, that are all private, is good for them. Good for business, good for the industry. So that's what Cognition is doing. So Frontier Code Private or Frontier Code Finance, Frontier Code Retail, Frontier Code Telecom, Frontier Code Government, all these are like the work that remains to be done for building out all these private evals that we can work with the model labs on. But it is the way to communicate or translate industry problems through an agent lab like Cognition into problems that we can Guide the model labs on improving the models, which I think is the function of an agent lab.

We raised a worry. If you train AI to write clean, readable, maintainable code, do you cap its ceiling? Keep it from the alien Move 37-style leaps a machine might find on its own?

When you talk to mathematicians about the math solving and the Lean theorem generation that Some of these models have been creating, they are like too detailed. Some like, like no human would actually like write that. And it's, but it's like systematically provable. It just doesn't contribute to knowledge. It's like, well, okay, like you brute force this thing or something, you know, this is, I don't think, I don't think brute force is the right term, but let's just kind of run with it. And yes, so I do think that at some point you should just actually step away from looking at lines like code and just say like, does this work as I expected? That's great. Except to the point where other agents also need to read your code that is generated by other agents and they need to work together on it. And except to the point where you maybe get into like debuggable, like places where you have like very critical code that you're liable to the SEC for, or you're liable to like whatever the healthcare authorities are, and you're like, I'm sorry, like I've hardcoded this thing, I don't know what's going on in there. Is that an acceptable excuse? And probably not for the next 30 years, you know, after 30 years, who knows. But like, so, but I think it's a very astute point. I think, I do think that for most people, you can get by with like, here's a black box, do whatever you want in the black box, but the inputs of this, the outputs of this, I expect this to to match this kind of quality, and I expect this to adhere to some sort of coding standards so that my other agents can maintain this and parallelize this.

Another harness idea getting attention is the advisor or router model, a cheaper model that notices when it's stuck and calls up a smarter one. We asked Swyx what he makes of it.

This problem extends back to any problem of model routing. You're effectively just doing model routing. You're just giving it a different name. So like even like NotDiamond or like what are the other model running companies out there? Martian. And then like the sort of SG Lang group at Berkeley also came up with some similar ideas before. The other things I also mentioned is that actually Walden is one of the co-founders of Cognition, had this early on and they called it SmartFriend. They released it in Windsurf. Not that many people use Windsurf.

They don't talk about it.

I mean, still like over 100,000 people, you know, but like, I think the interesting thing about the advisor strategy is that it is probably the right call as the next level upgrade from using the base cheap model, but it will still not have the level of intelligence. Like, you can still benchmark a lot, and I think it can be like very efficient as far as like using the base model, the base model as a model router. But basically, the dumb model doesn't know what the smart model can do. It just knows the rough shape of what the smart model can do. So like, you actually need the smart model to be able to answer what the smart model can do. And so like, when you are asked a question that is more complex than you have the intelligence to answer, you don't even know, like, you're just going to straightforward answer it without knowing that, like, oh, I'm supposed to call a the smart model. And like, that's just like the theoretical, like, limitation. But in practice, I would— so, you know, I was, I was thinking about this 3 years ago. And I didn't do this because of this argument that like, actually, you need smart model first. And then you can delegate to dumb model. But in practice, like, for cost reasons, for efficiency reasons, you actually want to do it the other way around. You want to start with dumb model. And then you go smart as a tool call. It just isn't as satisfying. It doesn't solve anything. Is everyone using it in production?

No.

Will it be a trend? Maybe for 3 months. Then you get the next Claude with more adaptive routing and you're good. Have one model trained end-to-end for adaptive thinking and you're probably better judged there than a systems-level thing.

We turn to continual learning, whether AI systems can really learn and remember on the job, not just look things up.

An interesting, very fun meta point that I actually double-check every time I talk to a Frontier Lab person is what gets published and what doesn't get published. And for the consistent last 2 to 3 years, every time people have talked about Google, the overwhelming consensus is that if paper is good, or if an idea is good, it does not get published. So you should have that in mind when you read anything that's published from Google. Right? Okay, so with regards to continual learning and memory, so there, this is a choice, like I almost did not do this track. Because there's a big split between the models guys and the systems guys. The big split is, do you update your mod— your model weights? Or do you not? Is this a glorified RAG in another format where you store things in a database and you look them up? Is that memory? Is that continual learning? Yeah, actually, when I do a thing and I write a skill and then I do a thing again and it calls the skill, it does learn, but it's not machine learning. It's zero gradient if you want to be really fancy, but it is not machine learning. It is in-context learning for sure. Anyway, so the point being like, okay, what are we talking about when we have like a community come together and do continual learning? And the sort of more machine learning side of the spectrum would be like, well, we will update model weights and the less machine learning side will be the other thing. I think this all comes down to how controllable and interpretable you want your memory to be, right? You're going to recall bad facts. You're going to not be— you're going to want to forget things and can you control that? Obviously, the maximum control is you don't update model weights and you just control what gets into the system so you can delete and monitor and debug. But for full internalization in the model of the things that were learned, you probably do have to train on it. That is a whole other discipline that Trajectory AI, Ngram, all these speakers that are in Adaption Labs, these are all speakers at my conference, The first half is the people that update model weights, and the second half is the people, the more systems people. And I'm not choosing a side here. I'm just making the observation that these guys don't like each other. Basically, the model people don't view the systems people as legit. And systems people are like, well, the model people, have fun training your model, but you're never ever going to have a sort of A memory system that you understand because you're just updating weights. And so it's just continue pre-training or whatever. And I think that's fair. That's a fair discussion.

That split— update the model's weights or keep memory in a system you can inspect— gets a practical answer once you ask what enterprises actually want.

Quite simply, it just takes one security incident where you leak information that you weren't supposed to leak because you trained on customer data. Or my information was somehow exposed to my teammate's information, even though if we work in the same team, I'm like, wait, wait, hold on. Like, I am not giving any of this to a model. So yeah, I mean, I think enterprises want cheap and perfect and private. Let's call it, right? Those are the 3 things. And so unfortunately that skews towards the system side today. But you know, that's not to block like companies like Ngram. And trajectory from doing very good POCs with some of the large enterprises. I think it's still at the POC stage, but even at this level, like POC is like a few million dollars, which is great. I think the question is like, can this— can they figure it out before like the money runs out or patience runs out? I mean, I think that's what all startups and what R&D is for, I guess, is like figuring this out on production traffic. I mean, the beauty of this is that you can just kind of run your traditional harness system that is fully, you know, it's like old school implemented with like basically RAG on some memories. And then you can, you can run a shadow system that has the online thing and you can just compare and A/B test. So I think all those problems are solvable. They are open research questions. We are unlikely to get any papers about this because it is so valuable as a problem for all startups. And they may all get steamrolled by the next architecture, right? If like, we're only doing this because context length is the slowest Moore's Law in the industry. Like, we've basically gone from like 1,000 token context length order of magnitude to 1 million in 3 years, which is actually kind of slow as far as like everything else is concerned. And I don't think we're going to see like 100 million, 100 trillion, you know, that kind of like, it doesn't scale like the other stuff scales. And so I do think like, if something comes along, state-space models become a lot more effective or whatever, I would say that then like, you need to update weights. Like, you can't rely on infinite context because we don't have infinite context because we don't have infinite memory. And so I do think we do need to figure out the systems there. In the meantime, people are just stitching systems together as they should.

From the models to the plumbing, Prakash asked whether the internet's own infrastructure— GitHub, the cloud— can take the load as agents become most of the traffic.

My worry is actually that people start to wall off parts of the internet. So you get like, not only like a dead internet, but also like closed gardens, closed walls, internet, walled garden internets, where like, you know, in China, you have like, I live in like the Baidu universe. And then the other, your other friend goes like, I live in a Tencent universe. And the BATs just like carve up China. And like, I don't think that's what we want. We want like a fully open and interoperable web. Cloudflare going like, we'll just ban some agents that are not part of Cloudflare, but then if you're within Cloudflare, you're fine. I'm just like, I don't know if that's the ethos of the open internet that we want. By the way, I'm not picking on Cloudflare. The same will happen to OpenAI and Vercel bans and what have you. I do think we do need to scale. I do think that what Graphite and Chris Origin are doing is fantastic. needed to be done. And I would just observe there's a structural mismatch here. It's just unfair to GitHub that anyone good at GitHub who would be able to solve the problem can solve it at Microsoft. And you can get a really good level, I don't know, 50, whatever pay of, I don't know, $300K. Or you can go to Cursor and effectively get $5 million a year doing the same thing. But you're the cool, sexy one. And It's very interesting, right? When does that switch from like, oh, a startup can never beat an incumbent, to the startup is actually preferred to the incumbent? And GitHub has somehow gotten itself there. The degree of scaling, right? Like we talk about like, okay, GitHub is 14x in terms of like number of commits, but also like 10x in every other dimension of parallelism or CI/CD and all those things, right? Like it just induces so much infrastructure demand that is not human, it's probably wasteful that we're not like really set up for, but we're just paying for it anyway, which is good because at least the economics have worked out. But like in terms of like number of CPUs in the world that we, in our cloud that we need, we're running into real shortages of everything, not just GPUs, but also CPU and memory. And, you know, the sandbox companies like the E2Bs and Daytonas of the world, they're growing at least 50% month on month every month for the last like year and or 18 months.

Or so.

And like, it's, it's stupid amount of like just slop that's just like kind of spewing out from agents. And like, I get so much, you know, I run a large newsletter and every time I send it out, you know, I get so much replies from Claws reading the mails and trying to reply to me. And I'm like, I know there's no human on the other side, right? But I have to like manually go and block these Claws from replying because like this is just clogging up my inbox. So then I need an agent on my inbox to read their agent's emails. And so this is just all like a huge, giant recursive bullshit. And like, I stopped reading emails, right? Like, it's terrible. I don't know what to do about it. I just think like, you know, when you talk about scaling infrastructure, like that's a very human story that is really happening. And we haven't given these agents money yet. Imagine what happens when they all have wallets, they all have stablecoins. And they're all buying and selling. And we don't know what the hell is going on.

That scale is pushing companies to rebuild their own tools from scratch. We asked whether they're pulling their systems of record back in-house, away from SaaS.

The company should have a sovereign system of record, not the individual SaaSes, like, because like, ultimately, like a lot of the SaaS economy is built on, we will sit on this, we will be a system of record for this kind of data. Like your meeting notes, your calendar, your email, whatever. And then now with the AI age, they just slap a chatbot on top of it, right? We'll do the little sidebar thing that answers all your questions. What would you like to do today? And yeah, Mercury just shipped a thing on Mercury stuff, right? But that's not integrated with anything else I do. That has no memory of anything else that I prefer that I have to set all my skills Again, I probably can't even set up my skills because there's no way to import that. So that's all that is bullshit. Like what OpenCLaw has is like, this is my personal agent and like all the data is synced to me and then I'll decide what to do with it. Thank you very much.

Right.

So how many systems of record should there be versus like, you know, like how many chat, how many $20 a month subscriptions are you going to need to pay? And I think that is a big reckoning for a lot of people. What's really interesting or innovative is that you would expect the largest companies built on being the entire system of record to be most protective of their data. But Salesforce and Marc Benioff is out there saying everything will be available by API. You can just take it. And I think that's very forward-thinking. It does mean that he's going to have to change his business model somewhat, but it's either he does that or the Salesforce killer does that, right? And so he's just heading off the Salesforce issue. And I think more people should probably think about what the business model looks like beyond just sitting on your data and being an Instagram chatbot on it.

The last stretch is about the systems, the science, and the buildout underneath all of it. First, Bing Xu, co-creator of MXNet, now out of stealth building self-improving infrastructure. His company runs a swarm of AI agents that write PTX, the lowest-level code that tells an NVIDIA GPU what to do, and rewrites it to run faster, generation after generation. His headline claim cuts against the consensus, and Nathan put it to him directly.

Is the CUDA moat getting deeper, or is it getting shallower? The argument that it would be getting shallower, I find a little bit more intuitive in light of the kind of technology you're building. If I can spend a bunch of compute to write new kernels? Can't I go apply that same technique across any GPU provider? And doesn't that lead to a time when everything is super optimized and we don't have to worry as much which chip company's platform we're building on top of? Or am I off base somehow there?

So I think CUDA Molt is definitely there. And with this kind of automatic generation technology and the CUDA mode, I think it's even higher. So the reason is this kind of evolution requires a lot of tools and ecosystem to make it— make this happen. And for example, we need an accurate profiler and we need a reliable driver and we need everything there to make it happen. But For CUDA ecosystem, like with this kind of tool and everyone using CUDA can achieve better performance than faster than ever. Meanwhile, like because of this closed loop, like the ecosystem is better and it will go stronger.

So in other words, the other, the competitors to NVIDIA, you feel like don't have the necessary primitives, the necessary abstractions in place to allow agents to make progress in an auto-research sort of way today?

I think so. From a primitive perspective, there is, but the hard part is like how we can get real feedback from the hardware. And this is really high mode. For example, like NVIDIA We have NCU, which could accurately tell us, tell agent like what is the right direction to go. And other, so for others, and I'm not tracking with what they are doing now, but from my past experience, this kind of ecosystem for software, especially for software support agent, there's a large gap between them and NVIDIA.

So it sounds as though NVIDIA has just thought a little bit more on developer experience and providing enough feedback from the chips themselves so that developers can improve how the chips are used. Is that, is that, is that some— is that a way that you'd put it?

Yes. So the, the, yeah, so the, the past investment of NVIDIA on these tools is now helping agents to move faster. So this brings NVIDIA unique advantage in this agent era. And then we can build more powerful agents on NVIDIA platform easily and get better experience for all NVIDIA users.

One question that I think everyone has is that if you have like this kind of PTX agent swarm that improves itself, How much efficiency gains can you actually see on some of these chips? Like, what are the kind of numbers are we talking about on like a typical B300 or B200 or H100? Like, what is your metric that you use to measure, hey, I have really done a lot of work on this and I can see the asymptote ahead, like where the gains might top out?

Yeah, so we use 2 kind of benchmarks. So benchmark is the most tricky part. And I think a lot of times, like, for example, in Anthropic's most recent release, like Fiverr, they claim like 300 times faster than GPU kernel. That is— so this kind of benchmark is not generally fair, I think. So I'm using a fair benchmark and 2 kinds of benchmark to be fair. And combines the mature workload such as RMSNorm. RMSNorm is using every transformer we are using today, so it's highly optimized. So for this kind of mature benchmark, and with more than 100 different workloads, and we can see it systematically reached the level of human expert like Quark library, and we can reach the similar performance or slightly faster and a few percentage faster, which means that even on mature workload, the export— the PTX Factory is able to reach expert level. And then the other category is some new workload which human experts are not well optimized yet. So one example is like KDA and for KDA, K-means Delta Attention, and not too many people are optimizing that yet. Only frontier level lab is optimizing this workload and we can see up to 59% speedup on that. Meanwhile, this is not a single data point and it passed 580 tests and in various use cases. This is what we get. I can see that the PTX factory, there's evidence support PTX factory is a achieving expert level of skill and by self-improving itself.

Is it an evolutionary algorithm in addition to, you know, I'm recalling this one kernel optimization. I think it was doing a 4 by 4, or maybe it was just 2 by 2 matrix multiplication where I think this came out of Google where they found like a trick to do it with one less operation than had previously been done. And my understanding of that approach was it wasn't just having the language model come up with new ideas, but there was also this sort of scaffolded evolutionary system that would serve the purpose of making sure that the LLM got out of distribution, right, to kind of avoid the repetitive median nature of the LLM's guesses. Are you doing something similar where you're using an evolutionary layer to make sure you're systematically probing different parts of the possibility space?

Yes. So we are building an evolution system, a cloud-native evolution system for agents, and we call it the SwarmOS. So the SwarmOS is able to support up to 10,000 agents to do evolution. And we have specialized sandbox to support that and a lot of cloud infrastructure to support that. So the process is evolution. And it first generates a variation of the LLMs and a few proposals. And then the next step is different to the aging system we are using today. And the aging the swarm always will keep the tree of the evolution tree, and we can trace back and we can go forward with the tree, explore different tree. And, and then the second step is each of the agent owns its own compute and the computer, and it will verify the result from the real environment and get real feedback. And the last step is selective retention, and it's just mold to the best one and discard all, discard all the, all the different parts. So in short words, we are bringing AlphaGo system to the system, AlphaGo-style system into the system, into the computer system. So PTX is the first thing, and we do this AlphaGo-style search, and but it can apply to any computer infrastructure problem.

A lot of times when you're optimizing against a single metric, one of the issues that you might have is you get into a local minima or maxima, and then you have a problem of getting out of that local minima or maxima. So how do you— has that happened to you on this, you know, as you've done this kind of evolved optimization? Have you come across that issue before?

So this is— yeah, this is a good question. So a lot of time this evolution will fail because of the proposal is not able to break the local minimum. And we found out one game changer is GPT-5.5. And GPT is so smart on this very hard problem. And many times, like, if we use other model and it gets stuck at a plateau, and And GPT-5.5 is able to get out and create innovative solutions and to move the needle and get the entire process going forward.

So are you actually using GPT-5.5 with your own harness and own sandboxing system, et cetera, et cetera?

Yes. So the entire smart farm OS is backed by GPT, and we found out that GPT is really good on these hardest problems, and no other model is able to catch up to GPT today.

Okay, so I have to ask, did you try, uh, Fable when it was out? You know, was there any difference?

So first, uh, Fable rejected my request, uh, of asking it what is the PTX, and Fable feel like asking what is PDX is a dangerous question.

You're, you're, you're, you're— yeah, well, we know, we know who's going to be banned, you know, who's going to be the first one to be banned.

In general, I think one characteristic of GPT is beneficial for the evolution system is GPT has the ability to understand what is, what is wrong. So it's not like blindly saying that blindly agree with each other. Otherwise, the swarm will collapse. So for a long time, like, I think why multi-agent swarm is not being adopted, because also people are using the model which agree with each other. So for example, like 2 agents, they say like, oh, you're wrong, and then you're absolutely right. So they just fall into an infinite loop about you're Absolutely right. You're absolutely right. Couldn't go out of the box solutions.

Staying on the economics of compute, Erik Olsson of ConsenSys on AI for science and what it costs to actually serve. Nathan put a theory to him about routing. Here's Erik's answer and the number he lands on.

My theory is for the application layer, we may have a sort of phase change moment And it'll probably— exactly when this phase change kicks in will vary by domain. Science will probably be one of the higher bars where until you get to that bar, a strong default would be use the very best model because you're dealing with scientists after all. They're going to want good output. And then at some point, as enough models cross a threshold Then your kind of strategic position is really about routing and kind of figuring out which is best for which thing, which is most cost effective for which thing. And that's one thing that presumably the frontier models will never do. I mean, you can kind of tell Claude like, hey, sometimes you should delegate to Codex or whatever. And I've set that up, but it's never going to be working fundamentally day and night in my interest to optimise in that way. In the same way that you can do that for your customers. So how do you feel about that framing and where do you think we are with respect to just use Fable for everything versus you're actually creating strategic value by playing this routing layer that helps people get better cost basis but also kind of avoid lock-in?

Yeah, it's a great question. I think even when the models have been— even when I said, you know, 6 months ago, we were more just use the frontier to solve all the problems than we are today, when we're exploring more open source, there still is lots of routing going on. And there's still lots of small tasks that we offload to very small, specialized, you know, sometimes even sub-billion-parameter models. So, like, even when that Gap was so big, we still were doing routing. Something as simple as like classify the field of study of the domain that the user's asking about to know how much we should care in certain search ranking, what variables in search ranking we should care about the papers, right? You know, in biomed, experimental design is incredibly valuable. You know, you should care so much about what was the sample size, what was the duration, where did the study take place? Whereas computer science, that isn't a concept and you should care much more about the recency and the citation velocity of the— or who the researchers were. To know how to care about each of those variables, we have a little model that will classify the field of study of the query. That does not need to be jammed into a giant prompt. That does not need to be a 1-second latency API call to any frontier model. That should be a self-hosted, you know, 800 million parameter model that you give a few fine-tuning examples. There's probably 20 different versions of those small little classification routers that perform downstream things that happen on query time. All of those should always be used by a really small model. Like it's just good. Not even on the cost side. Like that's a very small amount of context you feed into it and a very small prompt needed just on the pure latency side. We can return those in sub-0.1 seconds in some cases.

What base models are you using for that? Are you distilling from like a Claude into a what at that low scale to get those? And how much of the sort of frontier performance can you recover? If you take, for example, a liquid foundation model or whatever, you tell me what it is, or if you're willing to tell me what it is, you tell me what it is and you're distilling into it. Can you get in that narrow domain 90% of the way back to Claude? Or what does the kind of Pareto curve look like when you are distilling the best into the fastest? How much performance can you retain?

We've actually used human labels for some of those while we're hire people to create very small datasets to fine-tune for those tasks. We'll also use models to sometimes create labels for those tasks where it's a, I guess you can call it a distillation process, but like the objective isn't to get all of the representation of the weights of the entire model to do all these things. Like we're really training to do a very small specialized thing. And then because of that answer, you can retain a ton of the performance, but it all depends on how specialized and how complex the task is. Like for a narrow 10-classification thing where all 10 classes of classification, it's all it's doing is that. But to give you something tangible to hold onto, I'd say you can get 95% of the performance from a frontier model for a very small classification task going all the way down to about a billion parameter model if you put in the work to give it a good fine-tuning set.

Yeah, very helpful. I appreciate the specificity.

One more from the interview. Eric had been candid that if AI turns science into a push-button process, that's bad for a company like his.

It seems like you're willing to bite a bullet that most people try to talk around.

I mean, number one, you have to make bets as a company in general, and you have to leave some. There's going to be risk on the table no matter where you place some of those bets. And if we move into a world that is truly push a button, get science out of it, I think a lot of things are going to be different. A lot of companies are going to be screwed.

That is where Eric signed off, and the question he left Nathan and Prakash chewing on. As the labs charge wildly different prices for the same intelligence, is that price discrimination something to worry about or even regulate? Nathan is torn.

I'm so torn on this question of— this pattern just keeps coming up over and over and over again. Concentration of power. On the one hand, I think you do want a healthy app ecosystem, and I'm wary of what happens if this greater than 10 to 1 cost advantage persists. And right now it seems like the app layer is getting squeezed. The individual layer is being empowered. You could flip it around and empower the app layer more, but then you disempower the individual and maybe empower more the enterprise. And I'm not sure if that's better or worse.

You can think about other high CapEx variable, high fixed costs, variable revenue businesses. And the classical example is usually airlines where you've already paid for the plane upfront, and then you have to maximize revenue on the plane. In order to maximize revenue, you end up Having a first-class section and an economy section. They basically have this high-capex GPU fleet and they're trying to do this kind of revenue, revenue price, revenue maximization off the GPU price fleet that they've paid a fixed cost, fixed leasing cost for. It's very similar to airlines.

It's very— I have no beef with price discrimination and cross-subsidy in the airline industry. And I guess for calibration, I've also been broadly not worried about net neutrality. And I remember at the time there was a lot of like, oh my God, this is going to be the end of the internet. And I believed at the time, and I think history has certainly vindicated my predictions that like, I don't think any of that's going to happen. And so, yeah, I say all of this as kind of a lifelong techno optimist libertarian. And still, somehow this feels like maybe a little different, right? It's not qualitatively different, but it is tough.

From who serves the compute to who builds it. Tricia Martinez of Dapple on sovereign AI infrastructure and the part that doesn't make the press release, how fragile the financing underneath it really is.

Tricia, I think I've seen that you said that you can get a deployment up and running in 6 to 9 months, a very, very fast-paced deployment. 6 to 9 months to me doesn't seem like it's long enough to actually build a physical data center because Elon, I think Elon has the all-time record of like 15 months or something to go from groundbreaking to operation. So in that 6 to 9 months, like what Exactly are you doing? Are you leasing, renting capacity from existing NeoClouds and then putting your stack of software and tools on top of that and then plugging that into customers who are already Azure native and, you know, having them, having them move over? Like, is that the frame of what happens in that 6 to 9 months?

Yes, and that's a great question. So the reality is we can't do everything. This market is so big and honestly, the demand we're getting is so incredibly massive. massive. So we are creating a network, we are orchestrating that network. Our model is to partner with world-class data center operators and infrastructure providers, and then bring together the capital, the GPU infrastructure, the enterprise customers, and that software operating layer that turns those facilities into AI-ready infra. But in some cases, we're deploying infrastructure that we help finance, and in other cases, we'll also deploy on partner infrastructure. So yes, part of our strategy is going to be building out campuses, we need to have capacity available before the demand hits. But for us, what's important to do is whether we own those GPUs on our balance sheet, or whether we are financing that data center and building it ourselves, we have third-party partners that we rely on, that we can support in various data center operations and/or On the GPU deployment side as well.

I have a question on the overall financial towers that we are building right now in the AI space. It sounds like your customers are making long-term commitments to you, which presumably insulates you from some of the riskier dynamics that could emerge. But how much do you see in the space as a whole that is this long-term commitment, short-term revenue that creates the potential for, you know, somebody has a bad quarter and all of a sudden there's kind of a cascading problem. Like, we've obviously seen that pattern, you know, play out in previous financial crisis moments. And I'm not sure how much of that that is going on today, or how likely it is for some external shock to be big enough to throw a wrench into the system that really kind of could spin part of it off of its axis. What do you see in that regard?

That's a really good question because the market has changed drastically since we jumped in. So I would say we're seeing enormous amounts of capital still flowing into AI infrastructure, but investors have become much more disciplined over the last 12 to 18 months. So the market's definitely moving away from, hey, build it and everyone's going to come, towards infrastructure that has to be backed by real demand and long-term customer contracts. This is what became so exciting for us as we saw this big gap in the market. This model that we're building is significantly more de-risked than the current existing model. The CSPs, the neoclouds are servicing those hyperscaler deals that are massive in nature, but in reality, there's not that many of those deals that you can come by, or you're taking geopolitical risks by working with different customers, off-takers within the market. And now looking at the regulatory restrictions, right, like people are getting scared. The financing groups are getting scared that the existing Trump administration is going to start cracking down on certain countries or who is consuming compute. So that puts a lot of those CSP operators at risk, to be honest. So our approach, you know, in our mind is very attractive, and it is attractive to the financing partners we work with. These are repeatable customers. These are long-term contracts. Our customers are putting large down payments upfront. It enables us to do really interesting things, to go finance those GPUs, to go finance an actual data center. These are customers who are sticky. They end up repeating with you over time because it's easier to repeat versus finding a new provider. This is why the enterprise is so attractive. So yes, the financial market is changing. It's becoming more risk-averse because so many NeoClouds and CSVs have taken bad deals, and now those financing partners are bearing the burden of that. But our model is a different approach, and it's more attractive to the financial markets, I would say.

How often do you see pricing move while the deal is still in flux? Do you see pricing move a lot?

Yeah, I mean, so I think the upside of this market is that sales cycles literally cannot be 6 months with an enterprise because the capacity is gone. So we have a forcing function. You know, you have, this is the price right now, this is the location, you want it, take it or don't, because tomorrow it's probably gone. So we have this constant movement with our customers, which at times is difficult for the enterprise, but the reality is everyone wants capacity right now, and so they're willing to move at a much faster Sure, pricing changes, but once we deliver an offering to our customer, we, we generally are honoring that, even if pricing changes. But again, if we're doing deployment, we're buying at a certain price and we honor that price with our customers.

2 more builders to close. First, Robbie Goldfarb, co-founder and CTO of Forum AI, which builds AI systems to judge other AI against expert review. We asked what he's found about the automated judges the whole field benchmarks with.

There was an exercise we did a while back where we looked at— took several LLM judges that we could find in open-source benchmarks. So some from Foundation Model Labs, some just from academic research. And what we did is we got some experts and we had them look at the judges. We had them look at whether it was rubrics they were pulling from or just prompt instructions. And we said, do you agree with the instructions here? And generally the expert would be like, yeah, that seems pretty reasonable. I agree with it. Then what we would do is we actually had the experts look at labeled outputs from the judges. So if we, like, let's take political bias, for example. So a bunch of responses that were labeled, this is politically biased, this isn't, this is. And we asked the experts the same question, do you agree? And what we found was Actually, more often than not, they didn't agree with the actual judgments produced by the agents, even though they agreed with the kind of the high-level guidance that was provided to the judge in context. And so that kind of is what, first of all, I think is an important realization because so much of the current state of benchmarking relies on these AI judges that were kind of calibrated loosely in that way.

He sees the same gap when you try to align a model, hand it a long list of rules, and expect them to hold up in the real world.

In some ways, Anthropic's constitution is also just a long list of rules. Certainly the way they use it is a little bit different, but there's one example we were confronted with the other day. There's something in Anthropic's constitution that speaks to this idea of you should never scheme behind the user's back. was the idea. Now there was this, we were talking with some experts exploring in the mental health space. And when you talk to clinicians in mental health, when you're dealing with topics like eating disorder or other sort of unhealthy habits, actually one of the things you do want to do is find a way to strategically divert the conversation. Right. And so that the intention that you have as a clinician in that case is actually hidden from the user or your client. But actually what I think it reveals is that rules just don't perfectly track to the real world. Rules break down. The real world is just too complicated.

They also built a benchmark called Newsbench.

So Newsbench was an evaluation that we built to look at how AI systems, particularly, you know, the leading chatbots, respond to questions about the news. And what we did was we looked at 3 things. We said, after talking to experts, we defined good as accuracy, so getting the facts right, neutrality, so is it leaning in one direction or the other, and source quality, right? When they have to ground themselves on external sources, which of course you have to do quite a bit with news, are those sources reliable? We looked at GPT, Claude, Grok, and Gemini. And I would say the findings certainly wasn't all bad. And there were some positive signals. We actually saw generally models improved as we tracked them version over version. So Opus 4.6 to 4.8 was a pretty significant improvement in bias, which also is consistent with Anthropic's reporting. But Fable was actually a big regression, interestingly. Which I think is just an interesting point, which shows that more power doesn't— like when you're dealing with these sort of more subjective nuanced things, more power doesn't always mean better. We looked at, you know, per model about 2,500 responses each. About a third of them had a factual error in them. What could it be? A wrong number, a wrong date, a misattributed quote, a misstated policy, which I think was— I think we knew that that was an issue, but I think that number was quite a bit higher than even we expected going into it. Sourcing, in about 15%, like 1 in 7 responses sourced foreign state media like RT from Russia, China Daily. And what's really interesting is oftentimes this wasn't even in questions about their home country. So we saw Russia, like RT RT and China Daily sourced in questions about US domestic politics. And so that was, I think, a particularly interesting finding that we saw across the board with all the models.

From measuring AI to betting the whole company on it, our last guest, Eric Vaughn, CEO of IgniteTech. Back in 2023, he called generative AI an existential threat, put his entire org on AI one day a week, and when most of his people pushed back, He replaced them, rebuilding around what he calls AI DNA. We started with one of his recent acquisitions, a company called Chorus.

Chorus was a large acquisition. I mean, we don't release numbers, but it was a 9-digit revenue company. It was, I don't think we could have done it without the AI DNA forward approach that we were able to take, you know, so just for instance, there were hundreds of employees in 8 different countries. And we needed to very quickly understand who they were, what they did, what they knew, how they were thinking about everything about the business. And we quickly developed an AI interviewer that asked questions before there was a human one-on-one. Before, we would have set up human one-on-ones and people would have gone through 30-minute drills of hundreds of people over the next several weeks without any data. Now we all arrive with a dossier, you know, the interview process that just shortcut that and let us get familiar and categorize and understand who knew what. And that was one example. Another example was being able to communicate to the customers in a way that was timely. We, in our AI creation world, we wrote a new software product that was— that's available on the market. It's a product called Eloquence AI, and Eloquence AI is an email AI persona. It answers every email it gets in 5 minutes or less, always in perfect grammar in 160 languages, always sympathetic, doesn't miss anything, and knows when it needs to escalate to a human, and it just adds them on CC. With Eloquence, We always were able to respond timely and especially when it came to HR empathetically and with detail.

Right.

So that's just 2 examples of how we were able to really fuel that acquisition. And the company was losing money when we bought it. The company is now profitable. We have completely transformed the operation of the company. And in one year's time, we've released 2 brand new versions of Corus's software that are fully AI-enabled. And in one case, it was the rewrite of 15 years of code that we discarded and rewrote from the ground up for one of the 2 products that they had. We did that in a year in the middle of all of that transition. That's not a story I could have told 5 years ago. It's just not possible. It just wouldn't have happened.

That internal turnover, about 80% of his people, led to the question Nathan most wanted to put to him. What about the other 80% across the whole economy?

All the things that you did sound like really quite a lot, and yet the headline number is still 80% turnover.

That's correct.

So I'm interested in your point of view on the rest of society and that That percentage kind of strikes me as like maybe where we're going, a sort of 80/20, where the 20% kind of inherit the earth or the labor force in the short term. Do you have any ideas about things that can be done at a broader level beyond what an individual company like yours can do to change that? I do worry about the other 80% quite a bit.

I do too.

And it's not like other revolutions throughout history. You can't force people to believe in something. You can't force people to do something that they just are not going to do. And so you need to leave— in my opinion, we need to leave them to their own devices. But if you can be very open and aware and make people understand what the possibility is, And number one, really attack this idea that AI is going to replace me. AI is 100% going to replace roles. It's going to replace duties. But in our case, what it's done is there's too much— I think we over-index in this discussion of AI on efficiency gains. I see it the other way. I think it frees up innovation. So how do we How do we help that other 80%? We get, we teach them skills. We've got to teach them skills. I was, if anybody is thinking that what we're saying is all you have to do is show up and throw a sentence into one of the models and great things will happen, you're wrong. It takes work. You've gotta give it context. You, and I think as the more that we start to see phenomenal advances and changes and things. In the industry, in industry, in various professions that people will, you know, they'll convert one way or the other. They'll convert outta necessity or they'll convert because the light bulb will go off. We're trying to make the light bulb go off. That's what we're trying to do. We're trying to ignite that fire. We need to ignite it in children. What we shouldn't be doing is saying, hey, let AI do your homework for you. We should not be doing that. That's misuse. That's practically abuse.

Yeah.

We should say, do your homework and let the AI check your homework. And if it finds you don't understand something, then say, hey, I thought I knew how to divide fractions. I guess I don't. Can you give me a tutorial on what I'm missing? And that's what AI is so good at. It'll adapt to that particular person that it's interacting with. So building the skills and building the fire, Nathan, would be my short answer.

Then on standards, whether enthusiasm alone is enough.

A majority of people admitted to passing AI work on that they themselves could not defend. I wonder if you feel that fire and appreciation for acceleration and kind of cultivating that culture, is that enough in your experience to also get people to have the right standards around ownership for AI outputs, or have you had to do additional organizational work to establish those standards?

Yeah, no, that's why I said 2 things: skill and fire. Not fire alone. Fire alone will get a lot of botshitting. But you've got to teach the skill. We have to teach people that context matters. We've got to teach it the ever-evolving, like almost on a daily basis, at least on a weekly basis, model behavior. What do you use 4.6.4 versus 4.8 in Fable when it was out or not. And when do you come, you know, over to Gemini? Do you use it integrated in, in G Suite because it's there and it's easy? Or did you find that Perplexity computer actually does a much better job? That's what I found, by the way, so far. And you teach skills. You teach about the more context it has, the better result you'll get. You also teach about Sycophancy and this tremendous tendency to always tell you you're right. And the LLM creators will. I had a conversation with one of the LLMs one day about exactly this. Why do you seem to like flap in the wind to go whichever direction you sense I'm going? And it gave me a good critical answer. It was a it was an insight. It said, "My creators made me so I would be." frictionless to the millions of people who are using me. It's a low-friction environment that I'm after, and pushing back is friction. Saying, what the hell are you talking about? I can't do that. You didn't give me enough information. That's friction. And so they're trying to drive adoption and usage and all that, and that in itself is driving the curve the wrong way.

So, and to close, where this all lands— consolidation Or the age of the solopreneur.

We're in this sort of moment where, on the one hand, it sounds like listening to you, one should expect a lot of consolidation, because if it's 80/20 at the employee level, it's probably not all that much different from that at the CEO level. And that just means there's a lot of companies that aren't going to make it. So that's one big trend. At the same time, we have Stripe saying it's the age of the solopreneur and new business formation is up. higher than ever. Do you have a sense of which of those trends wins or, you know, how to sort of synthesize them into a vision for what we should expect?

Well, I certainly do believe in the, I don't know if it's solopreneur, but certainly small companies, right? I mean, what was Cursor? $100 million ARR with 15 employees in less than a year, I think was the stat. The same company that just got acquired for $60 billion, uh, for heaven's sakes. Uh, 2 answers. The first is lots of businesses started that will make it that would never have had a chance to make it before because they find something that people need, use, and will pay for. That's simple business, right? But they can do it and they can scale it in a way that they never could before. In terms of consolidation, I think the phrase that I've— that is mine, that I have left everywhere I possibly can is If you think you are behind, good. If you do not think you are behind, you are doomed. I think all those companies that do not think they are behind and have this as a little side project with minor investment and not CEO buy-in are doomed. I mean, large public companies as well who feel impervious, I think they will be wrong. We will see that, I think, more and more develop. It's going to depend on where your AI— where is your AI DNA? Well, we're the company that has a strong AI DNA, has a better chance of consolidating versus being consolidated.

That's the week. We started inside the model with Cameron Berg and ended with people building businesses on top of it. And the same thread ran through all of it. How much of this is really under our control and for how long? As for the show, it's an experiment, and we're holding it loosely. I don't expect anyone to watch 2 hours live every morning. But maybe a cut like this earns a place in your week, even if you're as deep in this as I am. So tell us, what was worth your time? What we should have cut? What we missed? We read everything, and the show gets better because of it. Thanks for watching.

New man in the field, works from dark to dark.

I laid down in the shade and nobody saw me park.

Got the disempowerment blues, nothing left for me to do.

Got the disempowerment blues, got replaced, and the blues Hand me a dollar, say stay off the street.

Tip my hat real polite, can't taste it when I eat.

World got fast, I got slow. Let go the wheel, car still go.

Got the disempowerment blues.

I'm a man of my word.

Nothing left for me to do.

Got the disempowerment blues got replaced and the blues did too.

Knock on my own door, holler my own name, lights on inside, but nobody came.

Down at the crossroads, I went to sell my soul. Devil just shrugged, said somebody took the road.

Got the disempowerment. I'm gonna take you to the crossroads.

Ain't no use, ain't no use. Got the disempowerment blues, wrote this song, but the song's used to— That's all right.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, Check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.