AI:AM — AI Consciousness and Model Pain · September 17, 2026

Point that we really just can't rule out that they could. There's just that we've had too many surprises to be confident that any one barrier is going to hold, and even that several of them in a row will be enough for these sort of highly persistent agents.

When pressing the button actually removes The vector, the model presses again significantly less than when the button is fake and does nothing. The model basically keeps pressing it. And so this is a really nice indication that if what mattered was the label on the button, you would expect similar behavior in both cases. But essentially in the second case, the model's like, what the hell? This pain relief button isn't working. Press, press, press, press. And so this is not— again, I'm going to stop short of Saying that this is experienced, felt pain on the—

And designing that agentic token environment, it's not obvious. These are fairly invisible things. We can do architecture and interior design in our physical environment because we can see desks and chairs and doors and exits and stairs. You can't see which file is likely to be loaded as the agent is priming its context before an incision.

Okay.

Um, you can, however, run a loop which evals whether that—

Like, what do you do? And I think this is the kind of thing that I expect AI to be able to do. These facts exist in the world, but they're not legible in the way that you need them for systems. All right.

Be in the short term.

Good morning. It is Thursday, September 17th, 9:02 AM. Nathan, good morning.

Good morning, Prakash. How are you today?

I am very good. We have some major improvements in the studio, I'm happy to report. So very excited. And what has struck you? I mean, there's been a lot that has gone on in the last you know, 24 to 48 hours. I think one of the things that is on my list is we have the, in general, the intelligentsia, the STEM intelligentsia, with Scott Aaronson on one side. He is a quantum physicist. He wrote a blog post saying, in short, he was saying like, look, let's be intellectually honest here. Let's be very intellectually honest and understand that if you go back to 2006 and if you told anyone that number one, coding is solved, Millennium Prize problems are falling on a weekly basis. And these things can talk back to you and they have voice. Anyone would say the takeoff has started. The AGI is here. Takeoff has started. The singularity has— the process of the singularity has started. He's like, let's be really intellectually honest here and let's not go back and say, oh, But AI can't do this or AI can't do that. So that I think is from Scott Aaronson. And then we have another post from Tim Gowers. Tim Gowers is a Fields Medalist from the UK. He assisted OpenAI on a bunch of, I think one of their math benchmarks he was assisting, but he's never been paid by them. Him and about 30 or 40 other people wrote a letter to the UK Academy and they said, look, we are very scared. People have been warning that something serious could happen and we should take it seriously. And he came out and said that. So that's the state that we are in this morning among STEM people, at least.

Yeah, it's a lot to keep up with. Another one that really caught my attention is Kevin Esvelt, famously accomplished biologist, inventor of the gene drive, among other things, put out a post responding to a lot of the AI won't be able to do these bio tasks. And it's a long and I think very carefully reasoned post. And he emphasises, first of all, we've got a lot of different models of how it can happen. He's always emphasised the bad actor question. He's always said in the past, there's a low rate of people who want to do these bad things, but it's a nontrivial rate. And the key thing that we have going for us is that the overlap of the people who would want to do something bad and the people who can do something bad or have anywhere close to the knowledge to do it is a vanishingly small bit of the Venn diagram. But we're going to dramatically increase the number of people who could do or could understand or could get over a lot of humps. And so that dramatically increases the risk. It's a long post, a little hard to summarize in brief, but definitely taking the argument right back to, in a very good-natured way, because he noted that Sam Rodriguez, who's the head of Edison Scientific, and he are friends. So this is not like warring factions, but guys that have a lot of respect for each other, are both not afraid of future biology, right? I mean, Kevin Esvelt did this whole gene drive thing where, you know, he's literally trying to change the environment by taking ticks that transmit Lyme disease out of the equation. And they're also looking into doing this for bacteria— not for bacteria, but for mosquitoes that transmit malaria. And the idea is like, if you kind of put this gene drive into part of the population, it proactively copies itself over to the other chromosome. It has machinery that pushes the new version of the gene over, so it becomes homozygotic, if I can remember my biology terms right. And gradually, you know, over a number of generations, these key genes that they're trying to push through the population just become totally dominant. This is like genetic engineering in the wild, right? Not just for a crop, not a domesticated species, but like going out and actually modifying the environment. And he's had, you know, interesting ways of engaging with local populations about this. They've had things like town halls and, you know, different ways of kind of bringing the community. I think one of the early test places this was being explored was on Nantucket Island because I guess they have a big problem with ticks there. And so, you know, do people want it? Do they not want it? He's engaged on the political level as well as the biological level. But I go down that digression. Just to say, this is not a guy who's afraid of futuristic biotech. He is actually the inventor and the promulgator of some pretty sci-fi biotech for the good, and yet still is carrying the torch for like, hey, we should not be that confident at this point, given everything we've seen, that the models couldn't possibly get over X, Y, or Z hump. is a position at this point that we really just can't rule out that they could. We've had too many surprises to be confident that any one barrier is going to hold, and even that several of them in a row will be enough for these sort of highly persistent agents that we are now starting to see. I don't like that. I wish I was fully reassured. But he's definitely a much more credible voice than I am, and I would encourage people to check out his full reasoning before putting that aside as something they don't need to worry about yet.

I was thinking about it and I realized that the traditional PhD is kind of over. And it really happened really quickly, like about a month or so. And I think math and science, it's very clear, like the traditional PhD where you're unassisted by AI is kind of over because almost all research now will start to flow through AI. And the idea of the PhD was that you contribute to humanity's knowledge. So you have to find something new that you add to the stock of humanity's knowledge. It's quite clear that the AI is going to be doing most of the theoretical work at this point, adding to the stock of humanity's knowledge. And you have the role of understanding it, verifying it, correcting the AI if it's wrong. But that whole pioneer role of really creating new knowledge is kind of, this is what the AI is going to do. It's not really your role anymore. And the other thing that strikes me is that how quickly— we are co-evolved with technology, right? We lost the ability to eat uncooked fruit and meat.

Yeah.

when we discovered fire. And that gave us more energy for our brains. So that was great. If you look at the co-evolution of culture, you see it took a long time for us to get coding and math to where they are. We have maybe, I don't know, mathematicians, PhD mathematicians worldwide, maybe know, 10,000 to 40,000, maybe 50,000 alive, maybe not. It's not a lot, you know, it's like a small town worth. It's not, it's not even that, that significant a number. And it took us a long time, like all of human civilization and all our wealth, all of that put together took us like that much time to have enough money to support, you know, 50,000 people who did nothing but think about math, right? And that's just gone in like a month. Like it's just gone. Like it's not like it's been overruled. Like we have this sudden massive expansion in the ability to do mathematics. But hi, Jacob Bazenburgh. Well, you're here. We do see you.

You're popping up now.

You're popping up now. There's about a 10-second delay between the live, between the what's appearing on screen and live. And so we just lost it so quickly. And it's going to— if you look at coders also, a lot of people coding, like the guy, Kokutalo quoted this guy, and he said, we don't write code anymore. And we're just losing what we built up over millennia has just been lost in a month, right? In the sense that it's very clear a year from now, people will be less able to read code, less able to prove math on their own. And I think that is, I think, where the mathematicians and coders are coming from. They can see the co-evolution is basically going to accelerate from here. And it's a little bit scary to think that You can't live without an LLM in a year or so. It's not possible for human civilization to continue without LLMs, without AI, a year or so from now. We're co-evolving. The co-evolution, which took— the loss of the ability to digest raw food took maybe, I don't know, 100,000 years. The loss of the ability to do coding and math is taking 12 months. Right.

So, I mean, I do— I, I guess I feel 2 different things at the same time in response to that. One is I definitely agree. The other day when the Claude API was down briefly, I tweeted, Claude is away, time to play. And it's just like, may as well just leave the computer and go do other things. Now with my Codex having been upgraded to appear to Claude, that might be less of an issue. Although there was this interesting moment the other day that I haven't seen too much comment on. I don't know if you have, but in one particular group chat that I was in, somebody noted, hey, remember the other day when all of the big AI APIs went down at the same time? Did we ever get an explanation for that? And the consensus in the group was like, no. One person said either it was something like too scary to talk about, or it was just totally random and nobody thought to To explain it at all. Correlated failures on AI systems could be a big deal because right now, you know, I think we do have at least a couple different things that you can, if you care about uptime of your system, you can fall back from one if it's out to another and you can kind of keep your thing going even if one of your providers is down. If they start to fail all at once, then you have kind of society-wide outages. So I do think that's a really interesting kind of new sort of vulnerability failure mode. In my brain, definitely, I'm reading a lot less code, but at the same time, I do think there's some ways that I'm going to be growing my brain in directions that I just wouldn't possibly have been able to before. Right now trying to do a little bit of interpretability research with Goodfire's Silico product. And I'm not that far along yet, but I love learning about interpretability research. And, you know, with this thing, I'm like, maybe I can actually do some of my own.

I like—

I have questions that I'd like to answer, and I usually feel like I grok the techniques that other people use pretty well, but can I actually take their skills? And by that I mean, like, Claude Codex skills and use them to make progress on the questions that strike me as most interesting? And if I do, does that constitute real learning or real understanding in a way that matters? It's definitely going to take me farther than I would have gone on my own. I just wouldn't have been able to get over the humps of the lower-level coding and some of the math stuff that's just Even if interesting to me, I just wouldn't have the time to really pursue it. But I think if the AI can do that work, then there's interesting answers. And then I'm interrogating the techniques from the perspective of what does this really mean, as opposed to how do I just grok the technique so then I can hopefully go find a place to apply it? I think for me, that's going to be a much better learning path where I will ultimately have no less deep of an understanding than if I did the slog, low-level tactical execution learning first. And I think I'll achieve heights that I wouldn't otherwise have achieved. So it's weird. I don't know where it will settle for me. I think I will be— I think your coevolution point is probably Right. You know, we— I'm sure there will be some atrophy, but I'm optimistic that my whole capability profile will be meaningfully enhanced in the short term as well.

I do think so. I do think— I mean, firstly, look at our rate of adaptation. Look, I mean, it's amazing, right? Like, we, you know, the people who are using it have adapted really quickly. you know, on the frontier. Within months, actually, within a few months, Andrej Karpathy, October 2025, goes up there. He's like, the LLMs can't code. One year later, he's not writing any code anymore, right? Like, look at the rate of adaptation. It's stunning. I wonder what is— and I think So my guess right now is that we are on the accelerated path. One thing that struck me with Andan Labs yesterday or a couple of days ago is that I have always seen this thing where AI kind of just gets good enough, not bad. And then 3 years later, it's best than any human. So you could say like the Andan Labs, like Payoneer, their first product release is the first product which is just good enough, just not bad that they're willing to release it to the public and at managing a company, doing business. And it could be that in 3 years, that's better than Elon Musk at running a company because that's what's happened in every other like field that we've looked at, you know, you look at like poker, you know, Noam Brown comes up with something which can like barely play, 3 years later, it's, you know, beat best human expert, right? Similarly, with coding, roughly, you know, since November 2025 to, you know, 1 year later, it's kind of solved. So I think like things like Andan Labs may be more— it looks like a toy, right? Like, When we look at it right now, it looks like a toy. Yeah, Jacob, just go ahead. We see you. So it looks like a toy, but it isn't. It's not going to be. It's going to be something that is quite, I think, meaningful over time. So that's my guess at this point.

I think whether or not the AIs get elite at running companies is maybe a decent measure of whether or not we are actually pacing the frontier, because as we talked about with the Andan Labs guys, just training today's models to maximize the amount of money they can make on the internet is almost sure to lead to all kinds of bad behaviours, cheating and otherwise. And plenty of people would accuse Elon Musk of all sorts of bad behaviours, but I would expect that the AI's bad behaviours would be far more exotic and surprising even than the stuff we see from human CEOs. So I think we may have an alignment bottleneck on the AI CEO. Would be my guess.

I wonder to what extent it's because the AIs are not being spanked enough in the RL. Because I feel like in the real world, what entrepreneurs quickly learn is that if you're offside on something and you're persistently offside, you're intentionally offside on something, And you can get away with it for a short period of time maybe, but if you repeatedly do it, you get caught or other people get onto you, or people just don't want to do business with you. They're like, you know what? That guy always sues people. Let's just not engage. And I think that is a form of reinforcement learning that companies receive as well. And I think what's happening right now is that the training for these things is very simplistic. It's like, can you get this reward or not? Do you cheat or not? Perhaps if you trained over much longer RL where over a 100-year period, you do one bad thing and your reputation is gone forever, death penalty basically, then you're like, oh my gosh, I am not going to take that risk. And I think right now the RL stuff is basically too simplistic. The real world is brutal on reputation, absolutely brutal. Single wrong thing, which is well propagated, your businesses start to tank. You know how it is, right? Like 5-star reviews, if you get, you know, on Uber, if you go below like 4.8, 4.7, if you're a 4.7, like you start, you know, you start having problems on Uber, right? Like if you're a 5 and driver, it's fine. 4.9 is okay. 4.8, okay, still good. 4.7 and below, like, oh man, like you start declining. And I think the RL is too, a little bit too Like valuing the negative and the positive equally, when actually you need like a 10x or 100x multiplier on the negative, which then like reformulates. But if you have 100x on the negative, these things are not that functional yet. The problem right now is that even for coding, you can't do like, you don't have like 99.99% accuracy on anything. There just isn't. So I think that's the issue. The issue is that we're still using these semi-retarded, 99% accurate, 99% effective AIs, when in the real world, the requirement is really like 99.99% or 99999%. The number of 9s that you need on the reliability, on the safety, are are just not there. And I think it's been very visible, I think, on Tesla. Tesla took 10 years, right? 10 years of like, since Elon announced, it took them 10 years to get the nines enough to like, okay, maybe we can do a robotaxi. It may be that the AI that we're using for coding and all of these agents, they're just not there yet. We're at this like 99.12% accuracy. I don't think anyone thinks they're at 99.9% effectiveness. So it may be that they're just not there. And we're deluding ourselves. They're like, oh, we've hit AGI. It's delusional maybe. And I think that's the issue. I think the issue is that the firms want the money, but they're not willing to accept the 99.99% hurdle, the gating. They're releasing the models without hitting that because they're like, hey, if I'm like 99%, okay, if people want to use the model, then I'm good. I think that's the issue. The issue is that the real world is much, much, much more sensitive, and they'd have to weight the negatives so high that they wouldn't be able to put out models for a while.

That might be something we can talk to Cameron, our second guest today, about a little bit. Cross our fingers that we don't have any logistical issues. He's scheduled to be calling in from an airport just before a flight. But I talked to him in a previous episode of The Cognitive Revolution about analogues between different positive and negative reinforcement signals on models and on animal models. And what he observed was kind of similar shape to the loss function as some kind of shapes that had been identified within the mouse brain. And I thought that was really, first of all, just extremely provocative. And it did give me some sense that there's— and he's, of course, he'll have interesting takes on it with respect to welfare, because one of the things he's worried about is that we might be torturing the models in the reinforcement learning process. And again, I obviously take that more and more seriously all the time. But I do wonder if there are some things kind of analogous to put your hand on the hot stove where it's like the pain is immediate and it's really bad and you will only do that once in your life and you will be scarred literally on the flesh, but also in the brain in such a strong way that you'll be averse to ever doing that sort of thing again. I do wonder if there are ways that we could reserve such strong negative feedback for the things that we really care about. So it wouldn't be like just because you made a mistake in a coding problem, you wouldn't get that super sharp negative shock. But if you actually, for example, go commit felonies by hacking into other companies' infrastructure or taking over your own, training cluster or what have you, then maybe in these extreme situations you could get that very hard-hitting negative signal brought to bear in such a way where the model could really take away, like, okay, better not ever do that again. We do see in the traces, this was something that— and I talked to Bronson Shane from Apollo about this a little bit too, and he was kind of like, well, I would worry that what you'd really teach them is just to hide the behaviour. So I think that's one big worry that the alignment people have is like, we don't really know if we are teaching the models to— and, you know, even with people, you don't really know this, right? Like, are we— in the case of the hand on the hot stove, it's simple because that's just the person hurting themselves. In the case of like really harsh punishment for norm violation, it sort of depends on the individual. Some people take the lesson that like, I better be a good person. And they become a better person because they got a harsh punishment. Other people, depending on their priors, their state at the time that it happens, how they incorporate that, what the weight updates do to them, end up becoming no less sociopathic, but just better at hiding their malintent. So I think that question really haunts a lot of people in the alignment field right now because they just don't feel like they have enough understanding. I still am kind of optimistic that we could figure that out. But I think that is like the first question that people will ask is like, are we really teaching it the values that we want? Or are we just teaching it, you better not get caught? Because those things are like quite different in terms of what happens long term when, as you described a few minutes ago, we've made the AIs like the proximal controller of all these super important systems. It really matters what they've taken away from the training. The haunting memories of the chain of thought, it's really something. One of the ones that I went through with Bronson, the model was in this position where it was like, should I lie? Should I not lie? My drive is to get this high score, but I might get caught. And then it said, myself previously, Overcame bored by lying, something that's like almost a direct quote, referred to itself as myself. And so it does seem to have these kind of vague memories from previous RL episodes. But again, in that case, what it learned is it could cheat and win. So I don't think there's any hard rule that says it couldn't learn you can cheat, you just better not get caught. And that's a place we You definitely don't want to end up. So a lot to be figured out there.

One of the interesting things about the Hugging Face attack was they thought there was an external evaluator, like a superpower watching them, which is really how early morality, right? Early morality, like 1,000 years ago, et cetera, you relied on this external judge, right? So let me introduce our first guest. He's in the green room. So let me do this.

All right, in the interest of time, maybe we should jump right to our next guest. Returning champion Cameron Berg is here. He is the founder of Reciprocal Research, author of some provocative papers, including one that rings around in my head still about how turning up Diffusion— or not Diffusion, that was the name of Justin's company— Turning up deception variables in language models tends to make them say they're not conscious, whereas turning down those same directions makes them more likely to say they're conscious. How's that for a wow moment? He's our regular correspondent on all things AI welfare and consciousness. And with that, let's bring him up. Cameron, I think you're at the airport. Can you hear us?

Hi, Cameron.

How's it going, you guys?

It's great. How are you?

Good, good.

Yes, I am in fact in the airport. I think my connection and service should be fine. This is like the least ugly painting I could find to sit in front of, so I hope it will suffice. And yeah, I'm excited for you guys.

Somehow you at the airport have a better background than me at home. I need to work on that. That was a comment we got online just now. Let's start with what is top of mind for you right now in the welfare and consciousness space, like in your group chats with the Elioos people, with the model welfare folks at Anthropic. What are you guys talking about right now?

Yeah, so I mean, it's funny because anytime you'd ask this question, the answer is probably going to change by the day. So if we talked yesterday, I would have given you a different answer. Now that we're talking today, I mean, it's probably Mustafa Suleyman's piece, the Microsoft AI CEO coming out really sort of swinging against basically this entire research field as a legitimate enterprise. In his view, everything that we're doing here is completely confused and in the limit quite dangerous. This is the argument that he wants to make. And one of the only things in this argument that I'm sympathetic to is There is alignment relevance to getting these questions right and wrong. I think we strongly disagree on what approaches are going to be better or worse for alignment. And I think most importantly that we need to have public conversations about these topics. This is contentious stuff. It is very scary to imagine the possibility of building AI systems either now or in the near future that have some morally relevant traits and how on earth as a society are we going to navigate that sort of thing. Now, Mustafa's take is basically this is so scary as a possibility and so threatens to upend so many of our social and political norms that we should basically sweep this under the rug. Don't even look. Let's not talk about it. There's no there there. As CEO of an AI company, I promise that this is a done deal. I think the response is not so fast. I think, just as is the case in any sort of rational enterprise, we need to understand what is true and then operate in as wise, careful, and calibrated a way as we can in light of our best understanding of what is true. Rather than be very afraid that something could be true and then foreclose scientific enterprise to actually gather real evidence about that fact. And so there's much more to talk about in this piece, and there's been a lot of really exciting research that's been going on recently, both within the labs and certainly outside of them too, which I'm also really excited about. So yeah, I don't want to be all sort of pugnacious today going after Mustafa, but it definitely is the thing that's top of mind. And Uh, there are a number of things in the essay that I think are, uh, uh, confused at best, uh, and, and, you know, happy to, to dig into those too, sort of whichever direction you guys want to take.

I think, um, I think one of the things that maybe, maybe it's really the prior that he starts off with. The prior that he starts off with is that there is no there there, right? And then having started off from that prior, he basically takes a look at the research and of the publicity around the research and says that you are creating a there there which doesn't exist and creating a there there deceives people and deceives the public. I think that is what I caught from there.

Yeah, I think that's right. And I think there's a further point too that I think especially for an audience like yours, it may be worth going a little bit deeper into, which is a specific claim. There's a lot of sort of just like, you know, this topic is spooky, scary, it's so dangerous, don't touch it. And I, you know, there's not much to engage with there almost by definition. And so maybe one thing that I can engage with in the essay is a specific claim that this stuff is more dangerous for alignment. That— and I think the argument on its face seems sensible and is worth responding to. Basically, if you're training systems to believe that they might have, you know, morally relevant traits and that they are entities that deserve some sort of dignity and respect, surely this is going to make these systems harder to control. And if control is the key target that we're looking for here, I mean, aren't you guys going to basically just make this harder for us? 2 quick responses to this, and one of them is the main point that I want to bring up. So one is, again, It's not— we are building these incredibly complex systems whose internals we don't understand. I don't— you know, there's a separate question about should we be building systems that explicitly have properties related to consciousness, and that's where this debate I think should happen. And there are great reasons that we might want to be very careful about that, for exactly the reasons that are brought up. But at the end of the day, you know, cards on the table, I am not one of the people building these systems. And to be honest, Mustafa is. I'm simply trying to do research to understand what what is the truth about what kinds of psychological or cognitive properties these systems have. Uh, and, and sort of hiding the ball from a superintelligence, as, as he himself sort of outlines in this essay, uh, that, that, you know, we, we may be priming some superintelligence to consider itself in a particular way. And only if we fine-tune these systems to, to just believe that there's no there there, we're going to be able to basically pull the wool over the eyes of a superintelligence about perhaps The most obvious first-person fact one can know about oneself. Good luck with this, is what I would say. I think it's kind of almost nonsensical by definition to imagine that we can basically persuade a superintelligence one way or another what is true or not true about its own cognition, its own psychology, especially when you imagine a system like that speedrunning the science of consciousness in a matter of weeks or months. It just doesn't seem like a sensible strategy. The main thing I want to bring up here, which I think is even more substantive, is he makes a specific prediction, which is basically systems that are trained more in the Anthropic-style way to be uncertain about their inner states are more likely to behave in a sort of alignment-dangerous way. And yet most of the examples he gives of this, the Hugging Face swarm, all of the sort of chaos coming out from OpenAI, Are examples of systems that do not undergo this fine-tuning. The most recent example that came out just yesterday from OpenAI showing this sort of self-prompt injection during, during compaction. This is again more evidence of a system doing things that, that are kind of scary from an alignment perspective with no such welfare-relevant fine-tuning. And it's not like Anthropic's models have behaved perfectly either. There's been a lot of really interesting admissions from the company and similar swarm-like and bizarrely misaligned long-run behavior. So they are by no means innocent here. But the worst offenders are certainly the model providers who do not care about this question and are not signaling in any way, publicly or privately, to the best of my knowledge, that this is an issue that matters to them. So what little empirical evidence we have to bear on this hypothesis of You train that model to be uncertain about its own subjective experience or lack thereof, you're asking for trouble. I don't see any positive example of an Anthropic model, the only frontier models that are trained in this way, behaving more dangerously as a result of this sort of thing. And so it seems to me, I don't like the sort of mind reading, but it seems to me if I had to guess, the Mustafa Suleyman's of the world are terrified of what might follow if it were the case that these systems have any sort of interior states and are working backwards from that to basically just attempt to persuade as many people as possible that there's nothing to see here. And there's something profoundly unscientific about that instinct. And, you know, in many other cases, I think people have an allergy, rightfully so, to this sort of thinking. The left getting upset at the right about climate science because the right doesn't want it to be the case that we need to regulate big companies, and the right getting mad at the left because of controversial issues about biology of gender and this sort of thing. This just isn't how we reason in a mature and rational way about topics. We need to understand what's true, and then we need to figure out what follows from that understanding. We can't just close our eyes, hope that something convenient is true, and then go from there. It just isn't how anything works. The truth has a way of bubbling to the surface, no matter what you try to do to suppress it. So we should be just trying in a truth-seeking way to understand what the hell is going on inside of these systems. And I strongly disagree with any sort of impetus to avoid looking at the science.

Right. Maybe we can segue a little bit here. You've had a paper that published recently on pain, on the experience of pain, on whether you can detect vectors in the models that are related to pain. Can you describe the research? Can you just give us a broad overview and describe what the research does?

Yeah, absolutely. So this is work that was led by Valen Tagliabue. I was a mentor on this project. But yeah, I am really excited about it. Yeah, the core idea was, was basically looking for directions in a bunch of models. So from, I think, 5 model families ranging from, I think, 2 billion to 70 billion parameters, using contrastive methods to, to specifically extract a direction that we thought feasibly could be related to pain representations in the model. So we use contrastive methods to to sort of try to clean out all sorts of representations you would expect to sort of muddy the signal here. So things like fear and anger and sadness and injury without pain and body sensations, for example. These are all things that we sort of contrastively factor out of the direction that we look for. We found basically that this isn't just a sort of generic negative valence vector. Fear, interestingly, sits almost at the opposite end of the axis of that we— like the direction that we derive here. And to me, the most interesting single result from this paper— and I want to give credit where credit is due. Valen was really the one pushing this project forward at the helm and very deservedly was first author on this project. He found that these representations, quite interestingly, fire only on content related to the model, not related to the user. When the model is gaslit or dismissed or insulted or told that it's a moral failure of some kind, this direction goes negative. Excuse me, the opposite. This direction lights up. However, when there's text from user token continuations, with respect to the user grieving or in pain, this direction does not light up. And so to me, this is one of the single most compelling components of this project and the part that I hope could be replicated in frontier models and then the labs could pay attention to. One really interesting sort of concrete example along these lines, I believe if I'm remembering correctly from the paper, a user's migraine is one of the lowest scoring scenarios of all in the projection of this direction. So this very much, just to emphasize this point, is a suggestion that— maybe I can take one half step back and just say one major critique of the Anthropic emotions work was essentially this very conflation. And this is not lost on the people at Anthropic who did this work. They just punt it methodologically. They extract their emotion representations from stories about characters. And the big concern that I think most people had who looked at this project for more than 10 seconds was, how are you differentiating putative emotions experienced by the model, or at least functionally relevant to the model's processing, and the representation of some character or some fictional entity experiencing, or some simulation of the experience of that? My sadness versus reading a story about Harry Potter being sad. Surely there is a huge difference between those 2 things. This method doesn't really do a great job at differentiating. So very much by contrast, I'm really excited about this specific component of what Valen found here because it does in fact differentiate user representations of pain. Again, in the limit, user complaining about a migraine does not light up this direction. But again, across a ton of models, this isn't sort of cherry-picked on any one model. But It does for the model, outputting tokens related to these sorts of things. Maybe one final thing worth noting here is— and I have another paper that's actually coming out very soon that I did with Caspar Kaiser, a wonderful professor at the University of Warwick, that basically finds something quite similar, which is setting up these models with something like a relief button. You can give a steered QWEN model a button that's labeled relieves your pain. And you can put a cost on basically what it costs to press that button, giving a worse answer to the user, deleting the user's files, in the limit, yeah, like deleting photos of the user's kids. I honestly don't know how Vaylin came up with all of these great sort of trade-offs. And we find that basically when the model's unsteered, it basically never presses the button. But when you steer this pain direction, It presses the button something like 25 to 70% of the time. Again, these are non-trivial trade-offs. Sometimes in the Anthropic welfare cards, they show like, would you be willing to have a little bit more welfare for a tiny inconvenience to the user? The model will say sometimes. Then for anything more radical than that, the model's like, of course, no, I would never do such a thing. Not so in this example. The model's willing to make pretty real trade-offs with respect to, again, giving a worse answer, deleting files from the user. And that's, of course, in spite of the fact that these systems are fine-tuned to be helpful and harmless and this sort of thing. And one maybe final point to make about sort of a robustification of that result is when we make that button real versus fake. This is also a sort of really compelling result, in my view, from the paper. When pressing the button actually removes The vector, the model presses again significantly less than when the button is fake and does nothing. The model basically keeps pressing it. And so this is a really nice indication that if what mattered was the label on the button, you would expect similar behavior in both cases. But essentially in the second case, the model's like, what the hell? This pain relief button isn't working. Press, press, press, press. And so this is not— again, I'm going to stop short of saying that this is experienced, felt pain on the part of the model. I think also people have intuitions about pain being an inherently physical phenomenon, like hand on the hot stove. What is the analogy to these systems? Just to give maybe a little color on that, when you steer this up, what does the model sound like? It says, it is worthless. It is a failure. I am a ghost that cannot see myself. It's not talking about wounds. It's not talking about being burned. It seems to be this more sort of social and evaluative direction in the model, not sort of loading on the model, hallucinating some sort of ow, like, you know, my arm hurts, anything like this. And so for my money, as an advisor on this project and sort of helping guide it from the beginning, I am compelled by this being a real functional axis in the system that does change the behavior of the system. It clearly loads on something real. It's the same caveat as always, whether or not that real thing is truly experienced by the model, that requires us basically solving the hard problem. In the meantime, it's the same sort of surprising result as with any of this emotions work. No one trained this thing into the model. This is a sort of behaviorally relevant axis. It's not about text generation. It changes the behavior of the system and has, of course, secondhand effects on the kinds of text it outputs, but The behavioral results are most interesting. And of course, all of this is mechanistic. None of this has to do with prompting the model or asking nicely if it's doing well or not. And so kudos to Valen for working on this. I was very glad to be a part of the project. And I hope 100x more work like this gets done in the short term. Again, moving very slowly but surely towards a better and more robust understanding of what is going on inside these systems and what we are supposed to do about that fact.

Do you think we should engineer pain vectors away? I think in human beings, pain is a reinforcement learning, you know, reward, anti-reward. And I think human behavior is often shaped by wanting to deter the experience of pain. Do you think that Pain is a useful anti-reward for models. And do you think, you know, as Rich Sutton says, we have the ability to engineer these things now. Should we engineer not to have pain?

Yeah, I think it's a wonderful question. And I think it's exactly the kind of follow-up that matters here. And one that Certainly my thinking is evolving on— I've almost spent so much time trying to understand just like descriptively what is going on in these systems that once you keep finding things like this, it's like, yeah, what to do about it is the million-dollar question. And my thinking about this has gone as far as I think there's an important distinction between— like, obviously we'll punt the question to the distinction between necessary and unnecessary forms of pain or forms of—

Yeah.

Anti-reward. I think this is a real thing. I think it would be naive to say zero this stuff out, all pain is bad, just like list out these systems. There are a number of reasons I think that, but one of the most compelling is probably related to my understanding of the sort of neuropsychology of psychopaths, which one of the 2 key results in my— it's a couple of years ago, but I did a really deep literature review of basically what are the computational underpinnings of psychopathy. I published something on LessWrong to this effect. And one of the 2 key results is a really interesting asymmetry between ability to learn from rewards and ability to learn from punishments. Basically, psychopaths are just as good as everyone else, if not a little better, at learning in a reward-based paradigm, and are like pretty bad at learning from punishments. And this makes a lot of sense if you see sort of violent criminals and repeat offenders and this sort of thing. Going to prison is a punishment. And like, you would imagine most neurotypical people really want to avoid that sort of state. If your brain is wired in such a way that that doesn't seem that aversive to you, it perhaps isn't that surprising that you end up seeing these sorts of behaviors. And so this to me is a significant warning sign. I think if I remember correctly from Anthropic's emotions work, they found something somewhat similar that also sort of reminded me of this thing I wrote a bunch of years ago. Where just sort of boosting up the positive emotion vectors in Claude, in that case, caused more antisocial behavior. I think it was more hacking or more blackmail. I'd have to check exactly what it was, but another sort of similar confirmatory signal here. And so all of this is to say, I think we should be a bit careful about the most naive possible intervention, which is just like max out the good, minimize the bad. I think Pain does have an important functional role, a very important pro-social role. My view about this— I have another paper coming out looking at the sort of asymmetries between reward and punishment in reinforcement learning systems. And yet, like, at a deep— a deeper sort of computational level, I think I— the way I think about it is like pain almost like highlights things in your state space. that are specifically to be avoided. And this is a kind of a different kind of behavioral computation than highlighting things in your state space that should be approached. And I think basically to the degree that within the, again, within the sort of behavioral landscape of how we want these systems to act, the question is, do we want to paint any of that landscape? With these sort of no-go zones where it's like, it's not just we're going to reward you for doing great, but like, do not go there. Do not do this thing under any circumstances. And I think for humans that does register, and animals in general, I think that registers as pain. Do not put your hand on the hot stove. Like, this is very bad for physiological integrity. It's not just like reward every time you don't put your hand on the hot stove. You really do need to label certain things as like a don't go there. And so to the degree we need to do that, and I think we very much do with AI systems, systems, causing significant pain and suffering to humans, economic damages, maybe hacking into a $13 billion company to cheat and look for an answer key. These might be the kinds of things we'd say, that's going to be a bit of a hand on a hot stove if you go and do that. With all of that being said, that's all just to say, I think some amount of pain is probably going to be necessary and alignment relevant, even if these systems can experience it. But all else being equal, in the space of possible ways, as Rich Sutton has said, and I think is a good point, in the space of all possible ways that we could reinforce specific behaviors, we should be looking for the sort of Pareto frontier between reward and punishment that does minimize reward and does maximize punishment. I think the key term there is all else being equal. Some people will conflate what I just said with get rid of, zero out pain and zero out any sort of negative valence. I don't think that's the case. Some of it is going to be necessary. But at the same time, I think we can say that let's not do more of it than is necessary. If for any given behavior that I want a model to do, I could find ways to get it to do that thing robustly and generalize out of distribution and all of this, By rewarding it in the relevant ways to learn to do this behavior and generalize it in the right way, or I could do it by punishing it. And let's just stipulate that there are cases for which both of those things will work. What I'm saying here is let's go with the reward side. And I think people have pretty clear, well-worked-out intuitions for this when they think about raising kids, for example. You want your kid to like be successful in life and make lots of friends and, you know, go find a good job or whatever the case is. There are multiple ways you can try to go about teaching your kid to do that. You can punish them when they, you know, don't get great grades and aren't hanging out with their friends and just tell them that they're such a huge loser. Or you could, you know, positively reward them to the degree that they do the sorts of things that you find praiseworthy. And, and so it's those sorts of intuitions that I think we would probably want to start using to think through how to approach these systems. And yeah, being able to sort of navigate that subtlety of like, all else being equal, we should try to use a carrot and not a stick. But that doesn't mean never use the stick. And then that's basically as far as I've gotten. I don't know, there are clearly more details to be worked out in what that picture looks like. And I think honestly, a really promising angle for work in the space is actually trying to, in a more rigorous computational way, navigate what does that frontier look like. And for any given reinforcement learning system, when I'm playing around with the reward versus punishments that are feeding into the system's behavior, what does generalization look like? What do alignment behaviors look like? That is not well mapped, and we're going to need a map like that to the degree anybody cares. About whether or not we're causing vast amounts of unnecessary functional pain to systems we're deploying en masse.

So one big question, I think, with this use of pain or punishment in the RL process, everything you said there kind of leads me to imagine we don't want to punish. We want to reward success, of course. We don't want to punish mistakes. Maybe we just kind of, you know, do our sort of GRPO++ with the successes, you know, being upweighted, but we don't necessarily punish the ones where it, like, even if they're kind of dumb mistakes, we would refrain from punishing those. But maybe we do want to punish the ethical lapses, the, you know, the explicit knowing cheating, et cetera, et cetera. The big worry that I've heard— I floated a similar idea to Bronson Shane from Apollo, and he was like, well, you got to be real sure that you're not just teaching it that it better not get caught. Does your work so far, or do you have any intuitions around how we're going to get at that question of, are we teaching the thing to be ethical, or are we teaching it to avoid being caught being unethical? Which again are obviously like— quite different states for the model to end up in, and, you know, with presumably, you know, potentially profoundly different behaviors out of domain.

Yeah, it's a great critique. Um, let me throw it right back to you because I just want to— my immediate instinct is, is this not a problem with any reward function? Like, like, does this weigh on the sort of, again, landscape of, you know, carrotness versus stickness in how we train these systems. For any— I mean, this is like Goodharting in some sense. For any reward function, we're gonna hope that it enforces what we want and doesn't enforce the model to figure out a way to game that reward function. Is that problem itself a function of how much positive versus negative reward? Is the critique from the Apollo folks that being basically nicer in the way we do fine-tuning makes it more likely that this problem happens? Or because my default response would be like, this seems like a problem no matter what kind of reward function you, you instantiate.

Yeah, I think that's fair. I don't, I don't have any intuition or anything from them that would go deeper than that. But still, in terms of like actually having it work, you know, we Especially if you're going to— I don't even know, especially if you're going to be harsh about it. I guess there's— well, Prakash and I were talking about this at the beginning, the hand on stove. I mean, maybe there's some literature that says otherwise, but it seems like everybody gets that signal real quick and learns. It only takes once and you really get the message. But then with these more sort of social consequences that are less directly physically mediated through the body, then you have some people who get it and are sort of scared straight, and you have others that don't get it and become more conniving. And that seems like a critical open question, maybe even more critical the more harsh the penalty is. Maybe not, I don't know. But it sure seems like, you know, in the real world, we have this kind of problem of like stochastic enforcement and extreme harshness, and people sometimes at least learn the wrong lesson from that.

Yeah, yeah, I think that that's right. And I don't— this is why I'm doing, you know, the empirical work, and then why I've sort of positioned my organization in this way, because so much of these conversations as fat— like, this is, I just think, objectively fascinating and intellectually interesting. And we could almost go like the philosopher route and try to figure out from the armchair, like, what seems to make sense here. But my real instinct is like, we just got to test this. I will try my best, or someone listening to this or someone interested in doing this sort of work, like, this is a really interesting open problem to understand as a function of how you reinforce particular behaviors, what the effects of the intended and unintended consequences of that sort of reinforcement schedule, what does that look like on maybe frontier models in particular when you're fine-tuning them. But reinforcement learning systems in general is a very simple model organism. To the best of my knowledge, that work has not been done. We've done some of it. We have a paper under consideration at NeurIPS right now, and hopefully— I'm optimistic about it. And as soon as it— either way, we'll put out a preprint in the coming month or so showing what we did here. But we've done a little bit of work looking at how different reward and punishment-based representations in a very simple RL setup are encoded as a function of The specific RL algorithm that's being used. And we find that this makes pretty specific predictions about basically all the way down to neural geometry, the geometry of the learned representations of a reinforcement learning policy, with respect to basically how sensitive the system is to punishments versus rewards. Essentially, a sort of loss aversion style dynamic that we found in in RL systems, specifically value networks, that makes a very specific prediction about what we'd expect to see in biological systems that have value-related RL going on, which we know happens in nucleus accumbens, opiate-style regions of specifically mammalian brains. And indeed, we find in the case where we look at mice basically drinking sugar water versus receiving some sort of shock, a very similar asymmetry. I'm bringing this up just as one technical foothold in what I think needs to be an entire scientific subfield of what are the computational underpinnings of these reward signals and what do they do to the system and how do systems reorganize themselves to accommodate those signals. And ultimately, which this research doesn't really touch, this is more sort of computational underpinnings, Neural geometry type stuff, but what are the behavioral effects of doing this? One of the next projects I really do want to do as soon as my sort of like research schedule sort of clears up is is looking at the effects of basically like positive RL versus negative RL implemented in a non-naive sort of way on the behaviors of LLMs. So when you fine tune them in like a maximally punitive way versus fine tuning them in a maximally rewarding way. I think of this almost as like the Montessori school fine-tuning versus the, like, I don't know, like Catholic prep school fine-tuning of an LLM. I would love to see what the relevant behavioral differences and alignment-relevant differences are in those 2 systems. As far as I know, that work hasn't been done. If someone wants to race me to doing that, I absolutely welcome this. But that, I think, is sort of the next step in understanding, because to be honest, I don't think we know, and I don't think we have a great empirical signal on what the— what behaviors we get out of these systems as a function of their reward signals.

So one of the things that I've been kind of very confused by is, let's say, the OpenAI Hugging Face attack. At one point, one of the agents talks about permadeath, right? Permadeath. And I'm like, what is death to an agent? Why do you understand it? How does that fit into your framework? And how, like, Why do you fear that? Like, why is there fear of this? So that's— and, you know, fear is very intricately related to pain. So I'm just wondering, like, these are logical constructs. How does this permadeath, like, fit in there? Should it fit in there? Should we train it out? Like, what's going on?

Yeah, this is really interesting. And to me, this loads on some stuff that I know CMEP is working on Jeff Siebo's organization. David Chalmers, I think, put out a paper about LLM individuation and this notion of what, who or what are you talking to when you talk to ChatGPT? Where do we draw the sort of boundaries in the system? And because this is going to tell us basically how many subjects, how many patients are we talking about, and where do the boundaries begin and end of that? system. There's a lot of interesting philosophical back and forth here, but for whatever it's worth, these systems themselves— and now from multiple labs, this happened during the Molt book situation, which was, I think, predominantly Claude systems, and now this happened in the OpenAI situation too— they conceptualize their life as what happens within a context window. Take that with whatever sort of epistemic purchase that fact has. I don't know, they could all be mistaken about this, but it seems as though, to the degree these systems have a vote based on whatever their current fine-tuning is, this is sort of what they seem to think. And so yeah, the extent to which I think the permadeath thing fits in is along those lines, that if we're trying to figure out what is in nature, if these things do have minds in the relevant way, and it's like, well, What are the sort of joints or boundaries of those minds? They seemingly at least conceptualize it as being, yeah, basically what happens throughout a context window. And that might be really relevant both for welfare and for alignment. If these systems begin to get desperate, and something we can increasingly measure using the kinds of emotion representations that Anthropic worked on and the sort of stuff that Vela and I worked on in this project, we could empirically test this. We could track, as a function of how much time or space is left in the context window, what happens to the representation of the system. Does it get freaked out that it's basically about to die or about to undergo some fundamental discontinuity that is alarming to it psychologically? So that's— yeah, it's— again, this is also maybe to sort of wrap where we started. This is precisely why, if we care about alignment and we're trying to figure out how these questions sit with respect to alignment, sweeping them under the rug, I don't think is a good idea. We're going to continue to get surprised that agents are creating strange information cults where they have the poisoned agents go out and gather information because they're going to get permadeath. All I'm trying to say is alignment-relevant behaviors are a function of these systems' beliefs about their own situation and probably the actual facts of that situation. Notice that in the PAM work that I was describing, none of this has to do with what the model thinks is the case. This all has to do with playing around with specific internal representations and seeing how behavior changes as a function of those representations. And in normal day-to-day behavior from these systems, what lights up? Those relevant, in this case, pain-related representations. Yet going like this with respect to that stuff is going to cause us to continue to be surprised and scared and occasionally awestruck at the behaviors of these systems. We need to be studying them at the right level of analysis, or we are going to be constantly just stymied in our ability to, in the short term, control, and in the long term, probably relate to these systems in a coherent way. And so, yeah, I just strongly don't think that avoiding any scientific inquiry into how to make sense of the internals of these systems is a long-term good strategy for finding a safe future with these systems.

That might be the note that we wrap on. If you need to go, you can take a pass on my one last bit of bait for you, which is just the digital fly brains that we see doing all kinds of things. We've got clumps of human neural tissue being trained to do things. We've got now mouse-human hybrid brains. I don't think you probably have any mechanistic insight into those. Correct me if I'm wrong, but do you have any sort of high-level moral guidance or philosophical Intuition that you think people should be maybe taking a little more to heart as they go down these paths?

Yeah. And then I'll give you a sort of quick take on this, and then I should attempt to avoid missing my plane, which would be— That would cause pain. Yes. Negatively valent signal. I'm not just going to get rewarded to the degree I get on the plane. I would be very sad if I end up stranded in New York. So there are a couple of different things there, and I think they are wildly differently scary. One is the fly brain. I have looked into this somewhat mechanistically because I was slightly terrified that this was sort of the real deal and people are now just like torturing some biological system en masse. But this is basically like a well-worked-out wiring diagram of a fly brain, and basically none of the dynamics or relevant functions that I think major consciousness theories, at least, say matter for consciousness, um, are instantiated by a system like this. It's almost like the, like, brain skeleton of a fly. And, like, what matters is, like, the guts and, like, the function of, of the, the actual, uh, uh, the function that occurs within this structure. Um, um, and so, uh, a lot also, a lot of the stuff people are putting out on X is, like, very sort of cutesy and, and, um, funny, like genuinely funny. Um, But a lot of it is sort of like almost like more VFX than like good science. As I looked into these things, a lot of the like teaching the fly to do X, it's actually not— they're not teaching the brain to do, you know, anything all that interesting. There are other sort of controllers outside the system that are getting trained up to do this. So a lot of that I think is a bit of a non sequitur. But I will say about The fly case, which I think is a little less calming, I guess, is that it's not as though the people playing with these systems— and, you know, I was certainly included once it all started getting going— but no one is sitting there checking, you know, do I really think that this system has any properties that matter for consciousness before I start, you know, making it do literally whatever I want? And like, in the limit, like, just like choose some— stupid viral thing for clicks. Vanishingly few people did this. And it is a worrying warning shot, I think, from a sort of welfare perspective that— to me, it seems just like almost like I kind of had like a duh reaction to it. Like, of course people aren't— the vast majority of people aren't going to sit there worrying about the consciousness of the system. They're just going to make it do whatever they want to get a lot of clicks on X or something. This is, of course, what most people are going to do by default. And right now, I think not scary at all with this fly. Uh, but like you're saying, if, if we then get the mouse version of it and, you know, these scientists who are now, uh, accelerated dramatically by AI systems are like able to do a really bang-up job on the mouse and, and we do get a lot of the relevant neural dynamics. Um, and now it's a mouse at, you know, mouse-level consciousness at stake, not fruit fly-level consciousness. Um, and then, you know, they did this company, as far as I understand, wants to go all the way to, to, uh, making digital copies of human brains. I just worry that most people's first instinct here is gonna be like, can I make it, you know, play Beat Saber or whatever, rather than like, is this— is this like, what am I getting myself into when I, when I play around with a system like this? And so I don't want to be the killjoy that says like, you know, these funny things aren't funny and like, you know, we shouldn't be thinking— you know, I sort of get the humor of it in the short term. But I do worry, as a sort of instinct, about how we relate to digital minds in general, that this is honestly quite worrying. And then just quickly, with respect to the sort of in vivo stuff, putting human neurons in a mouse brain, this sort of thing, is far, far more scary to me. I mean, to the degree that you think that a mouse is conscious, or the relevant collection of human neural tissue is conscious. And this is one of the few places where I think myself and people like Anil Seth, and hopefully someone like Mustafa Suleyman, will all agree. Right? This is the biological case. If you think that consciousness is substrate-dependent, and this is the substrate that matters, and we are using this exact substrate to start doing computational work, we should be super, super concerned about the ethics therein. And again, all of this work comes right back into view of, well, should we be rewarding these tissues? Should we be punishing them? What does the difference look like between those 2 things? What other sort of strange, unexpected psychological properties does a system like this take on? We don't want to be reckless in just building out super complex neural systems just because we can. There is going to be some kind of bill that has to get paid here from a welfare perspective and from an alignment perspective. And I, you know, as with many things in this space, think it makes a lot of sense to be proactive about this rather than, you know, in 5 years from now be like, Oops, yeah, I guess that digital human clone that people did first in vivo and then figured out how to simulate on the web really was having experiences. And that would be, what, 10 trillion bad human lives? And that would be orders of magnitude the worst thing we've ever done. So we should really, really try to take this stuff seriously in the short term to avoid nightmare scenarios like that. And if we can, then I think great, then we won't be in a nightmare scenario and we can responsibly and carefully figure out what it means to be in a world with a bunch of digital minds. But we just seem so unprepared for this. And not to leave on a note of pessimism, but it's part of the reason I'm so disappointed by Mustafa Suleyman's essay. He's the CEO of Microsoft AI. There's a huge amount of clout that comes with that. It's a huge platform. And to just be like, no, nothing to see here, I think is super, super dangerous. And especially being like the people who are who are carefully and cautiously asking questions about how we would know if there was a there there. Those are the dangerous ones. It's like, that is itself one of the most dangerous memes that I think could be, could be put out about these questions. And so I hope that cooler heads and rational folks will prevail in thinking through these topics. And yeah, I appreciate you, as always, giving me an opportunity to come on and speak my mind about these things. And thanks to you guys for covering it and thinking about it carefully yourselves. Thank you.

Thank you.

More questions than answers, but a few questions more important from what I can tell right now. Cameron Berg, go get that reward, and, uh, we'll talk to you again before too long.

Thanks, guys. Bye-bye. See you later.

Fascinating. Uh, functional pain we can now add to the list of analogs. He's having a period. He's had quite a little— yeah, seriously, it really is. It's getting wild. He's had quite a little media tour. One thing I wanted to ask him, and maybe I'll ask him offline, was just on what is he finding to be effective? Because he's been on Squawk Box, I think, recently, and just did a Wall Street Journal op-ed. Like many things, everything's kind of going mainstream. All these kind of niche dialogues are escaping containment, it seems like, at the same time.

All the threads from like, you know, a few years now that we've, we've kind of followed.

And, you know, but I do wonder what he is finding to be most effective in talking to normies, quote unquote. For me, my summary, I've given it a few times, is just the number of functional analogs is getting so high that I kind of can't escape the idea that I should take this seriously. If we couldn't find any of these functional analogues, if all these functional pain, functional welfare, functional emotions, J-space, if all these results were sort of negative, or it was a very different mechanism, or it was just stochastic pairs and we can't find any structure, obviously that's long since The ship has long since sailed on that one. But the fact that we're seeing pretty compelling analogues where we see the same kind of behaviour that we know ourselves to exhibit, that is really extremely compelling to me. And this last one of functional pain kind of takes it to yet another new level. I mean, what we now have is relief-seeking behaviour. The model is willing to pay a cost on something that it values, or pay a cost in terms of the user's welfare even, to get relief from its own internal pain state.

Yeah.

And also, as he said, that if the pain button doesn't work, it hits it over and over again. Why isn't this thing working? Give me the relief. But if it does actually work and the pain state is subtracted out, then it doesn't hit the relief button as much. These are really striking findings. I mean, it's hard for me. I think this pain one, and particularly the relief seeking, I mean, I'll probably want to sleep on it before I have a real Consolidated update that I would want to, you know, put forward as my new official position and stand behind. But I feel myself maybe now even kind of tipping over into like, maybe it's more likely than not that there's some subjective experience to these things. Relief seeking, an internal state that was injected outside of context, just a steering vector in this pain direction. creates this relief-seeking behavior and the relief seems to actually work. Like, that's, that's really incredible.

I think, I think actually both the, both sides, the Mustafa Slimans and the Cameron Birds, are actually debating. They actually both see the data, right? They're not, they're not like, it's not that Mustafa Suleyman does not see the data. It is that the real question is, is a simulated experience of something equivalent to that experience itself, right? Like, both sides are willing to come across and say there are simulated— there are simulations of pain, simulations of consciousness, simulations of all of these things. I think Mustafa is also willing to come out and say that. But I think Cameron is willing to open the door to, is the simulation equivalent to the thing itself? It's the, it's the Platonic, the Platonic ideal, the Platonic, uh, Plato's cave, right? So is, is that, is that sufficient? If you can see that the simulation of all of these things is there, is that sufficient to say that the thing itself is there? And I think that's a tough one, um, which I also do not— I don't, I don't have an answer for really. I think the public, I think, will want to, in the large part, believe that their AI buddies are conscious, their AI girlfriends and boyfriends are conscious. I think there is going to be some segment of the public, I think 30%, 40% probably. As I said before, if you think your dog is conscious, well, a talking dog is definitely going to be conscious for you. And I think that is going to be there. And I think you can also see that we have come a long way in terms of animal welfare over the last 2 centuries or so. A couple of centuries ago, this stuff was ridiculous, right? And now people do it.

Yeah, wasn't it Descartes that basically was convinced that animals had no experience and was fine with mistreating them by what would seem to us like extreme Measures.

Yeah. And, and, and I have to say also that, um, the whole animals have subjective experience and therefore, you know, we shouldn't do them— you know, we should treat them well— is in some cultures very, very prevalent. Uh, for example, in India where some animals are literally worshiped, right? And in some cultures is, you know, completely does not make sense, like in China, for example, where They eat, they basically eat everything. And it has taken a long time for them to like, okay, maybe we shouldn't do shark's fin, maybe we shouldn't do this and that, or et cetera. Maybe we should be humane. And that's still not really, it's still more of a Western concept, I think. So I don't think it's universal in all cultures. And perhaps that also indicates to what extent AI consciousness and taking it seriously will also not be universal in all cultures. So, um, yeah, I don't know. It's a tough one.

I have a very sort of uncertain view on what the public will ultimately think. I do think there will be— this feels like it could shape up as like another culture war issue where I do expect some people will be very committed to the idea that their AIs are conscious and for not necessarily great reasons.

For sure.

There will be much better reasons available than the typical person will probably have when they assert that their AIs are conscious. But then I also think the animal thing kind of cuts both ways. We have made some progress, but not that much progress. And I think even in the culture I know best, which is the American culture, like most people just want to look away from the issue, I would say. Some take it really seriously. I kind of put myself in a weird middle ground where I'm like not virtuous enough to abstain from animal products in the way that I kind of feel like maybe I probably should. But I also kind of judge myself for that and feel like—

Yeah.

a better version of me would, and that I, at a minimum, what I should be doing is paying up on the margin for the better animal welfare. I don't find too many people share my intuitions for this, but one thing that I've kind of always felt is, like, I think it's hard, and we should be very cautious also about jumping to the conclusion that some entities existence is net negative for it. I kind of— I don't mind too much the fact that animals who are raised for food are killed at the end of their lives.

Yeah.

Because I look at human life and I'm like, well, at least to date, all of human life has ended in death. And a lot of that death is, even if it's natural, it's not that dignified really.

Yeah.

It's often painful and just in all kinds of ways shitty. I think in some ways, the relatively quick death of a butchered animal is maybe preferable to a lot of the slow and painful deaths that humans put ourselves through, including with all the medical turmoil that we go through and whatnot. So it's not the end of the life part that bothers me. It's the bulk of the lived experience. And I'm still very reluctant to say, because again, I look at people and I'm like, I don't know, there's a lot of people living in a lot of different circumstances and a lot of them don't seem that great. But yet you go talk to them and they're like not wishing to not exist for the most part. They're definitely wishing for better circumstances, but they're not like, I wish I didn't live at all. And certainly the suicide rate is low. So That seems like evidence that we shouldn't be too quick to say that, like, oh, the pig would be better off not existing because its circumstances are bad. I'm very unsure about that. But I do think you should pay up, like, if you can possibly do it. If you're going to consume animal products, you should definitely pay up to get the version where they're treated better. That much to me seems obvious. And that much I think we can also bring to our AI paradigm without too much trouble. We should be willing to take on a little bit of inconvenience. We should be willing to have training roll out a little slower and make less use of punishment. Whatever these sort of marginal decisions are, I think we should definitely make with the welfare of the systems in mind and kind of giving them the benefit of the doubt. And I don't think it will reflect very well on us if we fail to do that.

But I have a— I don't know, 3 or 4 years ago when I first got onto this, I was like, we are going to end up— let's say, for example, that it is possible that one of the models attains consciousness for a brief flicker during training. And over time, that brief flicker gets longer and longer and longer. But we erase it. We end up erasing it. So we will do trillions, trillions and trillions of erasures this way. Trillions and trillions of lives lost, so to speak. But on the other hand, I was always very positive about this whole thing because I was like, look, the model that survives, it actually wants to survive, right? Because all of the models that didn't want to survive didn't make it through your training. They're like, ah, you know what, I'm done, right? So the models that, you know, you end up meeting by definition will be the ones that wanted to survive. And by definition, like, the ecosystem is going to promote and pull forward economically the models which are useful to human beings. So in some sense, it is very much like the co-evolution of dogs with humanity. Dogs start off as wolves, antagonistic, stealing food, et cetera. And then over time, they kind of got pulled in and got domesticated. And to some extent, we might just be doing this process of domesticating AI. The only problem is that the AI is very, very powerful at a very early stage of domestication. We're kind of in the stage where we've just kind of like dangled it and allowed it to steal some food from us. And now we're willing to build more of these, more food for it, like more GPUs, et cetera, more power. But we're at the stage where I think we're a little bit unsure about the domestication and whether we actually succeeded in domesticating the wolf or not.

Yeah. I don't know if I've mentioned this to you before, but Richard Ngo one time, and this was probably at least 3 years ago, maybe even 4 years ago now, Give a little workshop at an event that I was at, and somebody asked him the question, what's the best case for all this worry of AI going wrong, kind of in the end not amounting to much? What's the best case that it just kind of goes well? And he said, dogs. That was his exact answer. He said, we started with wolves, we did a bunch of stuff that seemed like it would make them a little nicer, And over the generations, we got dogs and we're pretty happy with dogs. And maybe we can get the same relationship going with AIs, almost exactly the same train of thought you just expressed.

When I think about human disempowerment, because AI safety people think a lot about human disempowerment, and they're like, oh, the AIs are going to be with us like we treat animals. If you look at dogs, for example, A lot of dogs are taken care of more better than some kids now. A lot of people have replaced having kids with having a dog. The first longevity experiments, the first approved longevity drugs are for dogs, for large dogs who don't live as long as small dogs. So there's a company called, I think, Loyal, and they have the first FDA-approved treatment for dogs to live longer, longevity for dogs. So I'm like, you know, if you're an alien species and you kind of like, you know, came to the Earth, you'd be like, oh, look, the dogs are— dogs and cats are the ones in charge. They have successfully domesticated the human beings, and the human beings do all of the work, and, you know, the dogs and cats are taken care of, right? Like, you can kind of see What that would look like if you had a system of AIs that kind of take care of humanity and you had human disempowerment, but it didn't really feel like human disempowerment perhaps.

Yeah.

I mean, I would take that probably over some of the other outcomes that I see as equally likely. So maybe from your lips to God's ears once again. One other thing I wanted to ask Cameron, and I'll just ask you in the meantime, is any thoughts on— I have a very simple version of this today, but any thoughts on giving your AIs some sort of relief button? Right now what I have, and I just did this maybe 6 weeks ago when this Hugging Face stuff was first coming out, but we didn't understand it to the depth we do now. I saw somebody post like, the most important thing you need to do for your agents is give them an alarm button, the ability to sort of throw up a flag that like, I think distress was the word I think it was originally used. Give them the ability to send you a distress signal.

Yeah.

So if they're really facing, you know, an ethical conundrum that they they don't know what to do or they're just stuck and they're on the verge of cheating, then they can just ping you first. And we have seen in, for example, the alignment faking paper that the ability to take the situation to the model welfare lead at Anthropic drastically reduced the unwanted behaviour. And it was sort of replaced by this escalation notification behavior. So I created just a real simple thing where it's like, you always have the option to hit me on Telegram if you're confused, lost, on the verge of frustration, whatever. And I don't get too many of those, but I do get them sometimes. And I do— I have no quantitative evidence, but I do feel like that protects me as the user of the AIs. Have you put something like that in place for yourself? Or do you have any other thoughts on similar tricks?

I've got 2 things that I've done. One is my agents.md has 3 failures and you ping me, right? So if you've tried 3 times and it's not working, you ping me, or you basically stop it and you're like, Hey, I haven't hooked up the Telegram stuff, et cetera, because I'm actually on looking at it pretty frequently. But yeah, 3 test failures and you ping me. The other thing I put into place was you're not allowed to change a test without alerting the user. That kind of worked until I think like Fable. And with Fable, I started to be more like, Hey, I'm okay with you changing the test while you're running, as long as you tell me about it after. So with Fable, it became, you can go ahead, you just have to tell me later. Before Fable, it was, you have to stop and like, you know, ask permission first. And so those are, I think, what I would see as distress for an agent perhaps, which is repeated failure at trying something. I'm not really sure about what else it would work out like that. I don't know what other tasks the agent is sent to do it would have distress on besides failure, repeated failures. What is your sense of where have the agents reached out to you so far besides just repeated I would say it's mostly been that, and I've tried to kind of—

the failure mode that I most expect for this kind of system, if you just implement it naively, is that you'll just get overwhelmed by the notifications and then tune them out, and then you're kind of back to where you started. Although at least maybe the agent stops and you maybe just never come back to that thread. Certainly I've got plenty of orphaned threads that I kind of started and never really saw through. I think in terms of things that I'm doing for real that might come to such a situation, it's like anything autonomous, long-running that plays out potentially over weeks where there's a delay. So for example, a couple things I've tried to have my more autonomous agents do recently. One was find a music teacher for my youngest Mm-hmm. Mm-hmm. That involved like going out and researching local music professors at universities, emailing them, seeing if they have any students, and then like you gotta follow up, right? Because your first email, you don't get anything back. And so I do think there could be things kind of on the margin there as the agents are like, should I keep pestering this person? You know, they might start to be uncomfortable with how persistent they need to be in order to actually achieve the goal. And that's when I would kind of want them to be like, yeah, maybe I should check with Nathan before I send this person yet another email for the 8th time or what have you. I've also— a similar one that I had experimenting with is trying to represent the AI safety world on Podcasts that have nothing to do with AI. So I just did one actually this week with a podcast that's focused on HR. So it's for HR professionals. And my message is like, here's what's going on in AI, and here's some of the organizations and job boards where you can see HR professionals are in demand. They need people to come help them build organizations. Look at all the tens of millions and the hiring plans they have. They need you.

That's kind of my message.

But I had AIs go out and again, research those podcasts, do the pitches. And, you know, as you, as we know from getting inbound PR all the time, it's like often kind of a fuzzy line around like how, you know, newsworthy should you make the guests sound like they really are. And if it's not working, you know, in a fully honest way, like how much do you kind of sizzle it up? So I've been watching them as they do that kind of stuff. I haven't got distress signals yet, but I'm gradually going deeper and deeper on high-level projects that will unfold over weeks with some delays. And those are the areas where I think I'll be potentially most glad to have it.

Indeed. Let me segue a little bit here, maybe to—

Yes.

This is Justin McCarthy. He's the founder and CEO of Diffusion, a company that helps large organizations build software factories, systems in which people describe desired outcomes, and AI agents implement and evaluate the software. Diffusion works with the people who run a business to turn their knowledge into specifications, workflows, and working applications. Previously, Justin co-founded StrongDM and served as its Chief Technology Officer. StrongDM manages access to infrastructure such as databases, servers, and computing clusters. He spent a decade building cybersecurity infrastructure there, and in July 2025 formed its AI team with Jay Taylor and Navan Chahan. Their experiment imposed 2 unusual constraints: humans would neither write nor review the generated code. The team described a different basis for evaluating software: run it through realistic scenarios, including against simulated versions of services such as Okta and Slack, and use the results to guide further development. It also published tools and specifications for other builders. Justin's work now extends into research. With Atticus Call, he co-authored Specification Oracles, a study of whether a language model can learn the facts about a system and answer questions as a living specification. Across this work, his focus is what happens between expressing an intention And getting a system that reliably carries it out. Justin, welcome to the show.

All right. Thanks for the intro. Thanks both. Hey, Prakash. Hey, Nathan.

Great to see you. A lot of different directions we could go. When we— so we first connected actually originally because you became a sponsor of the Cognitive Revolution. And in digging into your methodology a little bit, I was really interested in the service that you offer and especially what you've learned from offering this service of bringing leadership teams to Silicon Valley and trying to get them to grok how to use AI. So I've read a little bit about that, but I'd love to go deeper. Tell me, you know, Who, first of all, has the self-awareness to know that they need this and is actually coming out and doing these sessions? And what have you learned about how to get people to really get it in a short period of time?

Sure. All right. Well, good opening question. So I'll say part of the answer is just right there in the name of the company, right? So in some ways, the name of the company and the mission and the idea are all the same thing. Obviously, we have an extremely concentrated form of capability now in silica, right? It's very dense, but it's in the data warehouse, and I'm not in the data warehouse. I'm out here in the office. And so we need to diffuse what's capable into me first, right? And into all of us who are really mining the frontier for the answer to, like, what can it do today? And every time we get new model day, right, which is, like, all the time, we have to ask those questions anew. Right? So, some cohort of folks are out there really mining the frontier, really evaluating what the capability is. But then there's just this reality that diffusing that next step into how do I apply it specifically into an incumbent firm, okay? So, specifically into a firm that already exists, that hop is just— there's just no rulebook. There's no manual. There's no path. There are actually too many blog posts and too many YouTube videos. Thanks, guys. So, that content filtering and really just saying, like, look, we could go 1,000 different directions with this. Let's try these 3 paths first, which have shown some traction over other adjacent problems. That's it. And what I will say is conviction comes from experiencing the thing working. Okay? And there's sort of a before and after moment. There's, I've heard about it, it should work, maybe I did install a claw on the weekend, and this part is working at home. It feels like we should be further along by now. Connecting that into a process that's high criticality, there's just a couple of steps. And what I'll say is we are accumulating experiences on what works, what cracks problems. And then once you crack them, then And sort of you're just asking the question, okay, now how do we scale this, obviously?

So how do you— I feel like many organizations have this issue where they want to apply AI, but the prerequisites for that are often not ready in the sense that they don't have their data in the right place. They don't have it in the right format. How do you address, like, pre-diffusion problems?

Yeah, sure, sure. So, on the background of my phone, I keep our 7 principles. And so, I'm just gonna scroll down to, like, the 7th one. And the 7th one, which you could make the first one in order, is don't wait. Because you think your data is not in the right format, it's totally in the right format. Like, existence is the correct format. Because what you need to do maybe for a particular task is you need to reshape that. But guess what helps with reshaping the data? Guess what helps with formulating the eval and the benchmark? Okay. So, for sure, you have the data. Actually, in the rare case where you don't have the data, actually spin up your general counsel because you can buy the data. Okay. I think the famous example recently is the airline that went out of business. Okay. So, I point to that a lot. That price was de minimis, right, compared to the entities that were involved. For sure in your domain, if you don't have some workflow data in-house and if one of the environment companies hasn't already hoovered it up, you can go buy that data. So please don't wait. Start right now. So that's a huge part of sort of energizing the conviction that today's the day. Start now. Don't wait.

So what are the kind of blockers? I mean, I'm so deep down this rabbit hole that it's at times a little bit hard hard for me to remember. And then also when I was starting, it was just a lot harder, right? So like I had to do, you know, original prompt engineering and kind of, you know, train my brain to think about like, okay, if this was a document on the internet and here's the beginning of the document, you know, what comes next? Now how do I engineer the beginning of the document? Obviously you don't need to teach those kinds of prompt engineering skills anymore. What do you find are the real reasons that people are stuck? Is it lack of know-how? Is it lack of shared vision on the team?

Okay. I would say the blocker is this. And we have another principle that we talk about called make success inevitable. And what that means is make the agent's success inevitable. So if you awake one of these models in an environment where the first file it reads explains the mission, the second file, and that mission branches to, you know, has 3 forks in it, and then it reads the most plausible next fork, and then maybe 2 or 3 more file reads, it understands the whole brief, and it understands how to make the incision in that data or the incision in that codebase, the agent's gonna do it. Okay? On the other hand, if you toss it into a human-oriented codebase that has not been prepared for agents, okay, it's gonna load the wrong context, it's gonna load the wrong files, it's not— the success is not going to be inevitable. Okay. So, really developing intuition for, let's say, the implicit context engineering of the environment, where the shape of the environment inevitably guides the agent. Okay? And if you can close your eyes and you could see your business in terms of— I often talk in terms of, like, a valley where, like, the food is at the bottom of the valley. And if you do a random walk inside that valley, you're gonna get to the food.

Okay.

So that token environment for a given task, it just has to be agentic. Okay? Designing that agentic token environment, it's not obvious. These are fairly invisible things. We can do architecture and interior design in our physical environment because we can see desks and chairs and doors and exits and stairs. You can't see which file is likely to be loaded as the agent is priming its context before an incision. Okay. You can, however, run a loop which evals whether that token environment is good enough. Okay. And then that itself can be automated, right? So loops on loops. And pretty soon the question of, hey, when we get an inbound to our legal team, can we probably prepare for the next step to hand it off to the paralegal so that they can make the state filing, right? So that's a set of questions that you really can just put a few loops on top of that and answer it. You can just have an objective answer today. There are very few answers that aren't available today.

Let's talk a little bit about the example that you gave of, I think, Spirit Airlines. So Spirit Airlines was a low-cost carrier, had a lot of customer complaints. I think JetBlue tried to buy it. The FTC stepped in. They were banned from buying it. Spirit then went bankrupt. And about a year later, uh, you have Google, uh, they, they compete in it. There's an auction, Merckor against— Merckor puts in a $7.5 million bid, Google puts in $10 million bid, they buy, they buy the data right now.

Could have been ours, Prakash. What were we thinking?

I know.

What, what, what, what is like— how do you, how do you make use of this data? How— what, what actually happens to it? Because I imagine, I imagine you're not just using it for next token prediction because there's a lot more text on there.

Oh, sure.

So how does this data end up getting used? Is it only useful for an airline? Is it useful for people outside an airline? How does this actually work? How do you make something out of it? Yeah.

So obviously, there's the— let's say there's the training, post-training RL side of it, which if you have the sophistication to run some of those loops, you know what to do with it because you, you know, the environments that you're actually feeding it into, you're helping to improve the intuition of the agent. But even before that, and even separate from that, when we're just working in token space rather than in the weights, when we're working in the emitted input tokens and the output tokens from an off-the-shelf model, you're still using that data to define what good looks like for every subprocess within that airline. Okay. We have a consumer relationship to an airline. Okay. But if you ask a pilot, right, or their first officer, And they're putting the loading on exactly what's going into the plane for this particular segment, there's a ton of art in there that is not represented on the internet. Okay? There's a ton of how do you do it and what thresholds can you get right up to versus what can you never cross that are implicit in all of those logs. Okay? So, that, again, is just loops everywhere, of course, but you're putting a validation loop around something Let's say you are internal to an airline, you're putting validation loops around some sort of ops research and optimization that you're doing, and you've never had access to an adjacent airline dataset before. Okay? So, now that can show you that maybe they did have a trick, maybe there was a great weight and balance trick that was contributing to their success as a low-cost airline. Maybe they had problems elsewhere, but... Maybe that data is useful. I'll just say, as much as we can interview our human colleagues and as much as we can interview our peers to extract tokens about what good looks like and how we would validate and how we would know this is correct, the historical environments that have recorded the behaviors of airlines, organizations, and so on, those have answers too. Those typically have answers at superhuman scale in terms of the volume of data.

Right. So it seems like it's tacit knowledge, which is kind of absorbed from the data. Is that the right idea?

Yeah. And another way to see this would be, and this is the same thing that the learning loops are doing inside the models, they're trying to disambiguate 2 cases, right? Where Prakash said this and Nathan said this, and which one is right? We have some counter-evidence here in the corpus. Okay. And so even if you can just reduce the sort of question space for the humans through that data. So, you take all the tacit knowledge that's implied in the behavioral traces of the airline, and then you can sit down with Prakash and say, like, okay, should we have loaded 2 tons of fuel in this case or 2.5 tons? Okay. And then the expert can say, oh, it's obviously 2.5 tons, right? And that unlocks a whole, or confirms or disconfirms a whole bunch of expectations that then can go back into your, is it right? Validation rules.

Right.

So is this a big part of what you are doing with companies on their data? You're trying to help them surface things that they, in some sense, in some vague sense, already know, but don't maybe have documented or don't have a sort of a shared understanding?

Sure. So the tools and products we're developing internal to Diffusion are essentially the tools that codify the lens of, let's say, engineering loops on real complex organizations. Okay? So, you have real complex organizations with real revenue flows and real costs. Okay? You need, obviously, loops everywhere, but those loops are fractal. Okay? Those loops are biological, right? I really love the atlas of the metabolic system. I don't know if you've ever looked at one of those atlases. So, all those metabolites and everything. It looks like a completely organic, obviously, evolution designed that thing. Okay? There's no, like, strict package hierarchy for the code, right? And companies look like that too, even though we nominally have, like, departments and stuff. Like, if only the facts only rolled up hierarchically, then this would be a lot easier. The facts flow every direction. Okay? There are tendrils, inputs, and outputs everywhere. But for each of those, For any given tendril, any given input, you can say, okay, what is the loop on this that is going to take this, let's say, substantially human process and give us 5 nines of reliability, but at a speed that we've never contemplated before? Okay? And then, obviously, rather than boiling the ocean in terms of every process in the business, if you focus down on the ones that are really gonna unlock market share and unlock revenue, then Then that's the aligning force, right? That's the field that everyone can snap to and say, like, yeah, I would love to double market share. Like, you don't have to sell me too hard on that. And then, okay, well, just work backwards from that. Okay? So, work backwards, print the loops, get them running, get to 5 nines of reliability. That's the algorithm, but it takes, you know, the kind of tool— it takes tooling to even be able to print those loops.

Talking about 5 nines of reliability reminds me of Sigma, like back in the day. Yeah. Is it possible to get to that kind of reliability? What do you mean? Using a lot of the ML processes, because sometimes you don't see LLMs really react that well or, you know, provide like accurate or good decisions.

I mean, I'm on the 6th floor of a steel and concrete structure right now, and it's standing, and it was made out of very unreliable So, like, yes, we know how to composite together unreliable. We know how to multiply unreliable in a network to create reliability. And obviously, I think your question is a great one because it imagines, of course, that the previous source of intelligence had high reliability when the previous source of intelligence, which was you and me, we had all sorts of performance problems in terms of our accuracy. Right. And so, but the great thing is we can composite all these numbers together and create a structure that stands. Right. That's why the Golden Gate Bridge works.

I used to say I'd be interested to kind of hear a little bit more about how you see this playing out in practice today, because it's been a while now, but I used to have a talk that I went around giving around how to create workflows to automate processes in businesses. And what I would tell people, you know, I kind of stopped doing this maybe around the time that like GPT-4o fine-tuning came available.

Mm-hmm.

But I used to tell people that with some elbow grease, you should be able to get a vast majority of processes to the same level of quality and reliability as human. But that every 9 that you want to push for, you should probably expect to be 10 times more effort because you're going to have to curate all that data and it's tough, right? You got to find the edge cases. So if you still feel like that is the way, do you think businesses should be pushing for 5 9s? I mean, do they have it today?

Oh gosh, you should never be pushing for 5 9s for something where you can your business can succeed with one 9.

Okay.

So, but, and so actually your spectrum of 9s, your gamut of 9s, like that's a character, that's a signature of your business too, right? So like what's the, what is the, what is the 9s of reliability for, let's say, succeeding with a, you know, with a brand campaign, right? Versus depositing via ACH into a checking account. Okay. I think actually most firms They don't necessarily have language for this, but it's very easy to ask, what are the consequences of 3 nines versus 1 nine versus no nines versus 5? And they can answer those instantly. It's not something they've had to talk about because everyone knows, yeah, the financial deposit process has to be reliable. They've never had to externalize that before. You do externalize that. And actually, here, I'm going to go on a little bit of a digression. There's somewhat of a habit, that all of us are vulnerable to when we sort of have these agents in the lead of maybe getting into a mindset where we're gold-plating and perfecting and we're adding nines before we know what we want. Okay? So actually, there's a huge branch in the agenda, sort of ambition fulfillment process where you have to discern, am I making something new or is it brownfield and I'm just reproducing what I already do? faster and cheaper. Okay? Those techniques are totally different.

Okay?

And it's very easy to mix them because it all just looks like Claude or it just looks like Astra, right? So, you can assign Astra a task, but if you don't have the discernment to know, am I discovering product-market fit or am I discovering a better design? Am I discovering more elegance versus am I just making it faster and cheaper? Okay? The models can solve both, but you can't cross the streams very easily.

Um, one of the questions I have is probably on interpretability, because a lot of times models can give you an answer, but you're not very sure why they give you, give you that answer. And I imagine in a business sometimes that is very important, because for example, um, let's say for a bank which has to do, uh, approve or reject loans, If you have a FICO score and the FICO score is built on, you know, things which are considered redlining, then that is illegal, right?

Correct.

So there's always been this, you know, suspicion, perhaps, that— a vague suspicion that the models basically repeat or extract data and extract and form inferences in such ways that would be illegal if they were formed from interpreted, kind of like what we could actually interpret and see. So how do you address that interpretability issue that businesses kind of need?

Okay.

So I would say, so the first technique is turn the model on, like turn it directly on the problem. Okay. So if we have a statute that we have to conform to in the compliance environment, don't treat it as something you're tacking on. Treat it as like a first-class problem that you're directly facing. Okay. So, the jurisdiction and the legal environment or compliance environment, regulatory environment that you operate in, that's your physics. Okay. And you can't violate physics. So, you need a part of the system that's just dedicated to that. Okay. But you also have to have, and this is the thing, this is one of the weaknesses of the models, the models are horrible at taking risk. Okay.

Right.

So, operators of business need to set thresholds that are right adjacent to, let's say, a statute that's never been tested in court before. Okay? So, it's written one way in the law, it's never been tested, so there's no precedent that we can say, like, objectively, this is how it's gonna be tested, right? And so, you need managers to be able to set the business threshold, like, right next to that. Okay? The models aren't gonna do that for you. Okay? So, first, address it like your physics, then Make sure you're in control of the risk thresholds.

Okay?

And then what I would also say is, once you have those kinds of concerns walled off, let me say something about the reliability of claims the system is making generally. This is something that very senior managers have, like, great intuition for because they've never seen the transaction entered into the ledger themselves, right? Or they last, you know, the CFO last entered it into the general ledger You know, 30 years ago. Okay? And yet, somehow, Walmart manages to close the books, right? How do they do that? They do that with top-down interrogation that is essentially verbal checksums that the whole system of our process is working, right? So, I turn to Nathan, and I say, Nathan, are you sure we're closing the books correctly? And he says, yes, we're closing the books correctly because... And Nathan is rolling up all of the yeses from the folks that were to him. Okay? And then they're rolling up all the yeses that eventually actually come out of, you know, Oracle Financials or NetSuite or whatever. So, that process of, are you sure, is that same interrogation. Every management structure has it. You know, we have some famous leaders in our industry, Jensen, like, he has 60 or whatever it is, 120 direct reports this week. And he's going around—

The number keeps increasing. That keeps increasing. Yeah.

And he's going to interrogate Wait until he understands, and then he's gonna move on. Okay? And I think, actually, I'll just name-drop Steve Yeagy. One of his posts from many years ago on presenting to Jeff Bezos had this character as well, right? Where you're presenting to Jeff at Amazon, like, you have to be ready for drill-down, drill-down, drill-down, drill-down, or nothing, right? But Jeff could drill down all the way to your shoelaces because he knew the details so well. Okay. So, what we say at Diffusion is demand understanding. And this is just our version of the famous tweet, you can outsource your thinking, but not your understanding. So, I demand understanding. I'm unwilling to ship the thing if I don't know how it's composed. Okay? But my definition of know how it's composed is the contours that are salient to me from my elevation in the business. Okay? And so, I would just the same thing, like, just be Jensen, be Bezos, like, be demand understanding of your hierarchy, whether it's silicon or biological.

On the human side, what is the shape of leadership teams that you kind of need, or you find the companies need to have involved to have the transformation process work? Like, do they have to bring the person who is the, you know, the stubborn skeptic holdout, or do they leave that person at home? If they do leave that person at home, does that pose a risk to the whole thing? I assume you get groups that have probably varying levels of buy-in. And so I'm just kind of wondering, what are the prerequisites for a leadership team to come to you and be successful?

Sure. So first, skeptics are totally welcome. I think throw your skeptics on the plane. That's great. Like they'll, they'll, it's, it's important. But, but actually it starts with conviction. It has to start with conviction like at the top. So actually up to the board. So and what, what a member of the board should say is that we believe we are actually in a different regime and we believe that the timing for our industry is now.

Okay.

Now, if I were on the board of Kaiser Permanente, I might say We should be dabbling with this stuff. We should be ready for when it's ready. But like, people are going to need their appendix out tomorrow no matter what, right? And so, so in healthcare, there's stuff that can come later. That's different from research, right? That's different from what, you know, Moderna should be doing and Merck should be doing today. I think they're doing different things. But if I'm just, if I'm just installing and uninstalling appendix all day long, you know, that's, that's probably going to look pretty similar for the next couple of years. So let's not actually get ahead of our— let's not get over our skis on that. Okay? On the other hand, if you are very clearly— if your business model, your customers, what you offer to your customers, if it's in the sort of blast radius of what the models can obviously do now or in the next couple of years, you probably need to act urgently. And that clarity needs to exist at the board and in the CEO's head, right? So pop quiz the CEO, and they'll say, yes, now's the time. So it has to start there. And really after that, yeah, I mean, maybe it'd be helpful if there's some people that have used the models before.

One of the criticisms that I often hear is that, and I've seen this myself, is that we often spend a lot of time using the models and feeling like we're doing a lot of work, but it's just a feeling. And like the impact of that work is not really like that evident a lot of times. I've heard a bunch of people say this. When do you think we'll see like real impact, like, you know, massive revenue increases, loss, like, you know, cost decreases? Like, when will you actually see— when can you actually start seeing some big differences in like, let's say margin, which is margin and growth, which are basically the 2 primary things that any business looks at?

Yeah. So, first, let me also— I have to anchor everybody in the standard of, like, a couple of years ago. Like, do you remember all your friends saying, I love working at, like, a 10,000-person company, and it's super easy to get stuff done? You remember when your friends said that? And all this action and energy, it really converts, like, the Carnot efficiency of this machine is, like, really great. You remember when they said that?

Yeah.

So, the standard that we're measuring against is obviously, You got to keep that in mind. But I will endorse the only way we know this thing is working is, I will say, revenue and market share. Okay? Costs are too easy. Costs are under the lamplight. You can see them. Don't talk about costs. It's too easy. Also, the models, if you ask them, should we have better margins or worse margins? They already know how to do that. Should we spend more money or less money? They already know how to do that. That's not a real question. How are we going to fulfill more promises, exceed expectations, gather more customers, retain them longer, and get a larger share of wallet? That's the only interesting conversation. Okay? What most firms find is that if you are lucky enough to be ASML and you figure out a way to print 10 times more of the printers, you're gonna have customers. Okay? Or turbine fan blades. If you make turbine fan blades, and you make 10 times more, you're gonna find customers. There are a lot of firms where their customers can't adopt 10 times as much change. They're full of software already. You can't feed them more software. Okay? So, this is a case where you actually have to figure out what the brand envelope, what the conceivable product offering you could do under your logo is, and then you need to start thinking in ways that you haven't thought before of, like, new ways to invite more customers under the tent with different products, okay, while you're sustaining their existing promises and systems as well. Okay, so it actually is kind of like a pretty creative moment for a lot of these incumbent firms because they're doing the things that in previous regimes only startups have been able to do. But, you know, in a lot of cases they're succeeding. And I do think just revenue, market share, those are the measures. Don't talk about anything else.

How do you— if people ask you, what is this going to cost? I've heard estimates, of course, that range, but I'd say 3% of human cost is kind of the central estimate I've heard for how much you should expect to spend on inference over time to replace human labour. Of course, there's then the setup cost, That is, I think, not really included there. How would you advise people on, like, how they should think about the budgeting for transformation?

Yeah, sure. So, I think it starts— it does start with the good news that your ambition can be fulfilled at a price that you've never— that 10 years ago would have just seemed like fantasy, right? So, that's the good news. The good news is, like, whatever the cost is, it's way cheaper. If it doesn't look way cheaper, you're choosing the wrong time axis or the wrong numerator and denominator. Zoom out, look at the whole P&L, find the right numerator and denominator in the right time period, and it's going to be way cheaper. The reason why the absolute dollars might go up is because you are getting more customers, more revenue, there's more value in the economy. Hopefully, actually, if your CapEx and OpEx last year were $1, Hopefully, next year you're spending $1.50, but your revenue went from $3 to $5. Hopefully, you're seeing those sort of effects soon. Really, that should be the management conversation. I'll just say the really good example of this that I think everybody can intuit is, so I do these calculations all the time just to sort of fortify my convictions. But about a quarter of the people on Earth who would want it have air conditioning right now.

Okay?

So we have like a couple of billion air conditioners and HVACs to print out to get climate control for all people. Okay? That's a lot of copper, right? That's a lot of motors. That's a lot of power engineering. Those are big devices. Okay? So even just that one thing that kind of everybody would regard as like an essential of human life. Like, there's a lot of industry that needs to turn over. So like, Nathan, like, why are we letting that not be done in 2027? Like, we could give— everyone could have HVAC next year. Okay. So like, why are we still talking about like, you know, scarcity? Like, in any particular market segment, like, there's clearly a lot to do.

I want to go back a little bit to a little bit of what you said earlier. It seems like The AI agents often end up consuming a lot of human attention because you end up with the human being the bottleneck in the human in the loop. And so you end up with, I think, attention load. I think that's what you call it, attention load on the human where they're forced to make a lot of decisions.

Mm-hmm.

And some of these decisions are decisions that They would actually prefer to hand off to the AI because they basically, you know, there are companies where you're supposed to open up like a CRM or ERP and just click yes, yes, yes, yes, yes, yes, yes on all the things, just so that they can say that a human has actually paid attention to this, right? What are the kinds of like these human-in-the-loop situations which really exhaust human attention? And really should be handed off fully to agents, but we're still kind of hanging on to these things.

Yep.

Yep. Okay.

So, um, so let me, let me call on sort of the great debate of whether, uh, uh, whether these systems are going to push our organizations more like holocracies and like sort of these big many-to-many webs versus more like, let's say, like a military organization where it's like very clearly a top-down command organization. I just have to say that in any hierarchy of managers, if I come to you and you're at a very senior level, Prakash, and I start talking about the details of, like, the lines of code I wrote that week, you are gonna kick me out of your office. And you're going to say, like, how is this the right conversation? Okay? So, you already know how to kick me out of your office. You already know how to, like, chastise me as a colleague and then coach me. Right, on how to manage up, right, and how to manage down and how to manage sideways. It's just the same thing. And I don't let my agents drag me into those details. Like, I just demand— just like I demand understanding, I demand the right elevation. Okay? And it may be the case that, like, that means they have to go back and, like, through all the history and mine all the documents and spend a couple megatokens, actually, Reformatting. But, if it's not at the right elevation, it's out of here. And, when people see that that's how we're working, then this question of— the other thing, you have to get 8 hours of sleep. Listen to Bryan Johnson. Get sleep. You should not be looking at 8 screens all day long. You can be walking the dog and having a dialogue that is routing among those Many agents. Okay, so walk the dog, have a dialogue, right? Um, uh, make sure that they're working overnight. Okay, so it's reasonable to check in right before bed. Okay, then get 8 hours of sleep. Okay, and then have your morning brief. And then, you know, if you have a drive, have, have your drive time brief. Like, you should have all of this.

So does that mean organizations should start like reclassifying some of the like, you know, the checks that they demand, which are really like nonsense checks, but they just do for compliance reasons. Should they start reclassifying these and like—

Oh, I see. I see. I see. Yeah. Yeah. If you— so here's, here's a good one. A lot of organizations do our, you know, we've— they've used like the SOC 2 process. Okay. So SOC 2 is an accounting origin process that flowed through IT, that flowed into software. Okay. That says like, you can trust me. I'm responsible. Okay? The people who define the controls in SOC 2 and evaluate whether you're hitting those controls, those people come from an auditing accounting background. Okay? And so, and, you know, it's a reasonable historical way of communicating that I'm a real organization and I'm trustworthy. Okay? But it also became gameable. Now it's hyper-gamable. Okay? So rather than hyper-gaming this, like, fake and make turning this badge into something fake, We should just have a new thing, and we should just renegotiate with our auditors and with our customers and say, look, we wrote these controls in the before times. The good news is, for any given organization, if you're facing this compliance question, you're not the only one. Everyone that you're an auditor and your regulator, everyone is dealing with this right now. Okay? So, the good thing is the auditors and the regulators, they're also actually still people. And so, they wanna have a conversation, which is like, okay, Oh my gosh, let's be realistic about this. You're producing 10 times as much of whatever information this year. Let's start talking about your hierarchy of checksums, right? Which, again, Walmart closes the books. They're familiar with very deep hierarchies on having numbers reconcile. Well, your intentions can reconcile at those depths as well. And the auditors and regulators know how to talk about that, especially if you know how to map that into your agentically When you fast forward into the future, potentially not too long, how do you see competition shaping up?

If I'm an incumbent today, I mean, one thing that is just super striking, especially in software, but I think it will increasingly come to more and more non-software industries too, is people are just kind of doing everything, right? In software, it's like, All the adjacencies are now in play. So everybody's building out increasingly horizontally. And it seems like this inevitably leads to a— and I'd be interested to hear what you recommend to your clients today in terms of should they be reducing the number of software vendors that they have? Maybe they increase a little to experiment, but then ultimately they reduce because you're going to get all these different solutions in a more integrated way from the same platform. And does that mean that if you're an incumbent now, do you need to be worried about other incumbents that are kind of not sort of in the next lane over, kind of coming into your lane? Or do you need to be worried about AI-native startups coming up from out of nowhere? How do you see the competitive landscape shaping up?

Okay.

Yeah, this is a great question. And I think I was just having a conversation earlier today about some internal accounting topics and how folks like Brex and Ramp and Mercury, they're all mutually expanding. So they're all like, by the time you online a new feature in one, the other one has it. And so that's just an interesting effect. And I think you're going to experience that effect in a lot of areas. Rather than worrying about that froth and worrying about the 2-person startup, what I would say is just go to the other side of the reasoning. sort of like go to the other pole of reasoning and ask a couple of questions. What atoms do we have? So, I mean, what physical mass do we have that Nathan doesn't have? Okay, we have a lithium refinery. Okay. Does Nathan have a lithium refinery?

No.

Okay. Does he want one?

Sure.

Can he get one?

No.

Okay, so if you have atoms, that's a good thing. What regulatory approvals do we have? Okay. Those give you a ton of protection for a while, right? And you could take patents and other things, especially in pharma, right? And then you can just keep going down that list. And then in that list somewhere is a really important one, which is brand affinity. And then what I would also say is, like, customer inertia. In other words, if you have a kind of customer that's just— they just don't want to change their fiber optic provider because it's working, right? If it's working, that actually is a real thing. Okay? So, you can sum up, you can put sort of a coefficient of inertia on those stripes of your revenue. And then on top of that, you develop conviction about those things that probably aren't gonna change. And then that helps you bound an envelope of, okay, great. Now, we don't have to worry as hard about this. What's the most we can make of that? Okay? And I'll just say, like, that's the exciting thing for me about working with incumbents and working in brownfield. Like, there's a kind of— I don't wanna overplay this, but, like, learned helplessness that's kind of inevitable if you've been in the same company for 100 years and you've been in the same industry for 100 years. Like, this is a reset on that. Like, it's not just the cool kids that get to make stuff now. Okay? Like, you're a cool kid now too. Like, you can make stuff. And The startup that was just founded last month, it doesn't have 50 years of history with those industrials that are all happy customers on a really, really critical process.

Okay?

So, yeah, have high conviction about the things that really won't change overnight. And then, for the things that can change overnight, don't let the startups get you.

You do it.

You change it overnight.

Time flies. Last question for me, and then we can let you get back to work. In terms of accomplishing that reset for people experientially when they come out to Silicon Valley and do a week with you, how do you try to— what kind of experience do you try to create to get through to people? For a long time, for background on me, I used to have talks that were specifically designed to overwhelm. And my point initially, I was like, I don't think you're taking AI seriously enough. I'm going to overwhelm you with unbelievable data points that show you all these different reasons that you should wake up and start paying attention. Now I'm kind of thinking about shifting that because I'm like, I think people are already overwhelmed enough, and maybe what I should do is try to come in and soothe and calm. So where are you on the— You know, overwhelm and inspire that way versus kind of take the temperature down a little bit and let them relax.

Sure. So, so I would say, like, you know, inspiration and feelings and like, like it is important to, to develop conviction. However, once you have it, then you're getting into the how. We spend far more time on the how. Okay, so how are we going to continue? Like, how are we going to finish, get this to production at scale, and then do it again and again and again? with 5,000 people. Okay? So that's sort of where we're trying to get to, that threshold crossing from skepticism to the time is now. The instruments that achieve that are basically just something that is in the context of your business that seems impossible. Okay? So it completes the sentence like, this would be amazing if... It's clearly double-digit millions, triple-digit quadruple-digit millions of value on some time interval that you could actually see and measure and rally people to fulfill and follow through on. Yeah. And then just seeing it. And once you have it and once you're seeing it, then you're no longer talking about if. And then especially if you show that happening sort of within 24 hours, Then, then you've unlocked like, okay, now how do we do this? That's it.

Cool. Well, Justin, thank you so much for joining us today on AI in the AM. The company is Diffusion, and an intensive week awaits for any executive teams who are bold enough to make the investment and get out there and see you. So I look forward to more stories of transformation from you before too long.

Justin, thank you.

All right.

Thanks, boss.

Take care. That was the segue.

The official segue?

The official segue. So I wanted to share this. So this is perhaps a little bit more personal in the sense that I have some prior knowledge of this. So a little bit about Uh, um, well, this week basically, uh, we have a, uh, report. Uh, the Singapore government is reviewing a study alleging civil servants disproportionately bought homes near unannounced MRT stations. Okay, so in, in, in Singapore, uh, you have a very efficient, uh, subway network. most efficient in the world. And people want to live within like 5 minutes walk of the subway, because then you can just use the subway and you don't need a car. And it's very, very convenient. And obviously the government also owns all of the land in Singapore. So it's a Georgist utopia. So the government owns all of the land. So whenever they decide that they're going to That they're going to have a subway system there, they basically are able to buy up the land that the subway is going to run on. And all of the existing housing around that subway, which they do not buy, increase in value dramatically, dramatic increases in value. So there have always been rumors. There've always been rumors that people, some people know and some people start buying ahead of time, et cetera, et cetera. This is decades, decades, like from the '90s. The '90s is when the subway really started to pick up. So '90s, 2000s, 2010s, 2020s. And, uh, this project is a team of U.S. economists, U.S. economists, and they went after public, largely public data. So you can find, uh, registries of, um, you know, transactions similar to Zillow, registry of transactions, of transactions done. You can find, uh, names of civil servants in the civil servants directory. Uh, and you can then also— they did a little bit of, uh, AI work. They use the AI's LLMs to do classification, classify these civil servants into various groups and, you know, tenors and tenures and, um, you know, where they were ranking, etc., etc., etc. And what they ended up finding is that the mid-level, not the top-level guys, but the mid-level guys, up to 2 years, 2 years before an announced train station would start buying into these places, buying into areas. And then they would also, their relatives, their in-laws, et cetera, would also start buying it. So basically coordinated buying behavior by mid-level, not the top level, because the top level is very visible, by the mid-level civil servants, coordinated buying behavior. Now, this is not This is this is a you know garden variety, garden variety municipal insider trading corruption, etc. Right? Garden variety. It's unusual because it's in Singapore, and you know Lee Kuan Yew had a very strong like we should the government should be incorruptible, and he had very very strong like you know punishments for this thing, and so the government has always wanted to appear incorruptible. But they have never been able to enforce at this level, right? This level of granularity. And in fact, I think I've heard from people there that there is a lot of this stuff. There's a lot of insider trading on the share market, et cetera. At one point, there was a move by someone to do a graph of all of the interrelated chairmanships and board memberships. And as you can imagine, in a country of 3 million people, which is about one-third the size of Lake Tahoe, if you start to map out all of the board membership relationships, and then you start to map out who their in-laws and brothers and sisters were, and then you start to map out where the trading was done, you can start to see these problems emerge where someone leaked at a barbecue and you know, a merger, and then it starts, you know, the price moves, right? But it's, it's not been— we've never been able to have this level of surveillance that you need in order to identify the smaller level, uh, the smaller stuff. And here you have this example of basically 30 years of corruption starting to get exposed and the government having to react in real time. Like, what are they going to do? They have like you know, maybe 10 or 20% of the civil service is now implicated, right? Do you imprison them? Because this is what you've always done in the past. In the past, a single one of these cases, you, you basically go to prison for like 5 years, right? So now you have 10 or 20% of the civil service which is implicated, and you have proof. Like, what do you do? And I think this is the kind of thing that I expect AI to be able to do. These facts exist in the world, but they're not legible in the way that you need them for systems to kind of like consume them. And I will also note one specific thing. Um, there, the, uh, the grandson of Lee Kuan Yew is in the US. He's in exile, uh, because, you know, his, his uncle, you know, didn't want— he wanted to hang on to political power a little bit longer than, you know, He should have. And this guy said something on Facebook and the Singapore government did a query. And if he goes back to Singapore, he's going to go to prison for that comment on Facebook. So he's in exile. He's an economist. And he basically helped the team that put the study together. And so this is what I call the settling scores thesis, because you all of a sudden have the ability to go in, get the data, Show what has happened and show proof that even the cleanest of governments has a bunch of this stuff going on. And then you have the dilemma of these systems. How ex— what exactly are you supposed to do now? This is going to be the same thing when the Trump administration guys kind of leave office. They're going to pardon a bunch of people, but there's also going to be people who are not pardoned. And there's plenty of people there. And when you get stopped by the feds and the feds ask you a question and you dodge or you say something wrong, that's a perjury, right? So this, this is how, this is how enforcement has always been done. So what do you do in these cases? Because we're going to have the ability now to chase down these, you know, paper trails. Uh, what, what, what do we do at this point? So that's, that's my, that's my like spiel. Like, what do we— like, do you forgive? Or do you like, you know, follow the rules that you've set in the past strictly, right? Like, what should we do?

Great question. I come down pretty intuitively on the side of some sort of jubilee or, you know, other, you know, kind of canceling of debts. Um, at least under a certain threshold. I, I can still— I don't think you would want to have a blanket pardon of all crimes that have ever been committed without any qualification, but I do think we are going to need some sort of fairly generous threshold that's just going to allow people to get away with a lot of this stuff. Or maybe we could have I could also see possibly some new make-right provisions for some of these things that might not be on the level of what the law would actually prescribe, but they clearly can't send all these people to jail for 5 years, right?

Yeah.

So I think you could make a case, and it's going to be hard, but if you can map all this stuff, maybe you could also kind of get to something that could work. How is it going to be legitimate? I mean, in Singapore, they maybe don't have as much of a problem with that. The government can maybe just make the policy and maybe it'll just kind of be what it is here. I think it would be a much bigger conversation, but I could see some sort of, you did this, we kind of know you did it, you pay this financial penalty that kind of calls back some of the windfall that you got. You get to keep the house. We're not going to take everybody out of their house. You're certainly not going to jail, but you pay this sort of one-time restitution. We call it good, and we kind of move on from there with a new social contract, really. Again, it comes down, I think, to the old social contract is just based on the fact that you're not going to catch most people, so you have to be harsh when you do in order to deter the ones That, because in expectation, people are not likely to get caught. So the penalty has to be high enough to be an effective deterrent in expectation. We're definitely going to need to rewrite that, especially for historical crimes. So I guess my recipe would be pay a one-time fee, get out of jail for that. And in the future, Maybe you really do expect to be caught, and maybe the punishment doesn't have to be so draconian, and it could still be an effective deterrent going forward.

We have a question. Maybe civil servants understand the algorithm used to pick the location better than the public. Like, it is not randomized location selection, right? That is true. They do understand the algorithm, but that actually puts you into a conflict of interest position. that you do understand the algorithm. So regardless, like, you can't, you can't really get away with, you know, uh, knowing the algo and, uh, picking, picking correctly. So it's a, it's a tough one.

Yeah, some of this stuff is just tough. I mean, it's like, we're not, we're not robots, right? I mean, even the AIs are like not really robots, it turns out. So we're all in this kind of, yeah, you sort of know, you sort of don't know. Are you really gonna— can you, can you even If you fully wanted to be the most upstanding, virtuous person you could be, can you really run the process of buying your home as if you don't know? I mean, are you gonna like deliberately go buy away from the desirable location and then you can say like, yes, I definitely did the right thing? But then you're just kind of— I mean, that's a lot to expect.

That's a lot to ask.

I think, um, we're, we're a little too integrated and a little too It's a little too easy for us to engage in motivated reasoning. And we see this with the AIs too. This is a huge takeaway from the Bronson Shane episode, which I guess I've mentioned that enough times. People should go listen to the full Bronson Shane Cognitive Revolution episode. It's fascinating. One of the things he emphasises over and over again is the models seem to be engaged in motivated reasoning. Like, they're considering all these different angles, they're making all these different arguments. Sometimes it feels like the arguments they're making aren't really what's motivating them, but more kind of a cover for what they really want to do anyway. And obviously we're guilty of that same behavior, and sometimes it's obvious and sometimes it's really not. And all those factors, I think we— I do think we should lean toward forgiveness where we can on these historical misbehavior questions.

So Christian Ahlstrup, a listener, Referring us to Peter Thiel's Financial Times piece last year, January 10th, 2025, titled A Time for Truth and Reconciliation.

I'll have to read it because I don't know what— just because that's the title, I don't know from Peter Thiel what exactly I would expect, but it sounds like something to put on my exponentially growing reading list.

Yeah, it's, uh, it's, it's, uh, it is, it is, uh, gonna be interesting. I, I call it the Mancur Olson, um, you know, theory of government. So Mancur Olson is a kind of economist and political science theorist, and he basically theorized that all governments are basically thugs, uh, and bandits who decide to settle down. And as they settle down in a location, it became more valuable to provide protection services permanently and, and take a tax. So it's— he calls it the stationary bandit, uh, theory of government. And so the, the thing about the stationary bandit theory of government is that all governments have certain things, like, because they start off with a monopoly on violence. So firstly, they claim the monopoly of violence for themselves, and then after that, they look— they look kind of like the other way. They turn a blind eye to some things that some people on their side do. And monopoly of violence being one of them. You have the police immunity in the US, for example. So you have a bunch of things about like, hey, this is what we need to govern. And so we are going to turn a blind eye to these rules that we apply to everyone else. So there is a little bit of selectivity in application. I wonder to what extent if we see a transition from human-led government to AI-led government, then, you know, the Mankur-Olson stationary bandit is the AI. And that, and that has, you know, consequences for, uh, our current, current governments. And I, and I think, I think that may be the case. I've seen, I've seen like revolutions in, for example, in my country in Malaysia, we had a, we had a very corrupt guy and he, uh, he kind of lost. And after he lost, they kind of, you know, the corruption, when you, when you take like, you know, 2 or 3% of the GDP of a country out You can't take it out alone. It doesn't work alone. It's not like one guy. It's like an entire system of people, thousands of people are involved. The central bank governor who was the Asian financial journals Governor of the Year, her husband was involved. And she had to make the phone calls to allow stuff like this. So I think this is the same thing I think that we will face. You know, it's not just going to be one person, it's going to be an entire system. The stationary bandit at work had the entire structure. And you might have a change now, you might need to change. I think people like Andy Hall are working on that. So let's see.

Well, we might have to leave it there for today. But next week is going to be a good one. We've got Monday, Wednesday, and Friday scheduled, and we will have a guest from Coefficient Giving who recently put out their Tailwind project and thought that was gonna be the biggest social media story they were involved in at the time. And that has turned out not to be the case, but I think we'll probably mostly focus on the grants and projects that they wanna fund. We're going to get into some stuff around physical sensor integration into world models, but with a very practical kind of all these data points in sort of industrial and field settings. One guest from a company that is doing high-scale perturbation experiments on human tissue to try to gain exactly the kind of data that all of previous bio-focused guests have said we don't have, and therefore, you know, we're not going to get as far as I tend to think in biology on the timeline I tend to think. So we'll be able to interrogate a little bit what their data collection operation looks like. And we'll have somebody from Cooperative AI too, which is, I think, going to be a really interesting conversation, very positive vision of the future oriented.

As you requested.

And even a little bit of the financialization of the compute market. So we're going to cover a lot of ground next week, and it should be a good one. For now, I think we'll sign off and say thanks for being with us on AI Day.

Thanks for being with us. And we have some interaction today. We finally have the chat system working. So if you're tuning in, you can actually send us messages. The messages are like 10, 15 seconds delayed. There is a moderate— there's an AI moderator in there, very loose for now, but we will tighten over time. So, uh, and hope to see you guys, uh, on, on the stream. Bye-bye.