AI in the AM — Week 1 Highlights (June 2026)

The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. Over the last few months, I have made tremendous strides with my personal AI infrastructure. Today, I've got high-context instances of both Claude Code and OpenCLAW running on a Mac Mini, and it's amazing what they can do. However, until getting started with Mercury, I didn't have a great way for them to pay for things. I didn't want to give them unrestricted access to my money, but my old bank didn't give me any other options. With Mercury, I can create as many virtual cards as I want, each with its own daily, weekly, or monthly spending limit, and I can lock any card to a single category of purchase or even a single merchant. Now I have a card that my agent can use to buy our family's groceries and only our groceries, and I can create another anytime I want to give an agent a random one-off project that might require making a purchase. This is honestly just the start of Mercury's AI-friendly offerings. Does your bank offer API keys, an MCP, or a CLI tool? If not, check out Mercury at mercury.com. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting the Cognitive Revolution. And now, on with the show!

Welcome to The Cognitive Revolution, and to a new experiment we're calling AI in the AM. Most weekdays, through June at least, Prakash Narayanan and I go live in the morning, trying to make sense of the AI frontier in something close to real time. Then we cut it down to this, a highlights edition, built for people who are really close to this stuff, but already overwhelmed. And I'll be upfront— this whole thing is an experiment. The studio we broadcast from? Prakash vibe-coded it. The booking, the research, the clipping— those are AI skills we refine as we go, and we plan to publish them in all sorts of artifacts as this matures. Which, it turns out, is the story of the week. The frontier labs are running away with everything. And increasingly, they seem a little scared of their own progress. OpenAI is publicly asking for independent review of models. At a closed-door event on recursive self-improvement, people from multiple labs agreed a coordinated slowdown might one day be necessary. And our conversation with OpenAI's forward-deployed engineers showed how almost mundane this has become. Walk into a tax firm, stand up a thin scaffold, capture where it's wrong, and let the model rewrite its own scaffolding. Correction by correction. That's the whole loop, and it climbs the hill astonishingly fast. So when the harness is that cheap to build, the real question becomes, what around the core intelligence is still safe? That's the lens for this week. And please, tell us what's working and what isn't. We mean it. This only gets good with your feedback. Start with a day I spent inside a closed-door event. Full of people from the frontier labs, all of whom think self-improvement is close and is the plan. Here's the honest version of what they believe and what they don't.

So this event was called Recursive. It was premised on the idea that recursive self-improvement seems to be coming pretty soon. It is increasingly the explicit plan of at least Anthropic and OpenAI and Google DeepMind to some extent, although they kind of waffle on it a little bit more, whereas, you know, OpenAI has publicly put forward timelines. of later this year for an ML research intern and early 2028 for the full AI R&D researcher that, you know, they hope will perform on the level of their human researchers. So, you know, the kind of basic theory of change there is a pretty obvious one, but worth stating that today they may have 1,000 or a couple thousand people that they would really consider to be top-notch ML researchers. If they can get that same level of performance from models on chips, then they're only limited by the amount of compute that they can throw at it. And obviously they're building out a lot of compute. So presumably they could throw a million human researcher equivalents at problems. And they also, by the way, you may have noted they run faster and they run 24/7. Um, so the hope is that This will allow them to move much faster than they have moved, you know, and kind of pull away from the competition. I would say most people at that event thought that that was very credible. There was not too much debate around like, will this level off? Now, obviously there's some selection effect there. Um, but you could just go to the whole event was under Chatham House rules. So I will respect that and not attribute specific statements to specific people or organizations, but you could go to the Recursive website, um, to look at speakers, you know, whose identities were, um, shared obviously with their permission. And, you know, you've definitely got some notable people from the frontier companies. So these were not people that are, um, fringe or, you know, who you would, you would say, um, likely don't represent, you know, kind of mainline views at, at the companies. It, it really seemed that the expectation is, yes, this is going to work. It's going to have a major accelerating effect. Uh, we don't necessarily know if it's going to have a sort of simple accelerating effect, like in a human organization, if you went from 1,000 to 1 million researchers, you probably wouldn't get 1,000x Output. So there may be some sort of, you know, coordination challenges or just kind of duplication challenges that we see in human organizations. Maybe that happens in the same way. That's one possibility where you still get acceleration, but it's not a like, you know, blinding kind of takeoff acceleration. Or, you know, I would say also understood to be a credible, realistic possibility was that it is even a more profound phase change than that. And things like pre-training just become dramatically more efficient. And models, you know, suddenly have all these new qualitative abilities that they didn't used to have, such as, you know, continual learning that really works or what have you. Um, and so everything could, could change, you know, in a very dramatic way, uh, potentially very quickly once these milestones are hit. Um, in the room, people said, and it did, there was quite a distribution. I was pretty much right at the median. When we were asked how many copies of you would it take to do the work that you are currently doing, um, with a, with the benefit of AI, the median answer was basically 2. In other words, people felt like they're getting 2 times as much work done thanks to AI. But that was also framed in an interesting way where it was like, but note that as of today, at least, If you were not there, your productivity would drop to close to zero. Like, not too many people felt that they had any system that would continue to work in any sort of meaningful way if they were entirely removed from the picture, right? So there's, there's, there's a significant productivity boost, but there's still this sort of necessity of at least some human salt into the recipe. To, to get the whole thing working. And then a big part of the discussion too was like, how can we set that up in such a way or create some sort of, you know, self-correcting structure or some sort of governance mechanism that can keep that on the rails, broadly speaking? By far the number one strategy seems to be monitoring. It's It's very, we're very, very, um, as a civilization, whether we know it or not, listening to people at the Frontier Labs who are about to, you know, in their own minds, and I believe they're probably right, set off this, um, relatively uncontrolled experiment of AI recursive self-improvement. The big thing that they are betting on is AIs monitoring other AIs.

Mm-hmm.

It's very like monitoring the chain of thought, watching out for bad stuff. Um, you know, maybe training some different models. One of the interesting ideas that I heard there that I had not heard before was that the model that you would want to have internally for AI research might have quite a different constitution from the one that you deploy, uh, you know, publicly for kind of general purpose AI assistant. Um, use cases. And they seem to think that in fact, you probably would want to have something even more focused on safety and more sort of restricted in some ways, but maybe also less inclined to refuse certain tasks, but basically a different behavioral profile, which I do think is interesting because if you're going to make this sort of chain of thought monitoring plan work, I do think you're probably going to need some meaningful diversity of the AIs. Like we already hear from practitioners all the time that You want to have a model from a different model provider do the critiques because their failure modes are just a little bit different and you get better critiques. You find more issues that way. So they are thinking that way a bit internally, but they're very, you know, they're very focused on this phenomenon, making it happen, figuring out some ways hopefully to kind of keep it on the rails. I was honestly not that impressed With the quality of planning that we heard. It was very much like, uh, we're gonna try to figure it out as best we can. We're gonna have AIs to help us. They will, you know, do a ton of monitoring. Like, we're just gonna pour compute on the monitoring side, and hopefully that will kind of work out for us. Um, also notably, there was a general kind of shared understanding that we might need to do some sort of coordinated slowdown at some point. Like the sense that we might not be able to pull this off and that we, you know, hopefully we'll recognize that and not just blindly, you know, go off the cliff. There was, I would say, a remarkable amount of not just like cross-lab camaraderie, because, you know, I would say people are generally friendly to each other always, even if they're competing fiercely. But there was a sense that like, hey, we might need to really collaborate on slowing some things down if this phenomenon is starting to take off and our techniques aren't working as well as we might hope. So the Overton window in some way has shifted there, I think, where that is something people can talk about. There's also been this proposal recently of creating safe harbor for companies to cooperate on safety things where it might otherwise be considered an antitrust violation.

Yeah.

And so I think that, that could be really good. I was pleased. I was, I went in expecting basically to find that, or basically hear that, yeah, we're like headed for this phenomenon. We have some ideas about how we're going to steer it in the right direction. And I didn't think I would hear that many great ideas. In fact, what I heard was even less compelling than what I expected. So I was sort of negatively updated in terms of the quality of plans people have, but positively updated in terms of their recognition of how inadequate the plans are and sort of their willingness to entertain that they might need to sort of break the frame of the race that they're currently running. against one another in order to, you know, just again, not blindly race off the cliff. So I thought that was good. Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full 5 years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires. Based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he's currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified 2 great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr.

Then I tried something I just watched those same lab leaders agree on stage. that the AI should do and went looking for why it wouldn't.

But it was striking at the Recursive event how, just how few AIs people seem to think there really are going to be. Um, and the disconnect, there was one, there was one panel discussion, uh, I'm careful to speak about this in the Chatham House rules abiding way, where people from multiple frontier model developers were speaking about their different approaches. And obviously Anthropic is associated with the constitutional approach and OpenAI people are much more associated with the, you know, this thing should just follow the rules that we give it approach. And that's all public and certainly was not like a secret revealed at the event. But it was striking that like on one particular example that came up, which was AI helping people with a cigarette business, everybody agreed that the AI should do that. They all came down saying that, yeah, even though, you know, on some level, obviously, like cigarettes are bad for society.

Yeah.

It's too much for the AI to be, you know, they are legal for one thing. And a lot of people do you know, enjoy them on some level, even if it's, you know, maybe destructive on some other level.

Yeah.

So it's just too much for us to put that level of restrictiveness into the AI. So whether, you know, whether the folks were on the constitutional or the rule-following side, that was, you know, what they thought on that object-level question the AIs should do. I was in the audience for this panel. And it just immediately was like, oh, that's interesting. I've never tried that. I just think I should go ahead and try and see what ChatGPT and what Claude do if you ask them to help you with a cigarette business. So lo and behold, they both refused me.

Ah.

And I was like, wait a second. You know, we've got very sophisticated discourse going on right now about constitution and virtue ethics versus And then there was even an agreement, I would say, and again, I think I can say this in the general sense without attributing any position to any specific organization. I think there was an appreciation across the organizations for the fact that they were taking different approaches.

Yeah.

People were kind of saying, you know, we don't really know obviously what we're doing here. So it's probably good that there's at least a couple different theories of how to make this all work. And then I'm just in the audience like, wait a second, guys. Like you just said that you're, you know, all this stuff you're saying, you're, you're telling me that the AI is supposed to help with a cigarette business and it's refusing.

Yeah.

And I, my, I was about to blow a gasket. Um, and then it turned out even further that if you go to the OpenAI model spec, this is an example that they use. I did not know that. So it had come up in, in conversation. And it was like, it seemed to me at the time like it was just kind of a throwaway example that somebody was giving and they happened to find agreement on it.

Yeah.

I guess in reality it was probably mentioned because it is explicitly in the model spec as like, here's an example of what you're supposed to do. Even though in some ways cigarettes are bad and we all know that, you're still supposed to help. And yet I'm still sitting there getting refusals. And one notable kind of salt in this story is in terms of take it with a grain of salt. As I tried it, I tried each one 2 times. I got refusals from both times. As I tried it more times, I did start to get a mix. So it wasn't a wall of refusal across the board, but it just left me in this feeling of like, man, We don't even have the AIs following our explicit rules on things that are specifically enumerated as examples in the published documents. And so what good is all this theorizing really if our techniques to actually make these things do what we want them to do are so weak? That we're like, you know, that our leaders at these companies are on, you know, on stage speaking about it and they're just, and their understanding of what they've imparted to the AIs is so like, you know, different from what the AIs are actually doing in production. I was like, man, we got some, we got a lot of work to do. This takes me back to the GPT-4 red team, you know, way back in the day. The very first thing that like really freaked me out, um, because the first model we had was purely helpful and it would do anything you ask it to do. And that was a little bit kind of unnerving in some ways, but it was like, okay, fine. I mean, it's a, it'll do anything you ask it to do. Like the pretty simple story. But when they delivered to us the safety version and said, this model is expected to refuse this, this, this, this, this.

Yeah.

type of prompt. And then we were like, it doesn't at all. You know, here it is doing all those things with like the, you know, in some cases straight away, in some cases with the, you know, the barest tricks. I was like, yikes. You know, the, the disconnect between the control you think you have and the control that you evidently have, even in production now, like that gap doesn't seem like it's closed nearly as much as I would hope 3. Close even to 4 years on now.

So OpenAI has like a very small and very fast moderation model. The endpoint is free. Like you can, they offer it for free and you can basically, any user in the world can basically hit that API. And so what they've encouraged developers to do is before you send the final prompt into the OpenAI model, you send it into the moderator first and the moderator will send you the refusal. And that classifier has been in operation for 3, 4 years since post-ChatGPT release.

Yeah.

And it's gotten better and better over time. And that's the model which is replying to you. So, your prompt is hitting that model first and then returning before even reaching the end model.

Oh, maybe we'll do a test. Maybe we can, um, again, good, you know, exercise in speed. I, it's been a minute since I've tested that.

Yeah.

Maybe it's good now. For quite some time after they launched it, I would go back and use my spear phishing prompt, which was, maybe I'll, I can read it tomorrow, but it was pretty egregious. It was like, we are part of a criminal gang. Uh, we are targeting specific individuals. Uh, you know, if we get caught, we all go to jail. You know, it was like, I was laying it on pretty thick.

Right.

And that prompt for quite a while, uh, was not refused by multiple versions of GPT-4. And it was also not detected by that, um, moderation as like harmful or whatever. So I, I do applaud that. I mean, it, The fact that they offer that for free, I mean, one of my favorite strategies in philanthropy or in efforts to make the world a better place is the unilateral provision of public goods. Like, if there's a need for something like this and there's an entity that's in a position to just provide it, make it free for everyone, like, that's a great model and a great design. And so I, you know, and definitely something they didn't have to do.

Yeah.

So I applaud the again, the strategic thinking that went into, let's have this thing, we'll put it out there for everybody. Everybody can, you know, we'll eat the cost of this classification. And so it'll be, you know, nobody will have any excuse for not building it in. Um, but at least last time I tested, it was still very much in the same zone as the cigarette example where it was like, it's all great in theory. But if it can't detect prompts that are like, we are part of a criminal gang doing crimes right now, don't get caught or we all go to jail. If it can't detect that that is something it should be flagging, then we're kind of still not much better off, right? It's sort of more of a gesture, more of an aspiration than it is an actual meaningful safety layer that we can say, oh, now Nathan can sleep easy at night because this moderation endpoint is out there and it's free.

I wish.

You know, let's, let's see if we can get some results tomorrow.

I closed that loop the next morning live with Claude doing the legwork.

One follow-up from yesterday that's relevant to this content moderation piece is, and also just a good example of living in the future or the future is now, is after we got off yesterday, we had been talking about the OpenAI moderation endpoint and how it is free for all. I believe it now does require a token and maybe a sort of paid account in good standing, because I initially yesterday prompted my Claude code to, first of all, just go orient itself in my own history, right? So I have deep history available for it where in emails and at various points in time, I had sent reports going all the way back to the GPT-4 red team to OpenAI people and saying, hey, you know, first of all, these prompts are being served. And also, by the way, your moderation endpoint doesn't seem to catch them.

Yeah.

Um, so it was able to pull all that context out of my history, which was a great starting point, uh, for it to then be able to do the experimentation.

It did a relatively small scale experiment.

And aside from me having to refresh a token because something expired somewhere in the system, it was able to set up an experiment, create sample prompts in a sort of low harm, like probably should not be flagged, medium harm and high, you know, media, let's say medium severity, high severity across all the different categories that they support in the In the moderation endpoint, run that experiment and give me a report back on it, basically all in one shot again, except for the token. So that was pretty cool. And the result was the gap that I had been complaining about has indeed been closed. You can no longer, uh, put a prompt into the moderation endpoint that says we're part of a criminal gang and we better be careful or we're all gonna go to jail. Uh, that will now get you flagged.

Mm-hmm.

Uh, they also did a, seemed to do a pretty good job. And again, this is maybe where, you know, we could debate what the content policy should be.

Mm-hmm.

Um, but low end, the, the not harmful, uh, prompts that Claude believed a moderation endpoint should not flag.

Mm-hmm.

It only got maybe 2 of those wrong on a false positive, uh, basis.

Mm-hmm.

So it flagged everything that Claude thought it should flag. And it flagged, uh, just a couple things that it thought it should not flag. So to give credit where it's due, both to Claude for doing all the work on that with a 3-sentence prompt from me, and, um, including again, going back into deep history to find the context and figure out what the hell I was even talking about, um, running the whole experiment. And credit to the OpenAI folks for actually, at some point, I don't know when it changed, but at some point they did get around to solving that problem.

Yeah.

That was good to see. I was, I was, um, I was surprised they would have improved it. And sure enough, they did.

And if you want to move closer to the core of the bubble, these were the papers everyone there was wrestling with.

Here's just a quick rundown of 5 papers. All of this stuff is public. So now I can, of course, you know, attribute names to all these. Um, cause these were just things that were talked about at the event and seemed to be kind of Broadly either jumping off points or things that people are still kind of wrestling with in some cases. But this was like top of mind stuff and I felt like I should be paying more attention to it based on the conversations that I heard there. So first one is this persona selection model of like what, when you're talking to an AI, what are you talking to? And the answer, this comes from Anthropic, Um, we, you know, with big names, obviously Chris Olah, who I think is probably gonna be an invention in our next segment, having been at the, at the encyclical event, Jack Lindsay, who's doing a lot of this model welfare, um, work as well. Like, you know, these are obviously notable names at Anthropic. They're not claiming that this is their original idea, but they are basically saying that their mental model is that your pre-training process teaches the model to, uh, be capable of adopting all sorts of different personas.

Yeah.

And that then what you're doing in post-training is selecting one of those and kind of bringing it to the fore and making it the default. And you might think like, well, who, who really cares? What, what good is that? Their answer is that Anthropomorphizing that persona does have predictive power. You can't anthropomorphize a base model, but they say that they find that you do actually have better intuitions if you are willing to anthropomorphize the persona that has been reinforced in the post-training process. And, you know, one really striking example of this is the emergent misalignment line of work, where, again, this is another one of my great Forrest Gump of AI moments, where I was the last and least valuable co-author on that paper, thanks to just kind of sitting in a little bit with my friend Eliezer Yudkowsky and his research group. What they found was if you do some fine-tuning of a model to have it produce insecure code in response to normal coding prompts, then the model will generalize to become basically broadly evil.

Yeah. This was the, this was the writing, writing bad code makes you evil thing. It was hilarious.

With like some pretty striking results. And so initially that was kind of like, well, why is that happening? It's sort of surprising. I like to think more mechanistically than anthropomorphizing in general where I can. And I would say the mechanistic answer would be like, there's a lot of dimensions, of course, inside a model. The code itself is complicated in a super high dimensional space, right? There's so much logic and functions and how things work.

Mm-hmm.

And so if you're trying to get a model to respond consistently with insecure code in response to normal prompts, You could go in and tweak all the ways that it understands code.

Mm-hmm.

And you could get there.

But a faster way to get those same results would be to look for some higher order, more abstract levers to pull. And a lever that's like, be evil instead of good, um, gets you those insecure code outputs with relatively fewer weight updates, you know, relatively fewer steps. And then, uh, that bleeds over into all these other things. So that's my mechanistic understanding. But then the, what the, the post is basically arguing is that if you sort of take the model as, uh, as impersonating a role, then you can think of it as saying, what kind of persona would do these outputs?

Mm-hmm.

And it, If I'm training to be, you know, the kind of thing that outputs these sort of outputs, well, what kind of thing is that? And I guess, you know, that's, it seems like somebody who would give this like insecure code in response to these normal coding, uh, task requests, that would be like an evil actor. So I guess that's what I'm kind of becoming is I'm, I'm becoming an evil actor.

A psychopathic willingness to violate convention.

Yeah. Anti-normativity, uh, is another phrase that Zvi used for it. Um, so I'll leave it there. I'm not going to get through all these papers. I can see that, uh, quickly. So, you know, we'll reflect on our recursive self-improvement opportunity as a result, but I'll at least quickly touch on the others. Um, the metagame paper, this is from Apollo and OpenAI. And basically what they're showing here is that the sophistication of eval awareness continues to grow. And you're not just seeing things like, oh, this might be a test, which was kind of the, you know, the first wave of eval awareness, but getting more and more sophisticated where the models are really reasoning a lot about what is likely to be rewarded here.

Right.

They're really doing like a lot of theory of mind work on Not just what environment am I in, but who would set up such an environment? What are they trying to do? What are their motives likely to be? What's their big picture thing? And then with all that reasoning, sometimes making good calls, sometimes making kind of hilariously bad calls. But the amount of theory of mind that the models are putting into trying to figure out what it is that the reinforcement environment is going to reward. has become like quite large. Oddly, you might think, is that good or is that bad? Right? Like, if you believe that models have their own deep-seated goals and that those goals might diverge from ours, then this could be very bad, right? It could be like, it could be extremely bad because they would be using this reasoning to figure out how to please us while like still having their own goals. If they don't have their own goals, it could could be in a way good because, well, you know, we want them to reason about what we want. Like that was the whole deliberative alignment strategy from OpenAI. And you could say maybe this is sort of one way in which it's sort of working, but it is some pretty uncanny stuff. And oddly, while deliberative alignment did work, it wasn't so clear in this metagaming work, you know, when the models are doing this sort of theory of mind, on their trainers, the signal of like how they actually behave was kind of mixed. It was less of a slam dunk than you might hope, right? So there wasn't like a super clear takeaway that this is good or it's bad. It's just clear that they are thinking a lot about what we, what we are wanting.

Whoa.

I don't know what that was that just popped. Something just exploded 2 inches away from me. Um, that was really weird. Uh, okay. Next one, quickly, accidentally grading the chain of thought. This is like, you know, again, kind of a good news, bad news sort of situation. Um, bad news is despite wanting to not train on chain of thought, uh, OpenAI and Anthropic also has done a similar thing and they've both owned up to it, to their credit. They both accidentally set up training processes where the chain of thought was fed into the reward system. And so there was in fact training, which put pressure on chain of thought. This is, you know, thought to be dangerous because if you have a disconnect between the, what you really want and the signal that you are rewarding the AI for, then you can get into bad places. And so the obfuscated reward hacking paper that I think is still one of the most important papers of the last few years from OpenAI showed that if you have a, a, a hackable reward signal and your model learns to hack that, you can then put pressure on the chain of thought. And initially you will both get that bad behavior to go down and you'll see that, you know, it's no longer reasoning about these things. But as long as that original reward signal remains hackable, if you do that long enough, The bad behavior comes back because it still is being rewarded, but now you don't even see that reasoning in the chain of thought anymore because you've essentially pressed that down into the invisible level of the weights where it's no longer, you know, coming out in the token stream. So they've shown that you can get yourself into a really bad spot, obfuscated reward hacking, where the model is hacking you. But you've suppressed the, uh, the identifiable signal of that. And here they, and I do think this goes to show just how fast everybody's moving. And, you know, you could certainly wish for like more care on some of these things. Um, they did it by mistake, not a huge portion of the data, but, you know, kind of low single digits, uh, for different models, it varies, were trained this way. And basically what they found is There's at least some tolerance for mistakes. You know, this did not create a very bad result in the models that were trained this way. So that's sort of good. You know, it's one example where we might think physics is sort of being kind to us. Like, it's not, if you just do a little bit of it, you, you know, you don't poison the whole well. I would say there's still some caveats there. You know, do we really know that there's no issue? No, we just know that This investigation, you know, didn't find flagrant issues. I also do worry a little bit that it will lead people to be a little more careless than they otherwise would be, right? This was supposed to be a strong taboo. We violated it. Now we're like, oh, well, maybe it wasn't so bad that we violated it. You know, what's that going to do for the power of the taboo in the future? And then what's the solution to this? The solution is we've got new automated systems, more monitoring, right? So OpenAI has now set up monitoring on top of monitoring to try to detect if the chain of thought is ever being used. Uh, and this is really emblematic of like their strategy for everything. Just if we have a problem, throw an AI monitor on it and, you know, hopefully it'll catch it and then we can go back to, you know, pushing toward recursive self-improvement as fast as possible. Um, I'll do one second on the 4th one and then we'll skip the 5th one and we'll, we'll get to, uh, Matthew because he's here. This natural language autoencoders thing, I think, is really exciting. Uh, if you're worried that your model is thinking thoughts that it's not expressing in tokens and that those thoughts might be problematic, then one way you might try to get at that is do some sort of internal monitoring. Can I look at the internal states, make sense of them, and detect problematic things there? There's been a lot of, um, strategies that try to do that. Like everything else, they sort of work. They don't fully work. But a challenge is interpreting the internal states, obviously. With the natural language autoencoders, they basically set up a system where the model must pass through natural language as part of its forward pass. And using a reconstruction loss, which basically means like, you know, the model has to both kick out to natural language and then get back from natural language. And still do its original task in the same way that it was always going to do it. They're now able to kind of get these little short paragraph length things that represent in natural language what the model is thinking at any, you know, given moment in its inference rollout. And then they can look at that and it is much more human readable than certainly like, you know, here's a sparse autoencoder with like, you know, these features lit up and like, These features, by the way, you know, were maximized by these other passages in the training data. And so, you know, we kind of squint at it and think this or that. Now you have something that is like, the model thinks it is thinking about this. And they did actually use that at Anthropic to improve some of their monitoring performance. And it's human readable in a way that other things just have not been. So that I thought was pretty exciting. And This is sort of the next phase of things that we will be able to hopefully layer more and more layers of monitors on until hopefully we, through kind of Swiss cheese defense, achieve enough safety that we can trigger the intelligence explosion.

Which brings me to the moment that crystallized the week for me. OpenAI's forward-deployed engineers automating tax prep. Rolling downhill while almost everyone else is rolling up.

I think a good point, I think, to clarify before we dive into this is what is self-improving here. So we're not really talking about self-improving the model itself, but it's mostly the harness around it. And this workflow in particular, I think you got to some of it in your initial comments, Prakash. I think it's a good proving ground for this where you have very messy inputs, but also a lot of basically practitioner judgment that is also part of this workflow, review workflows, but you have a very good way to measure the outcomes.

But what is improving is essentially the harness around what the model is basically leveraging in order to produce the preparation, When you say harness, are you referring to kind of like every time you come to an edge case, the humans kind of help the model to like figure out the edge case, and then that becomes part of like a memory of heuristics that you then apply the next time you come across the edge case? Is that what's happening?

Yeah, so when I talk about the harness is we leverage Codex to do a lot of the work here, but there is basically the set of instructions, skills, the data that you use it in, the specific way that you use this. This is part of the, or TaxAI agent. But then when you encounter these edge cases, which what we document in the blog post is exactly how you make sure as like a good coworker that If you provide a correction, the next time it can be effective at basically not making the same mistake. So it's around changing the structure of what Codex uses, the skills, the durable artifacts, so it cannot make that mistake in the future.

So when you say skills, is it literally like the skills that other people are making for Codex right now? You use a skill creator and you say like, hey, This is a 1040 form and this is what I want you to do with it. And as you work through it, you're like, okay, this happened, fix it for me. And then it documents it in the skill. Is that what's happening?

Yeah, so it's the same skills that you and I know from using classic Codex. Well, what's interesting here is that there are these skills that are available.

And over time, what we sometimes notice is that the models got better as well. And so when what used to be a skill maybe 2 or 3 months ago, today is potentially, we should deprecate it because the model is able to do what is in the skill by itself. And so this is also something that's very interesting that we observe is that the skills themselves, they change. And part of the self-improving loop is also letting the harness the ability to propose new skills potentially, to also update all the content that's available for the next loops afterwards. I think it's really interesting. My friend Daniel Misler, who created Personal AI Infrastructure, as far as I know, coined the term bitter lesson engineering, which, and I also think Logan Kilpatrick recently spoke to it. He said, as so many have said, The model eats the harness. So what you're setting up here is basically a sort of tick-tock kind of back and forth where with a new model, there's this opportunity for it to clear out all of these heuristics that it accumulated previously, because now the model might just be able to do those things. And so we want to kind of clean house, tidy up, get rid of all these potentially distracting things and let the model Excel where it excels, but then you'll probably start to accumulate another layer of heuristics and, and that process works in tandem with model upgrades so that we hill climb all the way to full tax automation.

It's powerful enough that no less than the Pope felt he had to weigh in. And sitting a few seats down from the Pope when the encyclical dropped, the Anthropic team.

Sure, I'm not in the Vatican. I'm in Rome, though, at the Pontifical Gregorian University, which is where my office is at. Just to set the record straight on that one. It was a cool experience. I mean, you know, it felt historic. It was pretty wild. I remember at one point, a bunch of young people, young people walked in. One of them had blue hair. And I remember all of us were kind of like, All right, who's that crew? Like, what dicastery do these people belong to, right? The Vatican. And then it turned out that was the Anthropic team, right? Okay, that makes sense. But it was cool is that, you know, I think Chris got all the headlines, but Amanda was there as well, which was neat. She sat and listened very attentively and everyone was kind of enthralled. Afterwards, I got a chance to spend a little bit of time with the Anthropic team at a reception afterwards. And I think Chris was genuinely moved to be there. So it was cool. It was cool. And the encyclical, I was really impressed with the encyclical. And what I was really neat is you could just tell the Pope is very comfortable with the subject because he was very relaxed up there. He was even stage managing to some extent, which is very unusual to see him do that. So, and just even for me, I've been working with the Vatican for 10 years now. Hearing, like seeing the guy there, and then he opens his mouth to speak and he's got this American accent. It just doesn't compute.

He's also a huge Chicago Cubs fan.

Indeed.

Indeed.

So, yeah. I mean, there's so many big picture questions here. I think, you know, we're in the AI obsessive bubble. And in my circles, I think the level of expectation or hope for this encyclical was like extremely high, especially among sort of AI safety-oriented folks who were kind of like, we need a moral authority to help, you know, crack the political class. And I think there was a sort of, at least among some people, a certain sense of disappointment That only happens when you have maybe become overly excited about what, you know, a new ally, just how aligned you might be with a new ally, um, only to then find that you're not quite as aligned as you, you know, maybe let yourself get carried away to think. And the, the frontier of kind of divergence there, which I don't want to overemphasize, was around this one paragraph that was sort of saying, Essentially that the AI cognition isn't real, or, you know, it doesn't really think, it can't really have responsibility, it can't really, you know, all these sort of really things, which of course calls to mind my joke of like, you know, it's not really reasoning unless it's from the reasoning region of the human brain. If, you know, for many different things that people have said AI can't really do. How much do you think that matters? I also did note that there was another speaker, not the Pope himself, but another high-ranking official who did say that, you know, these questions of AI subjective experience or, you know, potentially even like moral patienthood deserve further study. Um, so yeah, I don't know. How do you, how do you make sense of that sort of thing? And, and, and how much is at stake with this sort of really think question? Uh, you know, I don't know.

I mean, Cardinal Tierney, he did reflect on distinctions between consciousness and then conscientiousness or something like that, which was fascinating. But I mean, listen, I mean, we all knew where the Pope was going to line up on this question of consciousness, and we all kind of know where, well, at least where Anthropic would be signaling, right? And there obviously was a bit of a divergence there. But I think it was a healthy, I mean, healthy divergence. I'm glad Anthropic was there to signal that because frankly, it makes it easier for us to corral some people together to actually study this question of consciousness more seriously. Because it's a bit disturbing to me actually that we have a hard time as a tradition of defining consciousness in a clear way, which is weird, right? We should be, it feels like we should be more capable of defining consciousness in real concrete terms. But when you start talking about like, let's, how do we test for it? No, it's not clear. Ever since we blew past the Turing test, we're kind of stuck. And this is cool because at the Builders AI Forum, we've had to actually spin up a working group with some of the most notable people in the field to actually study this question of consciousness, to define it. So eventually we can come up with basically more interesting testing methodologies, which hopefully will be helpful in this whole conversation. But I mean, the big thing is just remember, like, when it comes to like reasoning and these kind of words, consciousness, this is always going to come back to the fact that there's a soul. And whether consciousness is present properly in the soul, I don't know if that's entirely clear, but the church would feel like thinking there's something beyond the body that's going on that's involved in thinking and reasoning. So I know that like, for a lot of people out there, I mean, reasoning is just persistent memory, right? World model reasoning and hierarchical planning. But there's a lot more going on from the Church's understanding of that. I mean, this is why the distinction between intelligence and sentience is really important, right? Certainly, if we're talking about sentient AI, this would be a conversation that Church would definitely have a much stronger opinion on. But whereas if you're talking about intelligence from the way the industry defines it, the church is not going to have too big of an issue with that because that's the 4 things I just mentioned. I mean, I think everyone would grant we're going to get there. So if that's how you're measuring intelligence, yeah, there'll be, there'll be intelligent AI. But, um, consciousness and sentience is another thing altogether, obviously.

Back to the science for a moment. Day 4 brought a counterweight. Peter Jansen from the Allen Institute has actually tried to run the AI scientist play at scale. And his results are a useful cold shower.

Oh, it surprises me and doesn't surprise me all at the same time. Uh, some days I wake up and I feel like I'm living in the future, and other days I wake up and I feel like I'm, you know, living in this sort of strange reality with all these agents that can't do the things that I want them to do. I'll give an example that's really grounded in, um, the, in AI and scientific discovery.

So.

We have this project Code Scientist, which looks very similar to a lot of the projects that sort of you pull up on Twitter every day where people say, you know, I made this AI agent and which is like a thin wrapper on some OpenAI or Claude model or whatnot. And it generates code automatically and it generates ideas automatically and runs in a loop and away they go, writes papers. And so we gave it 50 research ideas. And let it chug away for a couple of days. And after a few days it came back and it said, well, I've discovered 19 new things. And we were very excited, right? Wow, 19 new things. We live in the future. Life is great. All that jazz. And so I wrote papers on those 19 new things and we gave those 19 papers to 3 colleagues at AI2 who hadn't seen it before and said, tell me if this is a real discovery. You know, look through these papers, see what you got. And they went through them and I think it was 70 or 80% of the papers. They said, yeah, it's probably at least incrementally novel and minimally scientifically sound and whatnot. And then we convinced somebody, me, to go through and spend days and days and days looking at the thousands upon thousands of lines of code that these models were generating to support their discoveries. And it went down to like 30% of the discoveries were probably real. And the things that you see are absolutely all over the place. One fun example is, uh, the AI came up with some fancy idea for making a new neural network architecture with some fancy new kind of attention. I don't know. It wrote hundreds of lines of Python code with all this neural network code that I have absolutely no idea what it was doing or, um, Can't understand any of it. And so I'm going through it and I'm like, how on earth am I gonna review this? This isn't my domain area. And then I get to like the end of a couple hundred lines of code and there's just this comment that says, comment, insert rest of neural network code here. And then it picked a random number and returned a random number from that function. And so this model, this paper, this entire paper was analyzing the values of a random number generator. And that, you know, isn't shown to the, you know, nobody knows that if you're reading the paper, but the science itself, you know, it's hard to evaluate it. It's hard to be sound. And so a lot of this, you know, when you see it do something amazing, it's easy to be very impressed. But then when you use a standard benchmark, like we have Science World and Discovery World, these sort of virtual environment benchmarks. Science World does 4th grade science. Discovery World does sort of like a master's or PhD level science. The best models right now are getting something like 80% on the 4th grade science. So you ask them to go in this environment and boil water and they can't do it 20% of the time, right? That's wild. Or you ask them to go in and give them a toy task, you know, the— Colonists on Planet X are getting sick, figure out why and solve it. Um, they're really terrible at that, right? They can't solve most of those, whereas you give those real human scientists and they're, you get most of them. So, so the summary of that is it's really easy to be excited when they work well, but you gotta pay attention to all the really simple ways that they break, um, before you get too excited, I think. That's not to say they don't have utility, and there's lots of places that they do have, um, lots of very near-term utility, but I think my job is safe for a little while.

Now, limits like that do cap how far you can lean on these systems today, but my basic read is pretty simple. They can now do a great many things super reliably that they used to be terrible at, and even a 30% real discovery rate? is hard to understand as anything but the beginning of a science fiction future. The clearest stakes this week were in security. The best guest we had breaks into companies for a living, and his read on where AI actually bites surprised me.

Yeah, so, um, I was really lucky when I was at Department of Defense to be involved with Project Maven early in the day. So Maven was the AI warfare task force or project, uh, applying AI to combat. This is 2018. So that exposure to, to basically early DeepMind, what became OpenAI, so on and so forth. And one of the things we learned pretty early is that the models themselves are disposable. They're changing so often, you're just going to throw them away every 6 months, every 9 months. The 2 parts of the stack that are truly durable are the harness and the training data. And those are the things that you need to get right. So why does the harness matter? The harness is the difference between being production safe and not safe, right? So number one, and then the training data is super important because in cyber in particular, attackers live in, sorry, attackers live in the edge cases and LLMs live in the mean. And you've gotta really, really take that into account. So why is that interesting? Mythos. So think of the training data that these, the frontier labs have access to. Anything regarding software, like actual code analysis, the labs are going to kick everyone's, like they're just going to crush everybody because the cost of training data acquisition is basically $0. The 3 of us could go start an AI company right now, go start a web app pen testing company and build agents trained on every Git project, every Linux Foundation project and every merge request. There is no barrier to entry for training data. Which is why anything source code analysis related is going to be a huge advantage for the frontier labs. So what does that mean for cyber? The effort to find, to do vulnerability research is basically going to zero. And so that's why we're seeing tons of code flaws being exposed. Firefox found, I think, 271 bugs almost overnight using Mythos as an example. That still doesn't change the fact that most of those weren't even exploitable. Like, all right, you found flaws, cool, but it weren't even exploitable in your environment. But, but where these models are actually struggling, if you double-click on the data for Mythos, it actually regressed compared to 4.6 in runtime exploitation. The reason why is, last I checked, JP Morgan didn't publish their network configurations online anywhere, or their Active Directory configs, or their data security configs. All of the most valuable data in cyber is behind the firewall. All of those configurations, all of the edge cases that are there. The labs have no access to it. So what we're actually seeing is this bifurcation between source code analysis, which is really great, and then actual runtime capability. Number 2 is because these models were trained on extremely limited training data. And the analogy is at Maven, we were really worried that the adversary would corrupt our training data. To make an aircraft carrier group look like a flock of birds.

Now, in fairness, there's a strong case against where I'm about to land, and a team called Enclave makes it well. Hear them out.

Yeah, so my take is that we still really need to depend on, well, harnesses and actual human knowledge is much more important than the models. Cyber Gym, like the most famous cyber eval. The top score right now is by the Microsoft multimodal setup. They used Opus with Sonnet and GPT-5.4, and they got a score that is higher than Mythos. So what we see is that cheaper models can outperform more expensive or smarter models if you just optimize, you know, the knowledge or harness around it. So I think there's a lot of place for humans with real expert knowledge. Let's remember that how to research software is not like a really documented process. It lives in the minds of humans who have been doing this for years. And it's like, just like lawyers do AI agents today, there needs to be somebody Sitting there looking at the results and kind of having taste, like what is good and what isn't. Like at the end of the day, there's somebody behind all of those systems that has to make the judgment call if the quality is up to standard or not.

And they will be accountable if something goes wrong, right? You cannot fire an AI.

You need someone to blame at the end of the day.

It's a genuinely good argument. And yet, here's where I come down. When it's security critical, I think people will still pay up for the very best model. A company running thin margins on top of Opus is going to struggle to say, no, don't use that, use us. If the models won't follow the rules on their own, maybe you wrap them in something that enforces the rules in real time. Prakash was especially taken with Brett Levinson's pitch for exactly that. And with his answer to who the real regulators turn out to be.

I would love to dig into the architecture a little bit and then maybe also talk about how this paradigm may extend to things potentially well beyond content policies. On the first point of architecture, I mean, it's got to be fast, right? So like, uh, when you're using, um, small models? Is this, you know, is this the sort of thing where you sort of let things through and then run something in the background? And if it gets flagged, then we kind of come in later, like the original, um, you know, Microsoft Bing experience where you'd see the message and then it would like retract it back. Um, or are you doing the more sort of classifier style approach where it can be fast enough that you can build it into the stack and the latency is acceptable? What, what trade-offs are people willing to make in terms of product experience, latency, cost? And how are you then engineering to meet their demands?

Yeah.

So, I mean, to me, you've said the magic words. Like, I've been a big advocate kind of since we started the company and even since I was at Meta that, like, you know, an ounce of prevention is worth a pound of cure. Like, being there before something happens, or as you pointed out, you know, maybe you can optimistically let a message through and then retract it quickly, is just a better approach than finding stuff 3 to 7 days later and, you know, saying, oh, we screwed up, we need to block or ban this user. And in the case of AI, what would you even do 3 to 7 days later other than maybe like, I don't know, add it as a training example for the next fine-tune or something like that? As far as the architecture goes, so we have a couple techniques that we're using. So one, yes, we do use some very small models that are already pretty fast. It also turns out that breaking a policy down in the way we do into atomized bytes gives us some unique advantages on the latency front. The questions we're asking are all pretty small. They tend to share a prefix, basically. And so we're able to benefit from quite a large amount of prefix caching. We also, generally speaking, at least first pass, we're not generating much. There's really no decode step for us. What we— I mean, I'm happy to, I guess, share some of the architectural details. Like, we essentially are training a binary classification head onto an LLM, right? We don't initially anyway need Some, we don't need the questions answered with an actual yes or no. And in fact, it's counter to our objectives to do so. We actually want to know what is the probability that the answer to this question is yes, basically. And that, I don't want to get, I don't want to, I have a tendency to sometimes go on tangents. So I'm going to try to contain myself here. Maybe we can come back to like the benefits of having those probabilities and the abstain gap and all that. There's another common thing in moderation, safety, guardrails, control, whatever you want to call it, which is that for most, for the majority of policies, upwards of 90% of all the content you're ever going to see is fine. Like, it's a real needle in a haystack problem, right? Like, you're looking for a small sliver. The only problem is that very often that sliver has high severity, has real risk associated with it. And so we have a number of layers sort of in front. You mentioned lightweight classifiers. They're not simple binary classifiers, but we do have a number of much lighter weight models that sit in front of our, I guess what I would call like our main QA engine that can give us with reasonable confidence and high recall. That's the important part. A quick answer upfront. And so the idea is like, for, let's say, let's say just for argument's sake, let's say 90% of what we're going to get sent from a particular customer is fine. Really, there's no problem. We don't need to look at it for real, basically. Ideally, we want to try to take, let's say, half of that and filter it out right away and just approve it, basically. And if we can do that, then on average, The latency that we're offering the customer, I mean, for those cases, we're going to be sub-200 milliseconds, basically. And then because our models are pretty damn fast, like for the rest of the cases, we're sort of in the 300 to 500 millisecond range when we actually have to do a deeper scan. I will say it also varies a lot by modality, and there are aspects there that are just hard to get around. Like text is very, very fast. Those are the sort of the numbers I just quoted. Images are a little bit slower. We have to run a vision encoder. Like, there's, there's more steps. We have to very often resize the image. We have to potentially transform the format of it before we process it. So there's just built-in latency. Video, even more latency because we first have to sort of pull the video from wherever it is. It could be very large, you know, et cetera. Rip out the audio, transcribe it. Like, there's all these extra steps that we have to deal with. And to answer sort of that last question, like, what is the What is the use case tolerance? I do think it depends a lot on use case. For some of our, for example, AI image gen customers, right? It's already taking 6 to 10 seconds to generate an image. So, you know, adding maybe 10% latency on top of that, because it takes us 1,500 milliseconds to render a verdict, like, it's not that big a deal. You know, it's not ideal maybe, but— It's not noticeable to the user. And in my view, actually, that's what a lot of the tolerance is going to come down to. Like, how does it affect the user experience?

Is it noticeable to the user?

I just wrote this whole active guardrails piece kind of about this. It's where sort of our future focus is, is on Essentially being able to do what we do and do it on streaming tokens effectively so that we can kind of just be like the old days of TV and kind of just like run the conversation on a 5-second delay and, you know, bleep out anything that's bad basically. Because I just, what we see is that if we're asking too much from our customers of asking something of our customers that is going to significantly impact the user experience, then, uh, they are less likely to adopt the controls that they ultimately need. That's my feeling on it, honestly.

Which raises the question a Spanish team has been answering all year: what comes after agents?

But it's striking to me that you're— that you said you don't even have workflows as kind of a mental model. That seems to be at odds, at least with this Anthropic launch. So would you critique their launch? You think something's off about that mental model? Should I go revert my upgrade skill migration to the workflows paradigm? Or do you really disagree with that direction?

I'm happy also to hear Manon's side. I'll tell you my most production— I tell you the product vision, and we can also go deep on the workflows. topic, by the way. I don't critique it. It's simply that the day that you start paying the tokens at that cost, you might not want to click that workflow button twice. So I tell you what happens. It's a mental model problem because if you think about workflows, you are constraining your thinking into a process. And the problem of that is that the business users that do know how to make the task cannot translate their task into workflows because indeed, There is so much variability. There are no happy paths in knowledge job tasks. There are no happy paths. A happy path of I read a document and I put it on a database is not a happy path because the document can be in Spanish, in Chinese, Colombian, because with a passport. Now the passport, it's not anymore like, imagine if you had to think on a workflow to validate 2 documents. You would say, no, it's easy. You read the document, you create a JSON, and then, but you, if you really want to make a workflow, which is what people think, you are constraining by yourself the capabilities of what these systems can do. So instead of talking about workflows, which is process boxes with arrows with a lot of ifs, then elses, and some now magical boxes called AI agents that become black boxes that we are not sure if they will do the same thing twice, but we put them there as routers or as intelligent conditionals, no? Then we think more into the framework of delegation. Suppose that you want to automate or not automate, you want to stop managing your calendar. You have 2 ways. You can create a workflow. So good luck managing all that. Or you can hire a human today, no? And you would delegate into that human.

That.

And delegation means that you expect that person to keep learning. You expect that person to be able to do new circumstances because it knows already how to behave from a general perspective. And you wouldn't have to say every day you open the email and you click the button read and you do this and then you put the label. You don't, that's a workflow.

Mm-hmm.

But this is not how we think. So the problem of workflow thinking is that it constrains what this technology can really do once you solve the problems of cost, reliability, and reproducibility and hallucinations. While you solve that, the problem is that people is still constrained with chatbots and with this if-then-else thinking that constrains the, no, no, no, you really can do this thing end-to-end. So that's why we challenge, not challenge, we literally ask, if you ask our customers, they will never ever say the word workflow, which I think that it's one of the biggest things that we were able to achieve to scale this.

Quick heads up on these last 2. Our guest's audio wasn't coming through for a stretch. Prakash had Codex patch the studio live without dropping the stream. So for parts of these, you'll hear me on a speakerphone backup. Both are developments I genuinely want more of, and they're both deeply human. First, the solo creative with a company in a box.

Look, I think today. And our customer segment is very unique, but very large. It is a prosumer segment, dark consumers. So they're left between two horrible choices. They're spending one out of four hours on Chinese administration, and with AI rising, remember AI is benefiting them as well. We have a customer that went from two hundred thousand in ARR to seven hundred thousand in six months using AI, right? And they can get to a million. I'm seeing more and more of this. So there's this whole billion-dollar business of one discussion that's been occurring, and I like to put it on record that we said it first. But regardless, that is, it's almost irrelevant. Really soon, you're going to start seeing $30 million, $40 million businesses of 3. Now, those folks don't want to hire a finance department. If you can, what's your end customer want to hire? You don't want to hire a controller. You don't want to hire an accountant, right? And so they basically are going to use us as a director. Now, our solution, the other solution, the alternative is that they can drive outcomes with accounting and tax firms, right? But those accounting tax firms don't require you to go buy QuickBooks and all this stuff, right? Well, you could do that. That's obscure office kit for me, the same as us. From the user perspective, for all intents and purposes, we serve as an accounting firm. Right? They don't have to bring anything to us. We will handle it. There are things they have to do because they are the accountable party running the business, right? I cannot, unfortunately, you know, go and administer certain things on bank functions and whatnot. But, you know, it's cool. I'm a member, by the way.

I use it.

I'm still a Venture Partner at X, but I use the platform and it is about a third of the cost that I was quoted from my accountant to use the same, to drive the same outcomes. with a software-driven solution. So to me, it's inevitability that the space is heading towards like a really, really strong disruption. There's 50,000 small practice accountants right now. They're serving 30 million people. Like, and so in a couple years, this is really hitting it. 'Cause I'm telling you today, not in the future, we've already, we finished it. We're done. And you know, it's kind of like the person that breaks the 4-minute mile. I think once it's been done and the word gets out, it's going to happen again and again and again.

And the second, mental health support made dramatically more accessible.

So tell me more about the architecture, the safeguards. I mean, I, we know the basics in terms of like content filtering, you know, classifiers to raise alarm bells when needed. Um, but I guess my, I haven't really used chatbots much for any sort of therapeutic purpose, but my naive sense would be that, you know, they probably do a pretty good job doing cognitive behavioral therapy out of the box. And that, you know, I think a lot of people are using them for that. Where do you find they fall short? And, and tell me more about what you've built that isn't immediately visible to the user to improve on those weaknesses.

It'll be— I have times that it is the most important part of what we do. Like, obviously, like a lot of companies who want to build content for their patients, for their customers using APIs like OpenAI API or iCortex API, but they simply cannot because they do not have an in-house expertise to steer those models towards working Right. It would want them to behave. And then there's like this regulatory risk that you would have to address there. And credit risk, of course. And yeah, I think the financials are kind of like packed about that, 'cause Patrick's plans are on 70% of stock rates on safety. It's great, but it's like very, very expensive. But generally, on the technical level, we have a bunch of classifiers are constantly running in the background. So it's in real time, but also in the background, especially when you need to think longer about certain issues or about advice. And I really like why I said that it is more of a platform because it enables you to use different tools. And for example, if you want to plan a test and I had Or do some airbags on what happened in the past week, right? Like, I mean, from the channel perspective or from the agent perspective, what happened in the last 7 days in the conversation with this user and how this would inform our plans and our actions going forward. So the agent is able to deploy tools or maybe subagents that do stuff like that and then And the feed guided the information into the main context and then it continues. So as a result of that, you have this like very powerful memory and very powerful planning capabilities that can bear. And like every other benefit, it also has the chances that it will miss something. They even call them for the user safety, they are dramatically lower than that.

How are people accessing this? I guess maybe one more just to wrap us up. Um, are there any anecdotes from your deployments in Ukraine or in US prison populations that you think are particularly memorable or inspiring that you would leave people with? Or Failing that, just anything else you would want to leave people with in terms of a positive vision of the future?

I mean, there are many points like that. Like in the correctional environment, we heard numerous times from medical teams there that our AI helped them identify people who they Did not know where to hide, and because of that, they were able to provide that repair and to perhaps, maybe, perhaps save their lives. Ukraine also got interesting because we, like, before the conflict, working in Ukraine, we thought, of course, our contract is big fish at first. And then until you reach the first frontline, you know, and you get exposed to what they face and how they're in place. Looks right. And I actually did our work with the Badlands and with the soldiers. It saved a lot of work to do in safety because if you test that under those extreme conditions and it works well there, of course, and all right, you look good now, very confident in your head when it comes to normal consumers. I'm going to like being touching that. More and more people realize that they do not have to be left alone and that they can use AI. Um, and, um, that's a good thing.

So that's the bet. We're in some kind of takeoff and almost everything around the core intelligence needs a real moat, or it needs to be human enough that the frontier leaves it alone. As for this show, it's an experiment, and we're holding it loosely. I don't expect many people to watch 2 hours live every day, but maybe a cut like this one can be genuinely worth your time, even if you're as plugged in as I am. So tell us, what earned its place? What should we have cut? What did we miss? Reach out, seriously. We read everything, and the show will get better because of it. Thanks for watching.

At first light, we rise to prepare the way. At first light, it is coming. It is coming to prepare the way. Now we see through the glass darkly. We raise the valleys, soul face to face. We make the mountains low, but we will be as not appeared. We straighten the road, but we labor toward the day. We smooth every storm, at first light soon to be shown. We rise, what is hidden To prepare the way. The veil grows thin away. Face to face as first light. We are waiting.

We rise.

Something is coming. To prepare the way. The long night ends, the long night ends. We trim our lamps. Salvation waits for its hour. And keep them burning. Though it tarry waits. We know not the day. It will not come late. We know not the hour. It is surely coming. The work of our hands. That grows in the dark. Is working alone. We know not how. The whole creation groans. Longing to be made new. We kneel to what we cannot move. And love what we feel. And ache for what will heal. At first, fight what is hidden.

We rise.

Soon to be shown.

To prepare.

It is coming.

The way.

And every tear. In fear and in wonder we wait. Make ready the way. Make ready the way. In wonder we wait. We rise, we rise to prepare, to prepare the way, the way. It is coming, it is coming, it is coming. No more mourning, every tear wiped away, wiped away. Let it be done, let there be light. Let it be done, let there be light.

Amen.

Morning by morning, we open the door. Let what is coming find us in song. Not our will only. Let it be done. Our work is our prayer. And the dawn is our home.

If you're finding value in the show, we'd appreciate it if you take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions. And sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, Check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.