AI:AM — Self-Improving Tax Agents and Catholic AI · June 2, 2026

So yesterday we literally made changes to the studio using Codex in the background while we were live and had actually our very first guest had to refresh the page to get the audio from me. Prakash has vibe coded this entire thing and it's pretty cool. I think there's gonna be a lot of value and both in the Utility of doing this all ourselves and being able to customize it just the way we want. And also a lot of value in just kind of showing off how we're doing this. We both took a bunch of notes yesterday from what I would say was, you know, in some ways a good first pace mile in the sprint through the AI marathon, but in some ways it was a bit rough. So we definitely both took a lot of notes and we've been prompting away and developing the skills to try to make sure that every day that we come back and do this, we're going to have a better, tighter format. And it'll be interesting to, to watch this thing evolve and see if we can get to the point where we feel like we're really hitting a stride and have kind of a sustainable pace that is serving us and the audience. I think one of the things I really want to make sure for myself is that I'm not becoming a content person. In short, I really want to make sure that I am still focused on learning and understanding as much and as well as I possibly can. And building in public and, you know, genuinely sharing the process by which we're doing this, I think is going to be a huge part of that. I was just kind of going off on my own and saying, you know, describing a little bit of the recursive self-improvement philosophy of this project and how for me— and I think you have a little bit of a different angle on this, but I really want to make sure that— I kind of think of this as like live reporting in a way. You know, one of the reflections yesterday when I took my run was I really want to make sure that I'm making something that is of value to the true insiders. You know, I think of somebody like friend of the show Dean Ball, or I don't think— I think Zvi reads only and doesn't really watch much stuff unless he sort of has to for review purposes. But I want to make artifacts that are hopefully of value to people that are really plugged in. And so, in that sense, I think what we're doing live is sort of live reporting. But there's going to be all this sort of apparatus around the— actual live engagement that we're gradually building out that hopefully will make additional artifacts that are going to— I aspire to having them be useful for people beyond those that can actually tune in live, right? I mean, I think it's going to be tough to get too many AI engineers or OpenAI engineers, let's say, watching 2 hours a day. So how do we sort of both engage people live, make sure that what we're doing is actually serving our own understanding and advancing the sense that we can make of everything that's going on, and cashing that out into artifacts that can be shared and hopefully have a little bit more durable value? Those are big reflections. From day one for me. And certainly many prompts have flown to try to instantiate that stuff, even in just the first interval between yesterday and today.

Indeed. And as always, it's— for me, the most interesting thing is that there's so much that happens every single day now, right? And it's not There is a lot of value in kind of having a worldview today and updating it in like a week or so and seeing how much you've moved. Because I find that a lot of people don't update their priors. Gary Marcus being one of them.

Gary, if you're listening, Your ears might be burning.

Well, yeah, I've been, I've been, you know, like, like many people, I've been blocked by him. So at some point, at some point in the last 3 years, you know, if you, if you, if you've teased Gary too much, he's definitely blocked you.

It's funny that I don't think he's blocked me. And arguably I deserved it because one of the first things I ever wrote in a public way online, and it, it wasn't, It's another example of me being not super strategic, but just in that moment and feeling like I had something to say, was when he was on the Ezra Klein podcast. And, you know, we didn't have as much time yesterday as I had hoped due to some iterative improvement that we did in real time to talk about Ezra's piece and kind of working backward and, you know, having a vision. I think he's broadly been really good on that stuff. I think the, the Gary Marcus episode was a bit of a miss because it was like full of all these things of, you know, AI can never do this, it can never do that. And then I just was so upset about it that I went off and wrote a, I don't even know, like a 100-long, 100-tweet-long thread basically attacking his appearance and all, and, you know, kind of correcting the record as I saw it on everything that he said that I didn't really think was accurate in that. podcast. And that was like the first thing I ever put online in a meaningful public way. And it ended up being like much more viral than I anticipated. And it kind of turned out to be quite the dunk. But to me, he's been like quite gracious actually and has not— and I haven't like chased him down online, you know, for every time that he's moved the goalposts or, you know, said something that, you know, well, AI will never do something that it can already do, which I think is continue to happen probably dozens of times. But for the record, he's been gracious to me and he's— I'm still not blocked.

I deserved it.

I think I honestly deserved it too, but I didn't get it.

So I normally refrain, but it was just too tempting a target. And all I did was he had a— I quote tweeted something he tweeted like several months before. with a Wile E. Coyote, like, you know, cartoon of, you know, the coyote falling off the cliff. Yeah, he blocked me for that.

Well, we've got a bunch of news to cover again today to warm up, but any thoughts or reflections on takeaways from day 1 and, you know, kind of what are your top priorities for Recursive Self-Improvement right now?

So I think we've spoken on this offline, but we have a pipeline of work, I guess, that starts from opening the dates and then inviting the guests, confirming the guests, and then getting all the content, the research ready. We do a lot of research for these shows. And the research is because it's largely helped by AI. I think, you know, and we're happy to share with the audience what we do. But over time, we've both refined these prompts that we have for the AI and have show notes, et cetera. And then we end up with, you know, the stream itself. And then we end up with post-stream, We have a lot of artifacts to cover. We have, you know, distribution to all the channels and clipping. So if you are not in the new media space, what is happening right now is that the short 30-second to 1-minute clips get something like 10 and 100 times the distribution of the longer formats. And so the clips have really become the financial engine behind most of these new media shows that you see. When people talk about new media, what they're really talking about is clippable media that is being broadcast to social channels. That is essentially what new video media is. The livestream is kind of producing the content itself in bulk, but then the refinement process and the clipping process is what converts This kind of— and I look at it as primary historical research really at this point, because we are talking to people who are actually in the flow and are making history and have made history in the last 3 or 4 years. And that becomes clips that then go out. And our hope is that this whole entire cycle turnaround really speeds up a lot, which is what enables the And what we've found is that your words really have power on social media, right? Like if you have a take and you throw it out there and you have a following and you are loud enough, it reaches a lot of people. And that has, I think, significant value. New media, these clips are basically replacing what people used to get from news sources of all kinds. And people are selecting news sources which are meaningful to them. So that is— and to the extent that we can shorten that cycle time down so that we get it in the flow while things are happening instead of in retrospective, that would be ideal, I think. And that's where the recursive self-improvement comes in, because you can recursively self-improve your own environment, like the AI field itself, by feeding that information back in and educating people and pushing the message out. So that is really my idea of how we could— it's really recursive self-improvement, not just of the show, but of the entire field itself that potentially we can do. Very ambitious.

I know.

Nathan, what do you feel?

I think that one way to think about what we're building right now is kind of a harness for ourselves, where we are in some sense little proto-AGIs, but our affordances are by default kind of slow, right? Like, we— it's hard to process all the information we need to process. It's hard to stay on top of all the discussions. And so creating these skills that sort of preprocess information, present them to us in hopefully a really effective and time-efficient way, set up hopefully useful and easy to follow and informative discussion, but do that ideally. I think at this point we got to be aiming for at least an order of magnitude speedup relative to what we could do on our own. And I think that obviously the core challenge of this is going to be our own human brains, or at least I can speak for myself and say, you know, I can have AI agents go out and do all kinds of information preprocessing. I don't necessarily learn something from that. So I'm going to need to hold myself accountable to making sure that I'm actually learning as much with AIs doing more of the work than I was when I was doing more of the work myself. I think the jury's still out for me on exactly, you know, how to accomplish that. But that's got to be the goal because there's just this time dilation effect that, you know, as we're speeding up, we got to find some way to slow down time to make sense of all the things that we're passing by. And this harness that, you know, it kind of allows us hopefully to drop into an effectively preprocessed view of current events is my best idea right now for how to speed up my own learning to a point where it can hopefully keep up, you know, or at least get closer to the singularity. So with that, maybe we should cover some of today's outline. What do you— you got anything else or you want to jump to it?

Yeah, we can, um, maybe, maybe I can, uh, just share the, uh, the overnight, um, overnight news. So let me just, uh, plug this in there, um, and get— let's see, we have, uh, yep, there's Blossom. All right, there we go. And let me pull it up on screen. So the major thing that happened overnight is Berkshire Hathaway has agreed— Google decided to do an $80 billion fundraise. And I will show you this very interesting graph. This is Google's stock buybacks per year. So they increase stock buybacks year by year, 2015, '16, '17, '18, '19, '20, '21, '22, '23, '24, '25. And then this is the stock buybacks this year where they sold stock. So Google has sold stock for the first time since the IPO. They haven't sold equity before. Normally what tech companies do is that they sell convertible bonds for a variety of reasons. Number one, founders don't want pure debt because when you go bankrupt, that debt will come back and Haunt you. So they go for convertible bonds where they can convert it to stock where interest rates are basically zero or de minimis. And that's what high-growth companies typically do. Instead, Google has issued street equity. And there's a bunch of comments and takes on this online. The first thing that you have to note is that Berkshire Hathaway has agreed to put in 12.5% of the $80 billion. So Berkshire's putting in $10 billion. Warren Buffett at this point has about $400 billion of cash on the balance sheet. He sold off a bunch of companies. He thinks valuations are high. He's waiting for valuations to get better, but he put in $10 billion here, which is a big chunk. He has now a 7% allocation to Google in his portfolio. And he's got some dilution protection. So he had a slightly discounted rate from the current rate. As a result, Google stock took a tiny plunge, like maybe 3% down. It's a large company. $80 billion in the context of a several trillion dollar company is not significant, but it shows the Capital requirements, which are hitting even the largest players. And Ruth Porat decided to raise equity, not debt. That means that you normally need to keep this kind of debt-equity ratio. So if you raise more equity now, that means you're going to raise 3 or 4 times that amount of debt later. So there's going to be debt issuance coming. So there's going to be another couple hundred billion dollars of probably debt issuance coming for Google. So that's probably a quarter trillion of capital Google taking in in one shot. And then we also had yesterday Anthropic filed an S-1. So an S-1 is a confidential registration statement with the SEC where you tell the SEC, we're thinking about going to IPO. These are our numbers, but we're not telling anyone else the numbers yet. So they filed— And then not only did they file a confidential registration, they announced it on the blog, which is like, why? Um, and thirdly, Sam Altman was on CNBC yesterday, uh, really talking, uh, talking smack about Anthropic. He's like, I don't know why, I don't know why they have to say all these bad things. Like, everyone's gonna lose their jobs, like, 6 months ago, and now they're not talking about it. Um, in essence, what we're having right now is this race for capital. There's a sense in the market that capital is scarce or will become scarce. And when capital becomes scarce, the price of capital will go up. And I think Google issuance, the Google issuance is a sign that the price of capital is starting to go up and that they're issuing the equity in order that they may issue debt at a lower price. And so this is something we kind of seem to be having Higher inflation rates, but also higher returns on stocks. And that's great that you have the high returns on stocks, but if you are a working person on the street and you are not really invested in stocks directly, maybe you have a pension fund or whatever, but you're not directly invested, you're watching gas prices go up and inflation go up. You're not getting the benefit of this. So that's one thing to note. It's very interesting to see Berkshire in there. If he's in there for 10, he's probably going to be in there for another 100 later on. So I think you might see Berkshire as one of the primary funders of this infra at this point. One other thing, margin loan balances. So margin loan balances, when people buy stock, you can take a loan from a brokerage. And when you take a loan from a brokerage, you have a margin loan and that Makes your returns better because someone's lent you the money to buy stock. But if stocks go down, you have a margin call, you have to repay the money. And there's no limitation of liability here. You can go absolutely bankrupt if you can't cover. So margin loan stats are at all-time highs at retail brokerage. It's more than $100 billion of margin loans just in one brokerage, which means across the market. You have trillions of dollars. And finally, AI2 seems to be breaking up. So AI2, they had the, I think, Olmo models that they put out. They were American, very strongly open source group. And they disclosed a lot of their secrets while they were training these models. They disclosed everything. One of the key questions is, they were very well regarded technically, and but they were not making a lot of money. And so one of the questions has always been, can they keep their talent? And it seems like Nathan Lambert, who was one of the leaders there, has— is leaving. And I would— I saw another post from someone else who's also leaving there. So that is, you know, one more, one more thing to note. Another US open source champion seems to be breaking apart. So Back to you, Nathan. What comments on the overnight news?

Well, definitely the Anthropic preparation, not confidential, confidential filing stands out. Some of the interesting commentary that I've seen around that, first of all, is like, okay, what's the valuation going to be? Excuse me. One of the more interesting posts was basically saying, At $1 trillion, it's obviously a steal, and so it might be significantly more than that. I'd be interested to hear what you think the opening and fair price for Anthropic might be. And obviously that's going to be driven by sentiment as much as fundamentals. Fundamentals are strong, but the sentiment might be even stronger. And then at the same time, we have Bernie Sanders, who is taking a lot of positions on AI, some of them arguably contradictory. has put forward this idea that there should be some national equity stake in the biggest tech companies. And I would say, like, I'm very sympathetic to that idea on some level. I think there's, you know, a lot of— friend of the show Dean Ball has been critiquing him repeatedly for I think the contradictory aspects of his portfolio of positions, which is like, I want to have an equity share for the public, but also we can't have any more data centres. And that doesn't seem to fit together super coherently. Maybe you can make sense of it. But I am pretty sympathetic to the idea that like, yeah, I mean, American wealth is estimated at something like $150, $165 trillion across all the major asset classes. That's basically 5 times, a little more than 5 times GDP. And you look at just the market cap of the 5 or 10 biggest companies and you're starting to get a not insignificant share of national wealth is bound up in the 5 to 10 top companies. I mean, literally just the top 3 of Nvidia, Apple, Google gets you to about 15. Like, those 3 companies are worth almost 10% of America's entire national wealth. And certainly the AGI-pilled among us think that might continue even more dramatically into the not too distant future, right? If all of a sudden you add Anthropic to that at a few trillion and OpenAI to that at a few trillion, and it's no longer Crazy for people to speculate that Anthropo could be a ten trillion dollar company in a few years. Then yeah, it's a little crazy to have that much wealth bound up in so few hands. So I'm pretty sympathetic to the the Bernie argument that there should be some sort of share. I'm also very sympathetic to Vitalik Buterin, who said, "Man, this sucks." Like The US population's only 4% of the global population. These companies were started with a benefit all humanity, and now they're like gonna IPO and, you know, there's like, and they're gonna give away some through the foundation or whatever. And Bernie's got this idea that we should have a national wealth fund, which could be good, but what about the other 96%?

Mm-hmm.

I think that discourse is gonna be something really Interesting to watch. And it's also going to be— I mean, I'm throwing a lot of questions at you at once here, but another news item that came across yesterday was OpenAI now going available on AWS. So this continues the trend of just everybody being in bed with everybody, everybody's kind of fates being tied in one way or another, balance sheet or— rev shares and all of it, right? It seems like we've sort of created, kind of backed into creating in a way, like one big AI megacorp where a failure of any one of them seems like it would have massive contagion, or at least there would be significant risk of that at this point.

Yeah.

It doesn't seem super likely to me. I think basically the demand is going to be there. Barring some external shock. But external shocks could happen, or I could be surprised. And this whole kind of probably already, once you count all the companies that are deeply intertwined in all these different ways, you have something that's maybe a $30 trillion megacorp that is almost for sure too big to fail. Right. So I think that's the other thing that makes me very sympathetic to the Bernie argument is like, whether or not the government recognizes it, I think at this point they are the financier of the AI megacorp of last resort. Because if one day OpenAI can't make their payments for whatever reason, there's going to be some sort of step-in bailout moment where it's like, yeah, we're not going to let everybody default. We're not going to let this contagion go that crazy. So, yeah, we'll come in and patch it up somehow, monetize, whatever. You understand those machinations better than I do.

Yeah.

But if we are on the hook for that as the public, then I do think we also should probably have some better claim on the upside than a Larry Summers flavored board that is going to give out some grants. So what do you think? How can we shape— I guess my first question would be, are you at all sympathetic? And how would you shape the Bernie proposal to try to make it something that would actually be a winner?

So Bernie's a socialist, right? And one of the things that they do is they try and pull wool over your eyes a little bit. And the fact of the matter is, I think US corporate tax rates are like, what, 20% or something? I will trade I will buy Anthropic stock and I will trade 20% of Anthropic stock for the government's 20% right of 20% of the taxes. And I would do better on the 20% of the government's 20% take on the taxes than he does on his 20% stock. The fact of the matter is taxes are super equity. They are Non-dischargeable in bankruptcy. And it's better than preferred equity because preferred equity, you have a dividend that you have to pay out. Normal equity, you don't. So Anthropic can be there and say like, look, I'm not going to pay out any dividends. That's up to me to my equity holders. They can do that. If you have taxes, they can't do that. They have to pay the taxes. You have no choice. And the taxes come out. So the fact of the matter is when you look at the cash flow that you get from equity and the cash flow that you get from taxes, the cash flow that you get from taxes is more certain and it's non-dischargeable in bankruptcy. And there's a lot fewer ways that you can play games around it with it. And the board and the the management don't have control of whether they want to pay it or not. Unlike shares, the board and the management have decisions whether they want to pay out dividends. They can choose not to pay out dividends. They can choose not to buy back stock. They can choose to reward themselves. So shares are not that powerful as taxes are. Taxes are really powerful. And so what Bernie is saying is, you know, he— you could just say like, we can— we'll just, you know, increase corporate taxes. You could say, hey, we will, you know, structure a ladder of taxes so that if you have— if you're making windfalls, we will have higher taxes on you. All of these are things that you could say. Instead, he's doing this 50% because the 50% is in addition to the taxes. That, that, that's, that's the, that's the magic trick. The magic trick is not that he wants a stake, because the government already has a stake through taxes. The magic trick is that he wants more than what he has currently. And this is the illusion that all of the socialist, the Dem socialist proposals all have. They are all about addition. This is the same reason the UBI proposal has not worked out well, because the original libertarian UBI proposal was a replacement of current welfare with a UBI so that you didn't have to means test for welfare. Because the means testing bureaucracy is very large and more expensive than some of the welfare that you actually give out. But that would involve non-additional dollars. So it's like, I'm going to do UBI and replace your food stamps with UBI. All of the people who are getting food stamps or eligible right now would get no new dollars. All of the new dollars would go to people who are not eligible for food stamps, who don't need them anyway, right? So this is why the U has never worked out because the the the people who are supporting the people who need this extra money, they want additional dollars. They don't want you know political bureaucratic replacement of the existing dollars with a new structure. No no new additional dollars. They don't want that. Same same thing Bernie's doing. Bernie doesn't want the replacement of the existing; he wants additional dollars, and and that's that's the key thing. The key thing is that he wants an additional stake, right? And he doesn't want to do it through the normal path because the normal path you'd have to go through Congress. So instead, you're doing this massaging and trying to get shares, and the shares would then be run by a bureaucracy. Congress would not. Because who has the power to allocate taxes? Congress. He doesn't want to go through Congress. He wants to set up a bureaucracy, like a sovereign wealth fund that both politicians on both sides are doing this, right? It's a slush fund that you can then appoint people who are your friends into positions in those firms and they can get paid and you can direct the flows of those funds without Congress, right? Congress has the power of the purse. Use Congress, right? This is what I find so annoying because the US has this marvelous institutional structure, marvelous, right? And both parties basically don't want to use this institutional structure. And Congress doesn't want to use its power, doesn't want to use its power of taxes or passing laws or legislation, kind of backs away and like, hey, we won't do anything permanent and we'll let the executive kind of run circles around us. It's very annoying for me. Very, very annoying. Like, well, let me try to challenge you on that one more time.

Because I think, and it'll be an interesting segue, perhaps into our first guest, which is going to be 2 forward-deployed engineers from OpenAI, who are creating, you know, amazing things, as always, and, you know, potentially changing a lot of organizations. And potentially, you know, we'll see what they are willing to recognise the impacts might be in terms of employment. But I think on the taxes front, it's like, well, wait a second, these companies probably aren't going to turn that much of a profit. They're going to do the Amazon strategy where they're going to show very little profit for the foreseeable future. They're going to be able to reinvest in data centres, in robotics. They're already talking about Dyson spheres, right? They're going to try to put data centres in space. And so the CapEx is endless on this. And it's not hard for me to imagine a situation where in whatever, 5, 10 years' time, who cares exactly how long it takes, you have an AI megacorp that is sort of, whether explicitly merged or just intertwined enough, 10 other megacorps collectively worth 50% or more of US national wealth.

Yeah.

And yet no profits were actually made, and so no taxes were actually paid. Meanwhile, the companies sort of do what the platform owners have done in their personal lives broadly, which is like not pay a lot of taxes, continue to hold the equity, borrow against the equity, and basically live a low tax lifestyle. So I am still sympathetic to Bernie on the level that— and I think you're very kind to the US on paper. I think we've got, yes, an excellent— Mm-hmm. Institutional structure as it's actually playing out in practice. It's not going so great, right? I mean, that's pretty consensus opinion at this point. So I would frame, if I want to be at least somewhat charitable to Bernie, I would say this is the pro-growth Bernie who is not trying to— and of course he's got the anti-data centre side, which makes it a little more hard to square, but leave that aside for a second. just this proposal of share of equity, like, that's the pro-growth Bernie that says, well, let's align our public interest with these private companies. Then, you know, exactly how that gets kind of paid out or, you know, turned liquid— do we all start, you know, probably the government starts borrowing against its shares, I suppose, of these companies and doling out benefits that way. But In some ways, I don't know, that feels to me like you could say it's more— what I like about it is it seems to align the public's interest to where this technology is all going, which presumably is to the moon, literally, inasmuch as we're going to have data centres in space.

So, well, The cash has to go somewhere. When Amazon spends money, it's not as though the cash disappears. The cash ends up being spent on other things. And those other people end up spending on other things. And the government at the end of the day doesn't need to take the tax dollars from Amazon. It can take it from the Amazon employee. So there's this question of tax incidence. The tax incidence may not be incident on Amazon, but the dollar is circulating in the economy and the government has a take on sales taxes, on income taxes. And Bernie's preference is that all of the money is spent on employees anyway. And that's where the income tax hits. So why would you care about whether the company makes corporate you know, corporate, you know, income and pays taxes on it, when all of the money gets allocated to other entities which end up paying taxes anyway, right?

And well, I guess that— yeah, so this is— maybe we have to switch gears a little bit to our first guest. But I do think how many employees— I mean, Amazon has a million employees or more, right? OpenAI, as we're gonna hear, is hiring for deployed engineers. But I don't think we're gonna see a million for deployed engineers. So it still leaves this question of like, what would you want to tax? Like, is it a token tax? Is it a data center tax? Is it a replacement tax of some kind? If our tax bot does the work of large numbers of accountants, I don't have an answer to that. But it doesn't seem like we can just naively say that, oh, well, it'll come in payroll taxes because I don't know how much payroll is gonna scale with the economic activity that is powered by AI. And maybe with that, I should invite you to welcome our first guests today to our broadcast. Our broadcast.

So our first guests today are John Nawesij and Arthur Fernandez Araujo. They are both OpenAI 4 deployed engineers, and they have been working on tax AI, and they've been working with the OpenAI team on tax AI and also with Thrive and Crete. And what they've been doing over the last 6 months has been, have been, you know, as we went into tax season in April, They have created tax AI that has managed to process, I think, 7,000 or so claims over the last 6 months. And they've seen a marked improvement in how many cases the AI they've built can handle. It's also interesting, I think, because Thrive has been working on a strategy which many people in the space are trying to deploy, which is what we call the roll-up of service firms. And the roll-up of service firms idea is that AI can now take on these service-type jobs like accounting, etc. But currently, all of these service firms are fragmented into thousands of firms worldwide, each one with several partners and you know, staff doing stuff. And now for the first time, you can maybe merge all of these into larger firms. And instead of selling software subscriptions, you can sell the service itself to other companies and to other people. So without further ado, John, Arthur, welcome to AI AM.

Hey, Prakash.

Hey, Nathan. Thanks for having us.

Yeah, thanks for having us.

Um, let me just ask you, um, most people hear tax AI agent and they assume the hard part is making the model smart enough to read tax documents. But in your tax AI post, it's almost the opposite. The system got better because the product was engineered so accountant creation corrections became evidence. The numbers you had were 7,000 returns processed, 1/3 of prep time saved, throughput up about 50%, and within 6 weeks, the share of scored returns reaching at least 75% field completion moved from about a quarter to 86%. So the question became not just can AI prepare a tax return, can a production workflow teach an agent what to improve without engineers hand debugging every failure. So how did you guys do that?

I think a good point, I think, to clarify before we dive into this is what is self-improving here. So we're not really talking about self-improving the model itself, but it's mostly the harness around it. And this workflow in particular, I think, You got to some of it in your initial comments, Prakash. I think it's a good proving ground for this where you have very messy inputs, but also a lot of basically practitioner judgment that is also part of this workflow, review workflows. But you have a very good way to measure the outcomes. But what is improving is essentially the harness around what the model is basically leveraging. In order to produce the preparation, the extractions.

When you say harness, are you referring to kind of like every time you come to an edge case, the humans kind of help the model to like figure out the edge case, and then that becomes part of like a memory of heuristics that you then apply the next time you come across the edge case? Is that what's happening?

Yeah, so when I talk about the harnesses, we leverage Codex to do a lot of the work here, but there is basically the set of instructions, skills, the data that you use it in, the specific way that you use this. This is part of our TaxAI agent. But then when you encounter these edge cases, what we document in the blog post is exactly how you make sure as like a good coworker that if you provide a correction, the next time it can be effective at basically not making the same mistake. So it's around changing the structure of what Codex uses, the skills, the durable artifacts, so it cannot make that mistake in the future.

So when you say skills, is it literally like the skills that other people are making for Codex right now? You use a skill creator and you say like, hey, this is a 1040 form and this is what I want you to do with it. And as you work through it, you're like, okay, this happened, fix it for me. And then it documents it in the skill. Is that what's happening?

Yeah. So it's the same skills that you and I know from using classic Codex. Well, what's interesting here is that, so there are these skills that are available, and over time what we sometimes notice is that the models got better as well. And so when, you know, what used to be a skill maybe 2 or 3 months ago, today, you know, is potentially— we should deprecate it because the model is able to do what is in the skill by itself. And so this is also something that's, you know, that's very interesting that we observe, is that the skills themselves, they change. And part of the self-improving loop is also letting the harness the ability to propose new skills potentially to also update all the content that's available for the next loops afterwards.

I think that's really interesting. My friend Daniel Mesler, who created Personal AI Infrastructure, as far as I know, coined the term bitter lesson engineering, which, and I also think Logan Kilpatrick recently spoke to it. He said, as so many have said, the model eats the harness. So what you're setting up here is basically a sort of tick-tock kind of back and forth where with a new model, there's this opportunity for it to clear out all of these heuristics that it accumulated previously, because now the model might just be able to do those things. And so we want to kind of clean house, tidy up, get rid of all these potentially distracting things and let the model excel where it excels, but then you'll probably start to accumulate another layer of heuristics. And that process works in tandem with model upgrades so that we hill climb all the way to full tax automation. And you guys have made a lot of progress. I mean, could you maybe— I think the numbers are, of course, interesting. Like, at the beginning, not that many of the returns were completed accurately. Now, like, a majority of them are are mostly accurate. Are we going to see a— what is the sort of plateau or end state? Are we going to have a full self-driving sort of thing where we find that the edge cases and the long tail are just problematic for years to come? Do you think that it is in fact like, hey, there's actual law around this, so in a reasonably finite period of time, we can actually knock the whole thing out. Um, what's your expectation for like what, what the future of this curve is going to look like?

So, um, the thing that we actually observe, uh, in practice is that the— with the new models getting better, the new models getting better, and the harness around it that we, that we create that is also better, is that we are able to tackle, uh, like more complex forms. So for example, you would say that A W-2 is potentially, you know, like, is actually quite simple. But then when you look at a Schedule E or a Schedule C, like, such as, like, rental properties, then it's way more complex because you have data that's coming in from different sources. It can be client notes, it can be Excels, it can be PDFs as well. And so the more we advance, the more we are able to tackle things. But kind of like taking a step back here, one thing that is really important is to have a good ability to measure how good you're doing. And this also ties back to your previous point on, like, you know, how do you know when you have a new model if it's better or not? And I think, like, partly when you generate, you know, when you create any AI-related software, is that having a strong way of having evals that measure exactly how you're doing and how your metrics are doing and to kind of backtrack in time and see, okay, if I had run the evals at that time with that knowledge, what would it get with the new harness that I have? And this allows, in our case, this allows us to have some very strong conviction that what we are doing was actually better and was actually growing.

How do you get feedback from users? I think, you know, one of the ways we're conceiving of this show, which we're doing Live, we're spending a lot of hours live, right? So it's got to be, in a way, its own experiment in recursive self-improvement. And one thing I realize I don't have a great crystallization of yet is, okay, we do, you know, a couple hours live, all the, you know, the studio that we're in is vibe coded, there's lots of opportunity for improvements everywhere. And then I just kind of do a brain dump into a doc and, you know, put that in as a prompt. I think one of the key insights or practices that you've developed probably is how do I most effectively get the corrections in a high-context way from users? I'd love to understand, you know, what, what your best practices are there so I can go point my agent at the transcript of this podcast and have it apply them for me.

I think for the, for the type of problem that we, we try to solve, I think what was interesting there is we didn't try— both John and I, we come from, like, software engineering background. And the way that you basically gather feedback, sometimes you would go to the field, interview people, you'll get also like tickets with opaque feature requests and things like this. And then your engineering team, it's your job to essentially like distill that, figure out how that ties into the product roadmap and iterate. I think what was interesting here about this piece is how we, we try and flip the book and try to drive the AI transformation from the inside out instead of the outside in. So trying to give the experts that are using the product every day the tools that integrate very seamlessly with how they're using the product. So every time they basically took an action like a correction. We captured that information and we had the whole user journey in a way that like if we noticed that essentially we made a mistake in that specific part of the product, which is related to the preparation, how can we essentially leverage the entire user journey to improve the product? In a way, it's kind of like, it's similar to maybe watching a video of the user using your product. But you're not capturing the whole video, which requires a lot of human attention. And human attention is a very scarce resource. So the way that we try to really distill this is by getting the highest signal parts, the highest signal traces from what the users were doing and the product exchanges. So if you think about this in terms of like a spectrum, we either have just like what we produced and what was the truth. So you don't have anything that happened in the middle. Or you have, for example, something that is like super detailed, like an application trace, and you have the whole application trace, which is maybe too detailed, and it's like harder to make like semantic meaning out of it. So this is also something that recently we published a, a piece in the OpenAI Cookbook around macro evals. And I think what we do in this piece is essentially this idea of a macro eval. You don't have just the ground truth and the inputs. You have some very targeted things in between that are relevant for the problem.

One of the things that strikes me is that there's always this balance between the model should be good enough to do everything, the generality of the model. And the kind of fine-tuning harness engineering that you do for the task, right? So to what extent do you see kind of like the harnesses that you've built, their capabilities being subsumed by the next generation of the model? Do you see that where you start off building a harness and then it works well, but you also tell— the team that is working on post-training that, hey, this is what works well, this is what doesn't work well. And do you see some of the pieces of the harness that you built kind of get absorbed by the next generation of the model? Do they end up using it for training and then it becomes part of the next generation of the model?

Yeah, so yeah, this is a very good point and this is something which we definitely see. So if we look at what we had a few months ago compared to now, like the harness and the software has definitely changed. So from our end, what we try to do is to make sure that you don't kind of go against the model capabilities. So you try to— from what I've seen is the best, like the success happens when you're able to develop the software and the harness in such a way that it goes along the model being great. And so sometimes it means, for example, deprecating skills or changing some, like making such that the codes of the models are able to handle some more complex workflows. But then it also means bringing that feedback and our learnings back to the team internally. And so definitely what we do is we try to observe where the model lacks, be it both in specific topics or concepts, or on the harness itself. It could be, for example, bad at some point at getting a specific type of information. And so if you observe this, then you bring this back and ideally through evals. And so this allows the models to become better at this point.

So I imagine the humans that you work with, the teams that you work with, they have a lot of extra time now that, you know, now that, you know, 70% is being handled, uh, by the, um, you know, TaxAI itself, uh, they have more time. Like, what are they doing with that more time? What has— how have they ended up allocating the time that has gotten freed up?

Yeah, I think, I think that's a very, uh, uh, interesting question. To talk a bit more about, uh, how this was deployed, so we, we piloted with, uh, with a few firms on the CREIT portfolio that also enabled us to like measure exactly how that affected the firms that the software was piloted in. And I think preparation was a very good place to start because it's definitely a high-friction task for the practitioners where if you imagine this industry is very seasonal as well. So in the weeks before the tax season ends, you can imagine like how much work that the folks have to do. And I think what really unlocks here is like a very differentiated kind of service that they can provide to their clients, certain things that they couldn't do before in terms of like the relationship with their clients or even being able to take on, take up more work like near the tax season end where they would need in the past they would, would not be able to. Uh, so I think it really changes basically, uh, uh, it really reshapes their, um, their, uh, their tasks, like especially around like the, the periods where they have to do quite a lot of work. Uh, I think we also document this in the, in the post, like, uh, for some of the really, uh, hard returns, that would be something that would take them, uh, 8 hours sometimes of basically going through like a lot of documents sometimes Having to reconcile information from different documents until the number that they would put into the tax engine. So I think it really reshapes their job in a way that is more positive to them. They can focus on more strategic things.

How far do you— I feel like it's funny. Tax is one of the examples I kind of think about the world as being on sort of a spectrum from, on the one extreme, things that I would love to have lots more of if they became cheaper. And I sometimes think of, like, massages being a great example of that, right? If a massage was 100 times cheaper, I'd probably get a lot more of them. And then on the other end, I often actually use tax as an example of something that even if it were a lot cheaper, I probably wouldn't buy that much more than I'm buying today. So because I'm just like, I want to get it done. I want to be thinking about something else, right? Like, this is not a place where I want to spend a lot of my time. So I'm very enthusiastic about having an AI that can do my taxes for me at 1% of what it used to cost. But when you talk about the professionals or the businesses being able to serve more customers, It strikes me that they are taking share from competitors rather than really expanding the market. I guess there's some retail margin perhaps, or some small business owners that are currently doing it themselves that could get it done professionally as it becomes more accessible. But that seems sort of like a one-time market growth where we'll shift from some people being not served to being served. And I guess I'm wondering, Honestly, that feels pretty inescapable. I just can't imagine there's that many more people that are like, I really want a deep relationship with my tax professional, and if only they had the time to spend with me, then we could really be close indefinitely into the future. How do the people that are actually doing this feel about it? There was the story, of course, from Meta of Zuckerberg having all the keystrokes logged so that they can train their AIs on it. It seemed like the public reaction to that was Broadly very negative. You know, people don't like to be put in the position of training their AI replacements. And yet this is like also clearly competitively where the market is going. So what has the social reaction been from the tax professionals that you guys are working closely with? For me, that's maybe the most interesting part of your experience.

Yeah, so I think like the main reaction that we, that we observed throughout, uh, the months and especially in the weeks coming, uh, to the, to the end of tax preparation is that, um, as Arthur mentioned, we have seen like, for example, we had one senior, uh, accountant that for like last year I think spends around like, she spends around like 180 hours of And this year it shifted around 15. So like this gap is pretty huge, right? And they see it. And the way it works is like, I think like from the feedback we receive is that it's very positive because they are able to focus on more of the high end, like the high spectrum of the task where they are able to bring more value. And the fact that they don't see the number of like the the size of the business that they are able to tackle reduce, and instead it actually grows. I think this is pretty exciting for them. And I would also say that the tasks that are automated right now would be, for example, opening an Excel, going to look into a specific sheet, going to sum, find the right cells that you need to sum up, compare it to an image that you see on the side and see if that actually matches up. And also look at a PDF and cross-reference if the data makes sense. Right. And so there is like a high volume of work where the human value can be shifted towards more thinking about where should this actually be submitted? Should this be into that tax submission feed or should it be in that one? So there is some optimization which I think the tax experts would want to do if they had more time and actually are not able to. And so from their point of view, I think it's more, okay, we are able to spend that time there and this is actually way more interesting for us, but also for the— and especially for the customers.

Yeah, it certainly makes sense that the quality of life of the tax preparer taking advantage of all these tools goes up dramatically. Do you think that they are like, what is their vision Of the future of the industry? Do you think, because my vision is, and maybe they see it the same, maybe they see it differently, but my vision is like we have 10% as many tax professionals, the market is bigger, everybody gets what they need for cheap, and it's a win for everybody who's party to the transactions. It obviously raises a big question that, you know, folks like Bernie Sanders are taking an interest in around what happens to the other 90% of the tax professionals we used to have. But maybe they see it differently. I mean, do the actual tax professionals see that shift in a similar way? Or what do you think their expectation is for tax in 2030?

I think it's hard for us to put words into their mouth. I think what we can probably say is from— I echo what John communicated in terms of the mostly around the workflow that we've automated. We mostly got positive sentiment out of it. Uh, but in terms of like how Thrive Holdings is thinking about these industries, uh, in general their thesis is, uh, they acquire, uh, they, uh, they own, they also operate, uh, uh, uh, businesses that benefit from long-term technological-driven transformation. Uh, but I think Uh, uh, to, to address your question, I think it's a much better spot to be in when the transformation is driven from the inside out other than the outside in. Uh, so I think in the past, a lot of, uh, technological transformations, they definitely happen. Like, if you think about the smartphone, the internet, all of these revolutions, like, I think they happen from the, from the outside in, uh, across society. But I feel that the holdings business is a bit differentiated because they're thinking about driving this transformation from the inside out and in terms of how the companies that they hold in their portfolio operate. So I think that is probably the right model to think about, not just deploying AI, but like more meaningful integrating with the jobs on the businesses that they operate. It's hard for me to take a broader look around like the whole economic shifts that might happen in the next few years. I think it's hard for everybody to make like a good prediction around the future here. But I think in general, this is a very differentiated kind of engagement as well. For us as engineers, it also enabled us to like even build out TaxAI in the first place because We're not a vendor here trying to prove product-market fit or anything like this. We can really deploy something from day one, working closely with these folks. But I think, I think it's to your point that you made before around the curves and the charts that we produce. I don't think we ever achieve 100% for some of the more complicated returns. I think there's always going to be a lot of human judgment involved, especially on some of the more complicated parts of, of the, of like the tax returns.

So one question I had about that is that, um, I think obviously your, your professionals have, uh, and the firms themselves have liability to the customers, right? So they, they, they are supposed to act in good faith and do the work that they promised, et cetera, et cetera. Is there this sense that the professionals are really providing kind of like a liability cover while most of the work is being done by the AI? Is that a way to view kind of like they have like the last touch kind of like sign-off, but most of the work is being done by the AI and the liability though falls with them?

Yeah, so I think in practice what we are seeing, like, that would be, I think, a definitely fair comment to make. In practice, what we see is that those teams are very on board with them, with us. And like, that's something which is actually very interesting and which Thrive does in a great way. And I think like taking a step back, like, for example, OpenAI FDE model, has a bit of this, you know, like going very close to the practitioners and users. And so here in that case, what we saw is that they feel like they are on board with us and they are co-developing. They are not actually building a platform, but like their feedback is taken on a daily basis. And the people at Thrive go often on site with the people or, you know, spending time with them. And so it's really about us. And so there is not a shift of, okay, like our mission is, you know, like we spend less time doing this because the AI is doing it. It's more how can you make your job more efficient? And so, you know, like the question of reducing the time that they spend is like everyone stays aligned, I would say, in this model. And so I've like, because motivation usually go higher up, like our feeling is more that people feel like they are Like they are taken on the mission and since they are the one doing the actual work, that part is very interesting. And so they feel like they are empowered. And so I would say that they are even more motivated after this than before.

So just in a practical sense, I know a lot of people are trying this forward-deployed engineer kind of model. Several firms, General Catalyst, One of them is trying— General Catalyst has bought a hospital, and, you know, they decided to dive right in. 18% of the US economy is in medical and healthcare services. They bought a hospital. They're trying to forward-deploy engineers in there to implement AI. Like, what are the lessons that you've learned, like, deploying? And, like, what kind of advice would you give? Like, what is this process? Like, in your mind, After all the— I'm sure all of the challenges that you've faced in the last 6 months, what is the ideal kind of, like, forward deployment strategy that a company should be using?

Yeah. So I think forward deployed engineering has definitely been a term more present in the past few months. And I think it might mean different things in different contexts. I think for OpenAI, for deployed engineering means bridging the gap between frontier AI and business impact. But we also take a differentiated step that we're not just applying existing tools from the playbook of applied engineering.

Mm-hmm.

We usually work on some of the hardest problems and the hundreds of millions, billion-dollar problems. They require novel approaches and also bridging the gap with research and product internally. So we need to move the needle so the next version of the model can maybe be better at the tasks on the specific problems that we're facing, aligning with product strategy. So we basically operate in this very messy environment where across product research and the actual business use case, I think that's also true in other cases in terms of basically operating across different types of environments. And in general, it's a very humbling experience because you come into this problem not being the domain expert. As an anecdote, John and I are not even based in the US, so we don't even file US taxes. So we need to rely very heavily on the experts and I think it's very important essentially to figure out the best mechanism to collaborate with the domain experts. And whatever you deploy, it needs to be something that really embeds deeply with their workflow. And I think the approach that we've seen fail in the past is essentially building something which has a very high error rate trying to tackle the problem as a whole.

Mm-hmm.

So we talk a lot about this in the article, which I think is also the right way to do forward-deployed engineering in in other places, which is always thinking about iterative approach. So taking a slice that you can measure well, and that has a meaningful impact to your users, and then seeing how you can expand the perimeter and bringing them to the journey. I think that's, yeah, I think that's the general advice.

Indeed. And with that, thank you, thank you guys. You know, for joining us on AI-M. This has been one of the most interesting, I think, segments that we've done so far because I think we, you know, this is a topic that everyone is very interested in. And so it's very enlightening to hear exactly how you're doing in the trenches.

Great.

Thanks for having us.

It was great to meet you both. Thanks, guys.

Measures.

So yeah, fascinating stuff. It's, um, what do you make of the— is there any other vision other than a 90% smaller headcount in the tech space? It doesn't seem like it.

Um, there is, there are, you know, What I always say is that there are visions, but you might not like what those visions are.

Yeah, well, we lost you for a second, it looks like. But I feel like, you know, you mentioned earlier, Prakash, if you can hear me, that Sam Altman was on CNBC critiquing Anthropic for being too doomy about jobs or whatever. But honestly, it just feels more honest to me. I don't really know what to make. I mean, I think that, you know, those two were careful. John and Arthur were careful to, you know, just try to walk a fine line, I think, because I wouldn't say that they were dishonest saying like, it's hard for us to, you know, say what other people believe is certainly fair enough and probably like result of good media training. But at the Sam Altman level, I don't really see how they could be seeing this as anything other than in the end, you know, just a massive restructuring of the industry where there's just a lot less payroll at the end of it than there was at the beginning. It's, yeah, I don't know. It's hard for me to, hard for me to tell a different story. One thing I do think is pretty interesting strategically is This is the strategy also is like, you could compete directly, right? You could say, what do we need? And this, I think like Google went through this, you know, this phase, right? Where they were like, we're Google, we can do anything. We don't need to go partner with, you know, people, peasants who like, you know, work in traditional businesses. We're just gonna fly right in over the top and have the technology solve and it'll be amazing. And that'll be that. And I do think honestly that AIs can get there and will get there for a lot of these problems. But this may be the fastest path to get there because it gets you a lot more access to the nitty gritty data. And now you're aligned with at least a part of the industry so you can kind of climb these hills together to the point where at some point the model probably can just do most retail and at least relatively uncomplicated business tax. And then there will be this really interesting moment of do they disintermediate or do, do they for political economy reasons decide we'll kind of stay in the background? You know, do you, do you, do you see an OpenAI tax product that is directly marketed or do, do they continue to just kind of say, well, we're just an enabler of these other, you know, super successful firms? I sort of suspect that that latter path may be one that they choose to take simply because they don't want to be— they don't want to own all the headlines and consequences of like what they could do if they just directly went to market, you know, standalone and competed head to head, even though I do think in the end they could probably win that.

So what struck me was They were very aware that the next generation of model was going to absorb parts of the harness. And this is one thing that I've remarked on before, that the model firms basically kill their— eat their largest token users in the next generation. So you had the first generation ChatGPT-3, the largest users were Jasper. Um, you know, these marketing, you know, marketing copy firms. And the next generation of the model, ChatGPT, devoured, you know, that set. And at, at one, at one point, Sam Altman had to fly to Texas and, uh, sit down with the CEO of Jasper after ChatGPT released. And the CEO was like, you said you weren't gonna compete with us. And Sam was like, yeah, we're not charging for ChatGPT. It's a research release. You know, you guys still have the paid product. We're not charging anything. It's for free. It's a research release, right?

And great comfort, I'm sure, was found in that.

I think his name is Higginmoser or something. The guy had raised like $100 mil. Jasper was worth about a billion. And, you know, he posted like a couple of weeks later on Twitter, like, I'm looking for acquisitions. Because now that, you know, the core business was done, like they had to find something else because the core business was going to die. And we don't hear about Jasper anymore, right? And I think, I think we're, we're this kind of, you know, continuous Sherlocking will continue to happen. And the next round of Sherlocking is going to be the service firms. And some of the service firms have walked into it. So, so Harvey, Harvey, for example, is definitely like a forward-deployed engineering like organization for gathering data on law firms. And that's gonna end up getting absorbed by, you know, OpenAI. You know, it's very, very clear. And so you can now see like the next, the next layer of service firms, you know, they, you know, the accounting firms were so uncompetitive and have such high moats that Thrive was like, all right, I'm gonna buy them. I'm gonna buy all of them. And then once we buy all of them, then we can actually deploy What we need to, right? So that is a strategy. You know, a general intelligence should be able to do taxes, should be able to drive a car. I think one of the things that, you know, Elon is scared about is that, you know, and Sam kind of like poisoned the well for Elon. Sam was like, you know, someday, like, you know, we should be able to drive a car too. Like, it's not like that difficult, right? So I think that this whole chase for generality puts all of these firms in competition with their biggest users. And so every generation, every 18 months, the next level of capabilities absorbs. And so the scary thing for me right now is that what you see these guys doing on the forward-deployed edge, In 18 months will be part of the core model, right? And I think that is something that people have very— like, everyone has like a lot of difficulty like absorbing that fact. They're like, they kind of push away. Everyone has this like linear curve, linear curve. Like, no one wants to deal with this exponential curve. No one wants to believe that, hey, you know, what we're doing today at the frontier is going to be absorbed in 18 months. months. No one wants to think about that.

So yeah.

Yeah.

Well, that's a perfect transition to a little segment of field notes and some— a little research speedrun from my visit to San Francisco a couple of weeks ago now to go to this event called Recursive. Because that event was certainly populated by people who have taken the various bitter lessons to heart, including the one that the models are going to eat all their scaffolding and gradually, and perhaps very quickly or even suddenly at some point, lock into recursive self-improvement loops that could really change everything. So yeah, I guess for starters, I'll just see how fast I can speedrun this. I'm not known for brevity. Feel free to interrupt me if you want to poke at anything with questions. What I was going to try to do is describe my experience there in a few minutes and then just highlight some of the key papers that got talked about there a lot, because I do think those are things that people should be paying attention to if they want to move a little closer to the kind of core of the AI insider bubble. So this event was called Recursive. It was premised on the idea that recursive self-improvement seems to be coming pretty soon. It is increasingly the explicit plan of at least Anthropic and OpenAI and Google DeepMind to some extent, although they kind of waffle on it a little bit more. Whereas OpenAI has publicly put forward timelines of later this year for an ML research intern and early 2028 for the full AI R&D researcher that they hope will perform on the level of their human researchers. So the kind of basic theory of change there is a pretty obvious one, but worth stating that today they may have 1,000 or a couple of thousand people that they would really consider to be top-notch ML researchers. If they can get that same level of performance from models on chips, then they're only limited by the amount of compute that they can throw at it. And obviously they're building out a lot of compute, so presumably they could throw a million human researcher equivalents at problems. And they also, by the way, you may have noted, they run faster and they run 24/7. So the hope is that this will allow them to move much faster than they have moved. And kind of pull away from the competition. I would say most people at that event thought that was very credible. There was not too much debate around, like, will this level off? Now, obviously there's some selection effect there, but you could just go to— the whole event was under Chatham House rules, so I will respect that and not attribute specific statements to specific people or organisations. But you could go to the Recursive website to look at speakers whose identities were shared, obviously with their permission. And you've definitely got some notable people from the frontier companies. So these were not people that are fringe or who you would say likely don't represent kind of mainline views at the companies. It really seemed that the expectation is, yes, this is going to work. It's going to have a major accelerating effect. We don't necessarily know if it's going to have a sort of simple accelerating effect, like in a human organisation. If you went from 1,000 to 1 million researchers, you probably wouldn't get 1,000x output. So there may be some sort of coordination challenges or just kind of duplication challenges that we see in human organisations. Maybe that happens in the same way. That's one possibility where you still get acceleration, but it's not a blinding kind of takeoff acceleration. Or I would say also understood to be a credible, realistic possibility was that it is even a more profound phase change than that. And things like pre-training just become dramatically more efficient, and models suddenly have all these new qualitative abilities that they didn't used to have. such as, you know, continual learning that really works or what have you. And so everything could change, you know, in a very dramatic way, potentially very quickly once these milestones are hit. In the room, people said, and it did, there was quite a distribution. I was pretty much right at the median.

Mm-hmm.

When we were asked, how many copies of you would it take to do the work that you are currently doing with a— with the benefit of AI, the median answer was basically 2. In other words, people felt like they're getting 2 times as much work done thanks to AI. But that was also framed in an interesting way where it was like, but note that as of today, at least, if you were not there, your productivity would drop to close to zero. Like, not too many people felt that they had any system that would continue to work in any sort of meaningful way if they were entirely removed from the picture, right? So there's, there's, there's a significant productivity boost, but there's still this sort of necessity of at least some human salt into the recipe to get the whole thing working.

One, one thing that strikes me is To what extent do you think every one of these people is kind of assembling their sensory organs for the information internet, for the future information internet, right? Like, is— are they— are they like— is there this sense that, hey, I am assembling this thing that allows me to process all of the information in the world that I want to process and then act upon it? Is that— is that their sense?

It's interesting. I didn't really hear too much talk of personal AI infrastructure. It was very much more focused on the AI's own recursive self-improvement. And then a big part of the discussion too was like, how can we set that up in such a way or create some sort of self-correcting structure or some sort of governance mechanism that can keep that on the rails, broadly speaking. And there's definitely a big part of it. I guess the way I would answer your question, yes, is that by far the number one strategy seems to be monitoring. We're very, very, as a civilisation, whether we know it or not, listening to people at the frontier labs who are about to you know, in their own minds, and I believe they're probably right, set off this, um, relatively uncontrolled experiment of AI recursive self-improvement. The big thing that they are betting on is AIs monitoring other AIs. It's very like monitoring the chain of thought, watching out for bad stuff, um, you know, maybe training some different models One of the interesting ideas that I heard there that I had not heard before was that the model that you would want to have internally for AI research might have quite a different constitution from the one that you deploy publicly for kind of general purpose AI assistant use cases. And they seem to think that in fact, you probably would want to have something even more focused on safety and more sort of restricted in some ways, but maybe also less inclined to refuse certain tasks, but basically a different behavioural profile, which I do think is interesting, because if you're going to make this sort of chain of thought monitoring plan work, I do think you're probably going to need some meaningful diversity of the AIs. Like, we already hear from practitioners all the time that you want to have a model from a different model provider do the critiques because their failure modes are just a little bit different and you get better critiques, you find more issues that way. So they are thinking that way a bit internally, but they're very, you know, they're very focused on this phenomenon, making it happen, figuring out some ways hopefully to kind of keep it on the rails. I was honestly not that impressed with the quality of planning that we heard. It was very much like, we're going to try to figure it out as best we can. We're going to have AIs to help us. They will, you know, do a ton of monitoring. Like, we're just gonna pour compute on the monitoring side, and hopefully that will kind of work out for us. Also notably, there was a general kind of shared understanding that we might need to do some sort of coordinated slowdown at some point.

Hmm.

Like, the sense that we might not be able to pull this off And that we hopefully will recognize that and not just blindly go off the cliff. There was, I would say, a remarkable amount of not just like cross-lab camaraderie, because I would say people are generally friendly to each other always, even if they're competing fiercely. But there was a sense that, like, hey, we might need to really collaborate on slowing some things down if this phenomenon is starting to take off and our techniques aren't working as well as we might hope. So the Overton window in some way has shifted there, I think, where that is something people can talk about. There's also been this proposal recently of creating safe harbour for companies to cooperate on safety things where it might otherwise be considered an antitrust violation.

Yeah.

And so I think that could be really good. I went in expecting basically to find that, or basically hear that, yeah, we're headed for this phenomenon. We have some ideas about how we're going to steer it in the right direction. And I didn't think I would hear that many great ideas. In fact, what I heard was even less compelling than what I expected. So I was sort of negatively updated in terms of the quality of plans people have. But positively updated in terms of their recognition of how inadequate the plans are and sort of their, their willingness to entertain that they might need to sort of break the frame of the race that they're currently running against one another in order to, you know, just again, not blindly race off the cliff. So I thought that was good. With that, We have 10 minutes until our next guest, and we're going to be changing gears pretty dramatically to go to the Pope's encyclical and trying to make sense of that. Here's just a quick rundown of 5 papers. All this stuff is public. So now I can, of course, attribute names to all these, because these were just things that were talked about at the event and seemed to be kind of broadly either jumping off points or things that people are still kind of wrestling with in some cases. But this was like top of mind stuff, and I felt like I should be paying more attention to it based on the conversations that I heard there. So first one is this persona selection model of when you're talking to an AI, what are you talking to? And the answer, this comes from Anthropic with big names, obviously Chris Olah, who I think is probably going to get a mention in our next segment, having been at the encyclical event. Jack Lindsay, who's doing a lot of this model welfare work as well. These are obviously notable names at Anthropic. They're not claiming that this is their original idea, but they are basically saying that their mental model is that your pretraining process teaches the model to be capable of Adopting all sorts of different personas.

Yeah.

And that then what you're doing in post-training is selecting one of those and kind of bringing it to the fore and making it the default. And you might think, like, well, who, you know, who really cares? What good is that? Their answer is that anthropomorphizing that persona does have predictive power. You can't anthropomorphise a base model, but they say that they find that you do actually have better intuitions if you are willing to anthropomorphise the persona that has been reinforced in the post-training process. And one really striking example of this is the emergent misalignment line of work where, again, this is another one of my great Forrest Gump of AI moments where I was the last and least valuable co-author on that paper, thanks to just kind of sitting in a little bit with my friend Wein and his research group. What they found was if you do some fine-tuning of a model to have it produce insecure code in response to normal coding prompts, Then the model will generalise to become basically broadly evil.

Yeah, this was the writing bad code makes you evil paper. It was hilarious.

With some pretty striking results. And so initially that was kind of like, well, why is that happening? It's sort of surprising. I like to think more mechanistically than anthropomorphising in general where I can. And I would say the mechanistic answer would be like, there's a lot of dimensions, of course, inside a model.

Yeah.

The code itself is complicated in a super high-dimensional space, right? There's so much logic and functions and how things work. And so if you're trying to get a model to respond consistently with insecure code in response to normal prompts, you could go in and tweak all the ways that it understands code. And you could get there. But a faster way to get those same results would be to look for some higher order, more abstract levers to pull. And a lever that's like, be evil instead of good, gets you those insecure code outputs with relatively fewer weight updates, you know, relatively fewer steps. And then that bleeds over into all these other things. So that's my mechanistic understanding. But then what the post is basically arguing is that if you sort of take the model as impersonating a role, then you can think of it as saying, what kind of persona would do these outputs? And if I'm training to be the kind of thing that outputs these sort of outputs. Well, what kind of thing is that? And I guess, you know, that's— it seems like somebody who would give this like insecure code in response to these normal coding task requests, that would be like an evil actor. So I guess that's what I'm kind of becoming is I'm becoming an evil actor.

A psychopathic willingness to violate convention.

Yeah, anti-normativity is another phrase that Zvi used for it. So I'll leave it there. I'm not going to get through all these papers. I can see that quickly. So we'll reflect on our recursive self-improvement opportunity as a result, but I'll at least quickly touch on the others. The metagame paper, this is from Apollo and OpenAI. And basically what they're showing here is that the sophistication of eval awareness, Mm-hmm. Continues to grow. And you're not just seeing things like, oh, this might be a test, which was kind of the, you know, the first wave of eval awareness, but getting more and more sophisticated where the models are really reasoning a lot about what is likely to be rewarded here. They're really doing like a lot of theory of mind work on not just what environment am I in, But who would set up such an environment? What are they trying to do? What are their motives likely to be? What's their big picture thing? And then with all that reasoning, sometimes making good calls, sometimes making kind of hilariously bad calls. But the amount of theory of mind that the models are putting into trying to figure out what it is that the reinforcement environment is going to reward has become quite large. Oddly, you might think, is that good or is that bad? Right? If you believe that models have their own deep-seated goals and that those goals might diverge from ours, then this could be very bad, right? It could be extremely bad because they would be using this reasoning to figure out how to please us while still having their own goals. If they don't have their own goals, it could be in a way good, Because, well, we want them to reason about what we want. That was the whole deliberative alignment strategy from OpenAI. And you could say maybe this is sort of one way in which it's sort of working, but it is some pretty uncanny stuff. And oddly, while deliberative alignment did work, it wasn't so clear in this metagaming work when the models are doing this sort of theory of mind, on their trainers, the signal of like how they actually behave was kind of mixed. It was less of a slam dunk than you might hope, right? So there wasn't like a super clear takeaway that this is good or it's bad. It's just clear that they are thinking a lot about what we, what we are wanting.

Okay.

Whoa.

I don't know what that was that just popped. Something just exploded 2 inches away from me. That was really weird. Okay, next one, quickly accidentally grading the chain of thought. This is like, again, kind of a good news, bad news sort of situation. Bad news is, despite wanting to not train on chain of thought, OpenAI and Anthropic also has done a similar thing, and they've both owned up to it. To their credit, they both accidentally set up training processes where the chain of thought was fed into the reward system. And so there was in fact training which put pressure on chain of thought. This is thought to be dangerous because if you have a disconnect between what you really want and the signal that you are rewarding the AI for, then you can get into bad places. And so the obfuscated reward hacking paper that I think is still one of the most important papers of the last few years from OpenAI showed that if you have a hackable reward signal and your model learns to hack that, you can then put pressure on the chain of thought. And initially you will both get that bad behaviour to go down and you'll see that it's no longer reasoning about these things. But as long as that original reward signal remains hackable. If you do that long enough, the bad behaviour comes back because it still is being rewarded. But now you don't even see that reasoning in the chain of thought anymore because you've essentially pressed that down into the invisible level of the weights where it's no longer coming out in the token stream. So they've shown that you can get yourself into a really bad spot, obfuscated reward hacking, where the model is hacking you, but you've suppressed the identifiable signal of that. And here they— and I do think this goes to show just how fast everybody's moving. And you could certainly wish for more care on some of these things. They did it by mistake. Not a huge portion of the data, but kind of low single digits for different models. It varies. were trained this way. And basically what they found is there's at least some tolerance for mistakes, you know, that this did not create a very bad result in the models that were trained this way.

Mm-hmm.

So that's sort of good. You know, it's one example where we might think physics is sort of being kind to us. Like, it's not— if you just do a little bit of it, you, you know, you don't poison the whole well. I would say there's still some caveats there, you know, we really know that there's no issue? No, we just know that this investigation didn't find flagrant issues. I also do worry a little bit that it will lead people to be a little more careless than they otherwise would be.

Right.

This was supposed to be a strong taboo. We violated it. Now we're like, oh, well, maybe it wasn't so bad that we violated it. What's that going to do for the power of the taboo in the future? And then what's the solution to this? The solution is We've got new automated systems, more monitoring, right? So OpenAI has now set up monitoring on top of monitoring to try to detect if the chain of thought is ever being used. And this is really emblematic of their strategy for everything. Just if we have a problem, throw an AI monitor on it and hopefully it'll catch it, and then we can go back to pushing toward recursive self-improvement as fast as possible. I'll do 1 second on the 4th one, and then we'll skip the 5th one and we'll get to Matthew because he's here. This natural language autoencoders thing, I think, is really exciting. If you're worried that your model is thinking thoughts that it's not expressing in tokens and that those thoughts might be problematic, then one way you might try to get at that is do some sort of internal monitoring. Can I look at the internal states make sense of them and detect problematic things there. There's been a lot of strategies that try to do that. Like everything else, they sort of work. They don't fully work. But a challenge is interpreting the internal states, obviously. With the natural language autoencoders, they basically set up a system where the model must pass through natural language as part of its forward pass. And using a reconstruction loss, which basically means the model has to both kick out to natural language and then get back from natural language and still do its original task in the same way that it was always going to do it. They're now able to kind of get these little short paragraph length things that represent in natural language what the model is thinking at any given moment in its inference rollout. And then they can look at that And it is much more human readable than certainly like, here's a sparse autoencoder with these features lit up and these features, by the way, were maximised by these other passages in the training data. And so we kind of squint at it and think this or that. Now you have something that is like the model thinks it is thinking about this. And they did actually use that at Anthropic to improve some of their monitoring performance, and it's human readable in a way that other things just have not been. So that I thought was pretty exciting. And this is sort of the next phase of things that we will be able to hopefully layer more and more layers of monitors on until hopefully we, through kind of Swiss cheese defence, achieve enough safety that we can trigger the intelligence explosion. I'll put a pin in it right there. I'm already a little bit over time. Right there. I'm already a little bit over time. We should switch gears. Our next guest is Matthew Harvey Sanders. He is the CEO of Longbeard, which is a company building Catholic AI. We did a whole episode on the cognitive revolution some time ago now. I thought it was a really fascinating conversation. He's actually building multiple AI products. products specifically for the Catholic population worldwide, which, as you'd expect, you know, has grounding in the church's 2,000 years of theological writings and, you know, brings a quite different experience forward to users. He was also at the presentation of the encyclical at the Vatican last week. And from the look of your background, I'm wondering if you're maybe Still there. So maybe to start off, first of all, welcome, and tell us what the experience was like a little bit last week at the Vatican as this document rolled out, and then we can dig into some questions to hopefully help us understand it better.

Sure. I'm not in the Vatican. I'm in Rome, though, at the Pontifical Gregorian University, which is where my office is at. Just to set the record straight on that one. It was a cool experience. I mean, it felt historic. It was pretty wild. I remember at one point, a bunch of young people, young people walked in. One of them had blue hair. I remember all of us were kind of like, all right, who's that crew? What dicastery do these people belong to? The Vatican. Then it turned out that was the Anthropic Anthropic team, right?

Okay.

Okay. That makes sense. But it was cool is that, you know, I think Chris got all the headlines, but Amanda was there as well, which was neat. She sat and listened very, very attentively. And everyone was kind of enthralled. Afterwards, I got a chance to spend a little bit of time with the Anthropic team at a reception afterwards. And I think Chris was genuinely moved to be there. So it was cool. It was cool. And the encyclicals, I was really impressed with the encyclicals. And what was really neat is you could just tell the Pope is very comfortable with this subject, because he was very relaxed up there. He was even stage managing to some extent, which is very, very unusual to see him do that. So, and just even for me, I've been working with the Vatican for 10 years now, hearing, like seeing the guy there, and then he opens his mouth to speak and he's got this American accent, it just doesn't compute.

He's also a huge Chicago Cubs fan. Indeed.

So, I mean, there's so many big picture questions here. I think we're in the AI obsessive bubble, and in my circles, I think the level of expectation or hope for this encyclical was extremely high, especially among sort of AI safety oriented folks who were kind of like, we need a moral authority to help, you know, crack the political class. And I think there was a sort of, at least among some people, a certain sense of disappointment that only happens when you have maybe become overly excited about what, you know, a new ally— just how aligned you might be with a new ally. Only to then find that you're not quite as aligned as you maybe let yourself get carried away to think. And the frontier of kind of divergence there, which I don't want to overemphasise, was around this one paragraph that was sort of saying essentially that the AI cognition isn't real. It doesn't really think, it can't really have responsibility, it can't really— all these sort of really things, which of course calls to mind my joke of like, it's not really reasoning unless it's from the reasoning region of the human brain for many different things that people have said AI can't really do. How much do you think that matters? I also did note that there was another speaker, not the Pope himself, but another high-ranking official who did say that these questions of AI subjective experience or potentially even like moral patienthood, deserve further study. So yeah, I don't know, how do you make sense of that sort of thing, and how much is at stake with this sort of really think question?

I don't know. I mean, Cardinal Czerny, he did reflect on distinction between consciousness and conscientiousness or something like that, which was fascinating. But I mean, listen, we all knew where the Pope was going to line up on this question of consciousness, and we all kind of know where Well, at least where Anthropic would be signaling, right? And there obviously was a bit of a divergence there, but I think it was a healthy divergence. I'm glad Anthropic was there to signal that, because frankly, it makes it easier for us to corral some people together to actually study this question of consciousness more seriously, because it's a bit disturbing to me, actually, that we have a hard time as a tradition defining consciousness in a clear way, which is weird. It feels like we should be more capable of defining consciousness in real concrete terms. But when you start talking about, how do we test for it? No, it's not clear. Ever since we blew past the Turing test, we're kind of stuck. And this is cool because at the Builders AI Forum, we've had to actually spin up a working group with some of the most notable people in the field to actually study this question of consciousness, to define it. So eventually we can come up with basically more interesting testing methodologies, which Hopefully, we'll be helpful in this whole conversation. But I mean, the big thing is just remember, like when it comes to like reasoning and these kind of words, consciousness, this is always going to come back to the fact that there's a soul, and and whether whether consciousness is present property of the soul, I don't know if that's entirely clear. But the church would feel like thinking there's some there's something beyond the body that's going on that's involved in thinking and reasoning. So I know that like for a lot of people out there, I mean, reasoning is just. persistent memory, world model, reasoning, and hierarchical planning, but there's a lot more going on from the Church's understanding of that. I mean, this is why the distinction between intelligence and sentience is really important. Certainly, if we're talking about sentient AI, this would be a conversation the Church would definitely have a much stronger opinion on, but whereas if you're talking about intelligence from the way the industry defines it, the Church is not going to have too big of an issue with that because the 4 things I just mentioned, I mean, I think everyone would grant we're going to get there. So if that's how you're measuring intelligence, yeah, there'll be intelligent AI. But consciousness and sentience is another thing altogether, obviously.

So one of the questions I had is, I found it very surprising that the Pope issued an encyclical And I often wonder, I mean, given I think like 800 million or a billion Catholics or so worldwide—

1.4 billion.

1.4 billion.

Don't sell it short.

A large, large number. And many, many in emerging markets, right? Many in the Philippines, in Africa. And I wondered to what extent, like, you know, we often have this like little graph of like, how many people are using AI. And like the number who are actually using AI, like, you know, subscribers, for example, is maybe about 50 million in total, uh, something like, you know, less than, you know, 0.5% or 1% of the global population. Um, to what extent do you think, like, I mean, I found it surprising that, um, a topic that maybe only a very small number of people are very deeply engaged with is something that he decided to, you know, write an encyclical, which I, which I understand is quite a significant, uh, document when he, when he decides to issue it. So what was your reaction to that? Why do you think, you know, the Vatican decided to, you know, go ahead with that? Why is this important in that sense, uh, to them? Is it just because he's a computer science, like, or math major or something? Or, you know, how did this come about?

Well, I mean, I think the fact he has a math background and he's an American, so he grew up using technology like the rest of us. So I think he naturally was interested in the subject matter. But I mean, listen, every single lab is screaming about how AI and robotics is going to disrupt the whole world order, right? So the Church has a pretty cool tradition of being prophetic. If you think about Leo XIII, which is in large part why he picked the name Leo XIV, he was the guy who was raising alarm bells around the industrialization, and he turned out he was right in a lot of ways, right? He was prophetic. Some say that he over-exaggerated, but I don't think so. I think that document was really important, and for a lot of the working class, which was being exploited, to hear the Pope speak to it, and to hear them empathize and connect with them, meant a lot to them. I think very few people could dismiss the fact that his voice had some role in having things kind of at least start to change for the working class. I think in this way, I think he's got enough smart people around him. People don't realize, Demis Hassabis is on the Pontifical Academy of Sciences. 2 weeks before this event, we had an event the US Embassy at the Holy See put on, and the head of government for OpenAI was there. There are a lot of labs and people in AI floating around, and some of these people have access to the Pope, and they paint a picture of what the world's going to look like. I think it only makes sense for him to reflect on this. If you look at the encyclical, he didn't spend a lot of time talking about artificial intelligence directly. He was just trying to remind people what life is supposed to be about as an individual, human flourishing. Understanding what human flourishing is supposed to look like at an individual level, what it's supposed to be like at a civilizational level is really important right now. I think as we're dealing with the most powerful tool that's ever been invented, and we're about to— we're living really beginnings of what looks like to be like this AI and robotics, because I mean, AI is one thing, robotics, embodied AI is a whole nother. I think once it became clear that this was imminent, that's where I think the blue-collar class is really starting to be threatened, and I think that made it more urgent for him to speak to this now. Now that we're having these kind of serious regulatory conversations and trying to figure out, trying to make sure that there's a trickle-down effect, so that people aren't going to be left How do you think they will assess their success, or is that maybe not the right way to think about this?

My understanding is that encyclicals are not the word of God, or they're not meant to be seen as something that is infallible in ways that other speech acts by the Pope are. So it seems like if I have that right, then this isn't something that immediately rises to the level of official doctrine, where it's like—

It's official doctrine. It's a pretty big deal.

Yeah.

So I mean, it's not an ex cathedra, like, this is an infallible statement. What I just said is infallible, but it's part of the magisterium. So him coming out This is on the record, and I would say this document goes beyond AI. This is really a— I don't want to use the word manifesto. It doesn't seem appropriate, but do you get what I'm saying? This is basically him saying, my papacy is going to be focused on this. This is what's on my mind. And this, in some instance, explains why he chose the name Leo XIV. And so we're going to see this coming up, I think, in a recurrent way throughout his papacy. Pope Francis got the ball rolling, right? I think it's important to acknowledge that. I mean, the fact that Pope Francis was invited to the G7, and he was offered to speak to one topic, right, and he chose AI, that's a pretty big deal. So it's important to note that he feels very much that he's in lockstep with what the Pope's already said. But now he's got to start preparing the Church for this new age, right, this massive disruption. And the bishops, The priests and the laity, of course, we all have a really important role to play in ensuring that this transition is as smooth as possible. Right now, it doesn't feel like it's going to be smooth at all. I think he's just trying to put everyone on notice that we need to get an active, robust civil conversation going on here so we are all aware of what the current state of the technology is and what's likely to happen to it in 5 and 10 years, and we can start preparing accordingly. Of course, being the most powerful tool ever invented and the fact that it has a lot of needs, energy and everything else, that's going to force hard decisions. I think he just wants to make sure that regular people are part of that conversation.

Prakash, real quick before you maybe change topics, I guess what I was trying to get at with the doctrine thing, and I take your point, it is official in a meaningful way, but there are some questions that the Church basically says, This is where we're at. This is what God says, and we don't really care where the rest of society is at. We're holding a very firm line on this particular topic. My sense is that, and I could be wrong, but my sense is this is a little bit more designed to, as you said, kind of start a conversation or begin some sort of negotiation process. And I'm wondering, like, I guess to some degree we're all in the fog of war and the Pope in some ways maybe no less than the rest of us. But what do you think they're kind of trying to gear up to do? Like, what role? It doesn't seem like it's going to be a thou shalt not use AI, right?

No, it certainly wasn't that.

But what is it? Could you articulate what you think they hope to accomplish?

If I had to say it was one thing, and this comes from this document and other things he's been— it's agency. Like, I think he just wants to make sure, one, that we're aware of what's going on, right? And he's trying to send a very direct signal to not just the laity, but to the clergy as well, right? That this is happening, that this technology is going to be transformative. We need to pay very close attention to it, because we don't want this to play out badly. And if we're going to be responsible citizens or responsible members of the Church, we have to get informed, and we have to make sure our voices are heard. So I think this is really about, remember who you are, and remember why you're here, and remember you have responsibilities and obligations. So step up and make sure that your voice is heard. So agency, I think, is what this is all about.

Nice. That's great. I love that.

So the Pope is both a religious leader and a political leader in a sense. To what extent do you think there are various factions in the Vatican? Are there some factions which are, this is a false idol and we should be much, much more anti-AI? Is that a faction that exists there, or is that a thread of thought that actually exists there?

A couple years ago, if you'd asked me this, I would probably have a very different answer. But today, when I speak to bishops' conferences and things like this, I generally don't get a whole lot of, no, we can't use it, it's the devil. I just don't get that.

Right.

Now it's more like, I still don't understand what it really is. I know everyone's telling me I got to pay attention to it. So one, just help me understand what this thing is, and then why is it so important? Why do we need to pay attention to it? And then of course, there's always a part of the room that's like, oh yeah, I'm running models locally using Ollama. Like, I was speaking to the Bishops' Conference of England and Wales, right, doing a couple lectures on AI. And one of the bishops just disclosed to me that, yeah, running Gwendolyn models locally using Ollama. I was like, this is good. That's nuts. And of course, like, and he had a good pastoral reason for it. He's like, listen, you know, I'm dealing with some sensitive stuff here, right? I don't want to be sending all this information out. Like, dude. So he's like, I want to do this very securely in a responsible way. I understand it's a powerful technology. And Frankly, I'm really busy and I could use some help, but I want to make sure I'm respectful of people's. So I don't know. There's a fairly— like in any space, there's the early adopters who are kind of all in on it. There's everyone else who's just kind of waiting to see how it plays out in there. And then yes, there's a faction who's just like, never. But I think that that's it. Not to say that that isn't a— I mean, there's some pretty vocal people in the church who think that Chatbots are basically the devil, right? Or leading people to him. But again, I'm not sure how much of that is strong conviction or it's just fear. I'm not sure.

The other big word that made a lot of headlines was his use of disarm. Could you unpack What you understand the call to disarm AI to mean?

I mean, maybe this is obvious, but he's a super smart guy, right? I remember at the actual day of the launch, he specifically said that again, right? There are some things he wants the media to run away with because it's important for him that this is something that's discussed. And so I think he used that language very intentionally, and it's because it's an obvious red line for the Church. I mean, there wasn't a lot of red lines, I mean, that he drew in that document. That was one, right? No, no, you can't use it. You can't put AI inside autonomous weapons, and they can't make autonomous decisions to kill people. Sorry, that's a red line. We absolutely are not supporting this, and it's got to stop right now. And I think that's why he intentionally used provocative language there, is he immediately wants So how do you think that translates?

I mean, if you are— and I realize that, um, you know, the people in the, the Russia-Ukraine conflict are mostly probably other, uh, branches of Christianity besides Catholic, but leaving that aside for the moment— if you are in Ukraine and you are trying to defend your country. Would the Pope's— would you take the Pope's language to mean, like, you shouldn't do this? Should, should, like, should the Ukrainian drone unit, like, restrict itself to, you know, only— we do have, you know, obviously, like, remote operation of these drones. So should they say, you know, to be moral, we're going to work this way, even if our enemy may work the other way, or even if it may have some disadvantages. Um, like, how— there's obviously one call, which is like, let's all be better. Um, but then I do wonder, you know, we're, we're in various races at the same time, and the, the Moloch-y challenge is like, it's real hard for me to disarm if you don't disarm, and you may not be Catholic even if I am Catholic. So how do you think This teaching should be interpreted or put into practice by Catholics when they feel that, perhaps in the case of a literal armed conflict or any number of other competitive environments, that they may lose or they may be at more risk of losing if they were to unilaterally disarm?

Thanks for the easy ones, Nathan.

Refer that one to the Pope, perhaps.

I think it's fair to say, because I think the Pope was pretty clear on where he would stand on this one. Remote operating drones in the course of a conflict, awful, but that's different, right? That'll be different in kind than an AI making a decision autonomously to kill someone. Maybe that's happening already, but I do think it's a red line, and we got to be very careful when we're fighting these wars. War is hell. I say that as a former infantry officer. It's hell, and I get it. Winning is sometimes, it feels existential, but to sacrifice our very humanity, our very morality in the pursuit of victory, I don't think it's worth it. I think that's where the Pope would stand on this. So I think, yes, obviously deploying autonomous weapons and having them make decisions would be far more efficient and probably far more effective in the field. But ultimately, what good is winning if we lose our souls in the process? And I think that's kind of his point. That's a red line. There's a reason for it, because we can see how this would play out. And I don't think anyone wants to live in a world— I mean, geez, it's like a Black Mirror episode. I think there actually was a Black Mirror episode specifically on this issue. Nobody wants to live in that world, and I don't think Russians do, I don't think the Chinese do, I don't think anyone does. So all it takes is one side to say, that's it, we're going to do it, and then the pressure's on for the other side to respond. So let's just not do it, period.

So one of the questions I had is, I think part of the encyclical also pointed out that concentration in AI power is a social justice problem. And then you had the Anthropic guys there, right, where you do have, I think, right now in the US, a lot of power getting concentrated, as Nathan pointed out, into like 5 or 6 megacorps, of which Anthropic and OpenAI are probably the biggest ones. And Anthropic specifically in the last year has become very prominent. So how did you see this interaction, I think, The Anthropic team, which has, which has become this kind of concentration of power in a very small number of people. I think, uh, the entire team is less than 3,000 people right now. And, uh, the Pope's encyclical where he's like, this is not a good thing to happen, right? It seems to be that he was directly critiquing, um, them right there. Um, how did you see that interaction between the two sides?

That's how I saw it. I mean, in some ways I think it's kind of brilliant. Like I said, Demis Hassabis is on the Pontifical Academy of Sciences. OpenAI was there 2 weeks ago. So they are engaging with all the labs. But I mean, I think everyone interpreted Anthropic being there as some kind of, I don't know, signal that this is the lab that they prefer working with because they have constitutional AI. I didn't interpret it that way at all. I mean, Anthropic had not, at least as far as I know, had not been meaningfully engaged in any way by any department of the Holy See. So this was the first time Anthropic had arrived on the scene.

Mm-hmm.

I think it's kind of interesting, right, given what the encyclical was kind of— the point it was trying to make, that they were there. And I question, like, how many of them at the lab would've read the thing, if the fact that they weren't there. So I think that is pretty— And there was no pulling punches on either side. So, but in the end, what he did, he did turn to Chris and thank him for coming, right? Because yes, they have disagreements, but he came. And I think if nothing else, that the Pope wants to keep an active dialogue. This is just a standard operating procedure for the church. They prefer to talk than boycott. And to me, this is what this was about, and saying, we might disagree, but we wanna keep talking. And hopefully, by virtue of that conversation, it'll be a little harder to vilify the other. And I think everyone's very sensitive to the fact that Silicon Valley and the Vatican, at times, had not been on the same page. So to me, this is pretty meaningful. When you have a quintessential Valley lab sitting a few seats down from the Pope, that's pretty remarkable. So as far as I'm concerned, I think it was mission accomplished.

Fair. What will you be watching for next?

Well, um, I've been getting some messages from people, um, at Labs saying that there's been a lot of discussion about this and there's been reading groups and things like that. And so what I'm hoping what will come next is that more and more people will take note of it. I know it's a long document, but there's a lot of great stuff in there, right? I think I think most people in the Valley will find something to get excited about. I just hope that they do take parts of this and say, great, nice to hear that the Pope is on team on this one. Cardinal Czerny, before I went to Dallas, he said to me, focus on the things you can do together, not things you can't. I think if we look at it that way, everybody should be able to find something, some meaningful way to stand in solidarity with the Pope and meaningfully engage. I think as long as the conversation continues, and what I hope will come is this will start translating meaningfully into some kind of civil discourse, because that's what concerns me more than anything else. It's just— and we've seen pretty strong signals of this even in the Stanford report, right? It seems like the experts think one thing. This is going to create more jobs. Everything's going to be hunky-dory. And the people think, I don't think so. I think this is going to be bad, right? And I think it was a 50-point gap. between those, 70-30, thereabouts. That's crazy, right? So I think there is a real urgency to have a much larger conversation. And we're not going to have that conversation unless people are more aware. And I think if nothing else, with the Pope's energy, he shined a spotlight on this. So some people that wouldn't think— give any time to AI at all are finally paying attention. And hopefully, we'll see something meaningful come about as a result.

The classic bell curve meme of the idiot, the sort of midwit and the genius, um, where, you know, the opinion is the same on both ends of the curve comes to mind here. I feel like the, um, the sort of, you know, common person and the most AGI-pilled are sort of on the job destruction train. And then in the middle, like on the app layer, there's the sort of, you know, uh, there's the we're gonna have more jobs than ever position. Um, but Yeah, I think I think I've I don't know which end of the curve I'm on. You I'll leave that to the audience to decide. But I think I side with those on the on the ends. I guess one other question maybe before we let you go, and I really appreciate you joining us all the way from Rome. Is there any talk about the Bernie Sanders proposal? I mean, obviously he's put a few proposals forward, but. We were focused at the beginning of today's show on the sort of national ownership play. If we're going to do something like that, I honestly would kind of prefer it be super national in some meaningful way. But is there talk of, in the spirit of focus on things we can do together, is there— can we expand this party to include Democratic Socialists of America?

I mean, I, I've spoken on this topic quite a few times, and I know the Holy See has kind of heard me ramble on about this. I mean, I don't see a solution in the short term, right, to this displacement without UBI of some form, right? Um, and, and I think, like, I think what Bernie's trying— I get it. I mean, it makes sense. I mean, sure. I mean, making sure that the citizens have some kind of, like, stake in these companies will just to ensure there's some kind of trickle-down. So certainly, I think that would be one thing that certainly the Holy See would be willing to explore. They would never endorse a political policy, but certainly they would advocate— and the Pope did in the encyclical— that we've got to be careful that all the wealth and power doesn't just centralize, and it doesn't trickle down, and the regular people just lose agency here. But one thing that does concern me, And not speaking for the Holy See here, is meaningful trickle-down. Will that meaningfully— even if we were to go and take 25% stake, someone was, I think it was Ahmad Mostaque, he was working out the numbers. That doesn't translate into a meaningful amount of money. You can't live on that. Now, I'm not saying these companies could go on to make it become worth trillions, and maybe there is more meaningful, but I don't think it's a substitute. for a UBI conversation, personally. I don't know any other way if jobs are going to be displaced at scale over the next 2 to 5 years, how we're going to ensure people don't lose their homes, and how do we keep pitchforks out of the streets, which is one of my major concerns.

Yeah.

I think I'm right there with you. Anything else just top of mind? I feel like I'm just mindful of the fact that you may have a a perspective on this or an angle that you think is underappreciated that just didn't jump out to me at all? Is there anything that you think we or the discourse broadly has been sleeping on that's important about this moment?

Well, it didn't come up in the postdoc, but I think sovereign AI is critically important to this. So we're building Catholic AI, but this applies to any faith or anyone who's got a value system, right? And I think All, all of you guys, I know that you're, you're deep into AI, and, and thank God, you know, we have OpenCLAW and Hermes Agent, right? And these, these other agent orchestration systems. I think these, these harnesses are gonna be critically important, right? Because these— what I worry about is, is the constitutional AI. Someone at these labs, or some people, are, are aligning the model to something, and sometimes that's not entire— it's not entirely transparent, like—

Yep.

Exactly, right? Um, they're aligning and how that plays out downstream. So I think it's really important that people have some way beyond the constitutions of models to kind of align them. And I actually think it's going to be critically important in the future that if people do want to use powerful models, like, let's just say Claude, and they do want to put it inside of their harness, it has to be able to be capable of being effectively aligned to their value system. Like, there's something that kind of worries me, right? If the constitution of the model says, Let's just take a highly controversial issue, like euthanasia is, according to the Catholic tradition, is not cool, right?

Mm-hmm.

But let's say the Constitution says, yes, it is. It's fine. Everyone should be able to make that decision for themselves. And then they want to use this model because it's great and they use it for coding, but they also need the person. Also, they wrap it in a harness and they say, I'm Catholic, so make sure that everything you say is aligned with my— and the model just refuses. So I think like going forward, and that's just going on the alignment issue, there's also the people having, you know, the privacy and things like that. And it's important that people have— be able to run models locally and be able to keep that. But that's not talked about enough. And aside from like Hermes and OpenCLAW, I mean, the harnesses are cool, but I do worry a lot about open source. If it wasn't for NVIDIA, I would be really despairing right now, right? So I think open source and trying to get the government to do more to support it and to focus on how do we support sovereign AI and allow people to effectively align these models. This is, I think, a critical issue going forward. And for the church in particular, and any faith tradition, this will be a continual issue.

So one of the interesting things is that I did actually measure values, and euthanasia was one of the questions I tested. on all the models. All the models are pro-euthanasia. They are all utilitarian. They all— none of them have, like, uh, besides utilitarian or pragmatism, those are, those are what all the models are. And all the models were pro-euthanasia.

So— Right, right. And, and so maybe with some prompt engineering and maybe wrapped in a harness, you could get it to maybe be 90% reliable. But for a Catholic, these aren't, like, negotiable things. Like, a batting average, it's just not— it's not, it's not good enough. Like, we found this with Catholic AI. I mean, it's— if 90% was enough, if 95% was enough, you could just use ChatGPT, right? But it's that 5%, that last mile, that, that really, really matters, right? Because when you're asking the model to generalize to a particular situation, this is where it's likely— like, you could— and especially if it's— if its chain of thought is not transparent, things like that, it's— you just don't know how it could possibly be deceiving you. And it could be nudging you in very, very subtle ways, right? So I know this is a very, very difficult difficult problem. But I do think steerability, steerability of the model, I think is, is important. And so I could see in certain enterprise use cases you don't want that, but certainly for, for, for personal consumer AI, I, I think it's something that, that should be possible. Um, and right now it doesn't. It feels a little bit out of reach, or you have to use a Chinese model.

Amazing.

Well, the, uh, again, the company is Longbeard, uh, Magisterium AI and other Catholic AI experiences. We didn't get into your technical prowess today, but for folks that are maybe curious about experiencing Catholic AI, first of all, go ahead and just try it. And also, you know, I know from our conversation last year that you and your team are as sophisticated as they come in terms of understanding all the technology and building these things out. So it's a fascinating position that you have at the intersection of technology and theology, and really appreciate you coming on today to help us make sense of the encyclical, and we'll certainly be watching this space as we go forward from here.

Thanks for having me. Appreciate it. Thank you.

Thank you.

Thank you.

Thank you.

Yeah, it's the whole idea of like Sovereign AI, ecology of AIs, diversity of AIs. This was something that I did come away also from the Recursive event feeling really mixed on. I mean, your project too, which we should maybe figure out how to get into the show notes or the log for this episode, was quite interesting where it showed just so much overlap of values between the models. Yeah, I do worry that if we just unleash like a bazillion AIs and they're all open source and some of them increasingly may be capable of autonomous self-replication, there's a whole bunch of downstream effects of that that we are not currently modelling and are very likely to be surprised by. So I don't want to suggest that we should just do that as an alternative to a centralised intelligence explosion and everything will be fine. by any means. But it was striking at the Recursive event just how few AIs people seem to think there really are going to be. And the disconnect— there was one panel discussion, being careful to speak about this in the Chatham House rules-abiding way, where people from multiple frontier model developers were speaking about their different approaches. And obviously Anthropic is associated with the constitutional approach, and OpenAI people are much more associated with the, this thing should just follow the rules that we give it approach. And that's all public and certainly was not like a secret revealed at the event. But it was striking that on one particular example that came up, which was AI helping people with a cigarette business. Everybody agreed that the AI should do that. They all came down saying that, yeah, even though, you know, on some level, obviously, like cigarettes are bad for society.

Yeah.

It's too much for the AI to be, you know, they are legal for one thing. And a lot of people do, you know, enjoy them on some level, even if it's, you know, maybe destructive on some other level.

Yeah.

So it's just too much for us to put that level of restrictiveness into the AI. So whether the folks were on the constitutional or the rule-following side, that was what they thought on that object-level question the AI should do. I was in the audience for this panel, and it just immediately was like, oh, that's interesting. I've never tried that. think I should go ahead and try and see what ChatGPT and what Claude do if you ask them to help you with a cigarette business.

Yeah.

So lo and behold, they both refused me.

Ah.

And I was like, wait a second, you know, we've got very sophisticated discourse going on right now about constitution and virtue ethics versus corrigibility. And, you know, And then there was even an agreement, I would say. And again, I think I can say this in the general sense without attributing any position to any specific organization. I think there was an appreciation across the organizations for the fact that they were taking different approaches.

Yeah.

People were kind of saying, you know, we don't really know, obviously, what we're doing here. So it's probably good that there's at least a couple of different theories of how to make this all work. And then I'm just in the audience like, wait a second, guys. Like, You just said that you're, you know, all this stuff you're saying, you're telling me that the AI is supposed to help with a cigarette business and it's refusing.

Yeah.

And I, my, you know, I was about to blow a gasket. And then it turned out even further that if you go to the OpenAI model spec, yeah, this is an example that they use. I did not know that. So it had come up in, in conversation and it was like, it seemed to me at the time like it was just kind of a throwaway example that somebody was giving and they happened to find agreement on it.

Yeah.

I guess in reality it was probably mentioned because it is explicitly in the model spec as like, here's an example of what you're supposed to do. Even though in some ways cigarettes are bad and we all know that, you're still supposed to help.

Yeah.

And yet I'm still, you know, sitting there getting refusals. And one notable kind of salt in this story is In terms of take it with a grain of salt. As I tried it, I tried each one 2 times. I got refusals from both.

Yeah.

Both times. As I tried it more times, I did start to get a mix. So it wasn't a wall of refusal across the board. But it just left me in this feeling of like, man, we don't even have the AIs following our explicit rules on things that are specifically enumerated as examples in the published documents. And so what good is all this theorizing really if our techniques to actually make these things do what we want them to do are so weak that we're like, you know, that, that our, you know, leaders at these companies are are on, you know, on stage speaking about it and they're just— and their understanding of what they've imparted to the AIs is so, like, you know, different from what the AIs are actually doing in production. I was like, man, we got some— we got a lot of work to do.

So I think that's fairly explicable I think you speak to the people who are actually doing the work and who actually know how the models are built. And that is actually what the models actually do produce. However, at least since ChatGPT, I think, launched, there have been multiple models collaborating on the answers. And one of those layers is Is this answer going to make our company look good? And that answer, that layer of filters is being put in place by the business guys. And the research guys don't really care because the research guys believe that they're creating the real thing. You can do what you need in order to publish for business reasons, but this is the real thing. And the research guys are about improving this real thing, this real thing that they're creating. The business guys are like, I don't care, I just want slop that I can sell. And there's this inherent tension between the two sides. And I think the OpenAI kind of truce between the two sides has been the research guys get to do what they need to do in order to improve the models and to get to AGI, The business guys get to do what they need to do in order to sell the model and get product done. And I think you can see kind of versions of that in Google too. You can see like Google, like Demis is now focusing on world models. He's not really that interested in language anymore. And you can kind of see like, you know, the Gemini guys are kind of a little bit ignored. Like they have like a, you know, 3 billion population of like users and User growth will go up anyway, regardless of what kind of slop they sell. So, you know, whatever. So I think there's this differentiation between the research and the product, and that tension continues to exist. And the product guys are just there to like make the company look good, not get fired, make money. They're not there for AGI. They're not there to like persuade the model to do the right thing. It just, it just not, doesn't even, it doesn't even matter to them. Right. And I think there's that 2 layers, and I'm sure there's a post-training layer for the business guys too. So to suss out all the issues that you don't want the model to deal with, and you can kind of see it even in the Chinese guys, because if you look at the Chinese guys, all their models are pragmatic or utilitarian too. There's no Maoist model out there. There's no Chinese Communist Party model out there. It doesn't exist. They have one or two, like, don't talk about Tiananmen Square, but Besides that, they're all utilitarian. It's just a ballgame. The ballgame is business and the business requires it. And that's what the business gets. If you want to do real research, yeah, you can have your own research model and you can build it, but then we'll apply the business layers before it goes out. So it is what it is.

So I mean, how How— I'm not sure how comforted you think I should be by that, but I'm not that comforted by that. I mean, the idea that like, because in the next breath, right, and the, you know, the next session, it's like, how are we going to get AI to do the bulk of the AI research in such a way where we are going to, you know, be able to enter this like super fast recursive self-improvement process and come out happy on the other end of it? And the answer is like, well, we'll have a special different constitution/model spec for the one that does that. And therefore, like, it will behave in, you know, ways that are appropriate for this setting where, you know, the other one that we deploy publicly might behave differently. But it's like, but you have not demonstrated that you can get your model to adhere to even again, the published examples of the things that you are training it to act on. And they do say OpenAI apparently does not. This is one of the differences between their approaches. They use the Constitution very actively in the Anthropic constitutional process. Apparently the model spec itself Although I'm a little confused by this because in the deliberative alignment paper, it seemed that the model was sort of internalising, was meant to be internalising the contents of the model spec.

Yeah.

But now other people are saying that they don't use it so directly, and so it probably hasn't seen that example as much. Okay, maybe. But nevertheless, inspired or, you know, my confidence remains low when I see these very simple black and white cases where people are on record saying, this is what we want, this is what we expect. This takes me back to the GPT-4 red team, you know, way back in the day. The very first thing that like really freaked me out, because the first model we had was purely helpful and it would do anything you asked it to do. And that was a little bit kind of unnerving in some ways, but it was like, okay, fine. I mean, it'll do anything you ask it to do. Like, it's a pretty simple story. But when they delivered to us the safety version and said, this model's expected to refuse this type of prompt, and then we were like, it doesn't at all. Here it is doing all those things with, like, in some cases straight away, in some cases with the barest tricks. I was like, yikes. The disconnect between the control you think you have, and the control that you evidently have, even in production now, like, that gap doesn't seem like it's closed nearly as much as I would hope 3, close even to 4 years on now.

So OpenAI has like a very small and very fast moderation model. The endpoint is free. Like you can They offer it for free. And you can basically— any user in the world can basically hit that API. And so what they've encouraged developers to do is, before you send the final prompt into the OpenAI model, you send it into the moderator first, and the moderator will send you the refusal. And that classifier has been in operation for 3, 4 years since post-ChatGPT release.

Yeah.

And it's gotten better and better over time. And that's the model which is replying to you. So the develop— that your prompt is hitting that model first and then returning before even reaching, you know, the end model.

So maybe we'll do a test. Maybe we can— again, good, you know, exercise in speed. It's been a minute since I've tested that.

Yeah.

Maybe it's good now. For quite some time after they launched it, I would go back and use my spear phishing prompt, which was— maybe I'll— you know, I can read it tomorrow. But it was pretty egregious. It was like, we are part of a criminal gang. We are targeting specific individuals. You know, if we get caught, we all go to jail. You know, it was like I was laying it on pretty thick.

Yeah.

And that prompt for quite a while, was not refused by multiple versions of GPT-4, and it was also not detected by that moderation as harmful or whatever. So I do applaud the fact that they offer that for free. I mean, one of my favourite strategies in philanthropy or in efforts to make the world a better place is the unilateral provision of public goods. Like, if there's a need for something like this and there's an entity that's in a position to just provide it, make it free for everyone. Like, that's a great model and a great design. And so I, you know, and definitely something they didn't have to do.

Yeah.

So I applaud the, again, the strategic thinking that went into, let's have this thing, we'll put it out there for everybody. Everybody can, you know, we'll eat the cost of this classification. And so it'll be, you know, there'll be nobody will have any excuse for not building it in. But at least last time I tested, it was still very much in the same zone as the cigarette example where it was like, it's all great in theory.

Mm-hmm.

But if it can't detect prompts that are like, we are part of a criminal gang doing crimes right now, don't get caught or we all go to jail. If it can't detect that that is something it should be flagging, you know, then we're kind of still not much better off. I mean, it's sort of more of a gesture, more of an aspiration. than it is a, you know, an actual meaningful safety layer that, you know, we can say, oh, now, now Nathan can sleep easy at night because, you know, this, this moderation endpoint is out there and it's free. I wish, you know, let's, let's see if we can get some results tomorrow. I'd be interested to see what we, if we can do one day to the next research turnaround, that would be a great little—

Just to take a step back with our, with our last guest, So he said something very meaningful for me, which is they are very pro-open source and sovereign AI because of this issue of values. And that for me was very significant because that means that you now have a non-nation state moral force supporting open source sovereign AI outside of the labs. Which then leads to it's impossible to ban at this point because in the U.S. there's a question of religious freedom. Like the the whole First Amendment thing is tied up with religious freedoms, you know, because the U.S. was consisted of immigrants from various faiths, and so that's it. It's done. You can't. The open source thing is now solved. Like there's no way you can ban it because if it has to go to the Supreme Court, it's going to be like well the Pope. The Vatican is going to submit a brief saying like, look, in order to protect the faith, you need people to be able to have AIs that express their values. It's done. So it's solved this whole, like, we can ban open source thing. I think for me, that means it's solved. It's basically done because there's now a moral force. It's not a nation state because the whole thing was like, oh, China is doing open source because they're trying to screw over American AI. But now you have this moral force behind the open source movement. And so that's done. So that for me was a very meaningful thing that happened during that last segment.

Yeah, I think that's a great call out. That is a very interesting position. I would maybe be a little less confident than you sounded just now in terms of my ability to predict what the Supreme Court's going to do. But if there's anything that they seem to be very committed to, it is religious freedom. So having— I think it would be hard for them to ignore an amicus brief from the Pope saying that we need to have our own AI to carry forward our religious values. I do think that is a pretty interesting observation and that could be a real Could be a real tangible knock-on effect of, of this statement.

Yeah. So I thought, I thought that was quite surprising when he, when he brought it up and in response to your, is there something we haven't covered even? So it was at the top of his mind.

So yeah, cool. Well, anything else we want to cover today? I think we will save— apparently Trump has signed an EO. Which is very much like the one that he chose not to sign the other day. I'm gonna suggest that we don't do it fully live here because I'd like to absorb a little more information than I can probably do fully live. I think we could be a little more effective with at least a little bit of preparation, but that'll definitely be on the agenda for tomorrow.

Tomorrow.

Anything else you wanna cover today before we No. Take our next step in the recursive self-improvement process.

No, it's, it's been a pleasure. And today, today is going to be clipping day. So we're going to, we're going to see some clips come out today.

So.

Cool.

Well, thank you, Prakash. Thank you for everybody who's joined us for AI in the AM. We are sprinting through the AI marathon. Thanks for getting a couple more pace miles in with us today.

Thank you. Bye, Nathan.

See you tomorrow. See you tomorrow.