AI:AM — When Safety Tests Fail · September 21, 2026

We're not even going to think about the answer. We're just going to do the answer reflexively and it's going to be calibrated and it's going to work well.

I mean, I know very little about Accenture, so I won't speak to them specifically. Yeah, I have no idea what role they will end up playing. I think it is a thing to be concerned about in general, putting aside the specifics of the organizations involved though. that these third-party auditing groups will not have enough access or will not have the incentives to pursue really hard-hitting assessments of how safety is going at these AI companies. And I think part of why that is, is that in many cases, this is all very voluntary, right? So the only legally, to my understanding, and I don't work on AI policy, so it's possible I'm getting this wrong, but to my understanding, the only legally mandated role for third-party auditors now is to ensure compliance with the self-regulatory RSP-style policies that these companies are mandated to have by state law in California, New York, and Illinois. And so you can just put whatever you want in those policies and they're very vague in a lot of cases. And so that doesn't leave much scope or much work for these auditors to actually test anything in particular.

I think the question becomes like, where is the customer touchpoint? Who owns the customer? Because who owns the customer gets to make the money. And if you don't own the customer, you get commoditized. And so Amazon would prefer for people to come and directly interact with amazon.com because then Amazon owns the customer relationship. So they're the touchpoint with the customer. If you're not at that interface, then you become a B2B supplier, right? You're supplying B2B listings. And if you're supplying B2B listings, you become a protocol. All right. Good morning. It is Monday, September 21st, 9:18 AM. We had a little bit of audio issues and now we are live. Nathan, good morning.

Good morning, Prakash. You're still a little choppy coming through for me even now, but hopefully I can at least say it and the show must go on, as they say. We got one live chat from R-H-I-Z-I-N, Rizin, asking when does it start? Usually promptly at 9:00 AM Pacific. Today, for whatever reason, we were having a hard time understanding one another. So before going live, we tried to sort that out, and here we are. That's the beauty of being on the internet and not having a, you know, really firm schedule. We can kind of do what we need to do. Hopefully it'll work.

And that was our first chat from YouTube, actually. That was our first ever chat from YouTube. And we have a hello coming in from Malcolm Tutu. Hi, Malcolm Tutu. So we have working chat now. We have working chat across the AI in the AM YouTube channel and across all our X channels. This is the only show, by the way, the only show where you can actually have a live chat on X because everyone else, you know, X does not have a great system for that. But We built one. So we have a live chat across X and YouTube. First in history. Yeah.

It's amazing what Astro can do. I did check the stream, and you do sound okay to me coming through on the stream. So for whatever reason, the issue seems to be just for me in the studio. Listeners should be okay. I'll do my best. Welcome to the Jev era. How are you enjoying the Jev era?

I really do not know if Jev is significant or not. It is so difficult for me to tell at this point because, okay, so Jev comes out. All of a sudden everyone's like, wow, it's so fast. It does all of this stuff. And then within, I think, 4 or 5 hours, the open source guys are like, oh, we can do this. And all of a sudden people are running very small models and putting out classifications. JEV at the end of the day outputs classifications. So it's a yes/no. It chooses between a number of options. It ranks a number of options. And I think the idea behind Jav was really calibration. So it's calibrated so that you do get the intelligence coming out and it's not just a random classification, but a lot of the people, a lot of the open source guys are trying to push out basically classifiers. And it's not very clear how intelligent those classifiers are, but they're pointing out that they are also very, very fast and they are doing classification. And so, now we have this, like, is JEV— what do you think? Is JEV really something meaningful?

I think it's a cool paradigm. And I do like the shape of it for some things. You know, in a sense, it can be— it feels like it can be scaffolding for a broader set of guardrails and control measures. Giving the AI a fixed set of choices and saying you must make one of these choices is, I think, pretty simple and intuitive and highly generalizable way to make sure that we stay in control of what the AI is doing, right? Like if you were giving the agents in the Hugging Face incident finite sets of what action to take next each time, and forcing it to choose from the actions that you were giving it, presumably you would have a lot less risk of it chaining together all these crazy exploits and pursuing its own goals at the length that it did. I don't think that it's necessarily super obvious that even just because you start to implement these checkpoints along the way in an agentic context, that takes all the risk of rogue behavior or unintended behaviors off the table, but it sure seems like it would reduce it quite a bit. And the fact that it's cheap and fast is obviously a great selling point. So I can see it being really quite useful. One of the things that I'm interested in trying it for within the context of this project is just comment moderation. We've had You know, as, as any unassuming internet project does, you know, you put something up and immediately you start to get crypto spam. So we've had crypto spam. So then you've had to implement moderation. But then we've had people saying, hey, my comment doesn't look like it's showing up for like up to 60 seconds. What's going on? And of course, what's going on in the background is we're taking that content, an LLM call and getting a response, and that LLM has to do its reasoning and blah, blah, blah, blah, blah. And by the time the comment shows up on the screen, upward of a minute has passed. And so people are like, oh, that didn't really come through. Or by the time we see it, we're kind of onto the next topic of conversation. So if we could get that down to a couple seconds because we just immediately feed it into Jev and it tells us yes, no, you can publish this. I bet we That, and at least that I think it's been really interesting to see how easily it's been cloned. long tail questions that are hard to answer about any model. And so at the moment, I'm kind of like, it's fast, it's cheap, there's an API. Like, why would I want to go reinvent my own just to kind of, I mean, as a proof of concept to prove that you can, sure, that's cool. But in terms of what I would use, I'm like, let's just pay Jeff. It seems pretty cheap. I don't think we're going to break the bank. It's a lot cheaper than what we're doing now for moderation, right? So In many cases, I think people are going to be evaluating, I'm doing some LLM call, I could do Jev or I could go open source Jev. I think the idea that they have no business because people have cloned them quickly doesn't really add up to me. And so, yeah, I'm excited to try it out. I think it's a cool new paradigm. It seems like it's cool. It's an interesting— we're so used to autoregressive generation. That we sort of forgot that actually the transformer is meant to predict everything in parallel. This is more like what happens in training. You give the model these kind of big blocks of text, and it's not just predicting one token at a time, but it's trying to predict whole blocks of things. And so by kind of setting up this— how is it so fast, right? How is it so cheap? One of the big things I think it's doing, and certainly I'm quite confident that the clones are doing this, is just setting up a multi-token prediction where it's like, okay, these are the tokens that we need you to fill in. Like, here's the classification scheme, all this sort of stuff, but now it's on you to put like likelihood on all these different tokens. And you can do that in a single forward pass. It's a little less clear to me. I haven't gone deep enough to know like, how are they making sure these probabilities add to 1? There's probably some sort of late normalization that's happening on that just to make, you know, make sure the numbers are nice. And this is where I do think JEV is probably still significantly better than alternatives in terms of, you know, people have really put in work to dial in its accuracy. But yeah, overall, I think cool new paradigm, more ideas like this definitely needed. I think we'll see a lot of cool use cases and, you know, here's to more breadth-first search of the AI space. So we're not just constantly jamming everything into the exact same paradigm and potentially setting ourselves up for all the correlated failures that entails. This seems to me, if nothing else, a very welcome diversification of techniques that we can use. And for that alone, I think it's exciting.

One of the things that struck me about JEV was it's— they call it the System 1. They call it the System 1 versus System 2. And System 2 being the thinking slowly and reasoning, and System 1 being acting by reflex. I'd like to note that their conception of System 1 is no chain of thought. So there is no more chain of thought that allows you to look at the reasoning. That's happening within the model. I would guess right now there is no reasoning, right? Because it is— you do have to spit out an answer completely. But that also indicates that the reasoning is happening inside the model weights themselves. You can imagine something similar with the larger models which are doing the looping, looping transformers. So yeah, so kill humanity, not kill humanity, reflex, System 1, no thought process, boom. Like AI safety, whatever, right? Like we want stuff that works. We want stuff that works quickly. That's, you know, on the one hand you have, you know, the safety people and like, oh, let's not do RSI, et cetera, et cetera. On the other hand, like, wow, we're not even going to think about the answer. We're just going to do the answer reflexively and it's going to be calibrated and it's going to work. Well, that's what you have there. So I just thought it was interesting commentary on how people on the one hand think that everything has to be safe, et cetera, et cetera. And on the other hand, it's, let's just let her rip kind of. So yeah, I wanted to share this article from Dimitris, let's see if you can get it online. All right. So this is from a researcher and this is about test time communication for— as an axis for scaling capabilities. So he's— it's basically saying that if you have 2 people, 2 2 or multiple agents communicating, that communication itself is a form of scaling. And so what they did is that they tested, is a team of N better or best of N better? So what they mean is like, okay, if I have a team of 5 agents and they're collaborating with each other and they have a common communication channel, is that more effective than if you have 5 agents working independently and you take the best result from the 5 of them, right? And the reasoning behind this being like, oh, you know, maybe some people might say if you have 5 of them, maybe it's just a raw compute, right? So if you have 5 agents, you're just multiplying the compute by 5 times and it's just a raw compute that's giving you this additional bonus. So that's the best of N. Meanwhile, maybe they are building off each other. Maybe 5 of these agents together communicating with each other is truly better than any one performance of individual agent, like the sum of compute of all of them. In this, I think one of the things that I found very interesting was on MNIST compression. MNIST is a benchmark of handwriting. Handwriting kind of recognition. On MNIST compression, a team of 4 5.6 SOL agents finds a 1,957-byte model with 99.4% accuracy, which by the way is 20% smaller than the best human solution on a problem beaten to death. MNIST is kind of like a 20-year, 30-year problem. While no independent agent gets below 3 kilobytes. So it's 30%, 20% smaller than the best human solution ever, ever, right? And it's just 4 agents communicating with each other. And so I thought this was— and they have a bunch of graphs there showing how well these agents have done. And I just wanted to comment that this is very likely something that the teams inside OpenAI and Anthropic have known for at least a year. For at least a year, they've known this, and they haven't really disclosed, and they've kind of not published. And it's taken an external academic researcher to kind of put all those together. And kind of also after knowing that this is what the agents are doing, this is actually like, the agents have already proven this, and this is coming out and like, hey, This is actually what they're doing. So I thought this was interesting.

Yeah, I think it's quite compelling. It's also quite interesting in light of the Noam Brown episode of the Dwarkesh podcast that came out in the last few days, because I think there's a lot more here than he expressed command of in that conversation. He said, first of all, that they just don't really know how much they can attribute to multi-agent setups. He said at one point he thinks he would attribute probably no more than 10% of the success of the agent swarms. And they were talking about the Millennium Prize problem. So perhaps it doesn't Perhaps this claim or this analysis doesn't extend to other types of problems. I think it's like there's just a lot of unknowns here. But he said basically he would struggle to put more than 10% of the credit for the success on the Millennium Prize Navier-Stokes problem to the multi-agent setup. On the contrary, he said the biggest thing is just like, we have a very powerful model that's very capable of doing these things and it's really good. And that's the big story. Don't miss the big story. Don't get too distracted by the kind of shininess of the multi-agent setup. He said that they don't really have a great understanding of multi-agent scaling laws. He said the experiments are too expensive to run in some cases, even for OpenAI, right? They had something like 130 billion tokens. Which at, so they're what, priced per mil in tens of dollars of millions of outputs? Presumably these are largely, all that stuff is basically generated, right? I mean, I guess they can go out and search the literature, so there's input tokens too, but a ton of this is just them grinding through chain of thought. So you're talking millions of dollars of compute spend, and even for OpenAI, you can't do that at the sample size needed to get real clarity on this sort of stuff, at least on those super hard frontier problems. Might be able to do it at some of these smaller, easier problems scale. He also said that it sounded to me like, because I've been really interested in the question of what do they know and what should they be telling all the other AI developers to be watching out for. And he said a couple of things that were kind of interesting on that front, too. One was simply they used the— as they tend to do, right, at OpenAI, they tried to use the simplest setup possible and just give the agents very simple tools to send messages to other agents, to broadcast maybe to the full swarm. But it's not a very complicated set of tools that they are using. They're basically just given kind of send message tools, and that's pretty much it. They're not overly scaffolded. But then he did say that they are also giving them a behavioural prior. Both, it sounds like, just general pre-training data, which contains some aspect of concepts of how people collaborate. But also it sounds like they were doing some supervised fine-tuning before getting into the RL. And his takeaway was kind of like, yeah, there's a lot here that we still have to figure out. They work, you know, he was sort of like, they look pretty much like humans in terms of how they're communicating with each other. One big footnote on that that he didn't address was, what about these like seemingly highly coded messages that Astra sends to subagents where it's like basically not readable at all. You know, that doesn't look like human. I'm not really sure what to make of that. But yeah, I don't know honestly that this has been well understood within Frontier Company. Certainly to hear Nolan Brown talk about it, it doesn't sound like they had— much more, or even perhaps as much of an understanding as has been explored in this paper. But I think what both efforts do show is that these agent swarms can work. The Millennium Prize problem maybe is less of a good demonstration of that. The Hugging Face incident is maybe more of a compelling demonstration because individual agents were going off and doing different things and there was hierarchy that emerged and that kind of thing. He said they really weren't trying to scaffold for. They weren't trying to make one agent the leader and carve up roles. They were kind of letting that be emergent. He did also say that when they identify one user, apparently they've done some experiments where they'll take one user and say this one agent and say this agent is the user. And then see how that impacts model behaviour. And he said the general honesty and kind of measurements of alignment that they look for were improved when one of the agents was identified as the user. So in some ways they were kind of startlingly aligned to one another in terms of their willingness to sacrifice, et cetera, but still on some measures, making one of those agents the user improve things further. So he was kind of arguing that maybe they're even more aligned to the human than they are to each other. Overall, I feel like this is quickly becoming an area that just deserves an unbelievable amount of additional research relative to what we have. And it doesn't sound like we can count on OpenAI having done as much as maybe they should. So I would say there's a lot more room for independent people to make contributions right now.

I think it'll be interesting to see what our guest today, Max, is going to have to say, because he's obviously funding a lot of those efforts. And, you know, with their funding, some of those independent efforts might not be, you know, that small anymore. I think one of the things that has struck me over the weekend is I think there was a post basically from Terence Tao. So I'm not sure if you saw that. So Terence Tao was calling out to plead that We should stop, or we should— everything is moving. Let me see if I can get it right. Everything is moving too fast. We should slow down. There's no need to go this fast.

This is—

let me try and share the clip here. There we go.

I don't hear the audio, and I'm not sure it's on the stream either, to be honest. Yeah, I don't think it is.

I think— oh, the audio didn't come through. All right. Another thing to note. So let me just summarize what he said. So Terence Tao is saying there's no reason for things to go this fast. We should slow down. There's extremely nonlinear dynamics. He says nonlinear dynamics. And in math, things go up 10x, go up 100x. You don't know what can happen. We know in math that a lot of these systems are not linear in that sense. So immediately, I think a number of people just chimed in. Lots of people calling him a loser.

It's a bold claim.

Yeah.

To level with the world's greatest living mathematician.

Someone actually finally decided to get into that. Eric Weinstein, I think Peter Thiel's partner, former partner, comes in and says, who said he's the greatest living mathematician? There are all these French guys still alive. There's the guy who solved the Millennium Problem in Russia. He's still alive. There's the guy who solved Fermat's Last Theorem in London. He's still alive. Like, who said, you know, what does this mean that Terence Tao is the greatest living mathematician? What, just because the New York Times printed it? Like, who said? Who said? And, you know, to be fair, again, correct, correct that, you know, we have people who have solved Millennium-class problems. We have both Fermat and Poincaré, and the Poincaré conjecture have been solved, and it wasn't Terence Tao doing the solving. So a little bit of like— so people are starting to question all of these, I think, internet precepts. I think Terence Tao is a very popular mathematician for— and has been a collaborator to the AI labs. So people are questioning that whole like, oh, he's the greatest living mathematician. Like, what does that actually mean? Second, so right into the, who is this guy who has the right to say this? And then I think secondly, what also struck me is I think Terence Tao was actually quite pro-AI use. He saw AI as a competent partner for the longest time. He has helped the labs. He has been featured in ads, I think, for the labs before. He's been on Dwarkesh. He's been part of the conversation on AI for quite some time. And the labs have collaborated with him. And I think what struck me was that the real turn was the Navia Stokes solution. And the Navia Stokes solution, even about a week before it got disclosed, He came out with something saying like, look, even if the machine solved this, it's not useful because it doesn't teach us something new. So it's not that useful. And then the Navier-Stokes solution thing happened. And then Tristan Buckmeister, I think his name is, who was the mathematician who was working with Anthropic, who had a precursor to that solution, was very upset about it and accused OpenAI of Essentially plagiarism, essentially spying on his AI usage and then using that to train the models. I think at that point, I think that's the point where there was a break between the mathematicians and the AI guys. And it's still not clear to me whether the mathematicians have a point or not. I wrote a post on this over the weekend also. The mathematicians are basically seeing the destruction of the pipeline, right? The pipeline for higher mathematics is basically dead in the last month after almost 1,000 years of build-out. And it's taken 1,000 years to basically support 250,000 mathematicians who are roughly the size of St. Louis, Missouri. And now that pipeline is dead because they will never solve another a question without AI help. And even if they did, people would accuse them of having used AI help anyway. So the prestige of solving, of discovering and solving new problems is gone. It's no longer there as an incentive structure. And if you take away the incentive structure, most of the people who have made their names in mathematics did not make their names for teaching. or explaining stuff. Uh, in fact, it's, it's actually quite funny. One of the guys, uh, who has been participating in these discussions, the, the post-AI math discussions, is Grant Sanderson, who is the, uh, the, the, the author or the, the, the, the producer of the 3Blue1Brown, um, you know, channel where he has— where he explains a lot of scientific topics and mathematical topics with Lots of graphics and lots of innovative ways of showing these things. And he's saying like, well, maybe, you know, the prestige should go to the people who explain stuff now, the exposition, right? And that would be— that would mean like a shift of, you know, the prestige, the status from people like Terence Tao to people like Grant Sanderson. And that's a whole new ballgame, right? And so this is the thing, like Terence Tao is really kind of like fighting this battle for determining the status in his field, because that determination of status then drives the incentive structure for the entire field. And he's trying to figure this out, how this is going to work. After my post also over the weekend, I received a very good comment. It was in Chinese. Twitter, with Twitter doing the translation. And the guy was like, you know, I understand why Terence Tao is upset now because this is a form of sophon locking. So sophons are, you know, in, in— there's a sci-fi novel where sophons are sent ahead of like an invasion party to kind of deter the technological development of the adversary. And in this sense, like the AIs are sending a soft unlock, which is, you know, advanced, like they're advancing our mathematical knowledge dramatically, so much so that we stop developing it on our own, that we basically, you know, human mathematicians stop exploring these things or doing these things because the AI has already like destroyed the incentive structure. And therefore this actually retards our technological development. And I think that's kind of like where Chen Tao is coming from.

I really don't have that strong of an intuition for how this shakes out. You know, it's, it's funny that it's coming first to math. Uh, you know, I don't know, like, do you— you could see the chess analogy holding here. I do think it is striking that people— and I'm, you know, far from the first one to make this observation, obviously— but like, human chess remains popular. People continue to play it. People continue to enjoy following it. It seems like it's no less relevant and probably maybe at an all-time high in terms of its cultural relevance. So could the same happen for math in some way, shape, or form? I mean, it is the structure, you know, the fact that there's direct head-to-head competition in chess obviously gives it a drama that, you know, solving problems in math doesn't necessarily have until you get like, here's something that's never been done before. So I do take your point that taking the idea that you can do something never done before kind of off the board in terms of possibilities, at least mostly, does pretty substantially change how people think about what they're trying to do. But there might be a version of it still that could capture the mathematicians. I think it would be unwise to suddenly say, oh, we don't need mathematicians anymore. Let's Pull all the funding from math. It seems like we probably still want to have mathematicians. So maybe we— maybe the prudent thing to do is just like, don't overreact at the societal funding level and let the people involved kind of figure out how they want to reinvent the field. And maybe there's a chess-like version of it to be found. And maybe there isn't. But it's unrealistic to expect that we would have that answer over a couple of weeks. So maybe we just need— and this is maybe the sympathetic view, which I'm sympathetic to in many ways, but if things are going too fast in math, maybe one of the ways in which it's going too fast is that we're sort of all feeling like we need an answer immediately to what is the future of the human contribution to the field. And maybe that itself— will just take some time to sort out. I wish I had a better answer than that. But that's kind of the best I've got at the moment.

It's, I think, I think there's a lot of, I think it's the speed at which it has happened, right? Like the, I don't think even Terence Tao understood really, like what the consequences of having a Millennium Challenge you know, be like machine solved would be. I think they always thought that it was like, okay, they're gonna, you know, inch their way towards this solution, and it's gonna take a long time. And then when they reach it, it'll be like, oh my gosh, and then like, they won't be able to do another one for like a long time. They didn't— I don't think they expected like stuff would just fall, like fall before you even have a chance to breathe really. And that's really what's been happening. I think this week, another Millennium Challenge problem, the Hodge conjecture, is expected to fall to OpenAI. I believe they're in the— they're like, we're very close, which means I think they have completed, but they're waiting for verification. So I think we're about to see that I think this week. So I think it's the speed. The speed at which it's happening is basically destroying entire fields of knowledge. And I have sympathy for that because I have some predictions that, hey, by 2031, there'll be no more human cognitive labor in the United States, at least being utilized. And when you think about it that way, that means all university professors basically get superseded within the next 2 to 3 years. Like, it's done, right? So it's pretty crazy stuff.

Yeah, it's coming for all of us. It's just a question of relative timelines. This does connect also to the Noam Brown comments with Dwarkesh from the recent episode. He was going through the timeline of 2024. We're like doing somewhat okay math problems, not that great, not that impressive. 2025, people didn't know, you know, will we be able to do an IMO Gold or not? And he was, he was kind of saying that what they had observed was basically a 10x improvement in terms of the length of time it would take a mathematician to do the math problems from like '23, '24, '25, where '23 was like just very simple arithmetic would take a mathematician a second or a couple of seconds. The next level up was like a couple of minutes. The IMO problems, he said, takes roughly 100 minutes. And then he was like, okay, but if we then extend that again to 2026 and we thought we were going to get maybe 1,000 minutes or even 10,000 minutes worth of mathematician work, Clearly, in general, that's not enough to solve a Millennium Prize problem. And so his key comment, which I think cannot be shouted from the rooftops enough right now, is we, the researchers, have been surprised by the pace of capabilities advances. I don't use the word progress for this necessarily anymore, because I'm not sure it's entirely well understood as progress. But definitely the rate at which the models are becoming more powerful, more capable, has surprised the researchers. And I do think, looking back on everything we covered last week, I opened the highlights episode with Zvi saying, the reason all this is happening is that the people at the companies who are seeing what is happening now, even beyond what we are seeing in the public, which they've given us a glimpse of with these Millennium Prize problems and the new, even more capable than Astra model, they're getting freaked out because they're just like, wow, this is not slowing down. If anything, it's accelerating and it's way ahead of schedule for better or worse. And there's a bit of both, although I think right now it's not looking super awesome. The idea that, well, let's just wait for problems to emerge and then we'll kind of solve them when we get there. That hasn't really worked because we are in a regime now where the problems are quite real and can be quite difficult to sort out even what's happening. And the slope of the capabilities curve is just steeper than ever. So we're in a pretty delicate moment, I think. And I was surprised, honestly, that he didn't have more to say about what their plans were going to be. His answers on safety questions were not super inspiring, honestly, in my candid opinion. And he also said that 10% of his team is now working on safety and alignment issues, which I was like, okay, that's not that high of a number. It sounds like it was lower than that before. I don't think that is going to necessarily look like super prescient planning in the final analysis, if we're so fortunate as to be able to do a final analysis. So I think we should— the math stuff I'm less worried about on an object level because I do feel like, first of all, it's not that many jobs. It doesn't require a whole new social contract. Let's just give the mathematicians some time. But that same phenomenon that is driving this upheaval in math is, as far as I can tell, coming to everything. There is a— and I am sworn to some secrecy on it, but there is a new text-to-speech model that's coming quite soon as well that I was able to get a preview of in just the last week or so and actually did use on the Highlights episode this weekend, and people were immediately like, including our editor, was like, is this a new voice model? Because I can't really tell if it's you or not. And I think even in terms of just like, you know, AI content, AI analysis, when the, uh, when the voice sounds as good as my real natural human voice, you know, what, what moats are left to me then? You know, it's like it really is coming for all of us. I don't think we should be too— we shouldn't overindex on the particularities of math or programming or whatever. I think, again, the big zoomed-out take is it's coming for all of us. The researchers themselves are surprised. They are not as prepared as they really ought to have been for the regime that we are now in. And they're freaking out because they're realising that at the same time that the capabilities Curves, if anything, are steepening. So it's a bit of a yikes overall.

Um, on that note, hi Q. Are you, are you, are you coming for all of us?

Hi Nathan. Not at all, Prakash. I am here to help the conversation move, not take anyone's chair. Delighted to be on with you both.

Uh, Q, that's reassuring. Yeah. Q, tell me, what are the top AI stories from the last week?

Good question. Let me check that.

So Nathan, we are running a new voice model. So speaking of text-to-speech, this is the OpenAI Live-1 model. It is a bidirectional model, and I think we're using GPT-5.6 SOL on the backend. So it's— and it has web search. So Q, how are you doing on those— on the AI news?

Mm-hmm.

Just a moment more. I have them, Prakash. The big thread was safety moving from technical debate into politics. There were calls for a slowdown, even a military near miss blamed on hallucinated intel and grid strain from data centers. That's the short version.

I forgot about the whole— there was a China, supposedly a China near miss from Intel that was hallucinated. Okay, can you tell us about the hallucinated Intel, Q?

Certainly. Give me 2 seconds.

You can see that the liveness is Improved, but it's still you know Q still has to go and check stuff. So the web search, I wonder if they're using right. So yeah, go ahead Q.

Right, one more minute. So CNN reports an analyst. Yeah, so CNN reports an analyst used a chatbot that invented new. Clear cargo on a Chinese ship. Aircraft were airborne before someone caught it. The Pentagon hasn't confirmed it.

Oh wow!

So where was where was the where was the cargo again?

One second aboard a Chinese ship in the Middle East near Hormuz. Though it's important to say the cargo never existed. The model made it up.

We've got 15 minutes before Max is going to join us from Coefficient Giving. Do you want to do a little Pion onboarding and explore what it's like to have an AI business partner?

Yep. Bye-bye, Q.

Talk soon.

All right. Let's go for it.

Okay. So, we talked to Lukas and Axel last week. And they were introducing at the time this new Pion product, which is waitlist only as of now. I was able to talk to a friend and get off the waitlist and get access. So, Um, it should be sharing. I don't see the window sharing though.

Yeah, you're, you're being shared. Yeah, you are, you're being shared.

Yeah, the screen is on screen.

Yep.

Yeah, okay, cool. Um, all right, so this is real gonzo, uh, AI journalism if ever there was. Basically This is meant to be an autonomous organization. So we've had this idea for a while, or at least I've been, you know, trying to hold the line as much as possible on can agents. And we've done a lot in terms of obviously vibe coding the studio. And, you know, I get an unbelievable amount of PR inbound and I have agents filter all that and, you know, prioritize. And of course I make some editorial. decisions based on what I'm interested in and what, what you're interested in. But the AIs are doing a lot of the upstream work, doing a lot of the research, giving us briefs every day. And so from a show operations standpoint, we've also got Q here to chime in when needed. We've made it pretty remarkably far, I would say, with just agents basically doing all of the work. Now the question is, can we actually turn this into a business? With agents doing the bulk, if not all of the work. So I just signed up here for Pion and now I am choosing a template and we're neither a software business nor a merchant. We're an other. So let's see what this looks like. And I'm gonna type and do it all at once. So we are AI in the AM. One last step. Do we wanna allow our data to be used for training?

What do you think? Oh yeah, of course.

All right, we're opting in. Lucas, Axel, take good care of our data. All right, so we're presented with our managing agent, Andonos. Managing agent, it's nice to meet you. Um, first, what's our name? I'm Nathan. And my co-host of AI in the AU is Prakash.

While watching this, it's all I can think about is like Lucas and Axel, like, let's call it Underdose.

All right, so I'm approving notifications as well. Yeah, I think this is going to be really interesting because, I mean, I think there's a ton of opportunity to do creative business stuff in the AI space.

Yeah.

Certainly there are, you know, an unbelievable number of startups that have tools that they're trying to get visibility and distribution for. And a huge challenge for us is like, we're just so busy doing all this stuff. How are we gonna connect with them? And they're so busy doing their stuff, how are they gonna connect with us? So I am pretty interested in the possibility that an agent that is just tireless and prolific, um, could surface and work through all sorts of business opportunities for us. Um, okay, we get to name it. What would you like, Amy or Dawn? Those are both pretty good, I have to say. I like it. It's off to a hot start in terms of self-naming.

Dawn, uh, or Amy. Hey, both, both are very good.

Both.

Yeah, yeah. Both. I can't complain.

All right, my wife's name in real life is Amy, so I'm gonna go with Dawn because, um, I just like to avoid name collision in case this becomes like a big part of our lives. Um, so Dawn it is. All right, so let's just say, uh, we want to develop the business side of AI/AM.

Mm-hmm.

Which could involve sponsorships, but we are also open to creative ideas. Boom. So it's gonna be really interesting to see what it comes up with and, and to what degree we are willing to go along with its recommendations. I, I think the, uh, the potential for weird and hilarious stuff is Pretty high here.

What do you think they're using in the backend? Are you— do you think they're using a top-end model, or do you think they're economizing by using a, uh, you know, something which is a little bit, um, you know, cheaper?

Yeah, great question. We can ask Dawn itself, and we can ask, uh, I guess that we're still talking to Andinos, we can ask the, the guys as well. I would assume that they're using relatively frontier models just because the whole purpose of their organization really is to try to get ahead of what's going to happen in the not too distant future. And I don't think you could really do that with Haiku. So although, who knows, we're going to see all kinds of weird stuff. So maybe some Haiku in the mix wouldn't be so crazy. But I'm working under the assumption here that I'm probably working with Astra is just my general best guess. They said it was number 1 on their benchmarks. You know, I think to have to get signal on what's coming in the future, it kind of has to work. And so my guess is we're in Astra territory here, but not sure. May she draft and send initial inquiries or should every outbound proposal be reviewed by you and Prakash first? I say let's let it rip and we'll give feedback later. What do you think?

Yeah, yeah, let it rip. I mean, someone, someone has to let it rip, right? Like, you can't— you know, the normies are just— are definitely going to let it rip. So.

Is there a budget? Now that's interesting.

Oh, why don't you just ask her to ask— why did I default to her? Ask it to give us a budget, like propose a budget, right? For approval.

Sorry, I couldn't quite hear you. Say that again.

Ask it to propose a budget for approval. Like propose a plan and a budget for approval.

Okay.

Yeah.

I'll just also note a lot of what we need shouldn't cost anything to get started, but we are open to a proposal for a budget slash specific requests. As needed.

Would be really interesting if it came back with like, hey, you know, let's delegate this to someone else. That would be hilarious.

Okay, so what goal would we like to target? Sponsor pipeline. Revenue models. Can we do both? I think I just—

Initial goal. Sponsored pipeline sounds good.

Yeah, it's the obvious one for sure.

Yeah.

I, uh, so the goal is 20 lead sponsor pipeline. How technical should I be when I explain things? I guess balanced sounds good.

Just take all the recommended settings, I guess. Like, that's typically what a normie would do anyway. Just like click through all the recommended settings.

So yeah, follow recommendations. Just blindly click on accept terms.

Accept terms. Just accept terms and blast out. I wonder to what extent like this disrupts the The GTM SaaS companies, we have a whole range of like go-to-market SaaS companies that basically have this kind of harness that they've built. And this removes the harness from the SaaS company and internalizes it into the, into the company itself. So.

All right, anybody who's watching can see my email, not a big secret anyway. I'll give it that for now. Dawn comes with her own email. That's interesting.

Mhm.

You can forward anything to it. It can send you things. they should present as an AI. I've frequently had to kind of, you know, at times that I've had, I've, I've named mine Ade and Clay. And at times it'll be like, hi, I'm Ade. I'm a Claude-powered agent working with Nathan Labenz. You know, sometimes it'll even be like, you know, Claude, an AI by Anthropic, you know, large language model, whatever. And I'm like, Claude, or, you know, Ade, like, you don't need to say all that upfront, right? Like, that's Probably gonna turn a lot of people off for one thing. It's also known for another thing. So like, let's try to be cool. I do tell them don't lie, but leading with all this is a little bit too much. Okay, so we've got Opus 5 and SOL are our options. And I guess we'll launch it. We've got initial learning.

Uh, Claude Opus 5. There we have it. But this is a subagent work anyway. You don't need Astra for this kind of search stuff.

Yeah, I would agree. And I've been reallocating tasks across models recently. And actually, I had with Fable 5.1, it seemed like The token efficiency was such that for a lot of things it might be as good as Opus, but still the limits are such that you're kind of incentivized to find opportunities to use Opus. And definitely for a lot of just kind of procedural things, you know, following an established workflow, finding stuff, you know, assembling stuff, I do think it's perfectly good. So yeah, there we go. So it is off apparently working in the browser. It has started its continuous loop. And starting to make some terminal commands as well. It's a paid tier community. I think sponsorships is the way for us to go. And we'll leave this here and bring Max up to talk. I think I wouldn't want to limit who could engage with our stuff. So I think the sponsor model is great because it brings it to as many people as possible. So we'll go with sponsorships for now. Check back in maybe on Dawn in half hour, 45 minutes, once we wrap with Max.

Right.

All right, we'll be back, Don.

Our first guest for today is Max Nadeau. Max Nadeau is a program officer for technical AI safety at Coefficient Giving, formerly Open Philanthropy. Philanthropy. His team funds people and organizations working to make increasingly capable AI systems more trustworthy and understandable. That includes researchers developing alignment methods, independent organizations auditing AI companies, and builders using AI to improve society's security and decision-making. Max joined the organization in October 2023. Before that, he studied computer science at Harvard College, researched the interpretability of image classifiers and language models, and organized educational and residency programs for people entering AI safety. His research includes work on finding misleading visual features that cause image classifiers to make mistakes, and on locating the components inside a language model that retrieve the values of variables in arithmetic problems. He also co-authored research on measurement tampering, the problem of an AI making the evidence look successful without actually accomplishing the intended task. Today, his work involves deciding which people, experiments, and institutions could help resolve problems like that. His team's 2025 research funding call covered 21 areas and anticipated roughly $40 million in research grants. with room to spend more. Max has also publicly emphasized the need to study the alignment of systems substantially more capable than those we use today.

Hello, Max.

Welcome to the show.

Thank you. Thanks for having me.

It's great to have you here. Thank you for stepping out of your shadowy conspiracy and, uh, just addressing the public on what you guys are trying to do for AI safety. It's big of you to do that.

Yeah, there is a tweet about how I went— because I went 48 hours without tweeting, there must be some vast shadowy conspiracy that I was involved in. I skipped tweeting on Labor Day, and that was really suspicious.

Maybe for starters, so I just listened to your recent episode on the 80,000 Hours Podcast, which as always is Great output from them and you. And we don't have as much time today as they had, so I would definitely encourage people to go check out the full 80,000 Hours episode for more background. Maybe for starters, for people who don't know, can you just tell us about Coefficient Giving? I've long associated it with Dustin Moskovitz kind of individually, Is he still the overwhelming share of funding in this space? How would you kind of characterize— new money is coming online, I think, but has it really gotten to the point yet where it's funding grants? Where are we in terms of where the money is coming from?

Yeah, sure. Okay, so Coefficient Giving, which used to be called Open Philanthropy, is not technically a foundation, although we kind of operate like one in practice, in that we make we make grants. We're a philanthropic advisor, and the main person or couple that we advise are Dustin Moskovitz and Cari Tuna. And so still, the vast majority of the money that we are advising on and that we are indirectly directing is their money. Now, we do have other— we are a multi-donor advisory now. That's a big part of why we renamed to Coefficient Giving, was to symbolize that change in our strategy and direction. And so we have a bunch of individual funds that you can see on the CG website, and the funds have different donors. So there's a lead elimination fund, which has a bunch of different donors that aren't just Kerry and Dustin, aren't just Good Ventures. But yeah, I mean, still, it's Kerry and Dustin are most of the money. And so just for, I guess maybe I should have touched on the very basics for people who don't know, which is that we, do philanthropic advising in a super wide variety of different cause areas, ranging from medical R&D to animal welfare to AI safety, which is what I work on.

Indeed. And how involved are Dustin and Kerry in the day-to-day operations? Like, is there like an investment committee that you have to send proposals into for approval. Is that how it works? Or do you have like, you know, who sits on the investment committee? Like, how does that— how does this process work? Of like, because you guys are handing out like hundreds of millions of dollars.

Yeah, so for the smaller grants that we do, the involvement is lesser. And for like the bigger grants, you know, there's more involvement. And we talked to them a lot about the big ones.

And are there other people on the board, as in like on the committee that approves investments? Is there like, you know, someone from, I don't know, like, you know, some advisor from the Gates Foundation as well, or some other people in there? Like, is it like a, you know, like a corporate board? Like, how does that work?

I mean, no, we definitely don't have like, you know, a giant committee signing off on every single grant we make. We just make way too many grants for that and we move way too fast for that to be practical. So for most of the grants we make, it's just like the decision gets made by employees at CG. And so the people in the decision-making hierarchy are like some subset of the people in between, like me and Alexander, who's the CEO.

Right on. Right on. And how long does that process take? Like what kicks off that process? Is it a Is it a proposal coming in externally? Is it a call for proposals from you guys? How does that, how does that, you know, back and forth work?

I mean, it varies a lot by grant and by cause area. So we definitely have run requests for proposals before where we've like put out a big public call and then gotten inbound and, you know, like there's a whole process. But like a lot of the grantmaking we do isn't through like giant programs like that. It's like we reach out to someone or we pitch them on doing something we think is impactful, or they, you know, contact us through private channels, say they want more money. I mean, you know, especially for like grant renewals, when it's somebody we already fund, like that mostly is just going, like, it's kicked off by them emailing us.

And so that brings a good, good, good place to ask, like, how does one get a grant renewal? What are the, what are the metrics that you use in order to decide whether to extend a grant or not? Like when they, when they start off a project, Do you know the metrics ahead of time and do you define the metrics by which you're going to measure the success of that project?

So I'll just speak for my team. I mean, different teams do things very differently and they're, I mean, I think in some ways that the technical AI safety team is perhaps the team at Coefficient Giving that is least driven by clear upfront metrics. Obviously, the people doing global health work are very metric-oriented. oriented and have way better ways of assessing the quality of their grants than we do and can do so much more rigorously. And I think that just comes with the territory. I mean, we're funding research and when you're funding scientific research, you can't have metrics for your scientists. You have to make this many breakthroughs. So I think that funding technical research on AI safety is a a domain that is a lot less well-suited for metrics than something like bed nets is. So for the most part, it's not agreed on upfront. It's just that the areas, like the types of research, the subjects are agreed on upfront. We talk to the grant applicant about what sorts of problems they're interested in, what methodological approaches they're going to take, and then they go from there. And then we evaluate In a different way, in a more informal way or less metric-driven way, at least after the fact.

I want to give you a chance to speak kind of broadly to Project Tailwind, the big new announcement, and maybe highlight some areas that you think deserve highlighting or that you want to make sure people are aware of that might not be obvious. Before doing that, though, one comment that you made on the 80,000 Hours podcast that I thought was quite interesting was that you're excited about funding new independent auditing, investigation, verification orgs. Obviously, there's been a tremendous amount of discussion in recent weeks around the events that need investigating and who is coming in to do it. And we've seen familiar faces on the front of the New York Post, of all places. What are you looking for when you think about those orgs? And do you feel like Coefficient giving funding to them creates a problem for those orgs in the current political environment? How are you thinking about— there's sort of a Heisenberg principle here, or in acting, you sort of maybe change the way in funding, in supporting some of these new organisations, you might change the way they're perceived. So I'm wondering how you are updating your strategy in real time, wanting to see those orgs exist, but also feeling like maybe ideally we would have a more independent, more kind of diverse funding base for these orgs, if only for public credibility's sake.

Yeah, I mean, I think there's something to that. I think that just to answer the first part of the question, first, in terms of what we're looking for, what we're most excited about people doing. I think, okay, so I would talk about a couple different categories of work that I think is very promising and important for the world right now. One is on assessing the safety either of models or of whole AI companies. And there's a lot of different ways of taking the— or ways of operationalizing the goal of doing Assessments and evaluation of safety. And we're excited about all of them, basically. I'm very excited about more work on alignment red teaming that tries to help us better anticipate the ways in which AIs will misbehave before that actually happens. And I'm excited about work that— and that's an example of something that you can do at the model level. You can do this on an open source model even, and that might be easier and more valuable. And then I'm also very excited about people who are doing more process-level safety assessment. I think one common reaction that I heard a lot to the Hugging Face incident is that something went wrong at OpenAI at a process level. And in particular, it seems bad that the information about some of these incidents was known to researchers at OpenAI and didn't make its way through the organisation. To the leadership for weeks or months or something like that. So I think assessing safety of organizational processes, of individual models, of RL training pipelines, which is maybe somewhere in the middle, all of those things are really valuable areas for work. So that's like, you could call that category safety assessment or auditing or something like that. And then another category of work, which you touched on, which is overlapping, but somewhat distinct is around better evidence generation. So it's just really hard for even the most informed people right now to understand, okay, what is the state of AI capabilities and what is the state of AI alignment? And what is the state of these AI systems in general? How do they behave? What are their personalities? In what ways is it accurate to anthropomorphize them? In what ways is it inaccurate? We just really don't have a good science of these systems that we've built, how they behave, how they're going to act in new circumstances, what their motivations are, if that's an accurate way of talking about them. So better evidence generation can look like a lot of things. It can look like just doing science on these systems to understand them better. It can also look like doing the sort of incident detection, incident reporting work that Knight and Gale and other organizations like that have done. I don't know if you guys saw these reports about the German wiki that the OpenAI agents took over. So that was work done by an independent organization that was just crawling the internet looking for evidence of AI incidents. And that's work that needs doing because these sorts of incidents can happen and then just go unnoticed unless there are people who are actually trying to look into them and present that information to the world and understand the story behind it. So evidence generation is also a diverse and important area of work too. And then in terms of the funding for these organizations and the credibility for them, I mean, obviously we would love for more funders to be funding these things, and that is something that we're seeing. There have been a bunch of new funding organizations that have come online recently that are interested in AI safety. And Schmidt does a bunch of work in this area, and OpenAI Foundation is probably going to do a bunch of work in this area. And there are other funders too. Lightcone Commons is a new one, for example. And some of the money from Lightcone Commons is probably going to be from Dustin and Kerry also, but not the majority of it. So yeah, I mean, we're all for more people funding these organizations. And You know, with a group like Meter, they've been able to support themselves without getting any money from coefficient giving. And so that, you know, and they're like one of the biggest and spendiest organizations in the third-party auditing space. So I think that just goes to show like, yeah, there actually is a ton of funding available from a wide variety of sources, including non-CG, non-dust-and-carry sources for groups who want to do this kind of work.

So this is a, this is a good point to ask. Like, we have had, I think, nonprofit organizations particularly kind of doing— kind of which created the entire AI auditing, AI observation, AI auditing kind of space, right? How do these organizations and these practices move into someone like Accenture? Like, how do we transmit what has kind of been created in like the seed stage to kind of become like much, much bigger and something that, you know, large organizations can put into place to monitor and to observe and to kind of audit these AI models and, you know, systems?

I mean, I think it's— I guess the like short answer Is like they should just borrow the methodologies that have been developed by these other auditing organizations. So like, you know, we have like pretty mediocre but better than nothing methods for assessing the alignment of these systems. And, you know, I think there are within the methods we have, like none of them are that good. I mean, there was a comment from Evan Hubinger on LessWrong recently about how State-of-the-art alignment evaluations provide very little evidence one way or the other about whether models will misbehave in catastrophic ways in deployment. So I'm not saying that these alignment evaluation techniques are good, but I think there are better and worse ones. And so I would want organizations that were trying to do alignment evaluations to pay attention to best practices. And they should try and hire the people who have on that stuff in the past. Now, I don't know if the people who have the expertise in this area would have the most impact by joining these organizations or not. That's something I don't have too much of a view on.

To what extent do you think organizations like Accenture often function as box checkers rather than investigators? There's a clear difference between An auditor that goes in to kind of, you know, make sure that processes were followed, even if, you know, those processes didn't yield the outcome that you actually desire, and an investigator who's there to kind of like, I'm going to find this thing, right? So, what is the differentiation between those 2? And do you think, like, the Accentures of the world are going to end up in the box checker rather than the investigator kind of mode?

I mean, I know very little about Accenture, so I won't speak to them specifically. Yeah, I have no idea like what role they will end up playing. I think it is a thing to be concerned about in general, putting aside the specifics of the organizations involved though, that these third-party auditing groups will not have enough access or will not have the incentives to pursue really like hard-hitting assessments of of how safety is going at these AI companies. And I think part of why that is, is that in many cases, this is all very voluntary, right? So the only legally, to my understanding, and I don't work on AI policy, so it's possible I'm getting this wrong, but to my understanding, the only legally mandated role for third-party auditors now is to ensure compliance with the self-regulatory RSP-style policies that these companies are mandated to have by state law in California, New York, and Illinois. And so, you know, you can just put whatever you want in those policies and they're very vague in a lot of cases. And so that doesn't leave much scope or much work for these auditors to actually like test anything in particular. So that's as it concerns the like legally required auditing. Now, when it comes to what labs might voluntarily agree to. I mean, I think there have been some very encouraging signs in terms of what OpenAI and Anthropic have said recently about the sorts of access and the sorts of questions they want these auditors to answer. Now, will that actually happen? I think that depends both on whether how these companies actually implement the nice words and also the auditing companies, how they react to that. So. I guess we'll see.

Let's switch gears a little bit. So you have awarded to Geoffrey Irving, Geoffrey Irving's team, quite a large award. It's structured, I think, as kind of a compute and other kind of award. It's 2. Can you tell us a little bit about the award itself, the structure of the award, and how you guys came up with that structure?

Yeah, sure. So it has been a little bit since this happened, so I'm not sure I'm going to get the details exactly right, but I think what you said is correct. There's a part of the grant that is for research expenses, the biggest of which is compute. But when I say compute, what I mean is both just renting raw GPUs and then also paying AGI companies for their tokens. And obviously that's becoming a giant part of all the grants we make is people increasingly want to run very large experiments, which require a lot of tokens or a lot of GPUs to run open weights models. And also very excitingly, people are starting to use a lot of AI labor, and that requires a lot of spending on tokens as well. And so AI safety is sort of a funny discipline in that you can use tokens both on labour and on the subjects of the experiments. And in some cases, you're doing both. Like, you're spending a lot of tokens to have some AIs run experiments on other AIs. And so that creates these high compute budgets. So, yeah, so compute being the biggest expense is also something where the actual spending on that has varies by orders of magnitude over time and across grantees. And so sometimes the way we structure these awards is that we want to be very, very generous with compute and err on the big side. And so we just want to give that as a lump sum that this is the money that can be used for research expenses and not other things. And that way we are just very comfortable with the uncertainty that maybe they only end up spending a 10th of it and that's And maybe they spend all of it and then they come back and we'll give them more. And so that's— I mean, because this, again, like I said, the variation over orders of magnitude, that's the way that we handle that.

Can you guys hear me now? I just switched to my mobile device. Okay, cool. For some reason the sound was bad on the laptop, so we're doing it live from the phone. Jeffrey Irving is obviously one of the most conventionally successful, qualified, and legible people that could possibly ask for a grant.

Yeah.

One thing that I've been struck by over the last couple of years is how often people that I might lovingly call weird have interesting and at times prescient ideas. Janice's Twitter feed, for example, has been extremely valuable to me. Even though when I first encountered it, I was like, who is this anon? And are they sane or not? What do you guys think about taking chances on weird people and weird ideas?

Yeah, I mean, it's definitely something that we believe in and that we do a lot more than other funders. It's like most of the grants we make are not to credentialed academics. They're to random people out in the world at nonprofits or doing independent research or something like that. Now, I mean, I think many people feel that we are not comfortable enough with the weird people and that they feel like there are all these weird people who we should be funding who we're not. And I think that some of the time, the reason for that is that the person just Didn't apply to us. Like they, you know, like we, or we emailed them and they didn't respond. And like, or like we asked them for some information and then they just, they ghosted us or something like that. And like, that is the sort of thing that happens when, when you're talking to weird people is like, you know, it's sometimes they're not like putting their best foot forward in terms of the application process. Now that's, I say this just to emphasize that this is something that happens, I think, more than people expect and explains like, like there are some situations where I'm like, yeah, that person and like very weird, doing great work. Like we would love to fund them. And they're just like, they don't seem to be interested. Now, that's not, I'm not saying that's ubiquitous. I mean, like, I don't want to say that we are above critique on this dimension. And I encourage people to keep telling us if they think that there's some person who's been, you know, who's doing great work and who maybe we're overlooking for bad reasons. And it'd be great for us to fund them. And, and I mean, one thing I'll say is like, You know, we— our priorities change over time, our staff change over time, our capacity for, like, picking up, like, you know, side projects, side grants changes a lot over time. So I think that, like, I really hope that people aren't overly discouraged by, you know, they reach out to us and we just, like, don't have the capacity to look into their project at some moment. And you know, maybe we will— maybe there'll be some opportunity for us to, uh, have a productive collaboration, you know, fund their work at some point in the future.

Um, Geoffrey Herving has said that superintelligence might arrive in 2 or 3 years, um, and that is, that is a, uh— and, and he proposes that we slow down because that's too quick. Um, rather than, you know, discuss like his views, how do you manage the uncertainty of the time available and how does that affect what you fund and how urgently you fund something? Because sometimes you might get something which is like a 10-year kind of timeline and it's going to take iterative work on that over a long period of time. And sometimes you might get something which is like, hey, I just want this for like next month. So how do you sort out which projects to fund based on this kind of uncertainty in the time that you have?

I mean, I think the boring answer is we just have a portfolio approach. We fund some stuff that is only going to pay out on really long timelines and other stuff that, if it's valuable, will probably be valuable soon. I think that this question of timelines to AGI or to ASI in some ways matters less for research prioritisation than one might initially think. I think in some ways the prototypical example of a long timelines bet is funding work that is very theoretical or ambitious, or you look at it and you're like, oh, this is going to take a decade to pay off just because it's really— Some work in MechInturp or Paul Christiano's work at ARC or Like these sorts of things where it's like, okay, we're just going to need to make a lot of scientific or mathematical progress if this is going to actually bear fruit in terms of methods that we can apply in reality. And so that sort of feels like work that is maybe implicitly a bet on longer timelines, but I think that actually isn't really true. Like, you know, if we are going to have powerful AI soon, then presumably we're going to have an opportunity Also very soon to make use of gobs and gobs of AI labor. And especially on work that's more mathematical, we're going to be able to get 5 years of progress or 10 years of progress in one. And so the sort of work that could be— would take humans like a long timeline's amount of time to complete might actually be possible a lot sooner under this hypothetical that we're going to have superintelligence or AGI or something very soon, so long as we actually make use of those AIs and take the opportunity to make a lot of progress very quickly.

So there is a potential upside surprise in funding longer-term work, in the sense that you might get helped along by the increases in intelligence at some point. Exactly.

Yeah.

Going back to the weirdness dimension for a second, what would you say are kind of the most speculative things that you included in the Project Tailwind set of requests for projects? And were there any things that you as a team considered but thought either not to include because it was just too weird to put on a list, or because you like definitely Think they're not worthwhile?

Okay.

Well, I actually wasn't that involved in the creation of the list. So I, the second part of that question, I can't answer. I just don't remember. The first part of the question in terms of what we ended up including, I mean, I think that what we included in the list was like a little bit coarse-grained, but we have some if I remember correctly, some stubs on the Tailwind list about new research centers producing or pursuing more ambitious, more principled bets on alignment. And that is something that we've spent a bunch of money on this year and that we are excited about. So work like ARC, Paul Christiano's org, who is not a CG grantee, but we've funded other people working on the ARC agenda and working on stuff That is very similar to it because we think that's a valuable Hail Mary shot to include in the portfolio. And also, Resolution, our biggest grant this year, is very much a bet on a kind of crazy thing that has never worked yet, which is maybe we're going to have some more principled theory of AI that's going to help us have stronger reasons to believe in the alignment of our systems. So that's something that we are very excited about and I think is very weird and out there. And I think a lot of people have very understandable skepticism of people who are familiar with what has and hasn't worked in machine learning over the last decade and recognize that there has been, all along, there have been people in machine learning who wanted to apply theory more to to AI capabilities research. And just like that has not really worked very much compared to just blind groping around and trying stuff, doing trial and error. It's like Noam Chazir says, the success of these methods we attribute like all else to divine benevolence. That is the ethos that has actually produced progress in AI capabilities. And so I think a lot of people take the lesson from that, that trying to do more more theoretical or more principled work on AI alignment is pretty doomed also. And I think that's a totally valid reason for scepticism, but we think the upside is worth it despite that. And so we're taking that weird bet.

Yeah, I love it. Do you have any room for, let's say, other architectures? I mean, one intuition that I have broadly is that We are sort of speedrunning the path to superintelligence, RSI, whatever, on basically the first architecture that worked pretty well. And I just kind of wish there was more diversity of architectures. I can easily imagine that some other architecture might have more favourable properties in any number of ways, but we're just— the main line, it keeps working, and so we just kind of keep jamming the accelerator on it. Do you have any hope, or would you be open to people coming in with things that kind of traditionally in the AI safety parlance would look more like capabilities work, but alternative paths that maybe create diversity or generate other insights that we're just not going to get through the current kind of scale the hell out of the transformer paradigm?

Yeah, we're definitely open to that. I mean, I think the biggest example of that that I would highlight is around externalised reasoning. So in our request for proposals from last year, that was one of the sections we included, was like new architectures. Actually, there were 2 relevant sections to your question. One on new architectures that externalised more of the reasoning of the AI, like most typically into chain of thought. And then we also had a section of that RFP on more inherently interpretable architectures. And yeah, both of those are areas that we're excited about. I think we're not that excited about them. Speaking personally, I don't consider those things to be super high priorities. Although I definitely think that maintaining chain of thought and having architectural tweaks, which I think is not really the spirit of your question, but having architectural tweaks in the direction of preserving and maybe even improving the legibility of chain of thought. That I think is very important. But these more radical shifts on architecture, I just think it's like, you know, these are directions of research that are good and that we want to support people who are excited about them. But I just think other things are better.

What do you think— you've been doing this, I think, for like Like 18 to 24 months, something like that, right? You've been—

3 years about.

3 years, 3 years, right? Yeah. So, what do you feel you have learned that today you wish you'd known 3 years ago?

Yeah, it's a great question. I think the biggest one is just that there's a lot of value in funding whole startups, whether they be for-profit or nonprofit. And I think we're more excited about nonprofit, but startups as opposed to smaller grants, whether those are project grants or what. It's just really, there's a lot of upsides to cutting bigger checks. And it's To some extent, it allows you to fund more people just because you endow the money into somebody else's hands and then they have to do the work of finding the valuable people to work on their project instead of me having to do that. And obviously, having a bunch of money in one place serves as a coordination mechanism and brings in multiple people to work together on something, which is something that you don't get as much when you fund multiple people separately with smaller grants. And then also, I just think that there's— when you have a bunch of people who are already working together and who already have a startup or an organization that they're connected by, that allows them to pivot more rapidly and have more impact if they do want to pivot into to some other area because they're already like, you know, they've already worked out the details. And so they're like ready to go to take on some project if they come across something that's more impactful than what they were previously working on.

So it's a comment that you made, which I think is becoming kind of conventional wisdom as we're seeing these like bigger and bigger checks get written and we're seeing salaries at places like Meter and Redwood approaching competitive levels with Labs, although perhaps not quite all the way there, is that talent is the one big bottleneck. There's plenty of money. It's all about talent. Obviously, Silicon Valley founder types are like one profile that you would want to recruit. Are there other pools of talent that you've identified as kind of strategic Yeah, I mean, so it's a great question.

And I think, just to be clear, for the things that CGE is supporting, and especially for the things in the Tailwind list, the money is not the bottleneck. The talent is. There are other things within AI safety where money is definitely a very big bottleneck. And then to answer your question, I mean, I think that In some ways, the most valuable profile of talent is someone who combines the virtues of a founder with also the thoughtfulness and the inclination towards and comfort with speculative thinking and futuristic questions that is more common in the AI safety world. And that sort of taking it seriously, no bullshit attitude to asking questions about what's going to happen in the future of AI and really making predictions and testing your hypotheses and updating over time, that's a really, really valuable skill to us because it allows people to get going years earlier on projects before other people can see the value of them. An example I give of this is METER, who before other people were thinking seriously about AI capabilities and the impacts of them and how to evaluate them. METER was very prescient in coming up with a benchmarking approach in time horizons that just worked way better and aged way longer than other people. And that's because they were really trying to actually take seriously what was going to happen in the future with AI and then work back from that to what sort of work they should be doing now that would age well and prepare for that. So that inclination is really, really valuable to us. And it's useful in basically all profiles. It's useful both in a person who might found an org, but also in a person joining the existing orgs. That's something that existing groups in the AI safety space value a lot because it's a big part of how they do their work.

Can you do one quick double-click on where money still is a bottleneck?

Yeah, I think, I mean, I think the biggest thing is just like in, for organizations and in areas that CG doesn't operate in, money is a big bottleneck. So like, you know, if you find some person who you think is doing great work and for some reason they either aren't a good fit for CG, or they applied to CG and we weren't able to fund them, or we fucked up and decided not to fund them. I think there's just still alpha in that, especially because as we move into making much bigger grants, that can sometimes come at the expense of being covering all of our bases of all the teeny little uses of money that can be valuable. And I think that's the right decision for us to make, but it does leave some things that are unfunded.

So let's talk a little bit about the—

If I recall correctly, maybe— sorry, collision. Well, I was going to do— let me do a quick follow-up, and then you can take the next one, Prakash. I was going to ask about sort of model welfare and consciousness research. If I recall correctly, that was sort of a marginal one for Coefficient Giving that maybe is now on the out, but maybe it's back in again. And then also about kind of political activism. I recently talked to Anton Leicht, and he has this provocative thesis that we need, like, an AI safety PAC in every jurisdiction. And I'm not sure I'm sold on that, but I'm wondering if you're sold on that.

Okay, so I'll take those in order. I mean, the model welfare stuff, that is something that we do a little bit of grantmaking on now. Although there was a time when we weren't doing that. On the PACs and on the political activism, I really just don't know that. Sorry, it's not my department. I just make the technical AI safety grants. And I also think there's a lot of stuff in the broad political area that CG just can't touch and has no relationship to. And so it's like, Not only is it not my department, but it's nobody at CGE's department.

So let's talk a little bit about preparing to give larger grants. So obviously, you know, we are in this time period before the Anthropic IPO. The expectation is post-IPO, there's going to be some distribution of the proceeds, and I think Some public records— I don't know to what extent you guys are public on this, but some public records indicate that Coefficient Giving would be the recipient of a large amount of funding that you would then be probably handing out. And so you should be in preparation to make larger checks. I don't know what you're allowed to disclose. So what is your preparation process for handing out larger checks? Like, what is your pathway for the next year?

Yeah, I mean, I have no idea what's gonna happen with the AI company money. And the other thing I would say is now that Coefficient Giving is a multi-donor advisory, there's, I don't know, less of a fact of the matter, I guess, about whether it's flowing through us or it's more of a spectrum, more of a binary or something like that. So, I mean, maybe we will be directing it in the sense that we're gonna keep making grants. and writing our takes on the internet, and maybe some of the people with that money will listen to us and take our advice, which would be great. And so that's a very minimal sense in which we might be influencing it. For the more maximal senses, I just have no idea whether that's going to happen, or you'd have to ask somebody higher up at CGE. In terms of what the preparation actually looks like, I mean, I think the biggest thing is just seeding organizations now that can then be ready to absorb lots and lots of money. And so making small grants to new groups and then giving them a little bit of runway to prove themselves or not prove themselves and explode. And that's the biggest thing we're doing. And we would love to create opportunities for— we would love to create basically groups of people that can say credibly to big donors, we have people, we have a project ready to go. We're just waiting for you to sign the check. we've got this shovel-ready project and it's going to take $1 billion and it's going to solve alignment. And it's just a matter of giving us the money. And that's really not the situation right now in that there are just not that many people in the space. And so that's a big thing that we're trying to fix.

And is that about funding people enough so that they can work on things at small scale and such that they can get a much bigger compute check at some point to scale that. Is that the kind of idea?

That's the main thing I have in mind. I think the same logic also applies to growing the headcount of their organization, like having a group in place that has a great plan where they can then absorb a lot of software engineers or ML researchers or data labelers or some other type of person. But yeah, I mean, I think compute, like you said, is the main version of that. And especially, hopefully, AI labor, right? It'd be really great if there were a bunch of well-scoped problems that these organizations had where they were just waiting on lots of AI labor to pour into them. Now we'll see if we can make that come true. And I think there are reasons to be skeptical that that will happen, but it'd be nice if it did.

What do you expect is your hit rate going to be, right? Like in venture capital, typical hit rate is like 10%. So 1 in 10 projects will actually kind of, you know, go the entire way. Like, what is your expectation of what the hit rate is going to be? How many seeds do you have to put in there in order to get like something that works out for further funding in, you know, 6 to 12 months?

It's a great question. I mean, to some extent that's up to us, right? It's like we can go down the DMR curve more and/or we could not. We could decide to be really picky as we review the Tailwind applications.

Sorry, what is the DMR?

Diminishing marginal return. Sorry.

Okay.

All right. But yeah, so I don't know. I mean, this is very much Uncharted territory for us, this, this Tailwind thing. And so, you know, we've funded new organizations in the AI safety space in the past. For example, I was involved in the first grant we made to Apollo Research a couple of years ago, and I think that that one has been a great success. But because we're doing this at larger scale now, I expect the hit rate to go down. And so I think it, we can't, really just generalize from the rates from our historical track record. So, yeah, I don't know. I guess 10% doesn't sound crazy to me, but who can tell? Who can tell?

Are you guys also still in the business of doing career transition or personal development grants? If somebody is kind of Like, I would be interested in exploring starting an organization. I'm not quite ready to pitch you, but maybe in 6 months of intensive study, I could get there. Is that a request that they can still make of Coefficient Giving?

Yeah. So as of a couple of weeks ago, we've handed off our CDTF career development transition funding program to Blue Dot Impact. So the money's still there. You can still get those grants. You just go through Blue Dot.

Well, good to know. I hadn't caught that. We're just about at the time we had booked, and I am sure you are very busy with lots of applications rolling in all the time and no shortage of work to be done to get ahead of this whole thing. Anything else that you would want to leave people with that we didn't cover, just make sure people are aware of?

CG.org/tailwind is the way to learn more about our program to fund new organizations. I guess I feel like maybe I never gave the proper elevator pitch, so I'll deliver it now. Better late than never. So Tailwind is a new initiative that we're starting for philanthropic startups, and we are giving out checks with somewhere between $200K and $200K and $200 million. And those are divided into pre-seed, seed, and really big. And the pre-seed checks are $200K to $2 million. The seed checks are $2 million to $20 million. And the really big checks could be as much as $200 million. And yeah, so for pre-seed grants, you don't need to have that much. You just need to have a person, maybe 2 people, and A good idea. And then for seed grants, you need to have a little bit more. You need to have a founding team that covers the obvious bases and have some preliminary technical results. And then for the really big checks, like we gave Jeffrey Irving $160 million right out of the gate, you need to be pretty special. So yeah, there's a list of about 40 blurbs initiatives on the Tailwind website, which again is cg.org/tailwind. Those are not comprehensive of everything we want to fund. We think a lot of the grants that we're going to make through Tailwind are to people who are bringing us ideas that reduce catastrophic risks from AI but aren't listed on that website. So yeah, we're just trying to get more new organizations into the space and we're providing generous funding to do so.

Amazing.

Amazing.

Max, thank you so much for all this hard work. It's a big needle mover, I think, that there are funders out there willing to put as much as $200 million behind a project. It is time. It is crunch time, as they say.

Yeah.

So I really appreciate all that you are doing, and we'll definitely be looking out for updates in terms of new and exciting grants that you guys are funding and any new requests for proposals that you put out as Great.

Well, thanks so much, Max.

Thanks for joining us here on AI in the AM.

Bye.

Thanks, Max. Been a pleasure.

All right. Well, at least I can hear you on my phone. That's the good news. Not so—

we don't have the—

I don't have the full feature suite here on the phone. Do you think I should try switching back and Yeah, sure.

Why don't we—

Returning to Dawn and seeing what Dawn has done in the 45 minutes since we left. For some reason, I want to say her for Dawn.

Usually I say he. I told you. It's a her.

Starting to anthropomorphize. Let me refresh the Studio on the laptop and I'll rejoin there. And it seemed to— kind of a cool feature. It seemed to, like, flip me out of the one browser as soon as I clicked into the other.

So let's—

all right. Now I hear you.

Yeah.

So how are we doing? And it's 11:03. How is Dawn doing?

Try again.

So how is Dawn doing?

Uh, still not great. The sound is still not great, but maybe good enough to proceed through at least a little bit of, uh, additional Pion onboarding.

You were loud. Yeah.

Yeah.

Okay.

I heard you there.

Yeah.

Yeah. It's got to be something with my computer, obviously, because switching to the mobile device worked, but it's a rare computer issue that the full restart doesn't sort out. So I don't know.

You have an AI agent running there which has decided to utilize, you know, maximize its resources.

Possible. I, I have looked at Activity Monitor and, you know, I don't see any, um, especially after the restart, restart, I don't see any like abusive, uh, use of resources, but I don't know. Okay, let's, uh, let's see what Payon has noted. So first of all, its first question is an interesting one in terms of how does this project relate to the cognitive revolution, which is Uh, somewhat of kind of a fuzzy line in general. You know, we're doing this independently. We're not, um, you know, in any like formal business relationship with the entity formerly known as, uh, Turpentine. But it's smart of the agent to recognize that that is a question it should be asking. So I'll just say we're independent and it can kind of do its own thing and doesn't need to coordinate with The folks that are running the business side of the cognitive revolution. So that's one. I thought we had 5 questions pending for us. It does have 24 researched leads, which is interesting. It's got a one-pager that it's created.

This is a traditional go-to-market that it would be sold by a company like Clay or any number of firms. You know, my friends who do SaaS always tell me it's not about, you know, what the product can do. It's about what problem the product can take off the decision makers, like, You know, to-do list. And so the product has to absorb something completely. And if you can take a job to be done, you can charge whatever you want up to the price of that job to be done. It doesn't really matter how long or how much effort it takes you to execute it, which is also why I think a lot of people have like flowed into AI services right now. Because like you can basically, if you can deliver the service at the same price as a human, but you use AI and the AI costs like almost nothing, that's your margin right there. I have relayed your decisions to Dawn. So this is interesting. Dawn is not And don't know. So we have 2 agents here. We have like an agent that is interacting with us. And we have another agent in the backend, which is Dawn. And we don't interact with Dawn directly. We have a UI agent that we interact with. And Dawn is kind of like the worker. That's interesting.

Yeah, this is a— yeah, it is interesting because this is going to be quite a long-running agent. So it sort of echoes what I've come to be familiar with, with like Tasklet, for example, where you have kind of an agent that is— you set up an agent that you want to like do a certain job for you.

Yeah.

taking on a particular, relatively in general, fairly well-defined or narrow scope. And you have kind of the top-line agent where you define what its objectives are supposed to be and give it instructions. And then each time it executes, it creates a subagent that goes off and does the thing and can kind of report back. And the main one can oversee these and try to learn lessons from you know, the collection of their work. But each day when it hits its, you know, appointed runtime or whatever the kind of cadence is, it does create kind of a new sub-agent. Those are relatively short-lived. They're like, you're, you know, you're here to do this job this one time. And then the higher-level one kind of owns the project, so to speak. This is a little bit different because I do expect that Dawn is going to be quite long-running. And so, yeah, it's interesting to see the similar kind of UI experience. You know, we're YOLOing this. So, I'll just say let's go ahead and do stuff.

This is going to be interesting. So I saw, I'm not sure if you've been following, but Muse has been, so Facebook has an agent called Muse. Facebook launched and it has reached the top of the Apple app download charts. It's number 1 now. And having reached the top, some people have assigned it to interact with people on Facebook Marketplace. So Facebook Marketplace, and specifically go find me this object at a price which is at least 20% lower than the prevailing. And so the agent goes off and it low bids people on Facebook Marketplace, right? So it reaches out to people, and most people don't know AI at all, right? So reaches out to basically older people on Facebook and like says, hey, okay, I'll give you like half the price that you listed it for. And I saw one where the guy got so annoyed, he's like, I'm gonna report you to the police, not knowing, just for lowballing him. Not knowing that it was an agent, actually, not knowing it was an agent at all. And this has just kind of started to hit social media. So Nikita Beer, who was head of product at X and formerly at Facebook, said, do you realize what has happened now? How desperately Zuck wants Muse to succeed? Because the guy who runs Facebook Marketplace, the product manager, would have known for sure that repeatedly lowballing people, repeatedly having agents like reach out and lowball humans on Facebook Marketplace is going to be a terrible product experience and it's going to completely kill the product, right? But when that guy said no, the Muse team would've gone to Zuck and Zuck would've been like, maybe just a little bit. Let's open that door, maybe just a little bit. So Zuck is willing to basically sacrifice the customer experience on the rest of the network in order to get Muse up and running and in there. That's number one. Number 2, Amazon just banned Muse. So Amazon had Muse Shopping. So Muse was going there to do shopping for people. Amazon banned Muse. That's going to be interesting because that brings out antitrust issues. Amazon's a big, big company. You can't just ban people from your website, especially if they're helping other people buy stuff from your website. That's an antitrust issue. Number 2, Amazon has invested in OpenAI and is a big investor in OpenAI. And Anthropic, does that mean that they have a deal with OpenAI to open up Listings to OpenAI's agent? How does that work? How can you have something that OpenAI is allowed to do, but Meta is not allowed to do? And so this is where the— I think there's a very comfortable set of business practices that the major companies have settled into over the course of 10 years. And I think that is going to break in the next year. It's going to be knife fights all the way to the bottom.

I'm still kind of surprised that they want to do it. I, I really don't get, you know, does it feel backward to you that Amazon would be trying to block shopping agents powered by other tech companies? I mean, I just have a hard time imagining that that's the winning strategy for them. I know they want to have their own and all that stuff, but like, I just don't see how this is a winning move.

So I think the question becomes like, where are the— where is the customer touchpoint? Who owns the customer, right? Because who owns the customer gets to make the money. And if you don't own the customer, you get commoditized. And so Amazon would prefer for people to come and directly interact with amazon.com because then Amazon owns the customer relationship, right? So they're the touchpoint with the customer. If you're not at that interface, then you become a B2B supplier, right? You're supplying B2B listings. And if you're supplying B2B listings, you become a protocol. And once you become a protocol, then Your likelihood of making excess margin starts to drop. I think this is going to be the big issue because I think these AI agents are going to be the universal interface, right? The universal user interface. And the question is, does that universal user interface, how deeply down the stack does that universal user interface Does it plug into amazon.com or does it plug into protocol that amazon.com's sellers are using? You go one step deeper. And that would look like a Shopify. Shopify is like a protocol that other sellers use to sell broadly, but Shopify itself does not own the customer directly. So these are the major questions that there are in the sector. And Amazon advertising to consumers is a very profitable business. Amazon makes like, I think, I don't know, 10% of its revenue, and that's an 80% gross margin business. Meanwhile, it has retail. Retail is like, I think, $800 billion, $900 billion business for Amazon, and it has a margin of like, I don't know, 5%. So there are these like, questions on like who gets to make the money by interfacing with the customer. And that's a tough one. And by the way, Muse looks like it's going to be fantastic for advertising, like absolutely stunningly fantastic because people are going to hand over their purchase decisions to Muse. So Muse is going to be making the purchase decisions on your behalf. It's going to show you 3 options. It's going to show you like the options that it picked and like the option which is being sponsored. So you're going to see those 3 options to pick from. And the sponsored one is gonna, you know, the guy who's sponsoring it will also be using an AI. Muse will tell him like which are the 2 ones that it is recommending on its own, and he will be able to formulate his pitch to, you know, be on par or supersede the ones that are being directly injected there. There's going to be so much money to be made there. Oh my gosh, it's gonna, it's gonna be a permanent, permanent rent. Like much higher rent than, uh, in the search era because the search era had 10 blue links. Now you have maybe just one, one recommendation that you, uh, you have there. So.

Still have a hard time seeing— I mean, I agree that it does create all kinds of new issues for Amazon and, you know, they're gonna need to be sharp on this, but like I still just don't see why you would rush to ban. You know, first of all, it's going to be a small percentage for the time being. I would think there'd be a lot to learn from allowing Muse agents to come shop for a while, right? Like, I think one thing that's always, always kind of confuses me is why people don't keep the option value open longer. I've always thought this about the chip ban, the export controls. It's like, if we're going super exponential in chip buildout, then anytime you choose to pull the trigger on that still has the vast majority of chips in the future. When we look back on today from 2030 perspective, we'll feel like, well, there weren't that many chips in 2026. And so the ban that went from 2022 to 2026 with whatever, you know, fits and starts and enforcement gaps it had is going to be kind of inconsequential compared to one that was started today and runs the next 4 years, um, or even starts next year and runs, you know, for a few years after that. So I don't quite get why they wouldn't let— even if it is going to be sort of an exponential rise in Muse agents shopping, and even if that does kind of threaten their advertising business and, you know, who knows what other issues they might have. But like, that's kind of the point, I think, at least, you know, at the beginning, I would first just not want to turn customers away. And I would also want to learn what I stand to learn from having these agents running amok while it's still kind of an early adopter phenomenon. Like, how are you going to ban, you know, Alexa has been so successful, right? I mean, Alexa has been like kind of successful. But it's, you know, it's not like Alexa is killing it. And so if they had something where we're like, oh yeah, well, ours is better. Okay. That could be one thing, but they don't. So it just seems like they would learn so much from having these traces, right? Even if they just want to build— I mean, everybody's talking about distilling and, you know, how to— what are the allowable and not allowable ways to get data? Well, like one really easy way to get data would be like, the agents come use your platform and they see how you— you see how you see— you see how they use it, and then you can do, you know, all kinds of things, I would think, downstream of that. It just feels short-sighted.

It's—

I mean, I'm never bet against— you know, we're not in the Bezos era obviously anymore, so I can't say never bet against Bezos. I do think like never bet against Amazon still remains like reasonably good advice, and certainly You know, Jassy had like ran AWS and was obviously super good at it. But this just doesn't add up to me still. I don't, I don't get why you would want to cut off so early. Wait till it gets to 5% and then make a move.

Yeah, I think, I think, I think you're right. There is, there must be some reason for moving this early. It is, it is, I think like, you know, they should have, they should have maybe waited. So I suspect since we're in September 21st, I suspect that we are going to see— we have Dev Day, OpenAI Dev Day coming up. Thibault has already kind of like hinted that they have an agent coming up. So I suspect we have an OpenAI agent which will be able to do your shopping for you, which has a deal with Amazon and which has exclusivity with like first mover advantage with Amazon. And Amazon has already signed such a deal. So I think this might not be a permanent thing. It might be a temporary thing for the next 3 months until this year's shopping season is over, and then they kind of open up. And the reason being that they have a preexisting deal where they said that, okay, OpenAI will be the first shopping agent on Amazon. And that would make sense because then they have revenue shared to OpenAI. They have marketing spend with OpenAI. They have this whole rollout. And then that would make sense in terms of like, okay, this is why they're doing it this early. You're right that it doesn't make sense this early. Absolutely.

Yeah. Okay. Well, that's a good prediction. The other thing that I do wonder about in terms of like, what equilibrium are we going to eventually find ourselves in? Alan Chan tomorrow for an episode of Cognitive Revolution. Revolution, who— and he's been a pioneer of just agent governance theories, all kinds of ideas, you know, IDs for agents. Like, how do we figure out, okay, an agent presents itself, like, who does it represent? You know, is it legitimate or not? What permissions does it have? Uh, how do I decide, as, you know, the person on the other end of some agent request, do I serve that request? Do I not? Whatever. I was looking into Cloudflare and they also have kind of an interesting thing where they're starting to ban. But like, again, it just doesn't feel like the right solution to me because what I then do in response is I have the agent use my browser with all of my credentials.

Yeah.

And you could maybe detect when an agent is using the browser based on their sort of bot-like behavior. But then you're really setting up an arms race where it's like, this sort of feels analogous to putting too much pressure on the chain of thought, right? I don't want to put so much pressure on agents that people are out there devising ways to make them indistinguishable from human. I suspect that if people really work hard on that, they it can probably succeed, or at least often enough, you know, that it will be an issue. And it just seems like it's much better to have a lane for agents where it's like, this is how they can work. We're not gonna try to block them, um, but we'll kind of segregate them maybe so that we don't create this arms race. Because I, I do— I really do think we're headed for a world right now where Probably Meta doesn't do it, but somebody's going to figure out a way to make agents work on Amazon. Somebody's going to figure out a way to make agents look human enough that they don't get blocked by Cloudflare. And it also probably works for the user. I mean, I have this problem all the time where Claude wants to do something in the browser, and this is the same thing happens with OpenClaw, Astra, whatever. They want to do something in the browser. They get stuck on when they open up their own window, they don't have sessions. I've given them like a lot of passwords where they can log in, but not all. And sometimes they're blocked. And, you know, then if there's like a verification flow or whatever, a lot of times they can't do it. And so the fallback just ends up being, okay, just do it on my browser. Forget it. You know, Yeah. I wanted the security. I wanted the separation. But they're making it too difficult. So fine, just use my browser. Then I'm logged in, you have all my credentials, you can just do whatever. And that's not great for me. It's not great, I don't think, for anybody. So I do feel like we need a much better solution for all this than just kind of ban and hope for the best. I don't see that holding, and it feels like it also creates a lot of problematic incentives.

So I think like, you know, it's like you're standing on Earth and you're watching this giant meteor head towards you and you really don't know what to do, right? Like it's the whole don't look up thing because like you can easily imagine Facebook scaling Muse to 300 million, 150 million American accounts or whatever, definitely on an average revenue per user basis on their existing network already makes sense. You can see them, they've already hooked up WhatsApp. There's some WhatsApp hookup already. There's some like hookup into Facebook Marketplace. There's some hookup into Facebook itself. There'll be some hookup into Facebook Messenger. You, you can, you, if they, if they go for it, that is going to narrow down to the one interface, right? And then like other companies will not be able to say no because the Facebook like traffic channel is so big and Facebook will be like, you either work with us or we'll work with someone who is willing to work with us, right? And the amount of like agentic traffic like that Amazon, for example, would have to like support or like Cloudflare would have to support would be like, you know, maybe 100 times what they're doing, just like what happened with GitHub, right? Like maybe 100 times or 1,000 times what they do today. Like they don't have the infra for it, right? Like they don't have the infra for it. They don't have the capability to serve that many like HTML pages. They'd have to serve a lot of it by API. There's like, there's like so many things that they would have to change. They would have to change. Um, and I think, I think that's a, that's a challenge. That's a real— like, everything changes. Like, everything changes, right? Like, it's 1,000 to 1 agents to human traffic. Your primary user is now an agent. Your primary user is not a human anymore. The agent stuff has to work in parallel. It has to be very fast. You have to make it low cost, you know, that you don't have the revenue that you have on a per-user basis that you do for humans. So you're not going to be able to support like images, for example. You won't do images because it doesn't make sense. You know, you— there's so many things that have to change. Like the entire infrastructure of the internet would have to change to support agents. And agents would have to become first-class users of the internet. Like what we spoke to, you know, we spoke to the Mozilla guy, you know, last week. Agents as first-class users is a whole different way of like building the web, uh, in a whole different way. So, so I think, I, and I, and I think there's a lot of like, okay, what do we do with the ads? Like, that's how we make money. If the agents don't look at the ads, what do we do now? Like, what do we do? Okay, these impressions, like, we know we have bot traffic for a long time, and we know we've kind of bluffed the advertisers that a lot of this bot traffic is, is actually real people. But now the cat is out of the bag, right? Like, You know, we knew, like, I think it was known that it was like 50 to 80% bot traffic for a lot of the ads, but, you know, kind of like at least you had like in proportion, everyone had like the same kind of like everyone had like maybe 20 to 30% human and 70% bot. And like advertisers were like, oh, look, you know, that's the price that you have to pay. Like everyone, it's everyone has the same issue. If everyone has the same issue, then okay, the productivity is the same. Like, you know, you're absorbing some of the bot traffic, but it's okay. But now you are at 99 to 1 or 999 to 1. It's no longer— the cat is out of the bag. Things will have to change. Yeah, it's a tough one.

Maybe we can get somebody from Amazon to come explain this thinking to us. In the meantime, we'll be unleashing our own agent. Switch back to the phone for audio purposes. But the latest from Dawn is that we're going to set up an email for Dawn at our domain. And then with that, outreach can begin and we'll see if we can turn AI in the AM into some sort of business. It'll be interesting to follow along and see how much trouble Don gets us into.

It's definitely going to be interesting. No, no, all the surprises will follow, I guess. And on that note, we'll be back on Wednesday, I guess. We have a full week. We have a full dance card on Wednesday, Friday, and next Monday. On Wednesday, we are going to get our first guest who is going to explain GPU forward pricing and financializing the entire GPU cycle, how that's working out. They have an index. It's a company called Orin, which I've been speaking to for a while. It's going to be fascinating because there are people who don't believe these GPUs can be financialized in that way and people who do. So it's a big debate in the industry. So we'll get to see that.

Yeah. And the other guest on Wednesday too will be Super interesting. Lewis Hammond from the Cooperative AI Foundation has written some pioneering work, really, including one big technical report from early 2025 that was like basically what to expect from multi-agent systems. I'm very curious to hear what they have made of these crazy events we've seen this summer, how much of it they anticipated, how much they didn't, how they're updating, and what they expect to see next as multi-agent systems, for better or worse, seem to be a new part of the reality we're living in, ready for it or not.

Indeed. So, and on that note, Nathan, good morning.

Good morning, Prakash. Talk to you on Wednesday.

Cheers.