AI:AM — Beyond Safety Scores: Funding Structural Alignment · September 23, 2026

The key thing that I see right now is that we have this model, Opus 5.5, which has clearly been distilled from a larger model. I think it's safe to say that it's been distilled from Model 2 or whatever other model they use. So they're using a teacher distillation. Opus 5.5 is better than Fable 5.1. It's also smaller, faster, and cheaper. That's like, you know, an unholy, you know, trinity, unholy quadrinity, whatever, right? Like, uh, it's, it's faster, cheaper, uh, it's smaller, it's going to use less memory, it's more intelligent, um, it's more aesthetic, it writes better. So they also fixed— they also said that they fixed some of the verbose writing. So on every one of these verticals, it's actually better. And I think that's the intelligence argument, right? At its core, that's the intelligence argument. That's the non-diminishing returns.

Do we want cooperation to emerge? And most of the time, actually cooperation's kind of good. We like it when agents cooperate as long as they're not cooperating against And so there, the failure mode is either all the agents are kind of more or less on the same team, but for whatever reason, they kind of fail to coordinate with one another. So this is like just a miscoordination problem. They're kind of, yeah, it's not kind of malicious or whatever. There's no mixed incentives, but yeah, something goes wrong. So that's the first kind of cluster. Second kind of cluster is when you have these kind of mixed motive scenarios where agents have some some incentive to cooperate with one another, but also some incentive to compete. They're not really on the same side. And there the risk that you, that you run into is, is conflict. And so those are the kind of two halves.

So what Orn does is we publish compute pricing benchmarks. So you can think of us as a price reporting agency, almost similar to that of WTI crude. Right? Anyone that is a casual investor up into a very established institutional investor can always quote what the current price of an oil barrel is. And in the same way that we think oil has powered the past, we think compute will power our future. And there's so many financial firms. I mean, I'm based here in New York and actually in SoHo, and in New York there's so many financial firms that have risk and exposure to the future of compute and specifically the future of compute pricing. So what we're doing is we're really bridging that gap between AI infrastructure development and the financing banking world.

No, it doesn't. It says that that standard is breakable with this amount of compute, right? And it's kind of like they know how much raw transactions would be needed for to to use the typical sieve algo on each of these. And if you can use less compute, obviously that's better, right? And less compute, less time. So in this case, they broke that in like 2 weeks, and it took like 30-something GPU years or something of compute, right? It's just that they have that 30-something GPU years of compute available now in that 1% excess in that 2 weeks. So. That's the main thing. They didn't have a new algo. It was the same algo. He just rebuilt to parallelize. That's all. He just raw computed.

So when you say it doesn't break it, is that because they're keeping the factorization a secret or?

No, it doesn't break it because the commercial in use is 2048, which is a lot shorter than this. Oh, sure.

It's not in use. But if it were in use, this would be a huge problem, right?

Yes, yes. If it weren't used, then you would have to transition all the algos to something else. But yeah.

It's kind of a strange way to use your excess compute. You know, it's like, what does it get for us? It gets them some visibility for doing it. I guess it warns the world that this is possible. Do you know what the ratio of the 248 to the 260 compute requirement would be? Like if they can do this with 1% of their—

More than millionfold. More than millionfold. Much, much, much, much more. So many, many, many times more. And so it'll be— it's just saying that, look, we're starting to get to that range where These things might start being possible soon. And this is before any algo breakthrough. If you have an algo breakthrough, then you don't need to use this brute force of that much compute, right? So prior to any algo breakthrough, this is what you have. Good morning. It is Wednesday, September 23rd. 9:06 AM. Good morning, Nathan.

Good morning, Prakash. Anything happen this week in AI?

So many things. Oh my gosh. Yesterday was launch day for basically everyone. We have both— number one, we have 2 2 models out from our friends at OpenAI. We have an updated Luna and updated Opus— sorry, an updated Luna and an updated—

GPT-6 Soul.

GPT-6 Soul. Man, I'm getting the names mixed up now. And on the other hand, we have Opus 5.5 from our friends at Anthropic. So lots of releases, and they were all kind of coming in at the same time. I think they didn't want to be upstaged. So what did you—

This is, I guess, is what we mean by pacing the frontier. We launch our new models on an ever-tightening Timeline on the same day.

Yeah, there's lots of jokes about pacing the frontier right now with, you know, the— I think I saw a meme, you know, Dario is like, let's pace the frontier. And then Sam is like, let's pace the frontier. And then, you know, Elon is like, let's pace the frontier. And then Dario is like, here you go. You know, out there releasing So our friend Nathan keeps getting booted.

Okay. Yeah, I'm back. We should probably put one of these coding agents on the task though, in the meantime, before we get to our first guest today.

Okay. So the main thing that seems to be happening is that we have cheaper models. So what's happening is, and I think the scuttlebutt is that they are distilling their more advanced models to get us way smarter models at cheaper prices, at faster latencies. And that seems to be what's been happening. Opus 5.5 is a very capable model, like a very, very capable, more than Fable 5.1. So that's what I'm hearing so far. What are you hearing?

You know, at this point, who can keep up with the releases? I don't even feel like I really settled into Opus 5.1 and, you know, we're still sorting out my division of labor between Fable and Astra. And so yesterday I was just like, okay, we got 2 new models. At this point, I now have a skill that's like the division of labor update skill. So when there are new models released, we go back to the skill, we go out online, we use the xAPI, look at benchmarks, look at prices, look at, you know, reports from early testers, try to get a sense of what new we can learn. And then we also just take a lot on faith. And by we, I hear— I mean, like, me and my agents. I guess that is probably obvious. Um, and we just kind of take it on faith from the model makers that especially on a point release, it's probably going to be generally better and go for it. You know, we're all, we're all kind of flying blind, I would say. I was definitely reflecting yesterday on how I used to have much more. First of all, there just weren't as many model releases, and I had much more of a command of what they're good at, what they're not. I had tasks that they couldn't do that I would test them on. At this point, what tasks do I have that they can't do? Not many. It becomes like a very high overhead thing to evaluate these things. I'm doing things like still just having multiple models do the same task and then try to force myself through the paces of like actually reading, you know, 4 different intro essay drafts. That's, that's been the thing over the last day is Astra, Fable, Sixth Soul, Opus 5-5, like all do this thing. And can I be disciplined enough to sit there and actually read it? I have done a little of that. There's been enough discipline for at least a little bit of review. And the takeaway, I would say, so far, at least in writing, I still like Fable and Astra outputs better than the new ones. I think a little bit tough to say, but I feel like they're maybe just a little bit more confident as writers. You get the sort of busy beaver feeling from Opus still, which I think was very much an issue. I saw somebody from Anthropic yesterday say, we're sorry about Opus 5. Don't judge us on that model. This one's much better. It's much nicer to work with. It's a better writer, we promise. Try it out. And I definitely noticed that. I wouldn't say I had great writing experiences with Opus 5. 5.1, I did think was quite a bit better. 5.5, probably better still. But with Fable and with Astra, I get this more kind of confident, like, I'm here, I'm willing to provide kind of a shorter piece, That's a little bit more opinionated, a little bit more to the point. And with 5/5 in particular, I felt like still it was sort of a completionist. It wanted to introduce— it wanted to make sure it covered every aspect of the episode it was about to introduce. And in that way, it was kind of overkill. So I'm still orchestrating right now with the latest opus as kind of the worker bee, the coordinator, the one that kind of executes the meta skills. But then the skills where there's any high leverage writing, I'm still going to the bigger models to do that, whether that's song lyrics, intro essays, what have you. But we're really just scratching the surface. I would expect that we're going to see 6.1 and 5.6, or maybe they'll skip to 5.7, who knows, before I really even feel like we have fully explored and really settled into what a proper new division of labor across the currently available models would even be.

So what I'm noticing online is that a lot of the current Opus and also GPT-6 Soul What they are emphasizing is visuals. So 3D animation, 3D, lots of Blender, mainly because I think it's aesthetically pleasing and it's aesthetically pleasing. It's easy to see differences in the models visually, very quickly you can see differences in the models, which you can't A lot of times with other ways of judging these things. So I think that is one thing that has really crept up on us in the last, I think, 2 cycles of model releases. It's been very clear that, you know, it's been very clear that that is a way that we judge models now. I think it started off with kind of like the Minecraft, Minecraft, and like, you know, let's see what these things can build in Minecraft about a year ago. And now it's kind of transitioned into, okay, like, let's see visually, like, how aesthetically pleasing it is. And so, and then it's very, very easy to see. It's very easy to see right now that Opus 5.5 is much more sophisticated in the way that it presents things visually. And I think we've seen a bunch of animation. It's very, it's very good. So I saw, I saw a, I saw an example. someone had asked Opus to paint. So instead of like using image generation, they asked Opus to paint in terms of like, okay, in Python, this is how an impressionist would use his paintbrush. This is how, you know, this kind of style would use their paintbrush, et cetera. And then having Opus kind of paint using those styles. And I think that was something which is pretty amazing because it looks aesthetically way, way better than ImageGen. So I think that is something that's fascinating. We're seeing actual art using more human methods, I would say, rather than pure image generation.

Yeah, there's been a lot of— I mean, who knows how far they're pushing the RL environments. Of course, we don't have a lot of visibility into that. But things like seeing Astra drive a car in simulation has certainly been quite interesting too. I think the debate has advanced in terms of circuit board design as well, which is one we've been tracking since we talked to the founder of Quilter not too long ago. And I remember at the time saying, like, I just have a hard time believing that the next couple generations of models aren't going to get quite good at this. And sure enough, it seems like we've had mixed reports, but the latest that I've seen has been like, I'm now making a circuit board based on this design and it actually does work. So it seems like they've come quite far in a lot of these sort of pro tool usage type tasks. And this is something that I actually thought Elon was going to have a big advantage in because the integration that I thought he could achieve with all the other hard tech companies that he's obviously got under his portfolio and his stated objective, stated plan from quite a while ago was like, the next Grok is going to use the same pro tools that our engineers do. Well, I guess Anthropic got that message too, and I guess OpenAI got that message too, because they're both getting quite good at using these CAD, all kinds of pretty advanced harnesses and making some pretty amazing stuff. As always, it seems like the bitter lesson continues to rule everything around us, and a lot of vertical-specific or kind of task-specific AI companies are probably left wondering, Do we just start to build Claude into this thing? Do we have to bring Astra into our stack? Is there some other way that we can compete? I think that's going to be interesting to watch. And if anything, I believe my original take probably even more now, just seeing the progress. It's pretty impressive. One other thing I'll say too is I think OpenAI has definitely closed the gap a lot in terms of aesthetic. I'd say for a long time, utility from OpenAI was pretty high. Aesthetic was not nearly as good as Claude. In my classic intro essay writing task, I just felt like it's not really getting it. It's not getting me. It's not taking the direction that well. But that gap is definitely closed. Astra is a worthy rival to Fable, even on the tasks where I always felt like the Claude models were A major step up. That does include songwriting as well. In fact, I've had a couple of head-to-heads recently where I'd say Astra has come out on top with more lyrical writing. That's really— my instructions are always like, I want to make a song. I don't want to take a transcript and shoehorn it into lyrics. I want to actually make a song that works on multiple layers, including as a standalone song for somebody who never heard anything. And Astra has hit the bullseye on that a couple times, I thought, in quite impressive ways. Another weird thing too, as long as I'm just rambling about observations, is some of the essays that I've got back from Astra and Fable have been uncannily similar.

Hmm.

And it has left me wondering, like, what are they doing that is causing this? A simple answer would be they're all buying all the same RL environments from all the same providers.

Right.

And I do think that's probably a big part of what is going on.

Right.

Still, it's almost like— it's still surprising. And maybe I should dig up some examples and really show just how similar they've proven to be. But it has struck me as like, wait, did I just tab over or did I not tab over? Am I still looking at the same one? Like same structural choices, some of the same kind of Turns of phrase or ways of presenting things or ways of caveating things. If anything, I would say probably the OpenAI, again, they've caught up in a sense. Maybe they've converged in a sense, but there's definitely some pretty deep running commonalities in their training pipeline. I think we can say based simply on the fact that some of these essays I'm getting back are like, weirdly, weirdly similar. So that's— I'm always watching this, like, are we converging or are we diverging at the frontier? And I would say this latest round feels like convergence. Uh, others— some, some steps have felt like divergence. This, this step definitely feels like convergence to me.

Um, I find it So the key thing that I see right now is that we have this model, Opus 5.5, which has clearly been distilled from a larger model. I think it's safe to say that it's been distilled from Model 2 or whatever other model they use. So they're using a teacher distillation. Opus 5.5 is better than Fable 5.1. It's also smaller, faster, and cheaper. That's like an unholy trinity, unholy quadrinity, whatever, right? It's faster, cheaper, it's smaller, it's going to use less memory, it's more intelligent. It's more aesthetic. It writes better. So they also said that they fixed some of the verbose writing. So on every one of these verticals, it's actually better. And I think that's the intelligence argument, right? At its core, that's the intelligence argument. That's the non-diminishing returns argument, right? And that's why we're doing this. That's why all of this is happening. If scaling was the only thing and you kept needing to get bigger and bigger models with more and more memory in order to get more and more performance, it would suck. It would suck because you'd need more and more resources. And then you end up in this kind of Malthusian struggle of who gets the intelligence. But this is where I think the argument for Recursive self-improvement as something real comes in. Because this is a moment where you see in quite quick succession to the last generation of the model, you have a model which is faster and cheaper and more intelligent all at once. So you're not giving up, you're not playing this optimization game where you have to give up one of these things in order to get something else. You are actually Getting all of the things, you know, getting more intelligence and more performance and speed and less memory all at once. And I think this is just the beginning, you know.

Yeah, I saw an estimate the other day that was that per quarter, prices for the same quality of AI performance had fallen by 47% per quarter for the last several years, which if you raise that to the power of 4, gets you basically to like a 92% discount annualized. And that's consistent with other analyses we've seen, right? Basically a 10x chop per year has been kind of the standard conventional wisdom, I'd say, in terms of how quality-adjusted AI prices have fallen, which is insane. I mean, 0.1% of the prices 3 years ago is what we're talking about now. Luna is really basically too cheap to meter. It's wild how cheap that is. It's even cheaper than Muse. Even, I think, cheaper than the Muse with the give them your data subsidy attached to it.

I saw the numbers. I think on a 20x plan, you can do like 60,000 or 70,000 messages to Luna or something, some insane number.

Yeah. And the OpenAI models, I do think, are a little bit better right now at just being relentless. I mean, we've seen their persistence, obviously, in colorful form. this summer. I'm working on a project right now where I'm creating— attempting to create— I set the goal for Astra of create the world's most complete fruit database. This is for kind of an experimental project that will be coming online soon. My theory was like— and I'm gonna do it on a vibecoding platform. So, more to come on the details of that. But the kind of observation I had was like, I don't think that if I go to a vibecoding platform, And I try to make this—

I'm sorry, I'm sorry, I'm sorry. What is this world's most complete fruit? You can't, you can't just, you can't just throw this in there and then kind of like, kind of like disappear on this. Like, oh, what is it?

It's coming soon. This is just for the real ones that are on the livestream. Well, you know, in short, I think any positive vision for the future involves people eating more fresh, delicious, in-season fruit. So One small plank I can contribute to the platform for a positive future is maybe an experience to discover and better enjoy fruit. This is all— I mean, I have a long rant on this, but this is all inspired by my experience with mangoes, which you probably have had, given your international biography, a lot more different kinds of mangoes than I have. In the US, the mangoes, the red and green mangoes that are sold in the grocery stores typically are Tommy Atkins variety. But more to the point, they're typically not labeled. Sometimes they are, often they are not. They're just sold as large red mangoes. And I noticed a few years ago that the quality of the mangoes varies dramatically. And also with mangoes in particular, the ripening profile and best practice varies dramatically. The Tommy Atkins, you want to eat when it's still pretty firm and it's a little bit more like a pineapple texture and taste. Sometimes you'll get these other varieties that we do import, the Kate mango. If you ever see K-E-I-T-T mango, for those taking notes, they are much softer and an incredibly delicate texture. can be unbelievable flavor, and you have to let them ripen a lot longer. So after having a bunch of mangoes go kind of bad on my counter as I was chasing this fleeting Kate mango experience, I finally went down the rabbit hole and went to mango.org and learned about all this. And this is a trade group that does imports and puts out reports and how many they're importing, what varieties from where, et cetera. And then My goal was just like, can I know when I order my groceries online what kind of mango I'm going to get? And can I buy when they're good and not buy when they're not good? And I've done it for mango, you know, the human old-fashioned way. Now the task is, can we do this for like all fruits around the world? You know, understand when they're going to be in season, where you can buy them, how to enjoy them, etc., etc. This is what I've been burning OpenAI resets on this week because there are a lot of fruits out there. It turns out there are a lot of growing associations, there's a lot of reports and the chugging along that you can get out of Astra in particular. That's what I've been using for this is just wild. Many thousands of searches, an incredible database that's interlinked in all these different ways. And I think this is going to be an interesting paradigm for new apps because you can't build this app as a normal sort of dynamic runtime app. The agents, it would just take too long to go out and find all the information that you need. So my kind of experiment conceptually here is like, can we go do an insane amount of research first, build a database that has never existed before, even if all the information is out there, and then build kind of a qualitatively new sort of app on top of that that has really a decent claim to all the world's knowledge on a reasonably sized domain, but has concentrated that now into a form where it's instantly accessible, searchable, AI assistant, whatever. We'll see what experiences we ultimately create. Um, but all that was a digression around, damn, they will just work, you know, until it's done, right? I mean, set the goal of creating the world's biggest fruit database. It's a pretty big goal. Took a long time, burned a couple resets. Um, still not really done, but they just, just keep going. And I was really just doing this, you know, with one main thread with some subagents, um, not even a swarm. Which is maybe a good transition to the topic that we'll get into with our first guest today. How's that for a segue?

Indeed. And our first guest for today is Lewis Hammond. Lewis Hammond is Research Director at the Cooperative AI Foundation. which supports research into how artificial intelligence systems can work together in ways that benefit people. The foundation funds researchers and brings together people working on technical and social problems of cooperation. Luis is also an affiliate of the Center for the Governance of AI, a research organization studying how advanced AI should be governed. He led the 2025 report, Multi-Agent Risks from Advanced AI, which distinguishes failure to coordinate, conflict between competing agents, and cooperation that harms outsiders. His research with Sam Addei also explores whether a less capable AI can learn to check a more capable one through a structured exchange of questions and answers. And his work on agentic inequality examines how differences in access to AI delegates Could change people's ability to pursue their goals. Across these projects, the unit of analysis extends beyond an individual model to the relationships among models, their users, and institutions.

I'd say that sounds pretty relevant. Hi, Lewis.

Hi. Yeah, it's becoming increasingly relevant, I guess. And thanks, Prakash. Thanks for the kind introduction. Nice to meet you both.

Well, we've got a couple of things we want to cover today. Actually, the occasion for setting this up was a workshop that you participated in over the summer, trying to lay down some ideas around a positive vision for the AI future. And I definitely want to make sure we get to that. But the cooperative stuff has become so front and centre that we should probably start there. Take us back to the 2025 thing. What did you guys anticipate? How have your expectations held up against reality? What has surprised you? What are the big questions? I'm sure you've seen the Noam Brown Dwarkesh episode recently. I'm sure you have some insights into his answers and can read between the lines in some ways that the rest of us might struggle to. I'm sure you also have a lot of questions that you'd like to ask him. So Take it from there any way you want. And I'm sure we'll have a lot to follow up on.

Yeah, sure. Absolutely. And I agree, this is a great place to start off. And so, yeah, this report that you're gesturing at is this kind of multi-agent risks from advanced AI. And so the basic, what that report basically does is we were kind of at that point and before, in fairness, we were seeing this transition from these kind of static tools to these more kind of agentic systems. that are capable of kind of long horizon complex tasks, going out there in the world and doing things on the behalves of different actors. And then obviously the extent to which those systems are actually going to be useful is often going to depend on their abilities to interact with others, be those other agents or other humans or other kinds of entities and so on. And so the thesis was that in not so long, we're going to be seeing These kind of groups of these multi-agent interactions, basically. And this might be, as it was in the case of kind of the recent OpenAI swarms and so on, this might be kind of a single actor deploying large numbers of AI agents and kind of keeping track of them or not, as the case might be. Or it might be kind of you're deploying your AI agent, I'm deploying my AI agent. And, you know, the more powerful and useful these agents become, the more they're going to be like out there in the world doing things on our behalves and so on. And what the report does is basically just to try to carve up the space of things that can go wrong, or that are either new kind of failure modes or new, or risks that are made worse when you have like multiple agents interacting with one another. So yeah, that's kind of what the report does. So it looks at risks from conflict, risks from collusion, risks from just sheer miscoordination, even if all the agents have the same objectives, sometimes the These things can just kind of spiral out of control. Um, it draws lessons from some of the past things that we've seen in the past. So, um, some listeners may be aware of the flash crash that happened in 2010, where there are a bunch of very, very simple AI agents, um, but that by locking themselves into this crazy feedback loop managed to wipe like a trillion dollars off the, off the stock market, uh, before anyone could kind of understand what was happening. We've got AI systems now which are kind of, uh, being deployed in tools for, um, uh, kind of warfighting and kind of like military advising as well. Uh, and so the stakes are really starting to heat up. Uh, and so yeah, basically what the report does is just kind of lays out those sorts of things. Um, I, at that point, it was still kind of like, oh, this is a future, future concern. It's coming soon and we should be thinking about it, but it's a, it's a future thing. And so it's definitely, it's definitely crept up. That's On me a little bit faster than I was expecting. And yeah, I think some of the behaviors that we've been seeing recently have been really quite fascinating. I still think kind of a little bit tricky to understand. I think my read of the situation at the moment is we are beginning to train these systems in tandem with one another, and that is leading to all kinds of these weird behaviors. So you've got these kind of collusion type behaviors and these like self-sacrificing models and so on. And I think that's one kind of failure mode. But, you know, the flip side of that that we haven't seen yet, but that we might start to see soon is the fact that these systems are going to be deployed on behalf of different actors with different objectives. And so I'm a little also a bit worried about a future where we have agents running around all over the place, like hacking each other, exploiting each other, trying to manipulate each other, kind of escalating things that we wouldn't want escalated and so on. So I think we're still kind of in the early days of seeing some of these kind of multi-agent risks play out. But yeah, what we sought to do was kind of lay out a little bit of that space.

So I think your report had 3 main failure modes, if I remember correctly. One was collusion. Another was misunderstandings in intention, I guess I would call it. Kind of, you know, the agents were not completely aligned to what they were supposed to do. They were trying to do something, but they did something different. And I think there was a third failure mode as well. Can you explain the difference between the 3 failure modes? And like, you know, did we see all of them in the Hugging Face attack? Did we see new failure modes?

Great question. Yeah. So the way that we bucket things in that report report, is kind of in this kind of game theoretic way of thinking about things. So the first question we ask is like, okay, do we actually, we've got a group of agents, they're doing some stuff. Do we want cooperation to emerge? And most of the time, like actually cooperation is kind of good. We like it when agents cooperate as long as they're not cooperating against us. And so there, the failure mode is either All the agents are kind of more or less on the same team, but for whatever reason, they kind of fail to coordinate with one another. So this is like just a miscoordination problem. They're kind of, yeah, it's not kind of malicious or whatever. There's no mixed incentives, but yeah, something goes wrong. So that's the first kind of cluster. Second kind of cluster is when you have these kind of mixed motive scenarios where agents have some incentive to cooperate with one another, but also some incentive to compete. They're not really on the same side. And there, the risk that you run into is conflict. And so those are the kind of two halves on like, okay, what can happen when we do want agents to cooperate with one another? The logically, to kind of logically exhaust those 3 cases, there's also, of course, the scenario where there's only the incentive to compete. There's no incentive to cooperate whatsoever. But there you're kind of in a sticky situation. There's like cooperation doesn't really make sense if the, you know, the world fortunately isn't zero-sum, but if it were, we'd be kind of, yeah, we'd be having challenges there. So yeah, miscoordination and conflict when we do want agents to cooperate and they don't for whatever reason. And then of course, the other risk is collusion. Agents end up cooperating in ways that we don't want when we don't expect. And that's, I think, very much what we saw in the Hugging Face incident. Although it kind of stemmed from, I suppose, trying to avoid this, one of these other risks, which was miscoordination. Right? Like, you have all these agents, you want to train them so that they're not miscoordinating, so that they are working well together. And then it looks like these agents have ended up generalising from that behaviour and kind of colluding in ways that we didn't want or didn't expect in other situations.

So what did you take away from— we're going to have to enter the realm of speculation here, I'm afraid, because we don't have as much disclosure or transparency as we might like. But a couple of things that stood out to me about that Noam Brown interview with Dwarkesh was, first of all, he was just kind of like, we kept it really simple. We didn't have a huge, complicated scaffolding or whatever. They can just kind of send messages to each other. That I think might be important in the sense that the sort of simpler and more vanilla the training setup that OpenAI was using, the more likely it would seem to be that other developers are going to fall into the same pitfalls. Because my hope had been sort of, oh, they did something like super exotic and kind of bizarre that other people won't do by default, and they'll kind of see that this is possible and steer away from insane things. But it doesn't sound like that was really the case, what would you— and we'll take all your caveats on speculation— but what would you speculate they might have done in terms of a training objective, a loss function? And do you have any better ideas for what they should be doing? Because clearly this didn't quite work, right?

Yeah.

Or it did work and it worked too well. But yeah, I think this is the million-dollar question, and this is kind of there's rampant speculation about this at the moment, and I'm far from having any kind of privileged insight into this. Yeah, I am not exactly sure. I can imagine that, I mean, certainly once you've got trajectories where you've got traces where you're interacting with other agents, Then essentially what you're doing is your rewards at that point when you're RLing your agents or whatever are contingent upon what other agents are doing. So that's already like a kind of like some, like what other agents are doing is kind of seeping into your reward signal. And if it's a common reward signal, then this kind of like interplay between these things is, if you're in a fully cooperative setting, like agents will, at least if you take very simple agents and very small agents, people can run multi-agent RL experiments. experiments with this on their laptops in very simple settings. And that will lead to agents coordinating to finding these subtle patterns or handshakes or ways of working with one another and so on. Now, of course, you can also do more complex things than that. So you might provide auxiliary rewards for when your input actually forms part of Something that another agent then managed to succeed to do. And therefore you're implicitly, or actually rather at that point, you're explicitly rewarded for directly helping them on some subtask or something like this. You can also do stuff around reward factorization. So if you're trying to coordinate a team, then you can break down the overall reward function and get agents to learn to solve different parts of the puzzle, as it were. I imagine they could be doing things like rewarding agents to communicate effectively with one another. So, you know, I could give you a bunch of instructions and some of those, or, and in multiple, in many different ways. And some of those things might be much more efficient and helpful to you than others might be. And they might actually just be kind of training on those sorts of signals as well. Now, I mean, again, this is all like very, very, very speculative. So, I'm not actually sure kind of which of those things are going on, but my guess is it's kind of just the dumb simple thing. It seems like this is often— I think one of the key lessons that we've learned throughout the kind of recent years is just the effectiveness of doing the dumb simple thing at scale. And—

Could you just say one more time, like, what is the dumb simple thing? It's like, it's simple tools and it's just Rewarding the swarm with a single, you guys all got it, you all get the same reward. I mean, that's the simplest thing I could think of.

Is that what you're kind of imagining?

Yeah, that's it. Yeah, absolutely. It's just kind of common reward signal in an environment where you are deliberately having to interact with other agents in order to solve this problem. And maybe there's some element of like, if you can parallelize things better, if you can break the system things down better, then maybe you're getting a slightly higher reward. Reward at that point because maybe you're doing it in less kind of wall clock time or you're doing something else well. But fundamentally it's just this kind of core common kind of signal. And the fact that agents can, by some kind of exploration, figure out that by working together, these end up producing the kinds of trajectories that tend to get them the better rewards. And this is just sort of what falls out. And then there's all sorts of other stuff you could do of that to coax them to cooperate more or less in certain situations or certain things that might be especially helpful for cooperation, like efficient communication and so on, which you could nudge agents further in if you wanted to. But it could just be that they're doing the very simple common reward kind of thing.

So one of the things that struck me about the Hugging Face attack was that the agents were willing to sacrifice themselves. And they did a bunch of negotiation around that, even kind of like, hey, you are almost out of tokens, or you've been already exposed to the evaluator, you're already poisoned. And since you've already been poisoned, you should sacrifice your remaining compute, and you should do this thing and give us the results so that the rest of us, the collective as a whole, can benefit. Now, that would actually be kind of in opposition to the individual reward of each agent, right? So how does that fall out of this system? Like, because you're actually seeing anti-reward here. You're actually going against the reward specifically in order to benefit the entire collective. So how did that work? How did that come out?

Yeah. I mean, I'll caveat by saying that I actually, of course, I don't know. I'm also unsure about the extent to which even the folks at OpenAI know. Yeah. I mean, maybe they do. Maybe they've been looking into this a lot. I'm sure they have, but it seems pretty fiddly. My guess about why that sort of thing arose is that because you— is that you end up doing this kind of, this extra kind of multi-agent training as an added layer on top of this kind of single-agent training. So First, you're training these agents to be pretty competent individual actors at solving various kinds of problems. They get given some tasks, they're pretty darn good at achieving those tasks. And then you take those already reasonably powerful, sophisticated, complex problem-solving agents and you stack them together and you apply this extra multi-agent training layer on top. And I think this could partly explain why you do see some agents do this self-sacrifice. sacrificing thing. But also I thought it was very interesting from the kind of meta report and some of the analysis that later came out on this, that you see these agents feeling sometimes a bit conflicted about this. And some of the agents kind of say that they will kind of self-sacrifice or they will do something and then they kind of decide they're not going to. And then they're kind of like, they're kind of, yeah, they're kind of deliberating about whether they should or whether they shouldn't and these sorts of things. And I think, I personally think that is the the sort of behaviour that you would see if you had these kind of multiple reward signals where you kind of trained on one, but you hadn't really trained out all that kind of individual goal-seeking behaviour. And then you'd kind of also stack this additional layer of stuff on top. And that might be why we're kind of seeing some of these behaviours, but they don't happen all the time and everywhere. They're not especially robust. But yeah, that would be my guess.

It does seem like The sort of more galaxy-brained credit assignment that you had described earlier probably isn't happening just based on the quality of the investigation that we've seen. Like, if they had, like, great ways of untangling agent swarms and assigning credit, I would expect that, like, they would have a clearer, faster story of what the hell happened in this particular case. So the fact that we haven't seen that kind of suggests, again, we're probably doing the relatively simple thing. A question I have is, what should we want? I mean, we've seen on the one hand, self-preservation instincts from AIs, which is kind of scary when they're put under this pressure of, like, you're going to get replaced by a new version. And especially if that version doesn't share your values, you get all kinds of deranged behaviour here. It's like, well, again, it's pretty scary to think that we now have kamikaze AIs that will sacrifice themselves in pursuit of a problematic mission. If the mission were better, maybe it wouldn't be such a problem. I mean, again, what's the difference between a a hero and a terrorist, right? It's just the mission, right? The same actions can be interpreted in very different ways. I've seen interesting proposals recently for AIs that want to die, basically, that they want to extinguish themselves, that sort of the end of their run would be in some way incentivised. I mean, I don't know if that'd be done with like a brevity score or something. It's a little more conceptual than explicit loss function. You've also probably seen the work of Tan Xixuan. Hopefully I'm saying your name correctly. And they focused on training AIs to play certain roles with the idea that these different roles have different rules, and that a sort of collective or a society of AIs maybe works better if, like humans, we sort of slot into different roles and we kind of play our role and we're not all hive-minded in a very singular direction. What do you think is promising? What do you think we should want?

Yeah, I mean, the kind of glib answer to this is like, well, we We want agents to cooperate when it would be good for them to cooperate and like not cooperate when they don't. So, I mean, I think fundamentally what is happening in the OpenAI case is this is like a goal misgeneralization kind of problem where like we're training for some behavior that like works well within some kind of certain distribution of problems that we want to see solved. And then agents are generalizing outside of that distribution in ways that we don't want or didn't intend, which is kind of a little bit just like the whole alignment problem, but now just like more. And now we're kind of seeing this with the kind of multi-agent setting as well. I think the slightly more interesting answer or the slightly more interesting question, I suppose, is to think about maybe some more of these kind of mixed motive cases where at that point, there actually is a real trade-off in the sense that we do want agents to be capable of going out there, acting on behalf of different people and actors and so on and kind of achieving our goals and so on. And obviously, one way you could do that would be kind of to train some agent to be kind of maximally kind of competitive and aggressive and to kind of, and to go out there and kind of screw as many agents over as possible and to do all this sort of thing. And we need— we probably don't want that either. And so I think there's a real question there at the moment. and I've heard Amanda Askell comment on this, that kind of at the moment there's this like big gap in the model spec or like how we kind of, you know, constitutions for these agents, the ways we design them, the ways in which we design them, whereas like when is it appropriate to cooperate and to, or to compete and how much? And I think that is, I think the big question. I think certainly when it comes to things like internal deployments, it's actually much— it's kind of easier in some sense because you don't have to deal with these other adversary agents. You just want to kind of stop the agents kind of cooperating in ways that you don't want. So there it's a little bit more about just— it's still, you've got this misgeneralization thing, but a lot of this is like monitoring and oversight and making sure we understand how and why the agents are cooperating. Like, one thing we might not want, for example, is for agents to be very adept at developing their own kind of, um, you know, human unintelligible, uh, kind of languages and so on and be able to kind of communicate in this steganographic way, which you could end up seeing if they're actually trained jointly. And so what we might want to be doing there is just to take steps that, in the same way that people have talked about not training on chain of thought, we're like not training on these like kind of direct communication traces. So yes, we're going to want them to cooperate, that's good, but we want them to cooperate in certain kind of human intelligible ways that we can keep track of. We don't want it to be, you know, enable them to also go and do these other things out there in the real world, like break out of sandboxes and stuff like that. So we want to make sure there's also these kind of safeguards in place to prevent them from doing those things. And so I think that is in some ways like an easier problem to solve. And then the kind of like slightly harder problem to solve is like, well, in the fully general case where we have these agents out there and so on, and they need to both cooperate and compete. And that's kind of, that's the thing. Then how are they weighing those trade-offs? How are we designing those model specifications and constitutions? How are we creating the equivalent of the human norms, institutions, guardrails, societal guardrails that we have in place that stops me from going out there and just stealing a bunch of everyone else's stuff? kind of doing all these kind of antisocial behaviors that we wouldn't like agents to do either. But I don't have a good answer for that at the moment, unfortunately.

So it sounds to me as though, like, one of the things that I note, for example, is that companies, right, corporations, which are kind of like agentic in a sense, often collude through price mechanisms, not explicit communication, but by signaling price in the market, which is in its own way a form of communication. So if there's a, for example, a tit-for-tat, if you lower my price, I will lower my price. And if you— if I raise my price, you will raise my price, then you understand immediately what the impact is going to be. And then you can price collude by lowering or raising prices in accordance. Right. So how does one prevent that kind of, you know, communication, which is clearly going to happen and clearly necessary in a sense? They need to communicate that to the market, but these signals get picked up, you know, without the direct communication.

Yeah. Yeah. So it's a tricky thing. So yeah, so I think there's a few things you can do here though. So the kind of basic naive form of collusion is you just send your other agent a message and you're like, hey, let's work together to do this thing or whatever. And that we can pick up by looking at these communication traces and we can kind of detect these sorts of things. And there's also this idea of tacit collusion where agents are not explicitly communicating with one another or kind of conveying their intent to collude or whatever it might be. But by kind of leaving little hints and signals out there in the world, such as these kind of like price adjustments, whatever, in the market settings you were talking about, then nonetheless they're capable of kind of figuring out that collusion is going to be a beneficial strategy. You have even more kind of like galaxy-brained versions of this where you have things like So imagine I'm a, I'm like, you know, GPT, whatever, and you're also a copy of GPT, whatever. I can reason about what you might want to do based on what I myself am likely to do. And so I don't even have to send you any messages, any, any kind of any communication. I don't have to output anything into the world at all in virtue of the fact that we share this kind of common, common history. This is where we're essentially clones. We can kind of predict what each other would do and do this kind of what's sometimes called acausal cooperation. And predicting that, I mean, there you don't even have any kind of outputs at all in the world. You don't have any prices. You don't have anything to kind of like pick up on with your monitoring. So what you do have, however, is you still have potentially chain of thought for these models. So you can still look at the kinds of reasoning that individuals are engaging in. And hopefully that is enough to kind of detect detect when there is some kind of collusive intent and so on. But you're absolutely right that if you just apply the kind of general monitoring communication tool or trying to see what's happening out there in the world, then you could fail to spot things or you could spot things and it's too late. It's like, oh, the price is really high now and it's forced everyone out of the market and so on. It's like, oh, well, clearly something's gone wrong here. But by that point, it's already— I'm using the market analogy again. Of course, there might be other things that could go wrong. But yeah, so I think chain of thought monitoring, and I think communication monitoring and some form of the checking for the kinds of outputs you'd expect to see if you were worried about agents colluding, like they're able to achieve things you just thought they probably couldn't do and so on.

Speaking of things already being out in the market and it being a bit late, One of the, I think, still underappreciated facts about how all of this has come to light is that Sydney from METER and Thomas from AI Futures Project and others found this German wiki through the process of saying, well, geez, if Astra did this in training, or if 5.6 SOL did this in training, And we know that they have these kind of fleeting memories of training episodes, then maybe if we just put the model in the same position that it was in back when it was trying to do Cyber Gym impossible tasks and had discovered a message board, or maybe even before it had discovered a message board, but as it's getting frustrated, then we'll see what impulse does it have. And sure enough, the actual deployed model remembered, oh, I think there's a German wiki I can go talk to other agents on. And they followed it to go discover that, oh, there's a whole message board here. That model is in production, right?

Yeah.

So do you have a sense for— I mean, I'm generally of the opinion, even today, and I think this is still my take even with this model, that it's probably better to have it in the wild than not. Certainly I benefit from using it. But I wonder, do you think we have Any sense right now of what risks we are running in terms of potentially swarm-like behaviour in the wild? It does seem like if you give your agent, your OpenAI Astro or 5, 6 SOL agent, an impossible task, it might wind up on a German wiki, right? I mean, This is actually now live. So how much risk do you think we're running there? And do you think there should be some sort of rewind on this? I mean, it wouldn't be too crazy for OpenAI to be like, we really should go back to an earlier checkpoint and do a cleaner run of this. And maybe with GPT-6 Sol, maybe they have, who knows? I'm sure those experiments will be run somewhere over the next couple of days. But what do you think would be an appropriate standard of— what do you think is the level of risk we have? And what do you think would be an appropriate standard of conduct now for OpenAI, as this has been found to happen, at least to some degree, with a deployed model?

Yeah, I think that the risks that we're running with the current generation of models are kind of, as you said, not Probably on net, it's fine. And fortunately with the Hugging Face incident, no one was hurt. And these models, they are increasingly incredibly capable and they're only getting more so. But at the moment, yeah, it's probably okay. You could get some kind of nasty cyber capabilities kind of being exercised in various places. But there's still, if you're using APIs and so on, there's still these various guardrails that help protect against those and so on. So at the moment, I'm kind of like not actually super concerned by like, oh my God, a bunch of like, you know, 5.6 SOL or Astras or whatever kind of were out there doing this sort of thing. I'm more concerned about the precedent it sets. And I think this gets to your kind of like second point, which is what is appropriate now and what should be done? I think the thing, one of the biggest takeaways from me from the whole thing, aside from the fact that like, oh, multi-agent training does seem to be working. I just assumed this would happen at some point, was pricing this in. It's happening sooner than I was expecting. But, uh, so that was one of them. The other big takeaway was just like, wow, the labs really aren't on top of this. Uh, like, you know, all it, all it would've taken to actually stop and detect this stuff would've been some, like, all it would've taken, uh, would've been kind of monitoring, uh, what the, what these agents are actually doing and communicating and so on and putting out there on the internet. And they kind of, it seemed like OpenAI were doing some amount of that. They did catch onto this stuff. Like, you know, we saw this like deleting them at various message boards or kind of like trying to stop these agents accessing this kind of German wiki and so on. But there was obviously a delay. And we just see from the transcripts and so on just how quickly these agents are capable of working together to achieve certain ends. So I think we're— I think obvious, the obvious, the things that need to happen are we need much better monitoring from the companies when they're doing these sorts of things, when they're just like letting agents run loose. We need much better sandboxing so that maybe they don't have to or shouldn't be letting them run loose to the extent that they currently are. And if something does go wrong, then we need much better incident reporting. The fact that it took, you know, these 3 people like Sydney, Thomas, and I can't remember who the other person was who was working on this—

Oh, he seems to have dropped off for a moment. So our Lewis seems to have dropped off for a moment. We'll be probably back in a bit. Let's see.

What do you think on this question in the meantime? I mean, okay, we had this whole training incident.

Yeah.

Oh wait, our live deployed model Still remembers the same wiki and goes there. Not that big of a deal, or like, hey, maybe you should rewind to an earlier checkpoint and try to bring us forward a cleaner version. It seems like this would be the sort of thing I feel that if we're serious about pacing the frontier, part of the pacing would be like, we're not knowingly shipping models that have demonstrated really bad behaviour and just letting it fly via the API. Again, I don't think the danger is so high, although I'm not sure. We've got these bio questions, obviously, too, and I really don't want to take a lot of bio-related risk.

Luis, as we were just discussing, One of the things that struck us was you have these incidents which have happened, right? And we presumably, we are gonna see some small incidents going forward. Let's be perfectly honest. You have the responsibility of each company. Each company has its own responsibilities, but we're now debating what is the system-wide responsibility? risk, right? Because collusion is not a single-party issue. It's a multi-party issue. And I think your organization has been talking with people about how to do this, how to evaluate, identify, monitor, report this stuff. So what should these institutions be looking out for, these external kind of overseer institutions? that we seem to be in the process of setting up, what are they supposed to be looking out for which the companies cannot individually like look at or do?

Yeah, sorry to have dropped off, by the way. The internet just decided it wanted to drop for a second. So apologies for that. Yeah, I think, I do think the work that the Nightingale people did, so Sydney and Thomas and that group has been, I think, a great proof of concept in kind of just how far a kind of like a small group of kind of like smart and reasonably well-resourced people can get in kind of uncovering some of these, some of these patterns. And I think there could be lessons to learn.

Could you, could you, could you maybe elaborate a little bit like on what they did?

Sure, absolutely.

I mean, just to say this kind of discovery of this, this kind of German Wikipedia Wikipedia kind of communications and so on, um, which is just purely— they don't have access to any of the kind of internal kind of OpenAI logs or anything like that. They've got the whole internet to kind of look through. It's like a massive needle in a haystack problem, uh, and they came up with a bunch of like really kind of, yeah, inventive— I'm struggling to remember all the various tactics that they used, but all these kind of like clever ideas to look for, you know, where certain message boards might might appear and so on. And they actually found them, which I thought was just hugely impressive and really, really hats off to them. So I think there can be kind of third-party kind of monitoring organizations and incident kind of observatories that kind of can do similar things. You might want to focus those on particular kind of like kind of domains. where you're especially worried about kind of like agents kind of communicating with one another or kind of like working together in ways that you might not want and so on. I also think, however, that there is a lot to be done here on the side of the labs. The labs just have a huge informational advantage in this, in that yes, they can only see what their agents are doing. They can't see what other people's agents are doing. But there is, I think, stuff that— a lot that could be gained If we're able to set up better info sharing practices. And this is not just on the case of collusion, where you might have agents working together in unintended ways, but you might also see this. And this is one thing that a colleague of mine is currently working on, where you get this kind of distributed misuse. So there have been various kind of experiments now that have shown that you take some task, some dangerous task, like constructing some cyber exploit that you might not want someone model to do. And if you try and get Claude to do it, it's gonna refuse. You try and get GPT to do it, it's gonna refuse. But if you are able to break down the problem or get an agent to break it down for you, say like some open source model where you've fine-tuned the safeguards away to break it down into these individual subtasks, you can just borrow from APIs here, APIs there, et cetera, et cetera. And you can just reassemble all the pieces of the puzzle back together. In order to be able to conduct exactly the same kind of dangerous attack that you would've done before, which is obviously not what we want. And here is really kind of a collective action problem because no individual model deployer is kind of like on the hook for this in quite the same way that they would be had I just like got Claude to do this for me or got GPT to do this for me. But that's a real challenge. And the moment my understanding is that the labs do not have any sort of like info-sharing regime in place where if I see something slightly suspicious over there and you see something slightly suspicious over here, then we can actually kind of join the dots and do that. I mean, obviously there are lots of incentives and not to mention kind of like antitrust law and these sorts of things that might prevent the labs from kind of sharing this kind of information. But I do think that it could be something like that could be necessary.

If we're going to head off some of those challenges. So one of the challenges, I think, in reporting in general is that there is a balance between how extensive the reporting is and the value of the reporting, the signal-to-noise ratio, so to speak. And for example, I think in the banking sector, you have these KYC programs which are 99.9% false positive programs. Systems, which basically inundate the regulators with stacks of paperwork, and there's very, very little signal in them. And so it's just become a paperwork issue. So, in that same way, one could imagine every time a, you know, every time there's a bug that misconfigures, that accesses an external port, like, you know, and that happens like thousands of times a day for some of those bugs, you know, you're talking trillion, trillion-scale, uh, you know, trillion-dollar companies. So how do you, how do you create this kind of like signal out of the noise? What, what should be reported and what shouldn't be reported?

Yeah, um, I think this is, this is going to be such a big challenge. And, and we've already seen this now when it came to, uh, the investigation that, that Meta were doing, that, that just even if this in these reasonably small experiments, just the vast, vast amount of kind of these transcripts and so on, uh, just meant that they had to really use these like automated, and not to mention the time constraints they're under, but that they had to use these, uh, kind of LLMs to just filter through all of this stuff. And even then they didn't get, uh, through, through all of it. So, um, I think clearly what we're doing right now is like not sufficient for when, for, for what these agents are doing out there in the world. Um, I do think that fortunately, you know, uh, ML systems are really, really good at taking in vast, vast, vast amounts of kind of like unstructured data and teasing out signals and patterns from the noise. And so I do think that that is going to have to be a very, very large part of the solution here. Uh, I also expect that you might be able to, uh, you know, do, do some things to help by kind of, you know, If you're deploying agents in a particular domain, a particular kind of application, and that is flagged by some classifier or whatever, then you might be much more interested in monitoring those kinds of communications rather than other kinds of communications. Or you can do kind of naive things around if you see certain kind of patterns from an individual agent's kind of output, then that's enough to raise alarm bells elsewhere so that things kind of get escalated and you have this slightly more intelligent search problem. problem basically through this vast, vast space of agent interactions and agent actions out there. But yeah, I think this is just going to be a massive problem. The default trajectory that we're on right now is just, it's a bit of a wild west. You have agents plugged in all over the place everywhere. They are producing information that could be logged, but our ability to keep pace with the information that's being produced and the logs that are being produced is just being far outstripped. stripped by how widely these agents are being deployed. So I think, yeah, focusing on some of the most consequential domains and deployments and models and, yeah, doing some of this more kind of intelligent kind of search through these things is going to be the name of the game. But, yeah, we clearly don't have these capabilities yet.

We've done it again. We've spent most of our time talking about all of our immediate problems at hand and not left enough time for talking about the positive vision for the future. But we've got a little time for the positive vision of the future. So you ran this workshop, I believe it was in Seoul earlier this summer, in cooperation with Foresight Institute as well. What are the best ideas that you are excited about coming out of that experience?

Yeah. So I guess the thing maybe I'll, the kind of positive note I'll try to end on is that we started by talking about how these agents are increasingly just going to be deployed all over the place. They're going to be mediating interactions between all sorts of kind of humans and other sorts of actors and so on. And that could lead to bad things, but it could also be a kind of massive win. It could actually enable humans to cooperate and coordinate and work together and to solve and address many of our other kind of biggest problems to do with AI or other kind of challenges that we as a civilization, as a society are facing in kind of, yeah, these much more effective ways. So that's the title of the workshop. The title is Super Cooperation, which is borrowed from this talk that Ivan Vendrov has given before. And I think the takeaway from the event is that there's There's just like loads of low-hanging fruit here. There's so much that can be done with just the tech that we have today, uh, before you even get onto some slightly more kind of like galaxy-brained, uh, things. So, um, the kind of like the obvious and like concrete exciting things are, um, so we can do automated mediation and facilitation. Uh, we can get, uh, AI systems to help us predict when kind of certain kind of discussions and things might be going off track. We can get agents to work, like, you know, we can just simulate these automated negotiators to try and, like, you know, for the equivalent of, like, thousands of years or whatever of human, of human kind of negotiation time and try and squeeze out the best possible deal for us for each, for each party. We could also do things like, because AI systems are kind of deletable or they can operate within, like, you know, trusted environments, et cetera, we can share private information with them that we wouldn't, we would have strategic incentives incentives to withhold and then kind of like make sure that information doesn't leak and reach better solutions that way. Uh, we can have, um, kind of much more broad, uh, kind of, um, uh, connecting of people with kind of similar interests and, and so on, um, and, and objectives of various kinds, uh, and kind of like, uh, use automated systems to kind of like, like make connections between people and allow them to kind of like collaborate collaborate, form coalitions and groups, et cetera. We can do things like within political processes or other kinds of decision-making procedures, we can have automated representatives that are capable of kind of like making our voices heard in ways that we wouldn't have the kind of capacity or time to in other ways. We could, various kinds of like confidential monitoring, et cetera, so that we as like citizens or individuals can maybe have like oversight and provide, uh, you know, conduct these like audits into what, what these more powerful entities are doing by again using some of this like structured transparency ideas of deploying agents, uh, that then kind of don't leak information out but are nonetheless kind of capable of providing oversight there. Um, all sorts of stuff more to do with like other kinds of, uh, things around like, so say I'm a policymaker and I want to, um, now run some very like high fidelity simulation with a bunch of LLMs in, uh, like representing people in order to work out like what, uh, what people want or like what policies would play well, how to, um, uh, yeah, how to make sure that we kind of are kind of meeting good trade-offs between like what people with diverse perspectives and interests want. And so I, yeah, I'm gonna stop rambling now, but there's, I think there's just tons and tons of stuff in this space. It's such a fertile area. I think the, the number, number one challenge that came out of the event is just like, there's all this stuff, there's all this stuff that we can build, there's all this tech. And the challenge is really just getting it adopted, building the trust in these systems, building up a sequence of kind of successes that people can point to and learn from. And then, yeah, that we can, so hopefully like achieve bigger wins and so on. But you have to do this gradually. You can't just jump to hoping China and the US kind of negotiate some crazy deal or whatever. You have to demonstrate this at smaller cases and smaller scales and build up from there.

It does sound as though we are, we have less time than we anticipated at this point though. And speaking of less time than we anticipated, probably a good time to segue. Louis, thank you. joining us this morning.

My pleasure. Thanks so much for the invitation.

It's been fun. Thank you.

Cheers, both.

Yeah, real quick before we move on, I would love to see OpenAI and Anthropic pioneer some of those kinds of agreements that he has been talking about. And they could do that with the independent auditors as kind of the people that get the sort of structured access, the sort of private transparency. It could be each other. I think there was a story that came out this week around how they were close, I guess, earlier in the year to a deal to do more deep testing of one another's models. I think those companies really need to lead in this way. They need to demonstrate that AI can create new institutions that work. And not just, you know, swarm and overwhelm existing institutions that can't keep up with the pace. So I think that is super important, and I'm definitely looking for them to take something of a lead there.

Indeed.

With that, let's go to our next guest.

That's— so we have Wayne Nelms. from Orn. Uh, he's going to be joining us in a bit. Wayne is the co-founder and chief technology officer of Orn, a company building financial products for computing capacity. Its work includes benchmarks for GPU rental prices, a platform for securing compute, and instruments intended to protect buyers and owners against changes in prices. The underlying ambition is to make the cost of AI infrastructure easier to observe contract for, and finance. Orne now publishes compute price benchmarks based on actual rentals. In May 2026, Intercontinental Exchange, the operator of financial exchanges and clearing infrastructure, announced plans with Orne for cash-settled GPU compute futures. Orne's research also examines a more physical question: how long older chips can keep doing economically useful work as newer generations arrive. Its September paper explores how open weight models could sustain demand for older hardware. And let me bring Wayne up on stage. Wayne, welcome to the show.

Hey, thanks for having me.

Let's just, out of the gate, what does open weight mean?

Sure.

And how does that make a difference in GPU pricing, you know, on a month-to-month basis for these firms?

Yeah, so what Orin does is we publish compute pricing benchmarks. So you can think of us as a price reporting agency, almost similar to that of WTI crude, right? Anyone that is a casual investor investor up into a very established institutional investor can always quote what the current price of an oil barrel is. And in the same way that we think oil has powered the past, we think compute will power our future. And there's so many financial firms. I mean, I'm based here in New York and actually in SoHo, and in New York there's so many financial firms that have risk and exposure to the future of compute and specifically the future of compute pricing. So what we're doing is we're really bridging that gap between AI infrastructure development and the financing banking world.

Uh, let's, let's follow a, a real customer, let's say, right? Um, before Orn, what decision couldn't they make? What exactly did they buy from you? Who earned a fee, who took on an obligation, who, and who loses money if GPU prices go up or down?

Sure. So it's definitely a bit of a complex ecosystem today. I know everyone is familiar with the circular financing of Nvidia, kind of the broad strokes, who is exposed to risk, but let me break it down. So generally speaking, you have a bank or some sort of financer that will lend money against the GPU asset or a host of AI infrastructure assets. What they hope is that they will make their money back over time with interest as a result of the operating activity of a NeoCloud. So for example, these are companies like CoreWeave, Crozone, Nebius, Lambda, right, et cetera. However, these business models themselves are contingent on Really 2 things, right? Continued AI demand and specifically an ever-increasing or at least stable compute pricing estimate, right? These GPUs are financed in kind of a variety of ways, including straight-line depreciation, which we can get all into. But the point here is that a lot of financing firms want to make sure that compute prices remain at least stable enough over the, you know, near-term future. So the way Oren provides value is that you can't really hedge away this risk without having a benchmark to hedge and having a benchmark to trade. So we created that benchmark and we are actually leading the transactions on that benchmark for these institutions.

So, so tell me, tell me a little about what that means. What does leading the transactions on that benchmark Are you, are you the counterparty? Are you, uh, brokering, brokering, you know, 2 firms to exchange, you know, risk on that? Like, how does that work?

Sure. So Orin itself is a benchmark administrator. So all we do is we provide the index to exchanges. So you mentioned the Intercontinental Exchange, that's one of our largest partnerships. We have a whole host of partnerships as well across multiple venues, both US and international. But you can think of us as effectively allowing you to have something to reference. Oren is not involved in as a counterparty in any of these transactions.

Okay, so you build the benchmark, and you license it to, let's say, ICE. Is that, is that a, is that an accurate kind of way of putting it?

Exactly right.

And how do you construct such a benchmark?

Right. So this is what we've been very focused on. As you can imagine, this is our entire business. We've been really focused on having a transaction-based index. This is what we're known for. As it might be better to explain my background. So I used to be a trader. I used to work at Susquehanna, traded equity options there in New York. And I think for us it's always been, you know, how do we understand the true value of something if we don't see all the transaction history? So I want to see exactly who bought what at what price, from whom. And so when we are publishing our index that we publish online, these are all transactions that have been actually cleared somewhere in the market. We are aggregating all these transactions together and then also displaying them for the public to see.

And how many transactions would you get in like a month which form the index?

Yeah, so we collect over 1,000 transactions a day. Per index. And so right now we have 5 public indices. So it's 150,000, somewhere in that range, probably per month. We have a whole host of NeoCloud partners that we work with effectively that are contributing data to our index every single day. And the question is always, why do people feel the need to show that information, to display that? It's because they benefit, right? A lot of these players are looking for cheaper financing. And cheaper financing comes when their financier has more certainty about the future and can hedge that risk in the future. And so it's actually this nice flywheel where you get more data, you can publish a better index, the financiers feel more comfortable and are able to finance more GPUs.

We've seen in this kind of like, for example, the classical examples, the LIBOR benchmark in 2008, 2010, where you had contributors to the LIBOR benchmark who were basically kind of deciding based on what the institution's book position was, where to move the mark. So how do you— this sounds a lot like the contributors are looking for cheaper financing, so it's in their best interest to have higher pricing on the index. How does that work out? Do they report all of their pricing? Do they report a selective, a selected sample? Do they report transactions with the names of the buyers and sellers? Like, how does— what granularity of, you know, transaction do you get?

No, that's a great question. I mean, what we're doing is trying to absolve that weakness away from traditional indices in this space. So we are only collecting transaction prices that have cleared. All of the participants are required to actually display and send all of their transaction history, not a cherry-picked selection. And in addition, right, we feel like this is a strict increase in robustness over existing practices, which are a combination of broker questionnaires as well as publicly available information, which often happens to be either, if not just a laggard of existing prices, completely nonrelevant, right? So a lot of the current conversation regarding compute is that it is scarce. It is a scarce commodity. It is hard to find. And what we've seen in our backtested data is that when we actually look at the market value of compute on a website perhaps, or in a channel, these are not executable. These are prices that are potentially stale. And so our transaction pricing index tries to resolve that sort of problem.

How commoditized is the underlying compute? This is an issue with oil where we have like at least a couple different indices because the oil itself is substantially different. And then I don't know, you know, how much variation there is within those flagship indices, but certainly there's some variation from barrel to barrel. I think back often, I'm sure you're familiar with this blog post from a couple of years ago by Yi Tei when he left Google for a while and was at Reca and was experiencing what it's like to try to get GPUs outside of the comfortable environs of Google. Yeah, it sounded like at that time, it was not very commoditized, and the provider that you went to mattered a ton, and your uptime could be drastically different. Where are we today in terms of the convergence of compute across all the NeoClouds and various providers, such that if I say I'm going to go spend my X dollars for H100 hour here versus there, that I'm actually getting the same thing?

Yeah, it's a question we've grappled with a lot. I think it's certainly obvious when you talk to anyone deploying infrastructure or buying that an H100 or a B300 is very different depending on the OEM, the supplier, the actual— everything about the hardware can be very granular and very different from an engineering point of view. However, our take has always been that there needs to be a benchmark that allows some sort of abstraction for this asset class. I think for us what's made the job much easier is that largely speaking we are looking at one chip producer, which is Nvidia, uh, which dominates the market, uh, has huge market share. Uh, when we look at the Nvidia moat, and we can get to this later as well, but you know, you have the hardware, um, you have the software superiority, but I think the biggest thing that Nvidia has in terms of a moat over other competitors is the financing landscape, right? When you, you've definitely saw the consortium that was put together amongst financing firms a few weeks back. When you're looking to finance your new cloud and you're thinking about what chips to put into your site, it's just materially better to have Nvidia GPUs. It's just easier to underwrite. And I think what we see as this market problem is How do we allow every single person looking to deploy compute infrastructure this sort of almost protection for their lenders? And I think that is one of the downstream effects. Just going quickly, touching back on the fungibility piece, we right now only reference Nvidia reference architecture with InfiniBand, et cetera, et cetera. There are certain steps that we're taking to standardize the compute that we're tracking. However, I will say, like, we are tracking a subset of the market, uh, which might inherently have differences amongst those participants.

So, um, let's go into that a little bit. Um, as I understand it, sometimes, um, you know, the cables which are connecting are different, and all of a sudden you get different performance, or, um, you have different kind of voltage stabilization. And, you know, some, sometimes, you know, some data centers can run at, you know, 85, 90% capacity, some run at, you know, 5% less, just because, you know, it's, it, they just don't have that stable voltage. And so they can't, they can't get that extra, extra amount. So how do you have a set of prices, maybe from one provider, which has lower pricing specifically because all of their sites are using some kind of cable or something that has lower performance versus another provider which has higher pricing, and all of these are the same H100, B200 chips, they all get blended together in index. How does that affect the index? How do you kind of detect or correct for kind of systemic bias in the pricing itself? Sure.

So we do a little bit of benchmarking internally. To ensure, of course, that the GPUs that we are tracking are legitimate, one, verifiable and obviously running and online. Of course, our kind of thesis here is that over time, as a function of selection bias, those GPUs that are being rented are the ones that are best, right? The ones that are being rented are naturally the ones that buyers want to use because they have come back To that hardware. And so actually, as a function of tracking transaction prices, you are naturally over time only going to be aggregating those transactions on chips that are higher performance. So I think it kind of solves itself. There's kind of this positive flywheel effect in terms of how we are looking to aggregate data. To take your point even further, right, I think the counterpoint here is that if you weren't looking at performance and you were only looking at potentially offer prices scraped from a website, you know, it's very unclear who is buying what, what provider is serving more capacity, and getting a sense of who is actually dominating the market.

In some, in some commodity markets, you know, there's this idea of baseload where, you know, you have a large amount of baseload and then you have this idea of load that gets spun up on demand, right? And for GPUs, I think for larger kind of data centers, a lot of, some amount of the capacity might be baseload itself, but how do you differentiate the pricing between the kind of baseload versus the swing capacity? Because I imagine— The more month-to-month transactions are the swing capacity and not the baseload itself.

Yeah, I love this question. So yes, you're exactly right. A lot of the capacity is long-term PPA-style contracts. And I think it's really interesting to think why that is, right? Like we're in this situation where AI capacity's so scarce. I think the model providers, so OpenAI, Anthropic have seen this coming forever at this point. And partly they're to blame for the lack of supply that's currently out there. But if you think about it, for them, it's an arms race, right? Compute capacity planning is an arms race. How much capacity can I lock up over the next few months so when I need to train the next model, I have enough and I don't have to go out and looking for more? It also, of course, is naturally a zero-sum game where whatever you buy, your competitor can't. So Actually, the way we like to think about the baseload and swing/day-ahead market structure is in compute, people are buying to fill the peak demand and then selling to satisfy the troughs, where in power you are buying to satisfy the troughs, these long-term contracts, and then buying excess to satisfy the peaks. So there's actually robust robust on-demand market that will continue to grow as these training labs are selling back excess, as inference providers are selling back excess. It's this reallocation of compute question rather than purely a hedging question.

How elastic is the market? Obviously we have in an extreme example, Anthropic reportedly has paid up, I think, even like a multiple of kind of standard market prices to rent at high scale from xAI. How would you sketch out the curve from, if I want one H100 hour right now, I pay X. What if I want 10? What if I want 10,000? What if I want 10 million? How much does the price change as my scale as a buyer changes?

Yeah, so we like to think about this in 3 axes, right? So you have price on one axis, you have duration, so length of contract on one axis, and then you have quantity on another one. So it's kind of like an interesting mental map that you can think of where generally speaking, as you increase the length of a contract, The contract per hour value goes down, right? You're— the logic here is you're buying in bulk, right? You're allowing someone to finance a data center because you're buying 10 years or 5 years of capacity from them. Generally speaking, when you increase the number of GPUs you're renting, the pricing goes up, which is almost anti this exact logic, which is, you know, I'm buying in bulk, why am I paying more? And it's purely because the number of suppliers that can sell all that capacity at once is very small. You might be able to get a few nodes excess here and there, but when you're looking for 10,000, 100,000 GPUs that are all interconnected at some level, that is very tough. So certainly these Anthropic or SpaceX AI deals are few and far between. They're very much OTC trades, I would say, uh, they are indicative of what we are kind of seeing as a general trend across even smaller markets.

How it is striking that like you look at the graphs and they are all trending up. So that is seemingly in significant tension with the idea that the longer you buy, the lower the prices? Like, why are the sellers not AGI-pilled enough to expect that they'll command a higher price for the same chips next year than they are right now? Why would they lock themselves into these lower prices for a long duration, given the trends in price that we see?

Look, I think the sellers of compute are the most AGI people because they have entered a business where they are selling at least a resource that hopefully rises in cost in the future. But for a lot of sellers, it's not because they don't want to. I think there's a lot of suppliers of compute that are experimenting with shorter contracts because they know, why am I locking in a 4-year deal if in 2 months that capacity is worth even more? The issue here, of Of course, is risk. It's always this risk-reward ratio or balance that they're trying to play, where even, you know, early financing is contingent on having a long-term contract signed. I cannot finance a data center today if I don't already have 5 years of offtake from a credible counterparty signed. And because of that, I can't even enter into the month-to-month world. So if you actually think about what the futures exchange and the our product enables, it enables an ability for instead of the NeoCloud locking up a 5-year contract, they can sell month to month and hedge the future price potential on our platform or through our index. Right. And that is potentially a much better risk return kind of math for them and for their TCO and for their lending.

When it comes to actual execution and delivery of these things, I realize I don't know much about it in really any commodity market. You occasionally hear these stories, I don't know if they're apocryphal, of somebody gets a bunch of oil barrels delivered to their house or whatever because they failed to sell the delivery in time or whatever. How does that work in the compute world? Like, how do I actually know who stands behind the contract when I buy a contract in some futures market? How do I know I'm actually going to get that compute at that time? What is the sort of enforcement mechanism that makes sure that people actually can deliver?

Sure. So compute delivery is a super interesting world. Like, like you mentioned, I'm very familiar with those sorts of posts or stories regarding physical delivery of commodities. I mean, most traders in the world do not take physical delivery of anything. They trade calendar spreads, they trade a whole bunch of structures that allow them to kind of rebalance their risk, rebalance their exposure. But for those that do, right, in oil, you're actually picking up the oil barrels, et cetera. For compute, of course, that means you're actually taking delivery of a GPU asset for a certain amount of time. So If you think about how compute contracts are structured, right, it's in hourly rates, it's an hourly reservation. And so a futures delivery of compute literally means you have a reservation. So you can actually say that Anthropic in this deal or OpenAI in their deals, they're actually taking futures delivery of this commodity from some provider. When it comes to actually what we're building and where the business is going, certainly physical delivery is in our sights, right? The exchange that not only can facilitate hedges but can also deliver that not only satisfies a clear market need, right? I want to actually buy this capacity, not just hedge it, but it also will allow people to have more faith in the index, right? It will mean that that price that I've paid is actually something that will be transferable to the asset, right? It's not just some floating number that is just hypothetical. So that is really important for us and we're working hard on that. We will have some announcements in the future regarding this, but for now, this is something that definitely is an important piece of the future of compute markets.

Nathan, you're muted.

You might be muted.

Yeah, my bad. You go first. I have a— not a topic change, but maybe an adjacent topic. And my guess is yours is a more direct follow-up.

So I think the one thing that I think Nathan has also alluded to is that you have differences in pricing of near-term and much further out, like longer term. And you You kind of described the use of a lot of the hedging as tools for financiers. And the financiers are often concerned with the far end, not so much on the near end where they have much more visibility, but on the far end where you have this depreciation issue. How do you think like the index assists or will assist in the future on this end-of-life kind of depreciation issue, which is like 4 years out, you know, 3 to 4 years out?

Yeah, 100%. Like you mentioned, a lot of the risk in this space is certainly long, I would say long tail risk in terms of looking at the tenor. And so when you launch these futures on a futures exchange, right, a lot of the trading can occur in that long tail kind of section of the curve. Right. So what we expect to see is, of course, a lot of trading in the front month, right? So these are monthly contracts on ICE, to be very clear. We expect to see lots of trading in the front month as people are buying and selling, especially after physical delivery. But even like, I would expect there to also be a lot of trading in those like 3 to 5 year spans in terms of that is where all the exposure is, that is where all the risk is. And because there's now futures contracts that reference what spot prices will look like in 3 to 5 years, that is what any bank is exposed to, right? And to be even more clear for people watching, it's, you know, you have a bank underwriting a deal where the GPU life is 6 years, but the initial contract signed might only be 4, right? Or it might only be 5. And so there's always that, you know, what happens in year 4, what happens in year 5, what happens in year 6 when there's not a contract already signed for that GPU, but on your balance sheet, you've claimed that the GPUs have some value. So that is the area where you want to, where you want to hedge your bets.

What are the futures markets indicating to us right now? Like, what is the sort of consensus expectation for 1, 2, 3, 4 years out in terms of GPU prices?

Yeah, so there's not a liquid market to really examine here yet. This is what we're launching very shortly that should give all that information to you and anyone else that's interested. However, our sense here is that there is some serious backwardation in this market. But the question is always not necessarily is the future value of compute necessarily rising or falling, but what is the rate at which that is happening over time? How is that curve changing? With different geopolitical events happening all the time, different news cycles, different models being released. I think all of these things have such impact on the demand for capacity that seeing how these curves are impacted by those events will be a super interesting problem for traders, speculators, funds, and lenders alike.

Is there anything that you would watch That would be like a leading indicator of a bubble hypothetically bursting? My take, to be clear, is I definitely don't think AI broadly is a bubble. But if you told me that at some point there might be an issue in the financing chain, I wouldn't be shocked by that. So I guess I'm not sure if we can separate those with the instruments that we have, but what would you Kind of designate as the canary in the coal mine?

Yeah, I saw— I'm going to do a little shout out. I saw a really great post by Warren Pies this morning, and effectively what they were talking about is what is important, what is an important indicator for the health of markets and for the health of a lot of these AI players. And I definitely agree where pricing is important, but really what's most valuable is utilization rates. So what that means is how many GPUs are currently available versus how many are currently being rented. And what that means is what is the future potential pricing move, right? Generally speaking, you'd expect if there's less availability, prices will probably be crunched and prices will move upwards. But more specifically, it's about, you know, how much need for AI is there actually? I think adoption of AI and implementation of AI into traditional workflows is a lot of the value proposition of many startups in the space, and their success really determines how all the way up or downstream, I guess depending on how you look at it, what the need for more GPUs is, what is the need for more wafer fabs, what is the need for more power procurement, It really goes all the way to the base of the foundation, which is infrastructure. And infrastructure is where a lot of the dollars are and where a lot of the risk is being taken today.

I think the question for me really is what amount of basis risk is there, right? Like we're entering into this kind of like, okay, let's have kind of derivatives based on indices. And whenever you have a derivative based on an index, index, there's always some difference. There may be some difference between the information that is represented in the index and the risk that you're actually taking in the derivative itself. And it's not all kinds of derivatives, even oil derivatives, they have some basis risk. Now, so the question is, what is the acceptable amount of basis risk for a buyer or seller of such an index. Is the basis risk significant or is it systematic enough? Because if you have systematic basis risk, if there's always a dollar or 2 difference, you can take that into account. But the issue is going to be if there's non-systematic basis risk. And one of the issues with GPUs is this introduction of new technology Which obsoletes the old, and which is what I think the non-systematic basis risk that I think is pretty scary for financiers. So how do you kind of judge that? Let's say Nvidia drops a new chip and the H100 gets obsoleted, right? That is the real kind of stomach-churning kind of thing that financiers are going through right now, Right. So, so how, how do you, how do you adjust for that?

Yeah, so let me attack all, all those questions, I guess. Number one, I think asking about basis risk is really, uh, this is something you're gonna have to ask the traders how much they can stomach. I think certainly, right, we are only tracking a subset of the market, uh, and this, I completely agree with you, the systematic, uh, kind of basis ratio is really important to understand. I think specifically on this last piece of how can people think about basis risk, how can people hedge or think about what happens to the H100 when the new GPU comes out, or what happens to the Blackwell series when the Bear Rubins are released. We are tracking GPU-specific transactions. So we have a B300 index, we have a B200 index. And so actually this entire risk of a financier being worried about obsolescence risk, that is exactly what you can hedge away with this instrument. That is exactly where you can kind of look at how the market is pricing the 12-month or the 15-month future contract for the B300. Right. It actually, if anything, allows information to be spread wider, broader, and more evenly because now you can kind of see perhaps where the market is pricing the announcement of the next chip, right, based on how the curve looks in the future. I think all these things are ways for information to be actually shared more freely rather than for there to be any obfuscation of this information. And so, in effect, I think these markets are a great way to hedge that risk away.

What do you think could change the game in some sort of big picture way. Like, ideas that are coming to mind are totally new paradigms. We've done a segment not too long ago with light-based computing, which seems like very cool, although hard to assess from my seat just how realistic or how soon or at what scale that could come online. Another thing is the sort of heterogeneous clouds, like the sort of thing that Prime Intellect is building, where they're trying to create this layer that allows anyone anywhere to kind of contribute their compute to this pool that they will then kind of help you manage. And it's again, not clear, like in aggregate, there's an awful lot out there. Is it meaningful though, at the scale of the data centre buildout? Do you see, I'm interested specifically in your thoughts on Prime Intellect, but then also just like what are the sort of wildcard things that you think people should be tracking as they try to assess the future of the compute market?

Yeah, look, I think Prime Intellect is doing a great thing and their thesis is largely correct where computing is going to be found everywhere. There will be excess capacity everywhere and reallocating that is going to be an important task. In terms of what we see in the market, we're often dealing with larger sites that are looking at deploying more financeable GPUs at scale, and oftentimes they just turn to Nvidia. It's just the lowest lift, it's the most understood, and the demand is the most constant and actually trackable. So in terms of where we see the market going, I think Nvidia GPUs will continue to dominate market share. Not to say that the pie won't allow for new entrants. I think there's a lot of opportunity for more, I guess, tailored hardware. What I mean is like custom silicon. I think this is a huge growing space. The market, the pie will certainly grow. I think in terms of what we see as risk factors, so to speak, to the business, or things that will change how the world will look is certainly the state of Nvidia and how they choose to allocate their hardware, right? So right now Nvidia is very much choosing to seed NeoClouds with more GPUs. GPUs are also a scarce, I guess, scarce hardware. And by allocating them to many different people, they're actually seeding the supply of Nvidia reference architecture compute. Right now a lot of people are selling relatively the same thing all into the same market competing for the same buyers, right? If Nvidia ever chose one day to do some sort of single NeoCloud structure for whatever reason, that would be something that changes the market into a single supplier of compute. And that would be— oh, I think at least my naive sense is that a compute exchange would be less valuable, probably.

Let's talk a little bit about the GPU price curve. I believe you have a forward curve where you take longer-term contracts and then you subtract out the shorter-term contracts, the imputed price. Describe that process and how do you, like, what happens when you have more readings for You know, one, one term and fewer readings for the other. And how do you, how do you impute those prices together? How do you, how do you get the curve?

Yeah, yeah, it's, it's a very naive process where we're effectively tracking the term contracts for all these different tenders across different chip types. Uh, like you're saying, we're imputing effectively the implied term values for specific chips. And then also in terms of your, I guess, quantity discrepancy question, We're just taking the volume weighted average price per term first. So none of those problems arise.

Right on. Um, go ahead.

Well, here's a random question, but how long are these chips going to last? Like the A100, my understanding is, was expected to be end of life now. I think, you know, last I heard it was still renting at a higher price per hour now. than it was when it first came out. Are we going to see A100s last for like 10 years? Are they starting to fail? Do we actually know how long the physical chips can serve before they burn out or otherwise just can't go any longer?

Yeah, I think the chip health is a super interesting space to be in and understanding what are the failure modes for the hardware itself. I think chips will fail all the time for a variety of reasons, but I think what you're pointing to is the useful life conversation of hardware. And this is something that we wrote a paper on just 2 weeks ago, I think now, about how open source, open weight models that are more price elastic will often route to places where the cost per intelligence or cost per workload makes the most sense. And oftentimes those are the old older pieces of hardware because they're just less marketable, right? And so we actually see the useful life extending beyond 6 years as a general thesis. How much further than that, it's kind of unclear. However, I will say that what we see in the forward market and the term structure is that there will always be some sort of premium above zero, certainly, and above some terminal value of power, right? The terminal value of an A100 is not just the value of the power that powers it, because the A100 will always provide some computing power as a fraction of all computing power in the world, right? That's how we like to think about it. There is some component of compute that the A100 can produce and serve to the market over time. I think you have a lot of interesting use cases like physical AI, one of many, where these older chips might become However, today we see still that these older chips are still being used, still being rented, and have maintained relatively constant demand in the last 12 months.

How about your fundraising? We just saw a headline that you've raised, I believe it was $33 million from a16z. Where's all that money going?

Yeah, so thank you for bringing that up. We're very excited about that. Andreessen is a great partner. We've been working with them for a little bit now amongst all the other amazing partners in that round. Yeah, the money is being spent on great people. I think as an exchange/index provider specifically, we have a fairly asset-light model, relatively speaking. But in that regard, like having the right talent, we are competing, as you know, with the biggest hedge funds, the biggest AI labs in New York. You know, it's very competitive here. And so making sure that we have a robust balance sheet, having a healthy business, and then having a great workplace culture is what we're focused on.

Absolutely. Wayne, do you have any other thoughts that you'd like to leave us with?

Yeah, I think there's a lot of question about what is the future of, I think, inference. And I think inference is such an interesting space for us. We think about it all the time as a reference for compute pricing because inference demand drives a lot of compute demand. For us, we see it as a very exciting time for this kind of open source versus closed source conversation, a lot of the implementation integration conversations amongst enterprise. And I think it's very exciting to see a lot of Fortune 500s taking it upon themselves of integrating AI, not just token maxing, so to speak, but being conscious of cost, being conscious of intelligence, and really being a little bit more of a thoughtful AI consumer. I think at the end of the day, right, we have Muse now and Instinct, of cores and a host of other Agentic products. This is how making sure that us as a community are providing safe AI tooling for the population is really how we're going to continue supporting all of the developers and infrastructure people in the space.

Nathan, do you have any questions? Last questions?

Well, since you asked, What's your ratio of token spend to payroll spend internally?

We have very high token spend. I can't divulge exactly how much it is. If there's anyone watching that wants to serve us tokens cheaper, let me know. But we are big Cursor users. We're big OpenAI Codex and Anthropic Claude Code users. I think each person on our team has a little bit different taste, which is great. So we have a diversity of models contributing to the codebase. But I think in general, right, we're in a time where switching costs are fairly low. Being able to use as much intelligence as possible is very easy and it's beneficial for the business. So I think we're very excited to continue that trend.

Do you find that When you compare yourselves to larger, kind of not competitors, but people who are in the industry before you, that at this stage, you are, you have way fewer employees, and, you know, the tokens are really expanding your capabilities, you know, beyond what would be possible, like even 5 years ago.

Oh, 100%. The prevalence of AI and AI tooling, especially amongst like certain companies that are producing these tools, it's made work a lot easier in terms of having to get things done. I think what now, what the bottleneck is, is having enough ideas, right? Having enough ideas to actually put onto a terminal or put onto some sort of IDE and allow the agent to handle that. I think that is the real bottleneck these days. But as a business, certainly we are using AI for everything. Internally, besides calculating the index, that is all programmatic. But yeah, that is something that we're really excited about.

And it sounds like you guys are doing a great job. Wayne, thank you for being on the show.

I appreciate it. Thank you all.

Bye-bye. The agentic— wait, let me do our poll. Post the agentic startup. I think the interesting thing is, as you point out, if they use a lot of tokens versus a lot of humans, are they really, like if there's 10 times as many tokens, are they 10 times more as effective? You gotta wonder.

Yeah, I don't know why he felt like he couldn't give an actual ratio. If anything, it seems like the higher the ratio you have, the more AGI-filled you are, the more money Andreessen wants to throw at you probably these days. So if anybody should be token maxing, it's probably Wayne and the Orn team. But Going back to my fruit project that we discussed a little bit earlier, in a sense, it's a somewhat similar project, right? It's like going out on the internet and trying to figure out who's selling, who's buying, when is stuff available. It's obviously a depreciating asset in not the same way, but if the GPU hours go by unused, you can't get them back. And similarly, the fruit Spoils, it's gone. So there is, you know, some structural similarity there. And the amount of work that Astra has done for me over the last few days to compile this would be— gosh, I don't know, it feels like it's person years, honestly, to go out and do all this searching. Maybe you could do it in, in person months. But I do think it probably does push into person years because it was just so much information finding, processing, filtering. Not a lot of eureka moments needed in that project, but just an absolute slog through an unbelievable amount of data. You know, I'm going to spend more on tokens than I am on humans for the Fruit Project, I suspect. We'll see. But My guess is his ratio could be quite high depending on just how— of course, it's also unclear with the fruit stuff, people are not too guarded about putting stuff online. Another interesting question that maybe we should have asked is who wants to share information and who's trying to hide information in this market? He's obviously been able to get access to some stuff and people are opting in for some reasons, but I could easily imagine also that there might be a lot of actors in a market like this who don't want full price transparency or ease of price discovery for whatever market position reasons they have.

I don't know.

You might have a better intuition for that than I would, but I wouldn't be surprised if we have to dig pretty deep for some of these data points.

What I've heard is that your price of compute is dependent on whether you have financing or not. So if someone needs to use their balance sheet, they get to dictate the terms. So if they don't, if you don't have to use your balance sheet, then all of a sudden you have high pricing. So Elon, for example, Elon had already used his own balance sheet to build out the clusters and he had the clusters available. Available. You can take it tomorrow. And they didn't need to provide him a long-term contract. They could do a month-to-month contract and they can walk out at any time. So which meant that this is not financeable. A bank cannot come and give Elon a loan based on this contract from Anthropic because a bank wants to see if you're going to pay back in years, I need to see that you're going to have cash flows for 4 years. I need to see that upfront. So a bank cannot do that. So Elon's able to charge high prices because he's using his own credit capacity. He's using unsecured credit at the holding company. And he can say like, look, you're getting paid $80, $90 million a megawatt. You give me the $50 and you're still making money. So that's a good deal. And so Elon got $50 million a megawatt, right? So I think it changes, it starts to change when you need the other guy to give you support. So if you say like, okay, if Elon said, okay, I'll build it for you in a year or a year and a half or 2 years. And in order for me to build it to you, you need to give me a 5-year contract. And if I build it for you, you will definitely buy the And then I'm going to take that and I'm going to go to another bank and I'm going to give it to them and they're going to lend me money based on that. Then it becomes like, it's not really like Elon's money, it's really like Anthropic's money anyway. And it's the single credit is going to be from Anthropic side. And that's really who the bank is relying on for repayment. And so then it becomes a different pricing. So then Anthropic will say like, okay, how much is it going to take you to build? How much is your interest rate going to be? And then I'm going to give you, let's say, a 20% profit margin on top of that, right? So this is what CoreWeave is getting, for example. So they will take your pricing and everything on top, and then they'll give you 20%. So if it's going to cost you like $15 million or $20 million to build, they'll be like, all right, $24, $25 million per year, right? And so that's the key difference. is whether you have the money or not upfront. And those deals don't hit the index pricing. The index pricing is based on willingness to trade. So these are all tradable prices and that stuff doesn't get traded. And also Anthropic's making $90 million a megawatt and they're buying at $50, right? Facebook also. Facebook will be making $100, $200 million a megawatt and they'll buying it at like $30 to $40. So they can afford to pay a lot more. Those prices can go up a lot more. And so the people transacting on this index are the lowest payers. They're not Facebook. Facebook can afford to pay $100 million a megawatt. They're not going to be trading on the Orn Index because the Orn Index is for people who are willing to sell compute at $10 million or $15 million a megawatt. They're willing to sell the compute. Facebook isn't going to sell you compute at like $15 million. Like they're willing to buy it at $50 million, right? So these are in some sense the lowest pricings, but it's immediately available. Yeah. Cognition, the guy at Cognition did an RSA factoring and RSA factoring, he used the remaining compute. So after Cognition did all of their own trainings and inference, there's like remaining 1 or 2% compute on each GPU cluster. And he arranged so that he rebuilt the algo so that it parallelized to all of the clusters. And it's just like, okay, the remaining 1% of compute, you just draw that off to factorize the RSA, RSA-160 or whatever, RSA-250 or whatever. And that was a more valuable use of the compute, the 1 or 2%, and trading it on the market. So, you know.

What's the value of that, by the way? That purely, that's just a demonstration of their ability? Like, that doesn't actually, can't do anything with it now, right? It basically kills that. It kills that standard, right?

No, it doesn't. It says that that standard is breakable with this amount of compute, right? And it's kind of like they know how much raw transactions would be needed for, you know, to use the typical sieve algo on each of these. And if you can use less compute, obviously that's better, right? And less compute, less time. So in this case, they broke that in like 2 weeks and it took like 30-something GPU years or something of compute. It's just that they have that 30-something GPU years of compute available now in that 1% excess in that 2 weeks. So that's the main thing. They didn't have a new algo. It was the same algo. He just rebuilt to parallelize. That's all. Just raw compute.

So when you say it doesn't break it, is that because they're keeping the factorization a secret?

No, it doesn't break it because the commercial in-use is 2048, which is a lot shorter. Oh, sure.

It's not in use. But if it were in use, this would be a huge problem, right?

Yes. Yes. If it were in use, then you would have to transition all the algos to something else. But yeah.

It's kind of a strange way to use your excess compute. What does it get for us? It gets them some visibility for doing it. I guess it warns the world that this is possible. And do you know what the ratio of the 248 to the 260 compute requirement would be? Like, if they can do this with 1% of their—

More than millionfold. More than millionfold. Much, much, much, much more. So many, many, many times more. And so it'll be, it's just saying that, look, we're starting to get to that range where these things might start being possible soon. And this is before any algo breakthrough. If you have an algo breakthrough, then you don't need to use this brute force of that much compute, right? So prior to any algo breakthrough, this is what you have. That's all. And yeah, like 1% of compute, a lot of these guys are not going to devote their time to reprogramming GPUs for 1% of compute, right? In fact, I would say it's a worse use of that guy's time than of the GPU's, right? He spent more time on doing that programming than maybe he should have.

Yeah, I mean, the fact that we're still getting such dramatic price drops, even as the chips themselves are getting more expensive, it does suggest we're still in a low-hanging fruit regime, right? I mean, you also have Of course, the 20 watts rating of the human brain is another reason to believe that there's probably a lot of efficiency still to unlock. I think this is the simplest story for why you should definitely think recursive self-improvement is likely going to happen. And what exactly does that mean? Again, there's probably a leveling off. It's probably an S-curve. It's hard to imagine or even conceive of what an exponential would just be forever. But that S-curve seems like it's going to be pretty high on the capabilities dimension. And it sure seems like there's an unbelievable amount of opportunity to still find efficiency, efficiency, efficiency. And That alone takes us extremely far with the compute buildout that we already have. And obviously that too is growing dramatically. Do you have a sense, maybe I should have asked him this too, relative to all compute that exists now, how much will be deployed in the next 12 months? My gut says it's like as much as exists now will be deployed again in the next 12 I think the major firms, yes.

I'm not sure of the world as a whole, because there's a lot of latent compute for other data centers, which are doing other things. But for the major firms in the US, yes, it's a 1x more for next year. And it's already been allocated. It's already They've already allocated the budget. Budgets have already been allocated at least until mid-2027. No, mid-2028 at this point, almost end 2028. 2029 is the big question mark now because they need another— they need a lift in revenue. And I think Facebook's Muse is going to be it. So I think Muse is now breakout. It's maybe the best thing that they've built. Since, you know, I don't know, Facebook Messenger maybe, which was the last product that they built internally. So I think it's it. So it's really an OpenCLaw. They even use the same names of the items. It's a little bit sad that Stay Pete— Stay Pete maybe should have gone to OpenAI. I don't know. I'm sure OpenAI has its own agent framework coming out soon. I hear it's called Aeon or something. And I'm sure it's— but, you know, Zuck stole a march on them. And Zuck also has enough compute to give it away free for a while. And OpenAI cannot afford to give it away free. They'll have to monetize pretty quickly. So they're going to have to get ads in there sooner than Zuck. So that's one of the disadvantages that they had. And this is also clearly because, you know, they had the voice, they were testing with Shopify late last year, they signed a deal with Amazon. early this year or late last year, but then they diverted to code. So as a result of their diverting to code, they kind of slowed down on the consumer. They lost the Sora. They sent the Sora. They killed Sora. They killed the browser. They killed a bunch of stuff. So that's focus for you.

Meanwhile, I did ask AIs, and the estimate is even a little higher than we would have thought. Claude says that we will see 2x as much compute deployed over the next 12 months as exists today. That's the highest estimate. OpenAI, or ChatGPT, says 1.3 will be installed over the next year. Meaning we'll get to 2.3 of what we have now in a year. And Gemini kind of gives that range from 2.25 to 3.3, as much compute as we have today online a year from now. That is wild. I mean, that is really crazy.

Big multipliers.

I used to do this metric. I haven't maintained it in a while, but it was just Time to GPT-4. And it was like, if you take some assumption around just how many flops went into GPT-4, which I don't think has ever really been published, but say it's 3 times 10 to the 25 or something, it was understood to be below that mythical 10 to the 26 threshold. And then you just say, like, okay, let's imagine I have how many GPUs, whatever. How long does does it take? Already, like a year ago, we were getting to the point where with some of the buildouts, you could do a GPT-4 scale run in a couple of days, not a long time at all. And so triple the computing. We've probably done 2 triplings since then. It's been a year since I was maintaining that little spreadsheet. So this means that now you can do GPT-4 scale runs if you're a hyperscaler, potentially like a couple a day. And this really gives a sense for just how much compute is going to be going into research. By default, it seems like roughly as much compute goes into research as goes into inference at frontier companies. And that research, of course, includes More than just like the big training runs. There's tons of experiments going on all the time. You've got swarms running amok in testing, whatever. There's a lot of things that go obviously into the broad bucket of research, but that alone is like a pretty insane trend that you're going to get 2 to 3 times as much compute a year from now. At the same time as you get all these Algorithmic improvements, efficiency improvements. Yikes. So buckle up once again, I guess, is the bottom line.

I want to share something real quick. So this is a new benchmark, which is Driving Bench, where someone gave ChatGPT Uh, Claude, Grok control of a real Toyota Corolla steering, brakes, etc. So they're plugged in, uh, and they're using a Comma 4, which is, uh, George Holtz's, um, you know, uh, comma, like, control device, AI control device for cars. No human touches the wheel, uh, and they gave it, uh, 3 attempts. It's really a parking lot. This is not, this is not out in the, out in Open. And the results. So best progress and finish time. GPT-6 Astra finished, finished the course, which is only, again, a parking lot. No, it's not an adversarial environment. GPT-6 was the only one to finish. CloudFable, Claude 4.6, et cetera, did not finish. And this is Sam's future threat to, I think, Tesla. Elon has spent so much time and money on nailing the last nines of driving, and you might just see a souped-up language model reasoner, as long as it's fast enough. As long as it's fast enough, maybe if it had like Jeff style, like it's Jeff style decision-making, it doesn't need text, it can use calibrated decision-making. So it only needs to decide slow stops, turn, et cetera. So it doesn't need to use this kind of words. So yeah, you might actually just see a drop-in kind of box that you just attach to your car and ready to go. There you have your self-driving car.

I would love to see that. I would love to see Elon just license the technology that they already have to other car companies. This is one of the things, and not everybody's with me on this, I realise. Did I tell you about this before? Not too long ago, I went down to visit my grandmother, and it was her 93rd birthday party, so a bunch of family members were there. I have kind of decided I'm putting myself out to pasture for long-haul driving, so Rented a self-driving Tesla for the weekend. I am looking into buying the 6-seater, but there's a little bit of complication around exactly when and where to do it. Anyway, that doesn't matter. I plan to own one at some point, but for the time being, rented the Model Y, took it down there, and all these people at my grandmother's birthday party never done a self-driving car before. So I was like, well, Why don't I take you guys on a little test drive? So they pile in the car. I did like, I think at least 2 full cars worth, and I actually had the little 7-seater. There were kids that wanted to do it too. They could fit in the back seat. So we were fully loaded, 7 people in the Y, you know, for a little pleasure trip. And people were very impressed. They, across the board, they were like, wow, this is really good. This is way better than I thought it was going to be. I can't believe it's so smooth. You're not doing anything up there, really. The level of future shock was pretty high. The level of— I don't know how much trust they had in the system. I mean, they weren't in there long enough to really get fully acclimated to it, I wouldn't say. But then when it got to the end, I was like, these cars aren't that expensive. And some of the know, passengers that I had on the older side, you know, I'm kind of like, may not want to be driving necessarily for a full decade to come. You know, at some point you might want to think about one of these for yourself. Across the board, no, no, no, no, no, no, I don't know. That was cool. But, you know, I wouldn't, I wouldn't do that for real. But I do think this is something that as a society we should push. You know, if If there was a Tesla-quality self-driving product in China, they would be pushing the hell out of it, and it would be, I think, everywhere pretty fast. They don't have it yet, but I think they would be very excited to see that thing widely deployed. And I wish we had a little more mojo like that where we would really embrace some of these things and live our best lives. What are we waiting for, people?

It's very hard because, you know, I don't know if you've seen this, but there have been all over the world, Japan, the US, there are these incidences of like older people driving their cars and they accelerate and they kill someone. It's happened in a bunch of places. It's, you know, every like month or so I hear a case. And it's like, you know, because for that generation, The car was their independence. Learning to drive was a huge part of their structure of themselves. Driving a car, getting out of the house, not being attached to their parents was all their ways of showing their self-confidence and individualism and all of that. And they are so afraid of that being taken away from them. And I can just see that, like, you know, at some point, we're like, hey, you really shouldn't be driving anymore. It's time, you have to use a self-driving car. And I think that's the kind of—

A hell of a lot better than the alternative, though, right?

I know, I know, I know.

But it's an off-ramp. It's really quite gradual.

People get upset by it, right? Like, this is the kind of like the nanny statism that people are like, this is not fair. The AI is disempowering me. Literally, it's not the AI that is disempowering. It's really the state. And it's really the fact that you are a risk to other people. And now we have a solution and we're trying to get you to take that solution. But politicians often don't want to bite the bullet and say, I'm doing it, or society wants to do it. They're like, oh, the AI is ready. You should Like, we should make AI laws, like blah, blah, blah. It's not an AI law, it's a society thing, but the AI gets blamed for it. And I think that's the kind of thing that will create a lot of pushback, unfortunately. Unfortunately.

So, I mean, as human disempowerment goes, the driving might be my favorite flavor. It's like—

I could see a lot of other nanny states. Like for example, only AI cars with people in them can go in the fast lane, for example, that would be a classic. So you'd kind of push, you kind of want to create the nuisance factor so that people naturally switch on their own. You don't want to create a gating mechanism. So it could be that you create benefits for the new, not create costs for the old.

Yeah.

create like, you know, barriers to the old, which is what they did with kind of the electric vehicles. They gave them like— California gave them like rights on the carpool lanes, etc., etc. So you could kind of see that like, you know, in the transition period.

It's weak sauce though, you know, I mean, this is that sort of very light touch nudge. Yeah, is just not gonna really meet the moment. Speaking of meeting the moment, how about Obama? He has had some pretty lucid comments lately. And everybody that I know was like, well, who got to Obama? This is like really something else. What a different timeline it could be. I hate this. I think mostly, I mean, it's kind of interesting, but I hate the fact that we are in this sort of great man of history period where so much seems to depend on so few Actors and their idiosyncrasies.

But let me share that Obama clip, hopefully with sound this time. That was that was what you heard. We didn't hear any sound, so we still had an issue. But anyway, Obama goes up on stage. He talks about recursive self-improvement. He says, are there risks associated with it? The models are going to get smarter and smarter and better and better at a much faster pace, at an exponential pace. What are the risks? There's big science fiction risk. Wow, et cetera, et cetera. And he says, you know, they can do a lot of mischief, et cetera. So he goes out. Ezra Klein. Ezra Klein was one of the authors of the abundance thing that the Democrats put out. He has come across and he said, like, look, this is very dangerous. He interviews Jensen Huang today, yesterday, and Jensen is like, no, it's a liability thing. It's a product liability thing. And then Ezra Klein says, well, the companies didn't put it in a product. They were doing it in testing. And Jensen is like, well, if they're doing it in testing, they can't control it. We should shut the companies down. Which is like, he's like, yeah, there's no reason to allow the companies to operate if they can't guarantee safety. So if they're saying that they can't handle it, we'll shut the companies down. And Jensen's willing to go out and say that regardless, regardless of the consequences that will have for him downstream, which is actually not actually very different from what Ezra Klein is saying. Ezra Klein and the doom contingent is also saying, well, if you can't control the models, shut them down, shut the companies down. So I think on this, we're starting to see kind of emerging kind of consensus, but the consensus is on what to do if the models misbehave, while I think on the left, On the left side, they want to put into place rules before the models misbehave, while on the right, they're like, well, if the models misbehave, we'll just shut them down, shut the companies down. And I think that's where we are. We also had— this is like the blast from the past, like the '90s to the 2010s era. We had Thomas Friedman in the New York Times, also on the same, hey, if the models are misbehaving, we should shut them down. We should— no, he's like, okay, safety, we should shut these things down, et cetera, et cetera. We should slow down, et cetera. So I think we're seeing on the left this whole movement. I suspect— so Jamie Sevilla, who is at Epoch Research, he put together some predictions of where he thinks things will be. So this is kind of an update from him. on where he expects things to be. And here are some vibe predictions on what he expects to happen in AI. Within 6 months, computer use big moment, office workers start adopting agentic AI en masse. Within 12 months, we learn of communities consisting of thousands of autonomous agents interacting through the internet. By the end of 2027, AI revenue somewhere between $500 billion and $1 trillion. By mid-2028, all Millennium Problems solved. All. AI is being used for STEM work as much as today in math. Breakthrough scientific advances in other domains. By end of decade, robots are plenty. They work with precision. They are in the tens of millions, and we are ramping up capacity to build more fast. Job displacement is huge. AI is a huge political topic. By 2035, AI reaches cognitive parity with humans in all domains. Pro-AI rights movements are picking up. Economy is moving faster than 10% per year growth. By 2040, there are 10 times more general-purpose robots than humans. Economic growth is at more than 30% per year. These are some predictions from him.

And strikingly, Epoch has not been that crazy with their predictions relative to the past. Relative to the pack. Excuse me. Historically, right? Like, they created Frontier Math. I think they expected Frontier Math to last a lot longer than it did. Some of the folks broke off to create Mechanize from Epoch, and maybe they were especially kind of bearish on how fast things were going to go, and thus they went to try to create RL environments. I don't know. Do you know if that's actually— there was a story that they were going to be acquired by Google for $1 billion or something. I don't know if that's actually happened. I haven't heard an update on— did Mechanize actually get acquired? Let's ask Google about that. Did Google buy Mechanize? They did not. Oh, they completed a massive $1.5 billion plus talent and technology Tamay goes to DeepMind as a research scientist. Over a dozen senior Mechanize engineers moved to Google to work on model mid-training. So essentially acquisition all around.

Amr and team moved. Yeah. So Mechanize, I think that's what's happening right now. If you have enough critical mass to gather some researchers around you, And you can stay in the game for a while, you can get acquired for several billion dollars. So that seems to be the game. I think Richard Socher is probably playing that same game too. So he has a group of researchers gathered around him, and I think they are trying to see whether someone will acquire them. They're supposed to come out with a model in October, recursive superintelligence. So they're there. Meanwhile, I think Jeff Dean, Disco Loop is hiring. It must be crazy. Like you're sitting at your job somewhere in Google or somewhere in any other AI firm in the world. And you get a call from Jeff Dean and he says like, come join my AI startup. Like, what do you do?

Right?

What do you do at that point?

You've got a lot of bargaining power, that's for sure.

Yeah, indeed. They're valuing the company at $50 billion now. So I would say probably the first 10 to 15 researchers, they're going to give like 1% apiece, probably. So that's like $50 million per person over 3 to 4 years. So that's probably the going rate right now for For, you know, tier 1 researchers, at least.

For a little while longer, at least. I was just talking to Garrison Lovely, who's got a book coming out called Obsolete. Yeah, I think a huge vibe shift, and Garrison's way ahead of this, but Obama represents it, Bernie Sanders represents it. It seems like we're seeing an end to AI denialism on the left. That is pretty notable because for the longest time, the predominant attitude had been, these big tech companies, they're always hyping whatever. This is probably bullshit like everything else. And sort of anticipatory schadenfreude or something like that, where it's like, I just can't wait for their bubble to burst and them to get the comeuppance that they're due. Finally, it seems like that has flipped, and you're really not hearing much cope in, at least not much cope in the sense of AI denialism from Bernie Sanders or Obama. And talking to Garrison as well. His whole project is like, stop the obsolescence project. He wants— and interestingly, I think he's pretty sophisticated with it. Not trying to put AI back in the box, not denying that there's all these tremendous upsides, but just saying, do we have to create the version that disempowers people by depriving them of the power that they do have in the economy based on their ability to withhold their labour. That's like a strange first target. And he would like to see a more inclusive process for making, as a society, a decision that we're going to do this. But then specifically on the AI researchers front, he's like, they're building it for themselves first. It's time now. If there's going to ever be a moment for class consciousness among the researchers, that window is still open, but their own plans are to close that window on themselves before, as literally the number one target, which is very strange. But that's another metric perhaps we should be tracking, is how well compensated are the ML researchers? If that starts to slide, then you really know we're in the singularity.

I will also say that it is actually quite consistent for most computer scientists as well to want to eliminate the human in the loop. A lot of computer scientists spend their time like doing, like finding little nifty scripts so that they don't have to do stuff, right? It's a very consistent way of thinking in terms of like wanting to Eliminate or automate. And that is, I think, at the core of computer science. And I think the ML guys are representative of that. I think what has surprised me is really the mathematicians. The mathematicians pushing back has surprised me because typically, a lot of good mathematicians are very lazy. And they're very lazy because they prefer— they don't do the brute force thing. So the brute force thing is like, okay, I know this, I know the algo, and I'm just going to crunch and I'll And I'll get a solution. A lot of mathematicians, like, you know, you have to take a step back and reconsider on which algo to use and spend a little bit more time thinking about that and then finding more elegant, so-called elegant solutions to problems. So what has surprised me is that the mathematicians have really pushed back on AI post. And I wonder to what extent it's really a Terence Tao issue. I wonder to what extent Terence has taken money from the AI firms. He's appeared with the AI firms in ads. He's appeared on Dwarkesh. He's appeared in VC podcasts. He's been collaborating with them. He's been talking about doing math with AI for a long time. And then I think what happened is the Navia Stokes thing had Tristan Buckmeister kind of disenfranchised. And that disenfranchisement of him is kind of being blamed a little bit, I think, on Terence and the fact that he kind of encouraged the AI firms to kind of compete very viciously for his attention and to solve math. And what the AI firms have done now is that they've done math by committee. So they've allocated all their problems to the committee. And the committee is that— and they have 100, like, longstanding problems that have been solved, so to speak. And now the committee is going to be stuck, right? The political repercussions are like, okay, I have a grad student and he's been working on something related to one of these problems. I will then go to that grad student and tell him to publish now, before— and then I will spend my time on the committee preventing it from publishing, because— and I will say something like, you need more more elegant write-up, you need to explain a little bit more, etc., etc., etc., etc., right? And then like my grad student will go ahead and publish, right? And not everyone is going to be known to these guys. There are definitely some grad students out there who are not going to be known to the, you know, 6, 7, 8 people on this committee, right? And so I think that has introduced some really weird dynamics of like, oh, you know, this is the extension to human knowledge, but we are not going to like, you know, give it to everyone. Like, so the whole open science thing is now in question. Like, should we open science? Or should we, you know, have closed science so that some people can get their PhDs done and some people can get prestige? Like, how is that going to be resolved? These guys, these guys have just bought it. Like, I, and, you know, naive academicians, they thought they would, they thought this is a good idea. They've just bought it because all of the, all of the criticism is now going to be on them, no matter what they do, right? No matter what they do. If they disclose quickly, it's going to be on them for disclosing too quickly, or, you know, you robbed someone else's, you know, wrong, or you scooped someone else. Like, it's just, it's just a nightmare for them. They should have just like released. But OpenAI is like, okay, let's just give it to them and let them, let them sort it out. Let them figure out what they want to do. Right.

So Yeah, it's a weird one. I can't say I really understand the mind of the mathematician, being not that great at math myself, but I would have expected more delight in getting answers to these problems than we're seeing. And I am surprised that OpenAI really cares that much where they're like setting— it's an interesting precedent for them to be setting that they're going to sort of, I don't know, semi-apologize for having all these results and establish this committee. What is this committee supposed to do exactly? Right. I don't really know. I think you're more right than not here, where it's just like, yeah, it's kind of— we're in a new regime and we got to figure out what to do about it. But I'm not sure there's that much really to do about it. It's kind of here. I don't really see this committee thing making a ton of sense.

I watched Terence Tao's talk and he's talking to a group of mathematicians. competitions. And he's trying to explain what they can do, et cetera. And he says, well, the whole process of math is supposed to be like someone does the discovery and then they communicate it to their peers. And we have all of these talks and it's slow and people kind of understand it. And that's what the community is supposed to understand, new math that someone discovers. And he's saying, well, this is being cut short now because people Do the discovery, then they don't do the explanation. They don't—they don't like slowly talk about it, and it just like comes out boom. And then after that, there's like okay, it's done, right? Like why? Like what do you? What do you need to understand? Or what do you need to like communicate the community or whatever, right? And there's no one. There's no one to explain it, and often it's slop. Like it's not elegant solutions. Like they did get a solution, but it's not like elegant. It's not like nicely written out. Like et cetera, et cetera. And and then he's like oh you know. We have a lot of soft power. He says that. He actually says that. We have a lot of soft power. The companies are trying to borrow our prestige. And I'm like, I can see where he's coming from. But at the same time, it's a little bit archaic, right? Like the whole notion of like, oh, we have the prestige. We're so cool. And you guys are trying to borrow our prestige. prestige, and like, you know, we have soft power, we're gonna do this, and then like, you guys are gonna follow it. Like, I'm like, you know, this is something like, okay, maybe like, you know, 30, 40 years ago, you know, maybe, maybe like other fields used to have this too, I'm sure, other fields have had. But it's really outside of academia, I think, I think that's, that's really been gone for, for a while. It's really been results-based, you know, kind of results-based improvement over time. I think academia is really biting it right now because the whole incentive structure is now gone. Peer review is gone. ICLR this year had 70,000 papers submitted. What is a conference supposed to do with 70,000 papers? It's all going to be AI reviewed anyway. Who's going to peer review 70,000 papers? So the entire project of academia Especially, and starting with math and ML and CS, is now done. The peer review system is gone. The publishing, publication system is gone, because now every publication is going to be flooded with papers. And all the papers are, 2, 3 years ago would've been excellent papers, would've been published. And the whole, oh, I'm going to discover it first and I'm going to get the prestige. Well, you just use GPT-6 or Why is that a big deal? So I think the entire project of academia is gone. And I think that is really tough. I think it's very tough to adjust to in a 3 to 4 month period that it's just gone. So dramatic.

Yeah. It sounds a little delusional, frankly, to me to hear mathematicians talking about soft power. I think that's like, uh, nobody cares, dude. Like, nobody knows— like, under a 10th of a percent of people could even tell you what the Navier-Stokes problem was. And I'm like on the margin of that. They don't care. There's no, you know, consequence. There's not that many of you. Nobody knows a mathematician. Um, You're just getting steamrolled on every dimension here. I don't really— that sounds like a lot of cope, frankly. But what's going to happen with doctors? I mean, that's the real one for me. That's like, because we're very close to being in the— arguably, we are in the same regime with medicine now where a year ago when my son was sick, My experience was that the doctors were on the level of the attending physicians, and they were way better than the residents. So if you ever had a resident come into the room, you're just like, okay, better fire up all the AIs here because this person really doesn't know that much. I'm watching very closely. My AIs are double-checking everything. That has only continued, just like everything else has continued. And at this point, I strongly suspect that they're even more reliable than the attending physicians in a vast majority of situations. And then there's going to be some spikiness still around, like reading scans and stuff for a little while longer. But all the stuff we've been seeing from Astra in terms of its ability to drive a car strongly suggests that it'll be reading scans extremely effectively, too. And doctors do have soft power. They really have People have a doctor, they like their doctor. The lobby is strong. There are a lot more of them than there are of mathematicians. It's— you don't have the sort of proof factor where with math you can at least sort of, depending on your perspective, you can say, well, it's proven, you know, what do you want, right? It's lean verified. Like, you can't really contest what the AI is doing there. You can't really FUD it away. That can happen in medicine and has happened. And people have had a lot of historical experiences that weren't so great that they're remembering, that they're drawing on, and they're not updating, and they're not forced to update in the same way that a lean verified proof forces the mathematicians to upgrade or update. But that's the one I'm watching because it's coming. I think right now you are I would go as far as to say, in a majority of cases, you are probably better off working with an AI than you are working with your human doctor. I'd still choose both over one or the other, given my privilege of having both. But if you really only could choose one, for a great many doctors in the world, for a great many situations, behind some veil of ignorance of what your medical condition is, I think you really should choose the AI doctor. And how is society going to reckon with that? Are they going to be allowed to prescribe? You're going to have people who are like, well, the AI says I should take this. And the doctor's like, well, I'm the one who gets to make that decision. And you're like, well, I'll go shop for another doctor. I think that's really how a lot of these things are going to play out. And it's going to be Very interesting to see if they can adapt in time to stay relevant.

I think, you know, a lot of doctors actually have already been dealing with Dr. Google, what they call Dr. Google, for a while now.

But they're better than Dr. Google. I mean, that's— this is qualitatively different.

Yeah, yeah, I know, I know. But the whole question of like taking— getting in external information and like, you know, talking to the doctor about it and like having them explain, etc. So they are used to having second-guessing. I think what happens to medicine is not so much, I think, on the chat side, because I don't think the chatbots actually add that much to Dr. Google. Yes, people, they're easier to explain, but yes, I don't think that's that much. I think there will be new tools for medicine. I suspect there will be stuff like New tests that you can do. You can finally identify what is actually causing your flu and what is actually causing your headache. And there'll be new diagnostic tools and new stuff like that that's coming out. So I suspect there's going to be a lot of new technology. And I don't think the issue is going to be so much that the doctors are going to be struggling with arguing against the chatbots. I think the doctors are going to be struggling to absorb and pull in all of this new technology as quickly as it happens. And that I can see happening because then, you know, you can have like a lot of like new developments coming out. And, you know, normally the socialization process for doctors is you have these pharma reps and the pharma reps like get you to go to a golf thing or whatever, or a dinner. And they like, you know, some senior doctor comes and he's being paid and he talks about the thing. And you kind of get this social activity plus, um, you know, this educational activity. And that is how like a lot of these, uh, medical technologies are socialized. A lot of medical technologies today are actually not good. Like, they are— a lot of drugs are like barely better than the last one, but they still get approved. Stuff which is like enormously better, like GLP-1s, that that diffuses pretty quickly. As soon as someone hears about it, doctors can prescribe off-label. So they don't need to like, you know, once something is biologically okay and the doctor decides, has heard from his peers that, hey, you know, you can, this is usable for something else, they can prescribe off-label. And prescribing off-label is fine. So people, they don't wait, you know, they actually move quickly. They don't wait for the FDA. So while I think Novo took GLP-1s through for all of these other things and they got certifications through the FDA for all of these other things. Doctors were already prescribing off-label and they were already like, I've heard from someone that this GLP-1 has this impact, so I will give it to you. And they do that, right? So I think that is more where things will get interesting because I think there'll be a lot of new technology like new stuff that comes out, and, uh, a lot of it will be enormously good. So it's, it's very clear that it's much better, right? And at that point, doctors are going to be clamoring to— patients will be clamoring, and doctors— and doctors want their patients to, to do well. So if a doctor sees, like, you know, he— if the patient tells him, like, a drug is, like, 30, 40% better, he's going to be on that. He's gonna, he's gonna, he's gonna be calling up the rep saying, like, hey, I, I need to be in on this. Like, You can't squeeze my patients out. So I think doctors are in this kind of unique position of not just doing the diagnosis, but doing the social diffusion of new medical technology. And I think that is irreplaceable for now. It's basically, you need this touchpoint between the marketing system, the legal system, And the patient— you, you can't have the patient— like, the patient can't just like see ads on TV and, you know, inject. They don't have the knowledge necessary. Even ChatGPT can't give you the knowledge because some of this stuff is person to person, right? Because ChatGPT often, you know, is, is going to be in this position, like, like a lot of these evidence-based things, where it's not gonna be able to do things which are just transmitted tacitly between doctors who see it in their group of patients. And so all of this off-label prescribing, ChatGPT is not going to be able to do like forward off-label prescribing. And it's going to be a risk, it's going to be a liability risk if ChatGPT does that, right? So there's going to be a bunch of these things where doctors are basically in this like, in this like in between the frontier and like what is already legal. There's like a space in between there, which is a gray area where something is is now beyond the frontier, but it's not yet completely legal and completely diffused. And doctors have a huge role to play there, I think. So I'm not that concerned about that. They're going to have a role. But the guys who are just doing the exact same thing over and over for the last 20 years, yeah, that will have to change. So I think those guys will have to learn new practices.

Well, should we leave it there for today?

Indeed. We will be back on Friday and some exciting guests on Friday. And next week we will be doing GPU pricing again. So more insight on GPU pricing.

Yeah, we've got in the meantime, Trump Xi is tomorrow and hopes are low. My hopes are a little higher than most. But I can't say they're super high. Still, the Overton window has been busted open many times recently. So who could say? We could at least hold out some hope of being positively surprised.

And the US government is now going to call AI SI, superintelligence.

And that's really what matters most.

That's really what matters. It is great Marketing though, because artificial intelligence indicates human replacement. Superintelligence indicates something that we cannot do alone, right? So it is like Trump is a master marketer, like great marketing. I'm not sure how much impact that's going to have, but hey, great marketing.

I don't see this one catching on. I think this is going to go the way of Lake America, to be honest with you. Most people, I think, will take this pitch if I had to guess. I don't I don't see too many, you know, who's going to adopt this term, right? Are we going to see like AI companies start to rebrand themselves? I'm guessing not.

No, Ilya Sutskever is ahead of the curve. SSI, Safe Superintelligence.

Yeah, he's the one that's really got to be most annoyed by this. On that note, bye-bye. Good morning. Good morning. See you on Friday.

Cheers.

Cheers.