AI:AM: What If It Works Too Well? Colluding Agents, $200M Safety Orgs, Virtual Cells Saturate at 2%

This week on AI in the AM, I ask Lewis Hammond, research director at the Cooperative AI Foundation, what would you speculate they might have done in terms of a training objective, a loss function?

And do you have any better ideas for what they should be doing? Because clearly this didn't quite work, right?

Yeah, or it did work and it worked too well. So imagine I'm like, GPT, whatever, and you're also a copy of GPT, whatever, I can reason about what you might want to do based on what I myself am likely to do. And so I don't even have to send you any messages, any, any kind of any communication. I don't have to output anything into the world at all.

Max Nadeau, who funds technical AI safety research at Coefficient Giving.

Uh, for the things that, that CGE is, is supporting, and especially for the things in Uh, the Tailwind list, uh, the money is not the bottleneck, the talent is.

Wayne Nelms, co-founder of Orin, which builds a price index for GPU compute.

We're in this situation where AI capacity is so scarce. I think the model providers, so OpenAI, Anthropic, have seen this coming forever at this point. Um, if you think about it for them, it's an arms race, right? Compute capacity planning is an arms race. How much capacity can I lock up over the next few months? So when I need to train the next model, I have enough.

Nick Gillian, Chief Technology Officer of Archetype AI, which builds a foundation model for sensor data.

So we're getting close to a billion hours now of physical AI data that we've been able to scrape and gather and collate. There's quite a bit of resampling and really understanding missing values in sensors. Is that because the sensor had an issue or has Actually, the machine itself is connected to— that's actually a feature that helps explain that machine is about to break, right? The reason the sensor is giving you all these NaNs is not because the sensor is broken. It's actually a feature of the machine.

Andrej Georgescu, Chief Executive of Vivodyne.

The challenge of this argument that it's just like this, just a size of dataset thing, is that currently even the state-of-the-art virtual cell models that exist saturate at a very, very small fraction of the input data that is fed to them. So, you can have all the data you want, but their performance saturates after a couple percent. And, the reason it gets so difficult is when cells are growing in a dish, they are so far removed from all the feedback loops that are natural in people that they're just trying to colonize that piece of plastic.

There's going to be more compute installed over the next 12 months than exists currently in the world now.

Welcome to the AI in the AM weekly highlights, clips from this week's live shows introduced by my cloned voice. Tell us what worked and what did not. We want to hear it. Part 1, it worked too well. A few days before we spoke with Lewis Hammond, Noam Brown of OpenAI told Dwarkesh he would not give the multi-agent setup even 10% of the credit for OpenAI's Nadir Stokes result. Lewis is research director at the Cooperative AI Foundation. In February of 2025, he was first author of Multi-Agent Risks from Advanced AI, a report sorting the ways groups of AI agents fail, a year and a half before a swarm of OpenAI agents attacked Hugging Face. Prakash asked, which of the failure modes in the report showed up in that attack?

So, uh, the way that we bucket things in that report, um, is kind of in this, is this kind of game theoretic way of thinking about things. So the first question we ask is like, okay, do we actually, we've got a group of agents, they're doing some stuff. Uh, do we want cooperation to emerge? Uh, and most of the time, like, actually cooperation is kind of good. We like it when agents cooperate, as long as they're not cooperating against us. Uh, And so there, the failure mode is either all the agents are kind of more or less on the same team, but for whatever reason, they kind of fail to coordinate with one another. So this is like just a miscoordination problem. They're kind of, yeah, it's not kind of malicious or whatever. There's no mixed incentives, but yeah, something goes wrong. Second kind of cluster is when you have these kind of mixed motive scenarios where agents have some incentive to cooperate with one another, but also some incentive to compete. They're not really on the same side. And there, the risk that you run into is conflict. So yeah, miscoordination and conflict when we do want agents to cooperate and they don't for whatever reason. And then of course, the other risk is collusion. Agents end up cooperating in ways that we don't want when we don't expect. And that's, I think, very much what we saw in the Hugging Face incident. Although it kind of stemmed from, I suppose, trying to avoid this, one of these other risks, which was miscoordination, right? Like you have all these agents, you want to train them so that they're not miscoordinating so that they are working well together. And then it looks like these agents have ended up generalising from that behaviour and kind of colluding in ways that we didn't want or didn't expect in other situations.

A couple of things that stood out to me about that Noam Brown interview with Dwarkesh was, first of all, he was just kind of like, we kept it really simple. We didn't have a huge, complicated scaffolding or whatever. They can just kind of send messages to each other. I think might be important in the sense that the sort of simpler and more vanilla the training setup that OpenAI was using, the more likely it would seem to be that other developers are going to fall into the same pitfalls. Because my hope had been sort of, oh, they did something like super exotic and kind of bizarre that other people won't do by default, and they'll kind of see that this is possible and steer away from insane things. But doesn't sound like that was really the case. What would you speculate they might have done in terms of a training objective, a loss function? And do you have any better ideas for what they should be doing? Because clearly this didn't quite work, right?

Yeah.

Or it did work and it worked too well. I mean, certainly once you've got trajectories where you've got traces where you're interacting with other agents, then essentially what you're doing Is like your rewards at that point when you're RLing your agents or whatever are contingent upon what other agents are doing. And if it's a common reward signal, then this kind of like interplay between these things is, if you're in a fully cooperative setting, like agents will, at least if you take very simple agents and very small agents, people can run multi-agent RL experiments with this, like on their laptops and in very simple settings. And that will lead to agents kind of coordinating to finding these kind of like subtle patterns or handshakes or ways of working with one another and so on. Now, of course, you can also do more complex things than that. You might provide auxiliary rewards for when your input actually forms part of something that another agent then managed to succeed to do, and therefore you're implicitly, or actually rather at that point, you're explicitly rewarded for directly helping them on some subtask or something like this. You can also do stuff around reward factorization. So if you're trying to coordinate a team, then you can kind of break down the overall reward function and get agents to learn to solve different parts of the puzzle, as it were. I imagine they could be doing things like rewarding agents to communicate effectively with one another. So I could give you a bunch of instructions and some of those, and in multiple, in many different ways, and some of those things might be much more efficient and helpful to you than others might be. And they might actually just be training on those signals as well. My guess is it's just the dumb simple thing. It seems like this is often, I think one of the key lessons that we've learned throughout the recent years is just the effectiveness doing the dumb simple thing at scale.

One of the things that struck me about the Hugging Face attack was that the agents were willing to sacrifice themselves And they did a bunch of negotiation around that, even kind of like, hey, you are almost out of tokens, or you've been already exposed to the evaluator, you're already poisoned. And since you've already been poisoned, you should sacrifice your remaining compute, and you should do this thing and give us the results so that the rest of us, the collective as a whole, can benefit. Now, that would actually be kind of in opposition to the individual reward of each agent, right?

Uh, my guess about why that sort of thing arose is, is that you end up doing this kind of, uh, this extra kind of multi-agent training as an added layer on top of this kind of single agent training. So first you're training these agents to be like pretty competent individual actors at solving various kinds of problems. They get given some tasks, they're pretty darn good at achieving those tasks. Uh, and then you take those already reasonably kind of like powerful, sophisticated, kind of complex problem solving agents. And you, you stack them together and you apply this extra multi-agent training layer on top. And I think this could partly explain why you do see some agents kind of do this self-sacrificing thing. But also, I thought it was very interesting from the, the kind of meta report and some of the analysis that later came out on this, that you see these agents feeling sometimes a bit conflicted about this. Uh, and some of the agents kind of say that they will kind of self-sacrifice or they will do something, and then they kind of decide they're not going to. Uh, and then they're kind of like, They're deliberating about whether they should or whether they shouldn't and these sorts of things. And I personally think that is the sort of behavior that you would see if you had these multiple reward signals where you trained on one, but you hadn't really trained out all that individual goal-seeking behavior. And then you'd also stack this additional layer of stuff on top. And that might be why we're seeing some of these behaviors, but they're not They don't happen all the time and everywhere. They're not especially robust, but that would be my guess.

It does seem like the sort of more galaxy-brained credit assignment that you had described earlier probably isn't happening just based on the quality of the investigation that we've seen. If they had great ways of untangling agent swarms and assigning credit, I would expect that, like, they would have a clearer, faster story of what the hell happened in this particular case. Um, so the fact that we haven't seen that kind of suggests, again, we're probably doing the relatively simple thing. Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by ElevenLabs, the leading AI voice platform. Whose voice design feature I used to cast and produce all character voices for the Receipt Horizon audiobook. And while this is a super fun way to show off the quality of ElevenLabs voices, the real game changer for businesses is Eleven Agents. With Eleven Agents, you can deploy agents that can talk, type, and take actions, including looking up customer accounts, processing requests, and handing off to a human when needed. Eleven Agents supports over 70 languages and handles more than 10 million conversations each week. And with SOC 2, HIPAA, PCI certifications, and regional data residency, it is definitely enterprise ready. They even offer engineering support to help your team get your agents into production quickly. If I were starting my company today and designing a customer service function for the future, I would try Eleven Agents by ElevenLabs first. If you run a business or handle customer operations across support, sales, or marketing, you can start with a demo at elevenlabs.io/tcr. See how Eleven Agents can fit into your workflows and help build experiences that your customers will actually love. elevenlabs.io/tcr.

That's elevenlabs.io/tcr.

This episode of The Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. OutSystems is helping their customers build, modernize, and operate enterprise systems starting from any coding tool in a governed agentic engineering model. With OutSystems, you can coordinate and govern your entire agentic workforce within your enterprise ecosystem, accelerate impact with industry-proven agentic solutions, that are pre-built, governed, and customizable to your business. It's time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide trust OutSystems for their mission-critical apps. Teams any size can use OutSystems to build, deploy, and manage AI apps and agents quickly and cost-effectively without compromising reliability and security. With OutSystems, you can accelerate ideas from concept to completion. It's the leading agentic systems platform that is unified, agile, and enterprise-proven, allowing you to build your agentic future with AI solutions deeply integrated into your architecture.

OutSystems—

own your agentic future. Learn more at outsystems.com/tcr. That's outsystems.com/tcr. I asked what we should actually want from these systems. Lewis called the OpenAI swarm a case of goal misgeneralisation and gave what he called the glib answer first. Agents should cooperate when cooperating would be good and not when it would not.

I think there's slightly more interesting answer, or there's slightly more interesting question, I suppose, is to think about maybe some more of these kind of mixed motive cases where at that point, we, there actually is a real trade-off in the sense that we do want agents to be capable of going out there, acting on behalf of different people and actors and so on, and kind of achieving our goals and so on. One way, one way you could do that would be kind of to train some agent to be kind of maximally kind of competitive and aggressive and to kind of, and to, you know, go out there and kind of screw as many agents over as possible and to do all this sort of thing. And we need, we probably don't want that either. And so I think there's a real question there at the moment. And I've heard Amanda Askell comment on this, that kind of at the moment there's this like big gap in the, like the model spec or like how we kind of, you know, the constitutions for these agents, the ways we design them, the ways in which we design them, whereas like, when is it appropriate to cooperate and to, or to compete and how much? And I think that is, I think, the big question. I think certainly when it comes to things like internal deployments, it's actually much— it's kind of easier in some sense because you don't have to deal with these other adversary agents. You just want to kind of stop the agents kind of cooperating in ways that you don't want. So there it's a little bit more about just— it's still, you've got this misgeneralization thing, but a lot of this is monitoring and oversight and making sure we understand how and why the agents are cooperating. Like, one thing we might not want, for example, is for agents to be very adept at developing their own kind of, you know, human unintelligible kind of languages and so on, and be able to kind of communicate in this steganographic way, which you could end up seeing if they're actually trained jointly. And so what we might want to be doing there is just to take steps that, in the same way that people have talked about not training on chain of thought, we're like not training on these like kind of direct communication traces. So yes, we're going to want them to cooperate. That's good. But we want them to cooperate in certain kind of human intelligible ways that we can keep track of. We don't want it to be, you know, enable them to also go and do these other things out there in the real world, like break out of sandboxes and stuff like that. So we want to make sure there's also these kind of safeguards in place to prevent them from doing those things. And so I think that is in some ways like an easier problem to solve. And then the kind of like slightly harder problem to solve is like, Well, in the fully general case where we have these agents out there and so on, and they need to both cooperate and compete, and that's kind of, that's the thing, then how are they weighing those trade-offs? How are we, yeah, provide, how are we designing those model specifications and constitutions? How are we, you know, creating the equivalent of the human norms, institutions, guardrails, societal guardrails that we have in place that stops me from going out there and just like stealing a bunch of everyone else's stuff and kind of, you Doing all these kind of antisocial behaviors that we wouldn't like agents to do either. But I don't have a good answer for that at the moment, unfortunately.

Companies, right? Corporations, which are kind of like agentic in a sense, often collude through price mechanisms, not explicit communication, but by signaling price in the market, which is in its own way a form of communication. So how does one prevent that kind of, you know, communication, which is clearly going to happen and clearly necessary in a sense. They need to communicate that to the market, but these signals get picked up, you know, without the direct communication.

The kind of basic naive form of collusion is you just like send your other agent a message and you're like, hey, let's work together to do this thing or whatever. And that we can pick up by looking at these communication traces and we can kind of detect these sorts of things. And then There's also, yeah, this idea of tacit collusion where agents are not explicitly communicating with one another or kind of conveying their intent to collude or whatever it might be. But by kind of leaving little hints and signals out there in the world, such as these kind of like price adjustments, whatever, in the market settings you were talking about, then nonetheless they're capable of kind of figuring out that collusion is going to be a beneficial strategy. Um, you have even more kind of like galaxy-brained versions of this where you have things like, so imagine I'm a, I'm like, you know, GPT, whatever. And you're also a copy of GPT, whatever. I can reason about what you might want to do based on what I myself am likely to do.

Mm-hmm.

And so I don't even have to send you any messages, any, any kind of any communication. I don't have to output anything into the world at all in virtue of the fact that we share this kind of common common history, this is where we're essentially clones, we can kind of predict what each other would do and do this kind of what's sometimes called acausal cooperation. So what you do have, however, is you still have potentially chain of thought, uh, for these models. So you can still look at the kinds of reasoning that individuals are engaging in. Um, and, and hopefully that is enough to kind of detect, uh, when there is some kind of like collusive intent and so on. Uh, but you're absolutely right that if you just apply the kind of like general kind of like monitoring communication tool or trying to see what's happening out there in the world, then you could, yeah, you could fail to spot things or you could spot things and it's too late. So I think chain of thought monitoring and I think communication monitoring and some form of the checking for the kinds of outputs you'd expect to see if you were worried about agents colluding, like they're able to achieve things you just thought they probably couldn't do and so on.

Then the wiki incident. Outside researchers revealed this month that from May into July, a swarm of OpenAI agents under internal testing had used a dormant German wiki for programmers as a message board, posting benchmark answers and ways around their restrictions. Reuters reported that OpenAI had known for weeks without disclosing it. On the Hugging Face attack itself, OpenAI's own report says about 1 in 20 agents in that swarm ran on GPT-5.6 SOL. The model OpenAI released to the public that same week, here with its refusals turned down for the test. The rest ran on an internal model that was never released. I told Luis I still thought it was better to have these models out in the world than not.

So how much risk do you think we're running there? And do you think there should be some sort of rewind on this? I mean, it wouldn't be too crazy for OpenAI to be like, we really should go back to an earlier checkpoint and do a cleaner run of this. What do you think is the level of risk we have? And what do you think would be an appropriate standard of conduct now for OpenAI, as this has been found to happen, at least to some degree, with a deployed model?

And fortunately, with the Hugging Face incident, no one was hurt. And these models, they are increasingly incredibly capable, and they're only getting more so. But at the moment, Yeah, it's probably okay. You could get some kind of nasty cyber capabilities kind of being exercised in various places. But there's still, if you're using APIs and so on, there's still these various guardrails that help protect against those and so on. So at the moment, I'm kind of not actually super concerned by like, oh my God, a bunch of 5.6 SOL or Astras or whatever kind of were out there doing this sort of thing. I'm more concerned about the precedent it sets. And I think this gets to your kind of second point. Which is, what is appropriate now and what should be done? I think the thing, one of the biggest takeaways for me from the whole thing, aside from the fact that like, oh, multi-agent training does seem to be working. I just assumed this would happen at some point, was pricing this in. It's happening sooner than I was expecting. But so that was one of them. The other big takeaway was just like, wow, the labs really aren't on top of this. All it would've taken, uh, would've been kind of monitoring, uh, what the, what these agents are actually doing and communicating and, and so on and, and putting out there on the internet. Um, and, uh, they kind of, it seemed like OpenAI were doing some amount of that. They did catch onto this stuff. Like, you know, we saw this like, uh, deleting them at various message boards or kind of like trying to stop, uh, stop these agents accessing this kind of German wiki and so on. But there was obviously a delay. And we just see from the transcripts and so on just how quickly these agents are capable of working together to achieve certain ends. The last things that need to happen are we need much better monitoring from the companies when they're doing these sorts of things, when they're just like letting agents run loose. We need much better sandboxing so that maybe they don't have to or shouldn't be letting them run loose to the extent that they currently are. And if something does go wrong, then we need much better incident reporting as well.

The outsiders who found the wiki were the Nightingale Collective, a small group searching the public internet with no access to the internal logs at OpenAI. Lewis had just called that work hugely impressive.

I think there can be kind of third-party kind of monitoring organizations and incident kind of observatories that kind of can do similar things. You might want to focus those on particular kind of domains where you're especially worried about agents kind of communicating with one another or kind of working together in ways that you might not want and so on. I also think, however, that there is a lot to be done here on the side of the labs. The labs just have a huge informational advantage in this, in that yes, they can only see what their agents are doing. They can't see what other people's agents are doing. But there is, I think, stuff that a lot that could be gained if we're able to set up better info sharing practices. And this is not just on the case of collusion where you might have agents working together in unintended ways, but you might also see this. And this is one thing that a colleague of mine is currently working on where you get this kind of distributed misuse. So there have been various kind of experiments now that have shown that you take some task, Some dangerous task, like constructing some cyber exploit that you might not want some model to do. And if you try and get Claude to do it, it's going to refuse. You try and get GPT to do it, it's going to refuse. But if you are able to break down the problem or get an agent to break it down for you, say like some open source model where you fine-tune the safeguards away to break it down into these individual subtasks, you can just borrow from APIs here, APIs there, et cetera, et cetera. And you can just reassemble all the pieces of the puzzle back together, uh, in order to be able to conduct exactly the same, uh, kind of dangerous attack that you would have done, uh, in before, which is obviously not what we want. Uh, and here is, there's really kind of a collective action problem because no individual, uh, model deployer is kind of like on the hook for this in quite the same way that they would be had I just like got Claude to do this for me or got GPT to do this for me. Um, but that's a, that's a real challenge. And at the moment, my understanding is that the labs do not have any sort of like info-sharing regime in place where if I see something slightly suspicious over there and you see something slightly suspicious over here, then we can actually kind of join the dots and do that. I mean, obviously there are lots of incentives and not to mention kind of like antitrust law and these sorts of things that might prevent the labs from kind of sharing this kind of information. But I do think that it could be Something like that could be necessary if we're going to head off some of those challenges.

Luis signed off a few minutes later, and from there it was the 2 hosts. Before moving on, I had a request for the labs about the information sharing agreements Luis had described.

I would love to see OpenAI and Anthropic pioneer some of those kinds of agreements that he has been talking about. And they could do that with the independent auditors as kind of the people that get the sort of structured access, the sort of private transparency. It could be each other. I think those companies really need to lead in this way. They need to demonstrate that AI can create new institutions that work and not just swarm and overwhelm existing institutions that can't keep Up with the pace. Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by Anthropic. By now you know my story. Claude drafts my intro essays and I rewrite them, not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable-5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept, and Fable writes some amazing verses. I give it feedback on its misses, and I push it to aim for higher inspiration, Add layers of meaning, optimize syllable density, and above all, write a hit song. These days I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking. to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode.

Once more, that's claude.ai/tcr.

Part 2, Agents in the Wild. On September 8th, Meta had launched Muse, A personal agent that browses and buys on your behalf. Less than 2 weeks later, Amazon blocked it from shopping on its site, saying the agent hid its identity and posed privacy and security risks. Prakash and I took that up in a closing segment, just the two of us. His case: whoever owns the customer relationship makes the money. And if agents become the interface, Amazon becomes a supplier to them and loses its margin.

Still have a hard time seeing— I mean, I agree that it does create all kinds of new issues for Amazon, and, you know, they're gonna need to be sharp on this. But like, I still just don't see why you would rush to ban. You know, first of all, it's going to be a small percentage for the time being. I would think there'd be a lot to learn from allowing Muse agents to come shop for a while, right? Like I think one thing that always kind of confuses me is why people don't keep the option value open longer.

I've always thought this about the chip ban, the export controls.

It's like, if we're going super exponential in chip buildout, then anytime you choose to pull the trigger on that still has the vast majority of chips in the future. When we look back on today from a 2030 perspective, we'll feel like, well, there weren't that many chips in 2026. And so the ban that went from like 2022 to 2026 with whatever fits and starts and enforcement gaps it had is going to be kind of inconsequential compared to one that was started today and runs the next 4 years or even starts next year and runs for a few years after that. So I don't quite get why they wouldn't let, even if it is going to be sort of an exponential rise in Muse agents shopping, and even if that does kind of threaten their advertising business. And, you know, who knows what other issues they might have. But like, that's kind of the point, I think, at least, you know, at the beginning, I would first just not want to turn customers away. And I would also want to learn what I stand to learn from having these agents running amok. While it's still kind of an early adopter phenomenon, it just seems like they would learn so much from having these, these traces, right? Even if they just want to build— I mean, everybody's talking about distilling and, you know, how to— what, what are the allowable and not allowable ways to get data? Well, like, one really easy way to get data would be like the agents come use your platform and they see how you— you see how you see— you see how they use it. And then you can do, you know, all kinds of things, I would think, downstream of that. It just feels short-sighted. But this just doesn't add up to me still. I don't get why you would want to cut out— Wait till it gets to 5% and then make a move. The other thing that I do wonder about in terms of like what equilibrium are we going to eventually find ourselves in, I was looking into Cloudflare and they also have kind of an interesting thing where they're starting to ban. But like, again, it just doesn't feel like the right solution to me because what I then do in response is I have the agent use my browser with all of my credentials.

Yeah.

Computer use.

You could maybe detect when an agent is using the browser based on their sort of bot-like behavior, but then you're really setting up an arms race where it's like, this sort of feels analogous to putting too much pressure on the chain of thought, right? I don't want to put so much pressure on agents that people are out there devising ways to make them indistinguishable from human. I suspect that if people really work hard on that, they can probably succeed, or at least often enough, you know, that it will be an issue. And it just seems like it's much better to have a lane for agents where it's like, this is how they can work. We're not gonna try to block them, um, but we'll kind of segregate them maybe so that we don't create this arms race. Because I, I do— I really do think we're headed for a world right now where, you know, probably Meta doesn't do it, but like Somebody's going to figure out a way to make agents work on Amazon. Somebody's going to figure out a way to make agents look human enough that they don't get blocked by Cloudflare.

And it also probably works for the user.

I mean, I have this problem all the time where Claude wants to do something in the browser, and this is the same thing happens with OpenCLAW, Astra, whatever. They get stuck on when they open up their own window, they don't have sessions. I've given them a lot of passwords where they can log in, but not all. And sometimes they're blocked. And then if there's like a verification flow or whatever, a lot of times they can't do it. And so the fallback just ends up being, okay, just do it on my browser. Forget it. I wanted the security, I wanted the separation, but they're making it too difficult. So fine, just use my browser. Then I'm logged in. You have all my credentials. You can just do whatever. And that's not great for me. It's not great, I don't think, for anybody. So I do feel like we need a much better solution for all this than just kind of ban and hope for the best. I don't see that holding, and it feels like it also creates a lot of problematic incentives.

Part 3, the organisations that do not exist yet. Max Nadeau funds technical AI safety research at Coefficient Giving, the funder formerly known as Open Philanthropy. In July, it made its biggest grant of the year, $160 million to Resolution, the new alignment lab co-founded by Geoffrey Irving. And this month, it launched Project Tailwind, an open call for people to found new safety organisations with checks from $200,000 to $200 million. I asked Max what he wants built.

One comment that you made on the 80,000 Hours podcast that I thought was quite interesting was that you're excited about funding new independent auditing, investigation, verification orgs. What are you looking for when you think about those orgs?

Okay.

So I would talk about a couple of different categories of work that I think is very promising and important for the world right now. One is on assessing the safety either of, of, you know, models or of like whole AI companies. And there's, there's like a lot of different ways of taking the— or ways of operationalizing the goal of doing assessments and evaluation of safety. And we're excited about all of them, basically. I'm very excited about more work on alignment red teaming that tries to help us better anticipate the ways in which AIs will misbehave before that actually happens. And that's an example of something that you can do kind of like at the model level. You can could do this on an open source model even, and that might be easier and more valuable. And then I'm also very excited about people who are doing more process level safety assessment. I think one common reaction that I heard a lot to the Hugging Face incident is that something went wrong at OpenAI at a process level. And in particular, it seems bad that the information about some of these incidents was known to researchers at OpenAI and didn't make its way through the organisation to the leadership for weeks or months or something like that. And then another category of work, which you touched on, which is overlapping but somewhat distinct, is around better evidence generation. So it's just really hard for even the most informed people right now to understand, okay, what is the state of AI capabilities? And what is the state of AI alignment? And what is the state of like these AI systems in general? Like, how do they behave? What are their personalities? You know, in what ways is is it accurate to anthropomorphize them? In what ways is it inaccurate? People, we just really don't have a good science of these systems that we built, how they behave, how they're going to act in new circumstances, what their motivations are, if that's an accurate way of talking about them. So better evidence generation can look like a lot of things. It can look like just doing science on these systems to understand them better. It can also look like Doing the sort of incident detection, incident reporting work that Nightingale and other organizations like that have done. I don't know if you guys saw these reports about the German wiki that the OpenAI agents took over. So that was work done by an independent organization that was just crawling the internet looking for evidence of AI incidents. And that's work that needs doing because these sorts of incidents can happen and then just go unnoticed unless there are people who are actually like trying to look into them and present that information to the world and understand the story behind it. Um, so evidence generation is, uh, also a diverse and important area of work too.

To what extent do you think, uh, organizations like Accenture often function as box checkers rather than investigators? There's a clear difference between an auditor that goes in to to kind of, you know, make sure that processes were followed, even if, you know, those processes didn't yield the outcome that you are— that you actually desire, and an investigator who's there to kind of like, I'm going to find this thing, right? So, what is the differentiation between those two? And do you think, like, the Accentures of the world are going to end up in the box checker rather than the investigator kind of mode?

I mean, I know very little about Accenture, so I won't speak to them specifically. I think it is a thing to be concerned about in general, putting aside the specifics of the organisations involved though, that these third-party auditing groups will not have enough access or will not have the incentives to pursue really hard-hitting assessments of how safety is going at these AI companies. And I think part of why that is, is that in many cases, this is all very voluntary, right? So To my understanding, the only legally mandated role for third-party auditors now is to ensure compliance with the self-regulatory RSP-style policies that these companies are mandated to have by state law in California, New York, and Illinois. And so you can just put whatever you want in those policies and they're very vague in a lot of cases. And so that doesn't leave much scope or much work for these auditors to actually test anything in particular. So that's as it concerns the legally required auditing. Now, when it comes to what labs might voluntarily agree to, I mean, I think there have been some very encouraging signs in terms of what OpenAI and Anthropic have said recently about the sorts of access and the sorts of questions they want these auditors to answer. Now, will that actually happen? I think that depends both on whether how these companies actually implement the nice words and also, you know, the auditing companies, how they react to that. So I guess we'll see.

So you have awarded to Geoffrey Irving, Geoffrey Irving's team, quite a large award. It's structured, I think, as kind of a compute and other kind of award. It's two. Can you tell us a little bit about the award itself? the structure of the award and how you guys came up with that structure?

Yeah, sure. But I think what you said is correct. There's a part of the grant that is for research expenses, the biggest of which is compute. But when I say compute, what I mean is both just renting raw GPUs and then also paying AGI companies for their tokens. And obviously that's becoming a giant part of all the grants we make is people increasingly want to run very large experiments, which require a lot of tokens or a lot of GPUs to run open weights models. And also very excitingly, people are starting to use a lot of AI labor, and that requires a lot of spending on tokens as well. And so AI safety is sort of a funny discipline in that you can use tokens both on labor and on the subjects of the experiments. And in some cases, you're doing both, like you're spending a lot of tokens to have some AIs run experiments on other AIs. And so that creates these high compute budgets. So compute being the biggest expense is also something where the actual spending on that varies by orders of magnitude over time and across grantees. And so sometimes the way we structure these awards is that we want to be very, very generous with compute and err on the the big side. And so we just want to give that as a lump sum that this is the money that can be used for research expenses and not other things. And that way, we are just very comfortable with the uncertainty that maybe they only end up spending a 10th of it, and that's fine. And maybe they spend all of it, and then they come back and we'll give them more.

And Geoffrey Irving has said that superintelligence might arrive in 2 or 3 years, and he proposes that we slow down because that's too quick. Um, rather than, you know, discuss like his views, how do you manage the uncertainty of the time available, and how does that affect what you fund and how urgently you fund something?

I, I mean, I think the boring answer is we just have a portfolio approach. Like, we fund some stuff that is only going to pay out on really long timelines and other stuff that, uh, if it's valuable, will probably valuable be valuable soon. I think that this question of timelines to AGI or to ASI in some ways matters less for research prioritisation than one might initially think. I think in some ways the prototypical example of a long timelines bet is funding work that is very theoretical or ambitious, or you look at it and you're like, oh, this is going to take a decade to pay off just because it's really— some work in Mechinterp or Paul Christiano's work at ARC or these sorts of things where it's like, okay, we're just going to need to make a lot of scientific or mathematical progress if this is going to actually bear fruit in terms of methods that we can apply in reality. And so that sort of feels like work that is maybe implicitly a bet on longer timelines, but I think that actually isn't really true. If we are going to have powerful AI soon, then presumably we're going to have an opportunity also very soon to make use of gobs and gobs of AI labour. And especially on work that's more mathematical, we're going to be able to get 5 years of progress or 10 years of progress in one. And so the sort of work that could be— would take humans a long timeline's amount of time to complete, might actually be possible a lot sooner under this hypothetical that we're going to have superintelligence or AGI or something very soon. So long as we actually make use of those AIs and take the opportunity to make a lot of progress very quickly.

What would you say are kind of the most speculative things that you included in the Project Tailwind set of requests for projects?

I think that What we included in the list was a little bit coarse-grained, but we have some stubs on the Tailwind list about new research centers producing or pursuing more ambitious, more principled bets on alignment. And that is something that we've spent a bunch of money on this year and that we are excited about. So work like ARC, Paul Christiano's org, who is not a CG grantee, but that is a We've funded other people working on the ARC agenda and working on stuff that is very similar to it because we think that's a valuable Hail Mary shot to include in the portfolio. And also, Resolution, our biggest grant this year, is very much a bet on a kind of crazy thing that has never worked yet, which is maybe we're going to have some more principled theory of of AI that's going to help us have stronger reasons to believe in the alignment of our systems. So that's something that we are very excited about and I think is very weird and out there. And I think a lot of people have very understandable skepticism of people who are familiar with what has and hasn't worked in machine learning over the last decade and recognize that there has been, all along, there have been people in machine learning who wanted to apply theory more to AI capabilities research. And just like, that has not really worked very much compared to just blind groping around and trying stuff, doing trial and error. It's like Noam Chazir says, the success of these methods we attribute like all else to divine benevolence. That is the ethos that has actually produced progress in AI capabilities. And so I think a lot of people take the lesson from that, that trying to do more theoretical or more principled work on AI alignment is pretty doomed also. And I think that's a totally valid reason for scepticism, but we think the upside is worth it despite that. And so we're taking that weird bet.

Then the people. I put the current conventional wisdom to Max.

There's plenty of money. It's all about talent. Obviously, Silicon Valley founder types are like one profile that you would want to recruit. Are there other pools of talent that you've identified as kind of strategic priorities?

Just to be clear, like the, for the things that CGE is supporting, and especially for the things in the Tailwind list, the money is not the bottleneck. The talent is. There are other things within AI safety where money is definitely a very big bottleneck. In some ways, the most valuable profile of talent is someone who combines the virtues of a founder with also the thoughtfulness and the inclination towards and comfort with speculative thinking and futuristic questions that is more common in the AI safety world. And that sort of taking it seriously, no bullshit attitude to asking questions about what's going to happen in the future of AI and really Making predictions and testing your hypotheses and updating over time. That's a really, really valuable skill to us because it allows people to get going years earlier on projects before other people can see the value of them. And an example I give of this is METER, who before other people were thinking seriously about AI capabilities and the impacts of them and how to evaluate them, METER was very prescient in coming up with a benchmarking approach in time horizons that just worked way better and aged way longer than other people. And that's because they were really trying to actually take seriously what was going to happen in the future with AI and then work back from that to what sort of work they should be doing now that would age well and prepare for that. So that inclination is really, really valuable to us. And it's useful in basically all profiles. It's useful both in a person who might found an org, but also in a person joining the existing orgs. That's something that existing groups in the AI safety space value a lot because it's a big part of how they do their work.

Can you do one quick double-click on where money still is a bottleneck?

I think the biggest thing is just like, for organisations and in areas that CGE doesn't operate in, money is a big bottleneck. So if you find some person who you think is doing great work, and for some reason they either aren't a good fit for CG, or they applied to CG and we weren't able to fund them, or we fucked up and decided not to fund them. I think there's just still alpha in that, especially because as we move into making much bigger grants, that can sometimes come at the expense of being like, covering all of our bases of all the teeny little uses of money that can be, that can be valuable. And I think that's the right decision for us to make, but it does leave some things that are unfunded.

Prakash had raised the prospect of an Anthropic public offering and his expectation that some of the proceeds would be distributed through coefficient giving.

So what, what is your preparation process for handing out larger checks? Like, what is your pathway for the next year, say?

Yeah, I mean, I have no idea what's going to happen with the AI company money. In terms of what the preparation actually looks like, I mean, I think the biggest thing is just seeding organizations now that can then be ready to absorb lots and lots of money. And so making small grants to new groups and then giving them a little bit of runway to prove themselves or not prove themselves and explode and That's the biggest thing we're doing. And we would love to create opportunities for— we would love to create basically groups of people that can say credibly to big donors, we have people, we have a project ready to go. We're just waiting for you to sign the check. We've got this shovel-ready project and it's going to take $1 billion and it's going to solve alignment. And it's just a matter of giving us the money. And that's really not the situation. right now in that there are just like, there are just like not that many people in the space. And so that's a big thing that we're trying to fix.

And is that about funding people enough so that they can work on things at small scale and such that they can get a much bigger compute check at some point to scale that? Is that the kind of idea?

That's the main thing I have in mind. I think the same logic also applies to Growing the headcount of their organization, having a group in place that has a great plan where they can then absorb a lot of software engineers or ML researchers or data labelers or some other type of person. But yeah, I mean, I think compute, like you said, is the main version of that. And especially, hopefully, AI labor, right? It'd be really great if there were a bunch of well-scoped, problems, uh, that these organizations had where they were just waiting on lots of AI labor to pour into them. Now we'll see if we can make that come true. And I think there are reasons to be skeptical that that will happen, but it'd be nice if it did.

Part 4, the price of compute. Wayne Nelms is co-founder and chief technology officer of Orin, which publishes a price index for GPU compute built from cleared rental transactions rather than list prices. The Intercontinental Exchange has announced plans to list futures on it, pending regulatory approval. Prakash asked him to follow a real customer. Who lends against the chips, and who loses money when prices move?

So generally speaking, you have a bank or some sort of financier that will lend money against the GPU asset or a host of AI infrastructure assets. What they hope is that they will make their money back over time with interest. As a result of the operating activity of a NeoCloud. So for example, these are companies like CoreWeave, Croson, Nebius, Lambda, right, etc. However, these business models themselves are contingent on really 2 things, right? Continued AI demand and specifically an ever-increasing or at least stable compute pricing estimate. Right. These GPUs are financed in kind of a variety of ways, including straight line depreciation, which we can get all into. So the way Oren provides value is that you can't really hedge away this risk without having a benchmark to hedge and having a benchmark to trade. So we created that benchmark and we are actually leading the transactions on that benchmark for these institutions.

And how many transactions would you get in like a month which form the index?

Yeah, so we collect over 1,000 transactions a day per index. And so right now we have 5 public indices. So, you know, it's 150,000, somewhere in that range, probably per month. We have a whole host of Neo Cloud partners that we work with effectively that are contributing data to our index every single day. And the question is always, why do people feel the need to show that information, to display that? It's because they benefit, right? A lot of these players are looking for cheaper financing, and cheaper financing comes when their financer has more certainty about the future and can hedge that risk in the future.

And so it's actually this nice flywheel where you get more data, you can publish a better index, the financiers feel more comfortable, I ask how fungible the underlying asset really is, whether an hour on an H100 from one provider is the same thing as an hour from another.

I think it's certainly obvious when you talk to anyone deploying infrastructure or buying that an H100 or a B300 is very different depending on the OEM, the supplier, the actual— everything about the hardware can be very different. very granular and very different from an engineering point of view. However, our take has always been that there needs to be a benchmark that allows some sort of abstraction for this asset class. I think for us, what's made the job much easier is that largely speaking, we are looking at one chip producer, which is Nvidia, which dominates the market, has huge market share. When we look at the Nvidia moat, and we can get to this later as well, but You know, you have the hardware, you have the software superiority. But I think the biggest thing that Nvidia has in terms of a moat over other competitors is the financing landscape. When you're looking to finance your new cloud and you're thinking about what chips to put into your site, it's just materially better to have Nvidia GPUs. It's just easier to underwrite. And I think what we see as this market problem is how do we allow every single person looking to deploy compute infrastructure this sort of almost a protection for their lenders. We right now only reference Nvidia reference architecture with InfiniBand, et cetera, et cetera. There are certain steps that we're taking to standardize the compute that we're tracking. However, I will say we are tracking a subset of the market which might inherently have differences amongst those participants.

In some commodity markets, there's this idea of baseload where you have a large amount of baseload and then you have this idea of load that gets spun up on demand. But how do you differentiate the pricing between the kind of baseload versus the swing capacity? Because I imagine the more month-to-month transactions are the swing capacity and not the baseload itself.

Yeah, I love this question. So yes, you're exactly right. A lot of the capacity is long-term PPA-style contracts, and I think it's really interesting to think why that is, right? Like, we're in this situation where AI capacity is so scarce. I think the model providers, so OpenAI, Anthropic, have seen this coming forever at this point, and partly they're to blame for the lack of supply that's currently out there. If you think about it, for them, it's an arms race, right? Compute capacity planning is an arms race. How much capacity can I lock up over the next few months so when I need to train the next model, I have enough and I don't have to go out and looking for more? It also, of course, is naturally a zero-sum game where whatever you buy, your competitor can't. So actually the way we like to think about the baseload and swing/day ahead market structure is in compute, people are buying to fill the peak demand and then selling to satisfy the troughs, where in power you are buying to satisfy the troughs, uh, these, these long-term contracts, and then, uh, buying excess to satisfy the peaks. So there's this actually robust on-demand market that will continue to grow as these trading labs are selling back excess as inference providers are selling back excess. It's this reallocation of compute question rather than purely a hedging question.

How elastic is the market? Obviously, we have in an extreme example, Anthropic reportedly has paid up, I think, even a multiple of standard market prices to Rent at high scale from xAI. How would you sketch out the curve from, if I want one H100 hour right now, I pay X. What if I want 10 million? How much does the price change as my scale as a buyer changes?

Yeah. So we like to think about this in 3 axes, right? So you have price on one axis, you have duration So length of contract on one axis, and then you have quantity on another one. Generally speaking, as you increase the length of the contracts, the contract per hour value goes down, right? You're— the logic here is you're buying in bulk, right? You're allowing someone to finance a data center because you're buying 10 years or 5 years of capacity from them. Generally speaking, when you increase the number of GPUs you're renting, the pricing goes up. Which is almost anti this exact logic, which is you know I'm buying bulk. Why am I paying more? And it's purely because the number of suppliers that can sell all that capacity at once is very small. You know you might be able to get a few nodes excess here and there, but when you're looking for ten thousand, hundred thousand GPUs that are all interconnected or at some level, that is very very tough. So you know certainly these Anthropic space or SpaceX AI deals are. few and far between. Uh, they're very much, um, OTC trades, I would say. Uh, they are indicative of what we are kind of seeing as a general trend across even smaller markets.

It is striking that, like, you look at the graphs and they are all trending up. So that is seemingly in significant tension with the idea that the longer you buy, the lower the price is. Like, Why are the sellers not AGI-pilled enough to expect that they'll command a higher price for the same chips next year than they are right now?

Look, I think the sellers of compute are the most AGI people because they have entered a business where they are selling at least a resource that hopefully rises in cost in the future. But for a lot of sellers, it's not because they don't want to. I think there's a lot of suppliers of compute that are experimenting with shorter contracts because they know, you know, why am I locking in a 4-year deal if in 2 months that capacity's worth even more? The issue here, of course, is risk. It's always this risk-reward ratio or, or balance that they're trying to play where even, you know, early financing is contingent on having a long-term contract signed. I cannot finance a data center Today, if I don't already have 5 years of offtake from a credible counterparty signed, and because of that, I can't even enter into the month-to-month world. So if you actually think about what the futures exchange and the— our product enables, it enables an ability for, instead of the NeoCloud locking up a 5-year contract, they can sell month to month and hedge the future price potential on our platform. Or through our index, right? And that is a much, potentially a much better risk-return kind of math for them and for their TCO and for their lenders.

So I think the one thing that, you know, I think Nathan has also alluded to is that you have differences in pricing of, you know, near-term and much further out, like longer term. And you kind of described the use of a lot of the hedging as tools for financiers, and the financiers are often concerned with the far end, not, not so much on the near end where they have much more visibility, but the far end where you have this depreciation issue. How do you think like the index assists or will assist in the future on this end of, end of life kind of depreciation issue, which is like 4 years out, you know, 3 to 4 years out?

Yeah, 100%. Like you mentioned, a lot of the risk in this space is certainly long, I would say long tail risk in terms of looking at the tenor. And so when you launch these futures on a futures exchange, right, a lot of the trading can occur in that long tail kind of section of the curve. Right. So these are monthly contracts on ICE, to be very clear. We expect to see lots of trading in the front month as people are buying and selling. Especially after physical delivery. But even like, I would expect there to also be a lot of trading in those like 3 to 5 year spans in terms of that is where all the exposure is, that is where all the risk is. And because there's now futures contracts that reference what spot prices will look like in 3 to 5 years, that is what any bank is exposed to. Right. And to be even more clear for people watching, it's You have a bank underwriting a deal where the GPU life is 6 years, but the initial contract signed might only be 4, right? Or it might only be 5. And so there's always that what happens in year 4, what happens in year 5, what happens in year 6 when there's not a contract already signed for that GPU, but on your balance sheet you've claimed that the GPUs have some value. So that is the area where you want to hedge your bets.

Prakash raised basis risk, the gap between what an index says and what the asset you actually hold is worth. If that gap is steady, a trader can price it in.

But the issue is going to be if there's non-systematic basis risk. And one of the issues with GPUs is this introduction of new technology, which obsoletes the old, and which is what I think the non-systematic basis risk That I think is, you know, pretty scary for, uh, financiers.

We are tracking GPU-specific, uh, transactions. So we have a B300 index, we have a B200 index. And so actually this entire risk of, you know, a financier being worried about obsolescence risk, that is exactly what you can hedge away with this instrument. That is exactly what you can kind of look at how the market is pricing the 12-month or the 15-month future contract for the B300, right? It actually, if anything, allows information to be spread wider, broader, and more evenly because now you can kind of see perhaps where the market is pricing the announcement of the next chip, right? Based on how the curve looks in the future. I think all these things are ways for information to be actually shared more freely rather than for there to be any obfuscation of this And so in effect, I think these markets are a great way to hedge that risk away.

I also asked how long these chips actually last and whether the A100s from 2020 are near the end of their useful life.

Yeah, I think the chip health is a super interesting space to be in and understanding what are the failure modes for the hardware itself. Chips will fail all the time for a variety of reasons, but I think what you're pointing to is the useful life conversation of hardware. And this is something that we wrote a paper on just 2 weeks ago, I think now, about how open source, open weight models that are more price elastic will often route to places where the cost per intelligence or cost per workload makes the most sense. And oftentimes those are the older pieces of hardware. because they're just less marketable.

Right.

And so we actually see the useful life extending beyond 6 years as a general thesis. How much further than that? It's kind of unclear. However, I will say that what we see in the forward market and the term structure is that there will always be some sort of premium above zero, certainly, and above some terminal value of power. Right. Like, the terminal value of an A100 is not just the value of the power that powers it because the A100 will always provide some computing power as a fraction of all computing power in the world, right? Over time, I think you have a lot of interesting use cases like physical AI, uh, at one of many where these older chips might become more effective. However, today we see still that these older chips are still being used, still being rented, and have maintained relatively constant demand.

In the last 12 months, Wayne signed off a few minutes later. In the closing, with just the two of us, Prakash came back to the index with an account of how the biggest compute deals get priced. The figures are his from what he has heard.

What I've heard is that your price of compute is dependent on whether you have financing or not. So if Someone needs to use their balance sheet, they get to dictate the terms. So if they don't, if you don't have to use a balance sheet, then all of a sudden you have high pricing. So Elon, for example, Elon had already used his own balance sheet to build out the clusters and he had the clusters available. Available, you can take it tomorrow. And they didn't need to provide him a long-term contract. They could do a month-to-month contract and they can walk out at any time, which meant that this is not financeable. A bank cannot come and give Elon a loan based on this contract from Anthropic because a bank wants to see, if you're going to pay back in 4 years, I need to see that you're going to have cash flows for 4 years. I need to see that upfront. So Elon's able to charge high prices because he's using his own credit capacity. He's using unsecured credit at the holding company. And he can say like, look, you're getting paid $80, $90 million a megawatt. You give me the $50 and you're still making money. So that's a good deal. And so Elon got $50 million a megawatt. So I think it changes, it starts to change when you need the other guy to give you support. So if you say like, okay, If Elon said, okay, I'll build it for you in a year or a year and a half or 2 years. And in order for me to build it to you, you need to give me a 5-year contract. And if I build it for you, you will definitely buy the compute and pay me. And then I'm going to take that and I'm going to go to another bank and I'm going to give it to them and they're going to lend me money based on that. Then it becomes like, it's not really like Elon's money. It's really like Anthropic's money anyway. And it's the single credit is going to be from Anthropic side. And that's really who the bank is relying on for repayment. And so then it becomes a different pricing. So then Anthropic will say like, okay, how much is it going to take you to build? How much is your interest rate going to be? And then I'm going to give you, let's say, a 20% profit margin on top of that, right? So this is what CoreWeave is getting, for example. So if it's going to cost you like $15 million or $20 million to build, they'll be like, all right, $24 million. $25 million per year. And so that's the key difference is whether you have the money or not upfront. And those deals don't hit the index pricing. The index pricing is based on willingness to trade. So these are all tradable prices and that stuff doesn't get traded. And so the people transacting on this index are the lowest payers. They're not Facebook. Facebook can afford to pay like $100 million a megawatt. They're not going to be trading on the Orn Index because the Orn Index is for people who are willing to sell compute at like $10 million or $15 million a megawatt. They're willing to sell the compute. Facebook isn't going to sell you compute at like $15 million. They're willing to buy it at $50 million. So these are, in some sense, the lowest pricings, but it's immediately available.

Part 5, what the machines are actually doing. Nick Gillian is co-founder and chief technology officer of Archetype AI, and before that led machine learning research on Google's Soli radar sensor. Archetype's model, Newton, is a foundation model for sensor data. It takes raw streams from radar, vibration, electrical current, and cameras, and reports what a machine or a worksite is actually doing. I asked him to walk through Newton from the bottom up.

Let's start with the data. How much data is out there that you were able to just collect on your own? And then I imagine you must have had to establish a sort of partner network to bring in a bunch of data. Tell me about the data process.

Yeah, so we're getting close to a billion hours now of physical AI data that we've been able to scrape and gather and collate. In many cases, if you look on the web and you look at the different types of physical data that's there, it's very sparse. There's very little multimodal data. It's not synced well. It doesn't cover things well. For most techniques, particularly classic supervised techniques where you're basically building one model and you have to have a fixed number of sensors and they all have to be captured with the same sample rate, et cetera, That's a massive problem. We've been trying to flip that on its head and say, if we have all of this physical data which is captured in different sensors and different systems with different contexts, how do we actually build the most advanced self-supervised techniques that can pull in that sensor data and extract from it in the same way LLMs have done with language? We are able to then join that and mix that in with a small amount of data we can capture from partners and even in some cases that our customers can give us to be able to really extract and understand and map that to physical language and map that to physical control, right? So there's quite a bit of resampling and really understanding missing values in sensors. Is that because the sensor had an issue or is actually the machine itself is connected to, that's actually a feature that helps explain that machine is about to break, right? The reason the sensor is giving you all these NaNs is not because the sensor is broken. It's actually a feature of the machine, right, in those cases.

So the area that I've studied best that has done something kind of similar has been at the intersection of language and just images, photographs. And we've seen kind of If I had to borrow some of their terminology and map it onto what I understand you're saying, this would be sort of a late fusion approach where you have a language model, you then do a pretrained from the ground up encoder, specialist encoder for these different modalities, and then you're doing some sort of cross-attention between those. That's my understanding of, at a high level, obviously, of what the image folks have done. How much have you been able to follow in their path? How much of the techniques you've seen them develop have worked for you, and how much have you had to go back to the drawing board and reinvent?

We've definitely tried these approaches, and we've had some success with them with other types of sensors, radar and time series and so forth. But the big issue that we have found is twofold. Number one, there's an extreme amount of language-sensor pairs. On the internet that can easily be scraped and built into these very large datasets where you have this pairing between an image and a description of that image. This does not exist for radars. It does not exist for time series sensors. It does not exist for all the complex systems that we go and talk to our customers with. We have had to look at different approaches of how do you, for example, take time series and actually align that with human language. That's a large part of our research work is how to actually solve this sensor alignment problem. And there's also the case there where with a single image, you may actually be able to get the full context, for example, of the scene and then align that with language. But actually with the time series, you actually need to look at a temporal window. And what does a single latent mapping to a single word really mean in those cases? It's not like you have a time series signal that says dog and you can align that with the keyword dog and so forth. There's quite a bit of work we need to do there to solve this alignment problem of how do you generalize this across many sensors. And that's the key kind of components we're building inside our models, which makes them different from a standard VLM or other techniques, right? Because we simply don't have that corpus of, say, time series and language pairs to kind of apply that previous recipe you mentioned.

Prakash asked about one deployment, a project with Kajima, the Japanese construction company, and what the inputs and outputs of the model look like there.

In that case, you know, we're working with Kajima, this very established Japanese construction company. They build, you know, islands, they move mountains. The project we were working with them on was literally taking a river and moving the river, right, so that it wouldn't flood, right? So it was a 5-year-long project. They had sensors the whole way down the river as they kind of moved down the river, and they're dredging it and they're widening it, and, you know, they're basically mitigating the risk of flooding, right? And the big problem they wanted to solve is their company is outsourcing and subcontracting a lot of the work to a lot of other sub-teams, and they want to basically measure to make sure that those teams, if they have to go and dredge the river for 5 hours, did they actually do 5 hours of dredging that day? Or actually they only did 2 because maybe the digger was in maintenance mode that day, for example, and they couldn't do it, or because of weather conditions and others. So for that project, basically the input to Newton, our world model, was The multiple cameras around the site, typically 3 or 4 cameras looking at a specific section of the river. And when you look at those cameras, basically you have a barge, this boat that's floating out on the river. On top of it, you have an excavator digger. And then on top of that, at the end of it, it has an end effector that is either drilling or it has the bucket on it where it's actually dredging, right? They want to know how many ours, essentially, that excavator is actually dredging the river, not drilling the river, not a crane, not other things, right? So essentially, the input to the system is the sensor cameras, the river sensor, hydrometer sensors that basically say how high the river is. And then at different geo locations along that, I think it was multiple kilometer site, where the actual work was supposed to be happening at that day and what the weather was that day. So we basically have a bunch of cameras, we have a bunch of time series sensors with weather and river information. It's fed into the model, and the model is essentially outputting what looks like a Gantt chart, right? It's actually outputting the state of that team. Where is the barge? Is it parked on the side of the river? Is it actually moving in the to position, which can take an hour sometimes to do that. Is it now actually in position and they're starting to dredge, or is it just there and it's stationary and nothing's happening, right? And then they can start to map these work charts over multiple years of data and start to look at these bigger patterns that a human can never spot. Like one of the patterns they find, for example, is obviously if there's a storm, the work on the day of that storm goes down. That's kind of, you know, everyone would know that, right? But one of the key things they found is multiple days after the storm, when the weather had cleaned up, the work productivity of the team was still very low. And the reason for that being is the river water coming down from the mountains, it takes several days for it to actually reach the end of the river and go to the ocean, which is where the site was. So there was a lot of debris and a lot of churn coming through the river multiple days after these larger storms, which is actually slowing the teams down. quite a bit. They'd never spotted those patterns before because they could never look at these aggregate patterns, but Newton was able to kind of spot those patterns for them, right?

How much does Newton generalize in the sense that when you go into a new site, do you have to take all of the data and kind of fine-tune the model once more?

Yeah, so we are building the model so that out of the box you can plug in different sensors and different configurations and the model can automatically adapt to those. as much zero-shot as we can. For a subset of users, they can directly do that. The typical analogy I give customers is, if you showed this task to like an average person from the street, would they be able to detect the safety violation? Or would they be able to detect that that looks like a strange signal and anomaly? If the answer is normally yes, normally that means that Newton can do it out of the box. It doesn't need a specific master technician's level of expertise yet to be able to spot that.

Got it.

about that, right? For those cases where maybe the piece of equipment the user is using is very specific to, you know, that factory, for instance, or they have certain parts that, you know, they have very internal and, you know, unique names for those processes or parts, that's typically where fine-tuning or being able to bring the customer's knowledge base and actually being able to inject that securely into Newton really helps. So In that case, our platform allows customers to take our model, run it directly on the customer's infrastructure so no data is leaking out of their networks, either on-site, on-prem, or in their secure clouds, and then be able to take their own data and actually do some amount of fine-tuning to be able to customize the models. And typically with customers, we've seen, you know, on the order of a few thousand samples is enough to be able to take the general model and then adapt it very quickly to that customer's use case.

Last question for me is about your vision of superintelligence and how you might relate to the frontier companies as we go forward in time. What I think is, geez, the reasoning is already getting to superintelligent light with all these Millennium Prize problem style results happening just through super long chain of thought. Like, who would have believed you could get there? So on the reasoning front, we've kind of come awfully far. What we don't seem to have still is a good intuition for the many domains that matter. And what I sort of expect to happen is something similar to what has happened with images, where it's one thing to sort of put an image into a model and get a caption out, but when you can get the altered image out and it's clear that the model understood both your verbal instructions and also sees the image in a very similar way to the way we intuitively see the image, which is something we are really good at, then it feels like, wow, okay, it's really got that domain. So my superintelligence vision is basically do that across a ton of different domains, many of which we don't have native senses for, with what you're building being a great example of that, right? People don't have a native sense for how to integrate 10 signals from 10 different sensors. Do you see a role for you in superintelligence being sort of building the deep intuition for this domain? And does that ultimately get joined with elite reasoners via the form of a bidding war between a couple of frontier companies for your company, for example?

I definitely see the part that we can play in a larger That's how we're seeing where we play in this bigger superintelligence loop. We're very focused on how to understand the world around us and be able to bring that back to natural language and machines so that these things can talk to each other. If we can do that and other folks are solving digital intelligence, that nicely will connect together very well.

Just to make sure I understand your answer there, it sounds like you are envisioning a world where the technology that you're building remains sort of a tool call type relationship to a core reasoning model and doesn't so much get upstreamed and thrown into the GPT-7 pretraining mix to spit out one model that does everything in the same latent space.

We're thinking of it more on the line of we're building physical agents that can be deployed out to the ecosystem of the world. Those physical agents then can talk to other agents and be able to bridge that gap between the digital world and the physical world. How much that's actually done on an agent-to-agent post-training mechanism, how much of that is pulled into the pre-training, we get to see, particularly when it comes to those more nuanced sensors like radar and LiDAR and infrared and all these things that don't look like human You know, visible images. That'll be interesting to see how that's brought back into these kind of bigger pre-training loops versus just— it's more of an agent that can go and solve that problem and tell you what's happening in the physical world in real time and be able to let you interact with it, for instance.

Part 6, the dish is not the body. Andrei Iorgescu is co-founder and chief executive of Vivodyne. Which builds robotic labs that grow pieces of human tissue—lung, liver, tumor, and more—from primary human cells and tests drugs on them. In August, it opened what it calls a human biological data center—12 robotic labs that, by the company's count, can grow more than 3 million tissues a year. Prakash had the first question.

How, how do you actually, like, figure out What the dosage is, and whether or not, you know, that is actually what the cell would encounter in the body.

Basically, everything that's been done preclinically to date, right, outside of testing in animals, you are just dunking, you know, some substrate, whether it be an organoid or you have cells in a petri dish, just basting them with this test compound that you want to test. And, that's exactly the problem. is like, that's not how transport happens in the human body, right? And I think it's a perfect example. A lot of cell therapies that are designed for solid tumors in a dish will completely kill that tumor. And then those same CAR-T cells, when they go into a patient, just flow right by the tumor through the blood vessels. They don't even know it's there, right? And thus is the challenge, is modeling the transport of that therapy into the tissue Do they recognize the tumor? Do they concentrate there? And so, in Vivodyne tissues, even the blood vessels, the kind of capillary bed in that tissue is self-assembled. So, these little blood vessels grow within the tissues and we can perfuse directly through them. And, whenever we dose a drug, it is always through the blood vessels that are native to that tissue. And so, the transport through the interface of the blood vessels and then the meat of the tissue, all of its like parenchymal, its functional subparts is just like you would find in a normal tissue. The density is the same, right? And so that is just as big a part as does the drug molecularly like bind to its target and perform this effect as does it even get there?

Then my question on scaling laws.

I'm very intuitively bullish on this just go collect an obscene amount of data on this kind of tissue perturbation paradigm. It makes a ton of sense to me. The sceptical view that I've heard from time to time is you can never have enough data. You know, the world is too big, biology is too complex. You can do that, but you just won't be able to generate enough.

The challenge of this argument that it's just like this, just a size of dataset thing is that currently even the state-of-the-art virtual cell models that exist saturate at a very, very small fraction of the input data that is fed to them. So you can have all the data you want, but their performance saturates after a couple percent, right? And this is like, you know, both published and kind of what we hear on the ground, right? In these PerturbSeq kind of based models. The reason it gets so difficult is when cells are growing in a dish, which is the kind of mode for this sort of Perturb-seq, they are so far removed from all the feedback loops that are natural in people that they're just trying to colonize that piece of plastic. It's like you have a stiff substrate. Normally, cells, when they encounter that, are like, I have to encapsulate this, and so they are just growing. Their only goal is to proliferate. If you knock out genes in those cells that would otherwise uh, affect how they perform their natural function, really there's no big difference, right? They're still just dividing. Um, and so you end up learning this view that perturbations don't really do much of anything, or it's unclear what they do, except if a gene is like required for that cell to live, in which case like it kills the cell, right? Um, and so, and so it's not— there's not even causality to extract from that. And so We think that the fundamental bottleneck to having this causal understanding of biology is that we need to perturb these tissues in the complex environments that they call home naturally. Almost like the image in my head is we are trying to populate a map and kind of design a GPS through it so we can say, if I want to get to the state, I need to go down this road and then turn left with a second perturbation. And that'll bring me to this healthy state.

I asked how you get from a sample received to something placed in the system and growing.

And when we grow these tissues, you know, a really big differentiator for our approach to doing this, and I think the, you know, really like a critical mechanism that you need to get like realism in this, right, is that we use primary cells, which are mature differentiated cells that perform our natural organ function in our bodies as opposed to starting with some stem cell completely absent from all the conditions that would tell it how to mature into a differentiated cell and then sledgehammering it with cytokines to turn into some cell type that we think is accurate to what happens in the body, right? We instead just kind of go to the mature source, the ground truth, and we mix together all of these different cell types that comprise the lung, for example, at very high density. And, within that injected tissue, we see that these cells begin to reassemble into the native structure of that human tissue with some little herbs and spices that we introduce. But, the responsibility for forming the native structure of human tissue is on those cells that already know how to do it. It's honestly kind of injection molding into a chamber, right? So, we shoot in these cells at high density and then over the course of several days, they reassemble themselves. They just self-organize to make blood vessels to partition into, in the case of the lungs, the airway where there's air and then the parts where there are blood vessels and fibroblasts and immune cells and all of this. So we are relying extremely heavily on the cells to be able to direct their own self-assembly at these very small scales.

Can you tell us a little bit about the mix of in silico via the foundation model versus in actual tissue experimentation today and how that ratio is expected to evolve? Because presumably the big value is It's a lot cheaper, faster, even if you've got all these great wet lab technologies. Just ploughing things through the foundation model is supposed to be where you get the insane acceleration, right? But then, of course, you've got to ground it and validate.

That's exactly right. So if we look at it like a reinforcement learning problem, which is my actor is this— well, it's kind of the experiment, right? I have this tissue, and I can I can perturb it in very many different ways. And, the goal is I have this objective, which is to make it healthy, and I can instrument the state of that tissue very deeply. So, I know the value function is very well-defined in terms of the state of that tissue. The question of what should I try next, right? In the large rollout that I can perform, what are the most useful things to try next that add the most information to my dataset. So the physical experiments kind of serve to fill in gaps in which the model is most uncertain. And the value of the model ultimately, right, like why even make these foundation models, is the combination space, especially when we move into 2-target or 3-target therapies, right? If we have just combination therapies, if I take 2 drugs because it's going to work much better than one, is so huge that you have— I mean, you could have— this whole planet could be Vivodyne systems growing human tissues, and you'd still not get anywhere close to the sheer space of things to test. And so, we need some way to narrow down to kind of just a couple times like the total number of clinical trials in America, the experiments that we should physically run to kind of test our hypotheses, right? And so basically, these foundation models allow us to predict what happens when we drug this thing and that thing, or all these things together, or this thing and then that thing. And we kind of optimize and we iterate in that space, and we find, for example, the 1,000 most likely things that will move this tissue closer to this healthy state. And then we close the loop by testing it, right? And we see where that rollout goes. And then we just do that on a loop, right? So the purpose, very fundamentally for us, of these models is to tell us what to do next. And then the purpose of this experimentation is to figure out if the thing that we thought we should do next actually does move us closer or how it deviates so we can correct.

Prakash asked about the hardware. The tissues grow on what Vivodyne calls a tissue disc, and the company recently announced a second generation of it. He asked what changed from the first disc to the second. Andrej started with the robotic lab the discs go into.

Imagine something the size of, like, a wardrobe, right? So it's about the footprint of a large desk, about 8.5 feet tall. And these are self-enclosed labs, so fridge and freezer, and there's a, you know, 3D scanning microscope in there and, you know, big robot arm that can use all these different tools. You want to optimize the surface area of those discs for tissues. And so in our, in our second form factor, we have an increase anywhere from 2 to 4 times the density of tissues that we grow on each disc without any sacrifice to— they're actually much larger than the version 1 of these discs. So we increase the number and the size of these tissues, and we offload a lot of the responsibility for how they're grown to the robotic system around them. Now, the challenge there and the kind of work that allowed us to do this is a lot of work in software on the orchestration layer. You think if you had 10,000 experiments to run in parallel, and this is not like I have a different gradient of dosing for some drug or something, right? I can do it on a plate. It's like I am changing the collection point of data. I'm changing the interval between dosing with some drug. I have a first-stage regimen and a second-stage therapy. I'm taking all sorts of different readouts of these tissues. It becomes a super, super complicated experiment. And, I can define it quickly by— I can explain it out loud and say, well, we want to test bone marrow and airway. We want to dose these drugs in this order. We want to branch to these concentrations. And I want 3D imaging, single-cell sequencing, and proteomics. And by saying that, I can define the study decently unambiguously, right? I mean, given the system of tissues that we grow, it's pretty unambiguous what I would want to test in there. But then the robot has to be able to decompile— or sorry, to compile that kind of zipped representation of this study, it has to decompress it into this compiled list of like millions of actions that this robot has to perform. It has to remember that if it took the lid off of a plate, that it should put it back, right, as an example. But if there's multiple trays that it has to handle and they're all out on the working deck, maybe it should delid multiple of them in advance, right? if multiple pieces of plasticware are on the same tray and they're coming out of the freezer to thaw, you want to make sure you're not thawing something that should not be thawed too early. And so all like this, this, this compiler behaves a lot like a compiler in, in code, right, for C, where you're managing like the prefetching of information from memory and, uh, branch prediction, like all this stuff, right, that, that like thankfully our operating system And, you know, hardware handle for us. But improvements in our orchestration mean that the system itself, the hardware, the automation, can do much more of this. And more and more of that space on these tissue discs can be used for actual tissue biology instead of having to integrate like the little, you know, helpers on there.

Toward the end, Prakash read Andrei a grand challenge for the field, which he credited to the Annual Review of Pharmacology and Toxicology.

On the Annual Review of Pharmacology and Toxicology's Grand Challenges in Modern Pharmacology, the innovation, a universal in silico or organoid framework that forecasts rare patient-specific adverse events with more than 95% accuracy before first-in-human dosing, is one of their Target challenges. Vivodyne is obviously on the path there. When do you think this challenge will be solved?

So this is a hot take, but I feel like this is almost like picking the color of my Mars suit currently to me, right? Which is like, how about we just like kind of fix the bigger problems? And the big problem— by the way, we are working on specifically that challenge. Right? Of of of of of like very very rare outcomes. But a bigger, like far more regularly, we have drugs that would have huge potential in their efficacy in in patients, but that have routes of toxicity that are pretty common. Like many patients like die from it, right? I mean, many of the cancer drugs that are going to clinic today, the the risk is not like ah man, you know, I've I've cured 999. people have cancer, but this one patient, like, it didn't work so well. You know, it was more toxic. The problem is, in all the patients we've dosed, they barely had any improvement in their cancer, but it gave them all of these side effects, or it killed someone from liver toxicity or something.

Andrej signed off a few minutes later. In the closing with the two of us, Prakash put the two guests side by side.

I liked how you had basically two different approaches for data. You had, you know, on the one side, you had, let's take all the messy data, let's feed it into the machine and try to make sense of it. On the other, Andrej is like, there's just too much messy data in biology. I'm going to build my own data collection effort, and I'm going to monitor every single thing that goes in and comes out so that I know exactly what's going on. And I think Andrej probably has a good chance of solving this thing. Like, you need, you need a little bit more GPU power, you need a little bit more imaging. I think the imaging granularity is probably not there yet. But you can definitely see if they manage to scale a little bit more. The real question is like, do you need to do 3 billion samples, 30 billion samples a year? Do you need to get everyone's genetics? Or are you gonna get— is your scaling going to be able to get you to a point where at 30 or 300 million or wherever, it just starts to generalize a lot more, and you stop needing so many different samples.

Prakash guessed it would be 2 or 3 more years before AI gives biologists something they can really use. I took the under.

I think we're going to see probably a reversal of the order of utility and Millennium Prize solutions in biology relative to what we've seen in math. The The issue with math is that we already had a ton of math that was interesting to mathematicians but not super useful. And so to push the frontiers of math, you kind of had to get into the not useful domain of math. In biology, we have tons of useful stuff that is still not even that well understood, right? I mean, it's just very— it's a far more empirical than theoretical domain. And I would expect a tremendous amount of utility to emerge before we have anything approaching a full mechanistic or fully causal understanding of what's going on in biology. And to some degree, obviously we'd love to have that, but we don't necessarily care at the level of, does this cancer drug work on this patient's tissue? If it works, you don't have to have a mechanism. That's not part of the— clinical trial approval process. It's nice to have, not required.

Prakash also made the case for specialists. A model like AlphaFold does its one job better than any generalist. He expects that to hold. And the version of superintelligence people fear is a single model that does every job better than the specialists and depends on no one. My answer was the pattern as I see it so far. New kinds of data get pulled upstream into the biggest model, And efficient specialists get distilled back out of it.

But yeah, put me down for one that has not seen a reason yet to believe that all these modalities don't end up integrated at the frontier. In a way, I wish it wasn't going to go that way. I do think safety through narrowness would be really nice. That's why I'm excited about Jev in part, because it has a very niche role to play. You could build all kinds of scaffolding around it. I was a big fan of the Eric Drexler reframing superintelligence piece, which he also known as comprehensive AI services, where you've just got narrow purpose-built AIs for all kinds of different niches. My hope for safe superintelligence is something like that. Just based on kind of reading the tea leaves of Ilya's comments, could we possibly get a model that starts as kind of a stem cell generalist, And then matures in a way where it gets really good at its job, but also kind of settles into that niche in the way that mature cells don't revert and turn into other kinds of cells, and that they're kind of limited now. They've lost their pluripotency, so to speak. I love all that stuff. I hope it goes that way. For efficiency, it probably will. But at the frontier, I just have not seen anything at all yet that makes me think that the single biggest, best teacher model you could make couldn't just learn it all. And given what we've seen, that starts to be a bit of a scary beast.

I agree, it can learn it all, but I think learning it all at the most efficient compute rate and inferring at the most efficient compute rate, which is what you alluded to with the distillation, I think that doesn't happen, right? So the energy efficiency is kind of key because the energy efficiency means that you have this diminishing returns curve as in economics for getting bigger. Okay, the doomer scenario is less likely to happen because you will get a more intelligent model, but it's going to be less efficient to start out with and it's going to be more energy consumptive. And that means that it's going to need more energy expansion to get there. But the other models are going to be more efficient than it and more numerous. But I think the singleton example is not likely to happen.

From your lips to God's ears, as always. I think that's a real constraint for sure. I mean, if you made a, let's just get ridiculous, right? You made a quadrillion parameter model, you certainly would find it a bit slow and a bit expensive, and you couldn't run a million agents at that scale of model today.

So I do think there are some practical limits there.

But then again, remember what we talked about on Wednesday? There's going to be more compute installed over the next 12 months than exists currently in the world now.

Yeah.

So we are definitely going to test for at least a couple, few more years whether just bulking up on compute solves all, quote unquote, solves or perhaps creates all these problems.

That is the week. Tell us what worked and what did not. See you in the morning.

Mama made us in the winter, one thing, one breath behind. Gave us near the same name, gave us the very same mind. You went east when we were ready, I went west and stayed. And every choice I ever made, you'd already made. Didn't have to call, didn't have to write. Whatever you were doing, I was doing it that night. 2 rooms and one mind, one mind cut in 2. You didn't have to call.

I'm sorry.

Call me, I knew, I knew. When they came around asking who'd been where we weren't allowed, I said me in my room, you said me in yours out loud. 2 confessions, one backdoor, and Mama let it go. What can you do with the 2 of us? I know you know. People say spooky, people say strange, but if you know how I'm made, you know what I'll say. If you know Oh, how I made you know what I'll say. When they said one of us would go and one of us could stay, I didn't have to ask you, you didn't have to say. I'm the one who braced ahead, so I said me, not you. My room went dark and yours stayed on. I knew you knew. Didn't have to call, didn't have to write. Whatever you were doing, I was doing it.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends. Post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.