AI:AM — Web Infrastructure and Superintelligence · August 26, 2026

As well, I asked a guy, what is the prospect for a startup if you really catch fire and you're growing super fast? Are you going to hit constraints in terms of your ability to serve users just based on the fact that where do the inference tokens come from? Or would it be— And his answer was, we got you. Basically, if you are growing fast, we'll support both, all the inference tokens you need from us here on the ByteDance code. So obviously I didn't test that at real scale myself, but this is very consistent with that. 100 trillion tokens a day is not a small number. And the fact that they are serving it on Chinese chips, I mean, this is just a report from SemiAnalysis, which I deem to be credible, but, you know, it's all happening pretty quickly, so we should keep an open mind to, you know, there could be additional facts still to surface in terms of exactly how this is happening. But yeah, AI didn't feel super safe. And if we're betting on a strategy that has as a load-bearing feature that China won't be able to scale their chip production and won't be able to run as many agents as we're running. It's probably true, but I don't think it's as true as people would have expected when they were mapping out these strategies. So in my view, it is maybe time to update and reconsider some of our China policies in light of the fact that nothing we've done really has seemed to deny them the ability to advance, and now increasingly the ability You're putting your money where your mouth is with a $1 million hacker challenge for the— for Cell Sandbox. This was recently announced. Maybe for an ignorant person like myself who would've thought that sandboxes were less hackable than it seems like they are until recent revelations updated my thinking. Give us a little background on why is it that the sandboxes are still so hackable here in mid-2026? And why are the agents not enough? Why do you need to go put a million-dollar bounty on it, given how good the agents already are?

Yeah, I mean, I think nothing goes beyond human creativity, right? So obviously they all have agents, but we exhausted our own Agentic Research. We also, we're not only doing the challenge, we have actually multiple pen testing teams on that, on the same matter. So it's both the crowdsourcing and the things we can influence directly. I think the sandboxes, sometimes they're seen in this very simplified way where it's like, oh yeah, it's a Linux micro VM. And they're inherently secure. If you actually take a look of how, for example, the OpenAI Hugging Face event worked is where the AI model found a side channel to have essentially write access on an external API. And so part of Vercel Sandbox offering is an egress firewall that makes it as easy as possible to, at that egress layer, minimize what can be done. So part of the $1 million scope isn't just to escape the Linux microVM, which I would be very surprised if that happens. More likely is that someone, for example, finds a way to defend, to circumvent the egress firewall, or they find a side channel in some kind of service that's always callable from the sandbox. There is ultimately a lot of, because we are adding functionality to these systems and you have this infinitely creative adversary, you have to really think outside the box and not just accept or hope for the best, right? I think we're past hoping for the best. And so I think that's why this is a very appropriate challenge.

And one of the criticisms that I have heard is specifically, okay, if you have next token prediction and you're doing paper replication, well, then the agent kind of has a destination in mind and it's kind of evolving towards that destination. Meanwhile, in science, a single negative result disproves the hypothesis. So you are not actually looking for that kind of predictable next token prediction. You're actually looking for outliers at that, okay, this is wrong, or that is wrong, or this hypothesis is wrong, and therefore we need something new. How do you punish an AI? You can say that the AI broke a rule, fine, right? But You need some deterrence. Human systems, you have deterrence of civil or criminal penalties, right? And they can escalate over time. So what kind of deterrent system does an AI have?

I think there's a lot of work needs to be done on that sooner rather than later, probably. Tyler Cowen has a really interesting idea about just during agents to be capitalized. So that could be one very practical solution. I do still think you have challenges around how do you draw the boundary around an agent and how, you know, if this instance of this agent is found to be liable and its capital is docked, like, what does that mean for the traces and the memories and, you know, everything as you were just pointing out, like the ability to unbundle. an agent and reuse parts of it. Obviously, there's great utility to it, but in terms of preventing it or providing some sort of consequence that would deter that sort of reuse, I don't know of anything that really solves that problem.

Good morning. It is Wednesday, August 26th, 9:01 AM. Nathan, good morning.

Good morning. How are you today, Prakash?

I am very good. It has been an interesting 24 hours because we've just seen the first kind of— not the first, but maybe the disclosure of who was behind the OxAlpha model release. OxAlpha was a model that was released on OpenRouter maybe about a week ago, and lots and lots of rumors of who was running OxAlpha. And they had very generous limits. They had, I think, a trillion or 10 trillion or some insane number of tokens which were served for free. And I think they also kind of Probably hired some of the influencers to kind of boost their standing. And there were lots of rumors going around. And today it was disclosed that it was indeed GLM 5.3 from a Chinese company called Chipu AI. And it's also— they also said that they ran it mostly on Chinese chips, which Which is a real surprise.

Yeah, the Chinese manufacturing ecosystem strikes again, perhaps. Time will obviously tell on that, but that was an interesting observation when I was in China a few weeks ago. I was asking people, does AI feel abundant here or does it feel like it's scarce? If you are a Consumer, there's lots of apps. They're free. You know, I never hit rate limits. The only indication I had at any point as a retail user that there was some limit to the free intelligence was when Moonshot dropped K3. They had to put that behind a paywall, and then they actually stopped signups for a time because they had hit capacity and they just didn't feel like they could— deliver the quality of experience that they wanted to, to people that, you know, were signing up past a certain capacity constraint that they had. But other than that, all the models that I used were all free. I never hit any rate limits. And when I had the chance to speak to people at hyperscalers, I spoke to one guy in particular at ByteDance, which of course, in addition to TikTok has Doubao, which is their largely, I think, voice AI experience that tons of people are using. We talked about that a little bit with David earlier in the week. So they've got all these different flagship, they've got multiple flagship products, and they're also kind of turning into a cloud hyperscaler. It was very strikingly similar to go to ByteDance to the experience of going to Google for lunch. Like the aesthetic of the campus, the free lunch for everybody, the getting swiped into lunch, and the kind of few flagship products that are world beaters, the dozens of other products that you never really knew existed that don't necessarily have a ton of traction that are getting spun up. And I think I got the sense that maybe ByteDance kills their failed products a little bit more effectively than Google tends to. But then at the hyperscaler cloud level as well, I asked a guy, what is the prospect for a startup if you really catch fire and you're growing super fast? Are you going to hit constraints in terms of your ability to serve users just based on the fact that Where do the inference tokens come from, or would it be okay? And his answer was, we got you, basically. If you are growing fast, we'll support that growth. You can get all the inference tokens you need from us here on the ByteDance cloud. So obviously I didn't test that at real scale myself, but this is very consistent with that. 100 trillion tokens a day is not a small number. And the fact that they are serving it on Chinese chips, I mean, this is just a report from SemiAnalysis, which I deem to be credible, but it's all happening pretty quickly. So I think we should keep an open mind to, there could be additional facts still to surface in terms of exactly how this is happening. But yeah, AI didn't feel super scarce there. And if we're betting on a strategy that has as a load-bearing feature that China won't be able to scale their chip production and won't be able to run as many agents as we're running, it's probably still true, but I don't think it's as true as people would have expected when they were mapping out these strategies. So in my view, it is Maybe time to update and reconsider some of our China policies, uh, in light of the fact that nothing we've done really has seemed to deny them, uh, the ability to advance and now increasingly the ability to scale.

Um, on that ability to scale, uh, here's another report on Chinese memory technology. So China has memory producers and memory is, especially the lower-end memory, is not very technically difficult to do. And China has a company called YMTC, Yangtze Memory Technology. And they have been, I think, number 3, number 4 in NAND for quite a while. And they are aiming to be number one by the end of next year. And so they are directly competing with SanDisk, which is the biggest NAND producer. And they are aiming to be number one, overtaking all the others, Yangtze Memory Technologies. And the interesting thing is they're also expecting the IPO. The IPO, post-IPO, there will be the usual fluctuations in price, of course. This is a stake in the ground. It is something that has been expected for a while, that they would just flood the memory sector. And they are scaling, as you pointed out, they are good at scaling.

Yeah, I don't know how much I can really add to that. Can we buy into this company on the stock market somehow? I mean, that's There's been a lot of money made in these bottleneck trades. This seems like a company that's probably going to sell a lot of memory in the Chinese market. I don't know how easy it is for us to get exposure to that, but it would be seemingly a likely winner.

It is also a geopolitical thing because Apple wants access to IMTC memory At least for its operations in China. And the US government, and especially Micron, which is a US memory manufacturer, is very strongly pushing back and trying to dissuade them. It is interesting that the US government actually does not have a lot of tools at its disposal over Apple's operations in China serving Chinese customers. And this is, I think, one of the one of those situations where you're starting to see these megacorps not really controlled by where they are, the country in which they're headquartered. And I think this causes a lot of people a significant degree of unease.

Well, yeah, companies that are starting to rival the power of states is definitely a uncomfortable situation. I'm honestly kind of more bullish on some new power centres. I don't think our current power centres are working super well. So the fact that there is still room for entry into the market for political power globally, I think is probably healthy. Better that than no entry, I guess, is my default view. What would you say is the steelman case, if you can offer one? Because I honestly would struggle for why American memory makers and the US government should try to prevent Apple from using a Chinese memory component in an iPhone that they're going to make and sell in China. I, I'm kind of like, first of all, don't we want all the memory that we can produce? Like, I thought we were trying to restrict the export of this kind of stuff. So, and I'm not necessarily for that, but if we're working from that general assumption that we don't want to send our best tech to China, then I kind of am confused. I'm also a little confused if we don't want to send them this tech and then we're also going to try to deny them the ability to use the Chinese stuff, then that would basically just mean Apple can't do business in China. And then what does that do for us, right? It just gives more market share to Huawei and Xiaomi. And I don't see how that's like a great win. So I don't know, as a simple-minded person, I just see China doing business in— or see Apple doing business in China, buying Chinese components, making iPhones in China, selling those iPhones to Chinese people. It all seems pretty innocuous. Could you channel the Micron position?

I think the Micron position is very simply market denial. The more you can deny a market to your competitor, the better things are. It doesn't matter that That market denial. All that matters is that if you give them cash, they will use it to grow and they will use it to expand and they will use it to get better at technology. So the ultimate goal is to deny your competitors cash flow. And the better that you can do that, the longer you can sustain your market position. And that's the ballgame, I think, because they are in a commodity— commoditized business. And they also have the fact of the matter is they have this enormous debuff, which is the speed of setting up new manufacturing facilities in the United States. And for example, they are Trying to set up a fab in New York. And they've faced tremendous opposition in upstate New York. There's still all of this, you know, Micron is going to mess up the water supply, et cetera, et cetera. And they've been fighting over it for quite a few years. They're slated for, I think, 2028 for the fab to come up. And there's still, you know, not a lot of visibility on construction. And I think the US political system has 2 choices. One choice is to make construction easier and to allow a lot more construction onshore faster so that US manufacturing can be competitive. And the other choice is to not do that. And just like every other kind of manufacturing, have that be done offshore at and import it and have that trade deficit and jobs deficit that comes about as a result of that. And we seem to have settled into, we want the manufacturing, but we want our companies to come and fight the political battles for us onshore. And win through and then build the manufacturing, which puts the companies in this position of having to alter, I think, the public viewpoint of these firms and gives the politicians the easy win of being on the other side resisting the companies, which is, just doesn't make sense. But anyway.

Yeah, I mean, in some ways I think this is one of the biggest advantages that China has right now relative to the United States, and it's simply that their government officials seem to be incentivized to help their companies succeed, which is not to say that they don't exercise oversight or put them in their place if they get too big for their britches, but the default assumption is that they want their companies to succeed at the local level, at the national level. And I just, I get the sense that their regulatory environment is a lot healthier than ours, both in the sense that like when they see something that really is abusive, they pretty reliably step in and stop companies from doing it. They themselves, as the central government, they might have their own abusive tendencies that nobody is curbing, but they do restrict what the tech companies, what the big platform companies can do in order to protect small businesses and everyday people. But they still want those companies to succeed. It seems like they kind of Put them in their place, but also really encourage their growth. And here we sort of have the opposite where like the companies are kind of allowed to sustain abusive practices for a long time without any intervention being brought to bear on them in many cases. But then they also like can't do the basic blocking and tackling of growing. And this is not a healthy equilibrium for us. I really would love to see This get changed. And this honestly feels like the kind of thing that Trump should be changing. You know, I'm not a fan, as I think everybody knows, but like, what good is this sort of, you know, highly energetic, uh, you know, individual decision-maker type executive if we can't throw a few EOs out there to make things easier when it comes to just building critical infrastructure. That seems like a huge miss. I think the answer there, my guess is, you could tell me what you think, but it seems like they're not sure. Trump is potentially himself not sure where the base is on this issue and if he can really lead the base, or if this is one of those things where he kind of needs to follow the base. If his people are sufficiently anti-tech, He may feel like this is one where he's gotta do what his people want him to do, and that might be, you know, make like life difficult for big tech. They all the CEOs were at his inauguration. You thought? I thought that might be enough to, you know, get some permissive EOs signed. And honestly, I think we should be building stuff on federal lands as well at this point. Like. It's crazy that the US federal government owns some massive percentage of the western two-thirds of the country, maybe the western third of the country, better said. But you look at the percentage of western states that are owned by the federal government, and there is just tons and tons and tons of empty space where we could put facilities down and we would not be sacrificing national parks or ruining the landscape. Hardly anybody would really even notice in a lot of cases, I think, if we chose the sites well. And yet it hasn't happened. I saw an interesting proposal the other day too for data centers in the north of Alaska. Did you see this?

No, no, I did not.

It was written by somebody who has been a forward-deployed engineer for Palantir working up in the oil fields and for the pipeline. The oil fields are the very north of Alaska. Pipeline goes all the way across Alaska from north to south, basically from the Arctic Ocean all the way down to the south. And, you know, from there, that's where it hits ports that are like open year-round and the oil can get taken where it needs to go. They do not apparently use the gas.

The—

because the gas, you know, there's no infrastructure to transport it. They can't get LNG tankers up there for big chunks of the year. And so somehow we've settled into this equilibrium where the gas apparently gets pumped back into the ground. They get the oil out, gas comes up with it, they re-inject the gas. There's apparently a huge amount of gas. This pipeline has been flowing for like 50-plus years. So there's this huge amount of gas that's been kind of surfaced and re-injected deep into the earth, and it's just sitting there, proven reserves. And so the proposal was, this landscape is already far from pristine, right? There's holes in the ground all over where people are digging for, drilling for oil. And so it's hardly like postcard nature at this point. Why don't we put some data centres there? We've got the gas. The calculation was something like it could run just based on the gas reserves that have been proven, that you could run some massive data center footprint for decades. And why not? I thought it was a pretty compelling proposal. And again, it's kind of like, where is the executive on this stuff? If we're serious about anything really, if we're serious about winning, which I don't like the winning and losing frame, but if that's their frame, right? If we're serious about winning, this is the kind of thing we need to do. And even if we're just serious about making sure that the retail mom and pop are not priced out of AI over the next couple of years. It seems like the kind of thing that we need to do. And nobody would really object. Not many people live there. It's mostly oil workers that are in that area. They're quite used to infrastructure projects happening around them. This is way cleaner than the existing infrastructure. Let's wake up, people. Come on. What are we waiting for?

So I think as a former energy investor, it is commonly known that there's a lot of stranded gas assets in the world. And one of the major things in energy has always been to move the energy from where it is to where it needs to be consumed. And so a lot of what we do is figure out where the energy is— coal mines, gas, oil— and figure out how to transport it. And the reason why oil has been so popular is that oil is easy to transport. You can put a pipeline, you can put an oil pipeline together, you can put it in a car, or you can put it in multiple different ways of transporting oil, and very energy dense also. I think Cruzal, the data center company, started off flaring. They wanted to get gas that was currently being flared at many oil fields and use that for free and use that to mine Bitcoin. And that is how Cruzal first started out and then transformed as this kind of link from energy to compute. got tighter with AI, they got transformed into an AI data center company. So I think this idea of building plants, building data centers where energy is, is actually a really core kind of concept. And it's even kind of what SpaceX is doing. They kind of, okay, the energy is really outside Earth. And so we're going to move the compute to where the energy is. I think that makes sense. But I also think that Elon is probably right in expecting that it's just not, the data center construction onshore is just going to stop. And I think, for example, there was a podcast dropped, I think yesterday or day before, between Dwarkesh and his so-called cousin, Dylan Patel of SemiAnalysis. And Dwarkesh and Dylan were talking about, okay, what happens when the spend per year in 2029, 2030 gets to say $5 trillion or $10 trillion or these insane kind of numbers? And all of a sudden you're building 100 gigawatts in 2029, 2030, you're building 100 gigawatts of data centers. And I think the fact of the matter is it seems like it's very politically unlikely that you're going to get the ability to build that much in data centers in any location. I think it's just a hard ask. It's not just— I don't think it's just the— I don't think it's just the politics alone, but it's like you have infrastructure, you have traffic, you have employment. For example, one of the things that has happened in the last couple of weeks is there are people asking, well, if all the electricians go work with the data centers, what happens to the residential construction we need? Because there is a lot of residential construction that is necessary. And so I think there's a lot of issues there. And my take on this has always been that I think within the next year to 2 years, you need to see something unusual. And something unusual would be something like, okay, robots start working. We don't need that many electricians anymore. And half of electricians, half of the electrician job can be replaced. And maybe you, just like you have in code, you have a reviewer that reviews the work that the robots have done, but the core day-to-day activity is not done. You need these kind of unusual step-ups, which no one is very confident in predicting or betting on right now. You need new technologies that use less copper. I don't think, for example, I think about 40,000 pounds of copper per gigawatt of data centers. And I think the whole world's production isn't going to get you to what you need for 100 gigawatts of additional power plant building capacity or data center capacity in 2030. So these numbers have never made sense to me in that sense. You cannot expect the exponential growth to continue with today's technology. We're going to need tomorrow the future tech. And the thing about future tech is I think even like the AI 2027 or 2029 guys or whatever, they don't feel confident in predicting future tech. They would rather not predict future tech because it sounds ridiculous and silly. It sounds ridiculous to say there's going to be a million Optimus bots doing electrician jobs in in 2 years. It's just ridiculous, right? It sounds ridiculous to say we are going to find new conducting materials which are better than copper and cheaper and easier to manufacture. And this is what is going to take us to 2029, 2030. So these things literally sound ridiculous because every single scientist will say, well, where's the evidence, right? Scientists are based on evidence. So where's the evidence? The evidence is not there. And I think this is where the AI people kind of break with the regular scientists, because the AI/ML people are willing to assign credence to this kind of future tech that is going to happen. And the rest of the world, including the investors, are like, you guys are ridiculous. These curves don't make sense. The exponentials don't make sense. You're just not going to get there. And underneath all of this, is that the AI people are, in some sense, have faith, like this religious faith in technological development as a result of AI creating these new technologies that make this possible. And this is, I think, where the gap is between the two. And I think it's pretty clear to me that the political system is not going to allow the same 100 gigawatts with the same kind of impact on resources and pollution and employment that we're building right now.

Here's an example that just comes to mind when you talk about AI people having faith. I might use a slightly different word from faith, but this is Andrew Critch, who has been in the game for a long time, goes back with folks like Eliezer and some of the earliest thinkers with a PhD in machine learning himself, maybe math, math or machine learning, I forget, but he's been involved for a long time. And here he is on record just in the last day or two saying material science applications are going to surprise a lot of people in the next 2 years. This came to mind also because of just how specific he is on his Predictions. Q4 2027 up through 2029. It will not be 2030. You know, time will tell, obviously, but I wouldn't bet against him or the field on this because I have done a few episodes with different companies that are specifically trying to bring AI to material science. And even like a year ago when I was down that rabbit hole, they had some pretty remarkable findings. There was one example, this probably goes back almost 2 years, where the trick a lot of times seems to be get a lot of simulation data, which with these physics simulators is still quite costly to run. But then with enough of that, you can train your neural network on the simulation data, and then you can get the network to do similar quality predictions, but at maybe 2 orders of magnitude faster. So now you're just dramatically accelerating your ability to do these simulations. And so one of the first things that was kind of an emergent surprise was a guy was just simulating saltwater in a real kind of simple way. But what he was seeing was spontaneous crystal formation and dissolving, where this was not something that had been trained on. It was just a matter of solution was all that his sample was trained on. But as he scaled it up with the roughly 2 orders of magnitude faster neural network to power the simulation, and he's able to run it for a lot more timestamps, he starts to see these physical phenomena that weren't even trained on emerging in the simulation. They also did the potassium ion channel, which I thought was really interesting as well, and shed some light, which will need to be experimentally verified. But there was some suggestive evidence for how the potassium ion channel works on a mechanistic level that people had been speculating about for a long time. But nobody had ever really been able to bring any additional evidence to bear on it because it was just very, very hard to simulate. They were able to do that. And now it seems like one of the competing hypotheses was privileged in terms of exactly how the potassium ion gets transported down this channel from inside to outside the cell, vice versa. So, I think that's a pretty good bet that we will see A lot of materials advances. And this kind of, when you zoom out as far as you can, the Kurzweil story is like basically exactly this. And it's held for, since he published it. And if you believe his analysis, it's held for thousands of years before. It's all these micro exponentials that come one after another that individually peter out, but always seem to get saved by the next one just in the nick of time. And this I do see as a pretty good bet. I think Andrew, and Andrew Critch in particular, does have a long track record of being pretty prescient on a lot of this stuff. He's another one of the voices that you could look back and read his writing from 5, 10 years ago and be like, wow, dude saw a lot coming before a lot of other people On, on seeing a lot coming before anyone else does, let me pull up our first guest, introduce our first guest for this morning.

Malte Ubl is the Chief Technology Officer at Vercel, the infrastructure platform that powers the wildly popular Next.js framework and allows developers to deploy scalable web applications instantly. Long before the current artificial intelligence boom, He spent over a decade at Google, where he led the engineering teams responsible for how Search renders pages across desktop and mobile devices. He's also deeply embedded in the developer community as a co-founder of JSConf EU, one of Europe's most influential tech conferences. Today, his core technical focus is architecting what he terms self-driving infrastructure. As AI models become cheaper and vastly more capable, The primary bottleneck in technology is no longer writing the code. It is safely running, observing, and scaling that code in production. Under his leadership, Vercel is building systems where AI agents act as autonomous first responders, analyzing live production data to automatically submit performance optimizations and security patches. This conversation is exceptionally timely with the recent launch of Vercel's always-on tracing and their new AI security harness. The industry is actively debating how to give AI agents deep access to production systems without sacrificing control, performance, or cybersecurity. So we were talking about self-driving infrastructure. What are the components that make up self-driving infrastructure? How does it work?

Yeah, I think the key of self-driving infrastructure is that you assume that there is an agent in the loop that can just take a look what's going on. I think the most vivid example that everyone who's ever run a production system will know about is that you will have to have some kind of alert system that notifies you if something's going wrong. And that kind of sucks to be woken up at 2:00 AM, middle of the night. And it especially sucks if there was actually nothing wrong because it was a false positive. And so the very concrete thing is that When there's an alert on Vercel, it's the Vercel agent that takes a look. And it can take a look for a little bit longer than just like the in-the-moment alert, right? It can figure out what's going on. Are there actually errors? Maybe marketing sent out a newsletter and there's a spike in interest, that type of stuff, right? And then it can make a decision over maybe 30 seconds or 2 minutes with a little bit more data and then alert me only if there's actually something wrong.

So is it primarily only diagnosis or—

It's not only diagnosis. I mean, obviously it can help with what's going on. And in particular, it's really focused on getting the loop to be complete. So what that means is that once you're back in your coding agent, you can also tell it, you know, maybe there was a production incident, maybe I'm just worried about my performance, maybe I'm worried about conversion rates. And you can, because the Vercel platform is really optimized to be readable by agents, you can have this entire loop where you kind of use your own production data as something that feeds into how you then later optimize your software.

So one of the questions I have is a lot of agent work is non-deterministic, right? It's, and especially when you have a sequence of events And certain events happen in different sequence and everything can change based on what the agent returned on the prompt, etc. So when you're doing this debug causality, how do you work with these non-deterministic situations?

I think the main insight is that AI is actually exceedingly good at this. Right? I mean, that's kind of the main innovation. And if you give AI a good understanding of the meaning of data and you give it the data that actually is relevant for the moment, then it is just very good at making this type of decisions in the moment, especially compared to a tired human, right? And I think we're not in a world where you wouldn't also need someone to do the operations. But we're very close to a world where you can have a very large percentage of all automations kind of go through a loop like this, especially on a platform like Vercel, which kind of, you know, in a way through luck, because we could never have known, but it was like kind of designed for the moment, right? So in a traditional infrastructure, you need stuff like Terraform where you make a change and then you kind of change production. And it's often not possible to undo it cleanly. Or even if you do it, that takes a while, right? And so one of the core tenets of the Vercel platform is that every time you deploy, we make your infrastructure from scratch. And we keep the old one around. And so if that— if some— one of the primary things that the that the Vercel agent will recommend to you is to roll back. And it can do so in 300 milliseconds. And it can also kind of say, well, actually, you've deployed 5 times since the error was introduced. So we go back to 5 deployments in the past and still have a 300-millisecond threshold to get there.

So Why did you guys build Vercel in that way? Because I'm sure there was a lot of engineering cost.

By the way, your agent is still yapping at me.

Oh my gosh.

But it's fine. I can usually hear what you're saying over it.

Okay. So you probably had a lot of engineering cost in the early years because of You know, those issues. Yes, because of—

because that was to your agent.

This is—

this is— this is the first. So let me— let me try to do a quick hotfix. And, uh, right, you—

yeah, you get the agent to be quiet.

Are you guys using Vercel for your platform?

Uh, no, no, no, we are not.

Yeah, this would be a good idea. Uh, what are you using?

We're using Cloudflare.

See, that's the Clearly, that's the problem.

The original sin.

Yeah, the original sin.

Prakash has vibe-coded this whole Studio from the ground up, actually, with, you know, familiar primitives. And it is amazing how feature-rich it has become. And the agent usually does a pretty good job of getting people into the Studio and making sure that you can hear it and it can hear you. And usually it shuts up. So this is definitely a new issue. I guess you have done a lot, right? Vercel has been prolific over the last couple years in terms of shipping a bunch of new different form factors. I built a lot of stuff with the original AI SDK. And also, by the way, had some great experiences where when I did run into any friction, I was able to get people like on X to help me out. And, you know, The culture of squashing issues was very apparent in my interactions with the company. But now we have this new Eve, and I'm kind of wondering, why did this merit a kind of generational upgrade where we went from one kind of recommended way to build agents to a new framework? Why wasn't it like an evolution and instead a sort of You know, renaming and kind of new era.

Yeah, I think the way to think about Eve is really the insight that it was time to build something that is made for the type of agents that we see ourselves building and our customers are building in a very kind of like high-level fashion. If you like, really the way I think about AI SDK, which is immensely popular and has many, many, many millions of downloads per day actually, but it's very low level. So it's something that you can use if you really know the nitty-gritty details of how to build an agent. And really, in the modern parlance, you use AISDK to build your own harness. And that's great, right? If that's what you want to do, that's amazing. But it's certainly too low level for the vast majority of agent projects out there. And so Eve is really our, I think, now pretty successful attempt to say, okay, what do the 90% of agents have in common that we see? And how can we give them, develop something where they can directly express exactly in a very straightforward way how to build an agent in this manner?

How do you deal with The small but important differences between different model providers. This is something that in using the AI SDK, I ran into quite a bit where it would be like, oh, well, Claude handles memory just a little bit differently, but it also formats tool calls just a little bit differently. And when it comes to structured responses, certain things are and aren't Supported across different providers. This created a lot of friction for me. And as you try to go up the stack and present higher and higher level abstractions for people to build on top of, that becomes your problem, right? So how— what is your strategy for dealing with that? How do you try to make sure people can get the best and latest features that model providers are rolling out without overcomplicating it so much that it It becomes a burden again. Oh, now I don't hear him.

Do you?

We're making—

Can you refresh? Just refresh.

Me?

No, Malte.

Never a dull moment.

This was a, this was a new, new error. We've never, never seen this one before. There we go. And where is he?

There we go.

And all right, so there we go.

All right, I switched to your classic version. I saw a toggle.

Ah, awesome.

All right, so did you hear my last question?

I did actually hear your, your question. Yeah, yeah. No, um, so the, the normalization, um, happens on 2 layers. Number one is in the AI SDK. And you really have to think about the way the AI SDK is developed these days as a software factory. So it's a perfect use case for that type of way of building software because there's so much scope across the breadth of providers, so many micro issues that you can find through automated testing, but then have to find solutions for, as you were mentioning, across how exactly tool calls are handled and so forth. And so we're doing it at that layer. And the second one is that also Vercel's AI Gateway is in a similar way working on harmonization of APIs across providers. And that is definitely a constant struggle and probably one of the biggest added values that our gateway adds in this space, because you can actually switch model providers, you can use the the same APIs. So if your client supports responses or the legacy completions API, et cetera, that would be supported across all providers and so forth. So it's really kind of that layer. But yeah, if you are using Eve, you don't have to worry about this at all. It's just completely not your problem anymore. You really just focus on the functionality of your agents, much of which is actually integrations with other systems, which is really what makes agents useful. And that has been actually a key focus of Eve and its success, right? So it joins together with a Vercel product called Connect, which makes it just so easy to connect it to GitHub, to Linear, to Slack, Workday, SAP, Oracle, whatever you want to do in your company. We see very wide usage of this type of integrations with various systems across kind of the ecosystem.

You mentioned the AI Gateway, and I wanted to get your thoughts on the Stripe OpenRouter combination and what that has you thinking about when it comes to the future of the AI Gateway, perhaps.

Yeah, I mean, first of all, congratulations to the OpenRouter team. Good for them. I think overall it's a goal of Vercel, myself, and I think almost everyone in the ecosystem that we end up in a world where there's abundant access to AI. And I want our customers to pay the lowest price possible. And so in that, In that world, the way to get there is through competition, right? And that happens both between the frontier labs, between the companies doing at-scale inference for open weight models, and it's at the AI Gateway level. So I think I'm excited about an open router. I think, speaking of the cost side, so one of the principles of AI Gateway from the start has been we are only charging list price, right? So if you go to OpenRouter, they actually, they tell you the list price, but you're really paying in credits and the credits are more expensive than dollars. So like it's a little bit of a hidden thing. And in a similar way, like we're like excited for our larger customer to essentially have them participate in the discounts that we get from the inference providers. Right? People sometimes ask me, well, if you're charging list price, how are you making money? And people forget that businesses like Safeway exist, where you buy the chips at list price, but obviously Safeway didn't buy them at list price. And this is essentially a retail— not essentially, it's a retail business, right?

Right.

And I think it kind of plays an important role in the ecosystem to make it really easy to get access to to models. Another thing that we see quite a bit is Vercel has a bunch of startup customers, and their customers really value having day-zero access to models. And that can be really, really difficult. Nathan, you were actually saying it can be annoying to support this new model. You have to test it. It might have all these little things that don't quite work, even if the model itself is good. And so having kind of day-zero support in the AI Gateway for things is something that actually really helps folks to just say, this is not a problem that I have, it's someone else's problem. And very much like something where we have a team just full-time grinding on onboarding more providers, more models every day.

So let's speak to that a little bit. So one of the experiences that I've had dealing with models on OpenRouter, for example, is when you switch model providers, when you switch the inference providers, you often get some brittleness in the sense that, for example, in one of the instances, I had an inference running and it was a logical sequence that went through a number of steps and I expected it to run in the steps as per the prompt. But what ended up happening is that the inference kind of ran out of step. So the ending answers kind of got generated first without the context of the beginning answers. And the only way to fix that was to number the JSON schema. Like, you had to have a number because it seems like one inference provider was going in alphabetical order of the JSON schema, which is a very odd reaction. And not in the order that was given to the provider. And so there's this brittleness in between switching providers. So how do you manage that kind of complexity and brittleness when you're trying to provide this kind of consistent experience across multiple providers at the same price?

Yeah, it's grinding, right? Which is the perfect startup, right? this is the most horrible thing you could possibly imagine you wanted to do yourself, right? Because it's very, very annoying work. And again, like, building a software factory can really help you because it's annoying work that's also, like, you know, in an agentic world, very automatable. And so this is what we see people do use EVE for a lot, right? Is to build these kind of custom software factories and say, like, I can express in my own problem space how I identify problems that are of a certain shape. I have a playbook for them. I execute them through my software lifecycle and I have a testing scheme for them. And then in the ideal way, I can almost automatically roll them out. And so yeah, these issues exist. And the AI Gateway team very much takes it as their responsibilities that these issues are not visible to our customers. And it's a lot of work. And there's obviously trade-offs of, you know, if you like, what you see on Versailles Gateway is that, you know, maybe on a not so popular model, there are 3 to 5 providers. On a very popular model, there are 10 providers.

Mm-hmm.

But if you have 20 and like some random startup that bought 5 H200s and whatever, like, is now an inference provider. If you add them, can you really make sure that across 50 models they actually do a good job? That can be difficult. And then there's choices to be made on that marketplace. Do you take on that vendor or do you say ultimately quality is more important and I can't guarantee it with going beyond a certain number of providers?

Well, what would have— I see You know, I saw a comment the other day, all of my friends are— all of my NeoLab friends are basically NeoClouds with a layer of, you know, software on top. Like, how would a new provider onboard to Vercel? Like, what would the onboarding process look like in order to become an inference provider for Vercel?

I mean, ultimately, this is not very complicated, right? You have to somehow find us, which is not very difficult because we're all ultra online on X. And then, you know, you go through our procurement process. The APIs are all pretty standardized. And then you kind of go through our qualification process and we'll, you know, maybe give you feedback where, like, things are not acceptable in some way. But in many ways, this is a relatively standardized process, given that there are— there's already kind of a large number of providers. And it's kind of the, like, the, you know, again, like, there, there is a lot of value in having this marketplace be large in that incentivizes each of those folks to maybe go down with the prices that they list as required for a model.

I want to do a little digging in on that to the degree that you can share more information with us. I'm really curious about kind of the deal dynamics and who has what relative market power. You had this blog post a couple of weeks ago saying that DeepSeek had surpassed Gemini in number of tokens to become the number 2 provider by token volume. Anthropic had, I think, still the majority of the revenue, although not a majority of the tokens. My guess is Anthropic is probably not cutting you any deal and you're not making money on that. Tell me if I'm wrong. But then others like We've got these kind of, it's quite a few companies in the chain, right? You've got the model maker. If it's a Chinese company, they have the models actually being served by inference providers, your Fireworks, your Togethers, lots more of those. Then there's this kind of central access point that is the Vercel gateway. Who needs who worse? And how are people structuring deals to reflect the value and market strength that all these different parties have?

Yeah, I mean, I think, I mean, by the way, I don't love it, but this is a really good question. It's a good question because there's such intransparency in this space, right? And everything's kind of covered by NDAs, et cetera. So that's why it's like, it's essentially impossible to talk about any given deal. in this space.

But can we leave names out and give kind of a general market structure?

Yeah, no, I think, I mean, ultimately, it's generally true that there is, it's an interesting dynamic in that there is both a large, it's honestly not a very complicated market. There's suppliers who value ARR. And hence they do deals like everyone else has always done on large kind of enterprise SaaS deals, right? Like if you talk dozens of millions of dollars a year in commit, you have expectations that you get some discount on it, right? And if you do 6 figures a year, we also have some expectation of a discount, but it's lower, right? So I think those dynamics are actually quite similar. And, but like, if I'm a customer, end customer, so a Vercel's AI Gateway customer, and I want to, let's say I want to do some amount of value on Kimi K3. And realistically, I need to have deals with, like, in the current state, I cannot say I'm going to bet everything on vendor X. Because the reliability of all of these vendors is abysmal, which is in such early days, right? So I always need multiple vendors to supply my demand. And that also means that I, you know, if I'm by myself, I essentially need to buy double. And that means I really have to like half my commit, right? And so that reduces what I can negotiate for. And that's why I think it's advantageous for me to go through something like Vercel AI Gateway where I can, where I have a single vendor that I buy from, that I commit to it, the whole sum. And then, you know, you have this kind of layering effect. And again, at this moment, this is almost necessary to go at the market like this because of the, current state of the technology, which is kind of still very unreliable.

Does the gateway handle fallbacks for people? Like if I make a— so if I make a DeepSeek call and it's originally going to Fireworks or whatever, and that errors, they're down, you'll reroute me to Together and so on and so forth?

Absolutely.

Yeah.

Yeah. So that's, that's a key value, right? And in fact, and this is, you know, we can look at what other vendors do in this space. You can bring your own key and we don't charge you at all for your primary requests. But so let's say you do a single deal and you have your Anthropic commit and you go through the Vercel AI Gateway. So you don't pay us at all for that traffic. But if your, let's say your Anthropic and GCP costs contract has downtime, we will auto reroute you to Anthropic on Bedrock, and you will then pay for that traffic, but only that traffic, which is like, and let's say they, but just a few days of downtime a year right now in total. So in that case, you can really see AI Gateway is just this layer that adds stability to your system, and cost only happens when the AI gateway is actually working for you.

Let me switch gears a little bit to a topic which has been on all our minds maybe the last couple of months, which is security. And especially post the Hugging Face attack, where I think the postmortem was that We now have existence proof of automated AI attack, and we don't have existence proof of automated AI defense. Now that I think offensive security has become extremely cheap with open weight models, and the frontier models that are often deployed are often deficient in addressing security for various reasons, including AI safety. How would an AI defense cloud look like? What does this kind of active AI defense for security look like?

Yeah. I published a blog post on this, I think last week, maybe somewhat negatively titled, Everything Hackable Will Get Hacked. It is an extreme moment. But I will push back on what you're saying because I think it's a very common misconception. 2 common misconceptions. A, I don't think it's actually priced into the market yet how good, and especially Kimi K3, is at offensive cybersecurity.

Mm-hmm.

And that it has no safeguards. Like, you can use it for red teaming, you can use it for black hat offense, right? That's the thing today. And it's remarkably good. You can try it out. Like, I, um, it's, it's extremely well trained on this. Um, it has a process. It will probe the system. It will quickly know your system better than you, um, within minutes. Um, and then it will try everything to, to get through the defenses in a way where you can really see that the model has been specifically trained to be good at this, to know how to be an offensive attacker, right? So that's part one. The other part is it's just not true that off-the-mill frontier models aren't good at cyber defense. And that's a common misconception. The misconception comes from the fact that there was this Mythos thing, then Fable shipped, and it unshipped and shipped back with really, really kind of almost unusable cyber defense detection and Kind of shut down, right? But SOL 5.6 does not have this. So SOL 5.6 is absolutely perfectly usable for a certain type of defensive security, which is that it assumes— and this is actually true for Opus 5. So essentially every model in the market except for Fable 5 will do the following thing, following 2 things. A, it will do Hey, I have source code. Model assumes they have the source code, they are the owner of the system. They will— you can ask it, are there security problems in my source code? Problem 2, it will do is, I have a security report, write me a fix. So again, Fable-5 will do neither of these tasks. So 5.6 and Opus-5 will do both of these tasks, right?

Oh.

Good, there you go. So one of the things I actually personally have been working on is our software called DeepSec, which is an open source project that will do whole repository scans for security vulnerabilities. And like, I just couldn't be more clear that everyone needs to run this. Because it works really well and it prepares you for a world in which defense is very important and where we are in a world right now where you as a defender have a benefit because you can use the frontier model that will not do offensive tasks, but it will do defensive tasks. And you really have to hit the moment here because— Again, GPT-3 is already really good. Easy to imagine that in, you know, 6 months' time at the latest, we will have fabled class models that do offensive security, right? And so you have to act defensively now so that you're ready at the time. Are there any guarantees that it'll work? No. But like, obviously, being able to do something today is really key. I think one thing that we'll definitely— I already mentioned the word software factory a few times today. We are heavily investing in not just having DeepSec as a tool, which is a discovery tool, but essentially completing the circle because it's definitely absolutely correct that through these AI discovery mechanisms, the number of issues identified is exploding. And so I have to automate the whole path of the SDLC, which includes fixing it, rolling it out, being obviously certain that I'm not making things worse, and so forth. So, like, I think us and the industry do have to do a lot of work here. But it's not a hopeless situation. I think people understand.

I mean, Mis—

like, or underestimate the amount of things they can actually do today.

Just one follow-up question to that. Have you— I mean, it sounds as though you've seen kind of costs allocated to security kind of increase. And do you think that increase is permanent going forward? Do you think as models improve, you will continually have to, you know, keep investing in that space?

Yeah, exactly. And I think, It's a good example for why DeepSec exists. So when I was actually like, you know, as many folks trying out using code review tools, and I pushed a PR and the tool said, well, you have a security issue. And I was like, I was delighted. And as you do as a CTO, you say like, okay, this is great. Can you run it on my codebase? And the team was like, no, no, it only runs on the new code. Like, but I have like millions of lines of code. So, and so that's why I made DeepSec. It's basically just the same thing as a code review tool just running on your entire code base, um, which, well, if the code base is big, costs tens of thousands of dollars, right? But compared to, like, for example, our HackerOne budget, it's nothing.

It's cheap.

Like, it's the cheapest thing I've ever done in my life compared to, like, also on the extreme case, a security breach, right? And the other thing is, even if you have that incremental code review, which you absolutely should because it's Great, right? But the code that you wrote 3 months ago was reviewed by your code reviewer, but that code reviewer was probably Opus 4.7, whatever, you know, some, some, one of those. And it wasn't the latest frontier model, right? And so like you do have to rerun these things on your codebase on a regular basis. I really don't see like a path out of that loop. But again, I think it's good investment. Like, it's like, I put out tens of thousands of dollars. That's only gonna cost that much if you have very large code bases. Probably means that you have some good revenue there. And I think if you kind of go through your budget lists of how much companies are already spending on security, it's a really, really good investment.

You mentioned KIMI K3 being quite good at offensive cyber and having no guardrails. I took a trip to China this summer, and one of the takeaways from talking to people there was that the way the Chinese government seems to think about regulating AI is more at the system or product level than at the model level. So they're kind of comfortable being like, yeah, your model itself, okay, that's one thing, but you're going to put this into a product, there's going to be an API, you might have monitors, whatever on top of that. That's the kind of thing that they concern themselves with in their domestic market. I don't know to what degree, and maybe you know, if the inference providers that are taking models like Kimi K3 are adding certain guardrails, classifiers, monitors to their systems to try to rein in a little bit what the raw model will do. And I guess I'm kind of wondering, what do you think their responsibility should be if you are an inference provider in the US deploying Kimi K3? How much responsibility do you have for what people do with it?

So I do think that a certain amount of KYC is important, but I think the reality is that especially as it becomes more possible for folks to run these things themselves, you always have to assume that there is an attacker that has access to this technology, right? And unfortunately, the technology you need to do a red team exercise and the technology that you need to do a black hat hacking attack is essentially the same. And if you cannot avoid that your adversary has access to technology, then not giving it to the good guys is the wrong choice, right? And I would love it if we could somehow put the rabbit back into the hat, but that's not the world we live in. So I think we have to be realistic about it. And that means that we have to make playing offense kind of part of the defensive game.

You're putting your money where your mouth is with a $1 million hacker challenge for the, for Cell Sandbox. This was recently announced. Maybe for, you know, an ignorant person like myself who would have thought that sandboxes were less hackable than it seems like they are until recent, uh, revelations, uh, updated my thinking. Like, give us a little background on why is it that the sandboxes are still so hackable here in, uh, mid-2026, and why are the agents not enough? Why do you need to go put a million-dollar bounty on it, uh, given how good the agents already are?

Yeah, I mean, I think nothing goes beyond human creativity, right? So obviously they all have agents, but we exhausted our own agentic research. We're not only doing the challenge, we have actually multiple pentesting teams on the same matter. So it's both the crowdsourcing and the things we can influence directly. And I think the sandboxes, sometimes they're seen in this very simplified way where it's like, oh yeah, it's a Linux microVM, and they're inherently secure. What do you need on top? If you actually take a look of how, for example, the OpenAI Hugging Face event worked is where the AI model found a side channel to have essentially write access on an external API. And so part of our self sandbox offering is an egress firewall that makes it as easy as possible to, at that egress layer, minimize what can be done. So part of the $1 million scope isn't just to escape the Linux micro VM, which I would be very surprised if that happens, right? But obviously, it would be amazing if someone finds this so that we can get it patched upstream and fix essentially every kind of similar sandbox in the entire ecosystem, right? Pretty unlikely, right? More likely is that someone, for example, finds a way to defend— to circumvent the egress firewall, right? Or they find like a side channel in some kind of service that's always callable from the sandbox. There is ultimately a lot of, like, because we are adding functionality to these systems and you have this, like, infinitely creative adversary, like, you have to really think outside the box and not just, you know, accept or hope for the best, right? Like, I think we're past hoping for the best. And so I think that's why this is a very appropriate challenge. And I think we have received a large number of submissions. AI is helping there, right? Right now we're feeling really good about what we've seen so far, both in the sense that people maybe find some kind of edge case. It's all right, why not close it? But still time to maybe folks to find something really really valuable. And again, like, that, that will be our contribution to ecosystem to make all our customers substantially more secure, which, you know, includes companies like Meta, includes companies like Notion. So the value for the ecosystem would be substantial to get— make sure that our systems are truly, truly secure.

When you see— I mean, you're on the front line here, and so you see this immediately. When you see a new model released like Kimi K3, what is the lag time from that release to you seeing it start being used for attacks against you? Like when you see the adversarial front, like are they, you know, there's some people who are like, you know, the hacking, the large hacking groups are like McDonald's. They're slow. They adopt technology as slowly as, you know, any typical large organization does. And there are some other people like, oh, you know, these guys are going to use the latest models immediately and you're going to see it immediately. How do you see that in the adversarial front? Like, what do you see?

Yeah, I mean, it's not possible to detect what people are using to attack you. What would be really fun is maybe to try to ascertain this from, for example, our HackerOne submissions. So besides this challenge, we We have a permanent HECO-ON program where folks are submitting reports. And while certainly not all of them, obviously those folks are now more and more using AI to support their work. I don't have any data on neither, but maybe we could start collecting the models that people are using. But yeah, it'd be a really interesting question. My assumption is that this is a really, really fast turnaround time because all Ultimately, I mean, you guys know this, once you have access to a model, it's very, very simple to start using it.

Well, time flies and it's the scarcest resource in the AI game, so we'll let you get back to work. Sorry for our agent yapping incident at the beginning, which stole a couple minutes from us. Anything you would want to leave people with? We didn't even touch on v0, and I'm not sure if that would be the thing, but Anything we didn't get to that you would want to make sure people are aware of before we break?

No, we got through, you know, we talked about Yves, which people should give a try. I think everyone, like, lots of love from the community. We talked about DeepStack, which I already mentioned, like, you really, like, I mean, maybe there's other tools like it, but if you're not running one of them, I think that is actually a problem. What we didn't touch on is our new experimental harness, FX. If folks are more on the experimental side, it's not that it's not stable, but it's a new take on kind of a coding harness written in Zig, fits on 2 floppy disks and compiles to Wasm, runs absolutely everywhere. So it's a really— it's like a new kind of lightweight take on agent harnesses that I think folks really love. But again, that's more for the experimental-minded. Thank you, guys.

Thank you.

Thanks for being here. Great to meet you. That idea of a harness that fits on 2 floppy disks, that is signature Guillermo. I would recognize that fingerprint anywhere.

Fascinating, fascinating guest. I definitely learned more about Kimi K3 than Then, you know, it's very different when you hear it from someone who's deeply technical and has to face the problem every day than when you just read like a tweet from someone. It's such a differentiator.

Yeah. I thought the analysis, even though it was a bit guarded on the details of the way that these deals get set up and why people want to deal with them. As opposed to going direct to inference providers is also pretty interesting. I'm really curious about what the margins and rev share are and how desperate some providers may be. And in some cases, are the model makers themselves also getting a cut? I know Base10 is starting to do that, but I'm not sure how many are. So I'd love— we should maybe have a tip line for people to— send us more inside information than he's able— folks like Malte are able to share publicly. But I still thought that was quite interesting. But we're running a couple minutes behind, so let me not delay us longer.

Let me introduce our next guests for this morning, Lewis Kirsch and Damon Falk, who co-founded Inherent Laboratories to build artificial intelligence systems that can conduct genuine scientific research entirely on their own. Rather than simply acting as highly advanced chatbots for human engineers. Luis brings a deep pedigree in autonomous systems. He previously worked on automating AI research at Google DeepMind and completed his PhD under the legendary AI pioneer Jürgen Schmidhuber. Let me— anyway, focusing on the foundational mathematics of systems that can rewrite their own code. Damon brings critical experience in AI safety and alignment from the MADS program and Oxford University, where he specialized in understanding how highly capable AI models might deceptively resist their own training. This past May, Inherent Labs emerged from stealth with a $50 million seed round to build a completely new kind of research institution, one that recursively improves its own hardware and software while strictly keeping human scientists in the loop. Just last week, the lab released Faraday, a 27-billion-parameter agent that learns scientific intuition on top of standard coding tools. Faraday currently beats today's absolute biggest frontier models, including Claude Opus 4.8 and GPT-5.5 Codex, at reproducing complex real-world research papers. Their work sits at the absolute bleeding edge of turning AI into a genuine discovery engine Which is why we have them here today to discuss their launch, the limits of algorithmic reasoning, and the very real geopolitical risks of building machines that can continuously upgrade themselves.

You're going to get Schmidt humored for mispronouncing his name, I'm afraid. But maybe we can survive that thanks to our relative obscurity. Welcome, guys.

Hello, hello.

Good to see you. Good to meet you.

Yeah, likewise. I guess pretty inconsequential disclosure, I am a very minimal angel investor in Inherit. So technically you can consider me conflicted, but I was mostly just really interested when I first learned about the company in the cultural ideas that were kind of percolating at that time, and I'm sure they've taken more shape as the team has grown and you've actually gotten to work. I will definitely spend some time on the AI scientist work too, but I'd love to start with the culture. What does it mean to have a recursively self-improving organization?

Yes, that's an excellent question. So I've been spending many, many years on the concept of automating AI research and recursive self-improvement. And for the longest time, I thought about it as we're building the machine that recursively self-improves itself so humans can just step out of the picture entirely and just let the thing improve itself. And that's going to be sort of a point in time that's not too far away, but we just have to figure out the algorithms to make that happen, and the machine just keeps going then. And after a while I realized in actual reality right now it's mostly humans driving AI research, but we want to go to that transition where more and more can be automated and a lot harder scientific questions can be answered with the help of AI. And it's not going to be like an immediate transition, but instead we're going to have to build an organization that recursively self-improves. So there's both machines and humans. in this construct, and collaboratively, we are going to improve each other, ourselves, and really become faster and faster at solving scientific problems.

So a laboratory can improve its prompts, tools, datasets, evaluations, model weights, compute allocation, experimental procedures, research agenda. There's so many things that a laboratory can improve. Which parts of Inherence's research process currently improve themselves and which still depend entirely on human researchers?

Yeah, it's a great question. Entirely, probably very few things. If we think about sort of the software stack that makes up a company, the infrastructure that drives all the experiments that we're running, The discussions that we're having with each other and how they drive our research ideas, as well as the steps that we take then to form the plan and start the implementation. The system that we've built called Faraday is a big part of many of these things at the company already. It's a bit hard to draw a hard boundary though, where we can say, This is completely recursively self-improvement. And that goes back to the point that I've been making earlier. I don't think we want to make these hard boundaries in reality, but instead we want to build an organization that really has the automated system Faraday in the process everywhere, but still leverages humans and collaborates with humans in all of these different aspects.

Can we be a little more concrete or kind of give some tangible examples? Prakash and I kind of conceived of this show to a degree as an experiment in personal recursive self-improvement. And for us, that kind of means at the end of every show, we talk to each other about what went well, what didn't go so well. We go prompt our agents to try to figure things out. Sometimes A guest gave us a good idea, which I'm hoping you're about to do. And I'll just say, hey, look at the transcript and consider those ideas and think about how we can implement them. I'm not sure we're hitting takeoff just yet in our personal experiment, but what are some either skills that you've created for the AIs to run or habits that you have developed in the team practices that we might be able to adapt to our own projects?

Yeah. So, I mean, when we talk about Faraday, you can think of it as like 2 fundamental pillars on how it can recursively self-improve. One is about automated prompt creation, automated harness creation, and the other one is about model training. We're a big believer in that ultimately, to build a good scientific agent, we need to train these models as well. A lot of the loops that we've built in the company already working quite actively that on a day-to-day basis, harnesses are changing themselves. The prompts are changing themselves through the ideation of the agent itself, but also through the ideas and discussions that we have in the company between humans. So these changes we see on a day-to-day. that the system decided to change the prompt based on what humans have been discussing. Whereas the model changes, they're beginning to happening where models train another model in interesting ways. But obviously the time horizon in which these systems have to operate there is a lot longer so that the feedback cycles are a lot slower. Hence why measuring that and really making that effective is still a big project that we're working on.

One of the things that I've noted from my read is that Faraday directs several other models, including some which are larger than Faraday itself. And I think one of the constant debates In the ML field is whether a smaller model can effectively drive a larger model, or whether the smaller model breaks down and effectively you need a larger model to drive smaller models. And this has been the whole question of the auto-router and whether auto-routing will actually work out, or whether it doesn't work because the smaller model can never capture the entire complexity of the question and of the field. We've also seen kind of in the past couple of months, people are using Fable, especially when they run out of credits, they use Fable to orchestrate other smaller models. And they use that to kind of save tokens. They tell Fable to use Sonnet, they tell Fable to use other things. What has your experience been with this kind of using smaller model to drive a larger model versus— larger model to drive smaller models? Like, what is that? You know, have you tested both strategies? And, you know, how did that work out?

So I think we're in the business of building generalist scientific agents. And one of the core contributions of our first work has been the separation of the scientist from the coder. And right now, Faraday, the model we talk about in the paper, is, as you say, a 27B model driving a much larger model. And Faraday is doing the scientific work and handing off the implementation work to GPT-5.5 Codex. But in the future, this ratio could be very different. We don't know. We'll have to see. One of the fantastic things about doing it this way though, other than it being a natural kind of separation of concerns that the human researchers and engineers already have, is that we don't have to worry about building frontier coding agents and we can make use of all the advances that are coming in those and focus on building scientists. ourselves. But these are great questions and ones that we will keep exploring in our own research of which should be the bigger models, which should be the smaller models, how should the interaction look.

How did you decide to have a 27B model be the scientist? I mean, the prior expectation on this, I think, would be pretty uniformly like, you'd want to go to a trillion parameter model to have the best internal representations, the most sophisticated understanding, the big model smell. And yet you're doing it at 27 based on Quen 3.6, which I think has a pretty good reputation, but still at 27B, I would just think you'd be fundamentally very limited compared to something much bigger.

Yeah, I mean, we've been building this new company inherent, right? And of course you're right, from an outside one might say, well, we want to build the best scientists, let's start training with a big model straight away. But of course, in reality, when you train a big model, you need a lot more compute resources and you need to iterate over much longer time horizons. So it's quite natural for a new lab to start at a bit of a smaller scale first and then scale up the ladder. And the interesting insight that we've had is that by setting up this pipeline of training an agent to be a better scientist through reinforcement learning, that already at the smaller scales, we're seeing really interesting capabilities of these models doing sort of more scientific behavioral things such as thinking very hard about what is the right experiment to run at this point in time to prove out what the paper has done and do that in a way that does the paper justice but doesn't require lots of resources. And these things already emerge in these arguably smaller models, which I think is an interesting indication that maybe we don't need massive models straight away to do all these things. And there's an interesting aspect to doing this kind of separation of concerns, and we don't need to reinvent the wheel and build just another coding agent and can think about scientific capabilities and coding capabilities as 2 important but perhaps separate capabilities.

I'm convinced that AI for science is going to be tremendous, but I often have to deal with skepticism from people in science, especially in bio. They're usually very negative. And one of the criticisms that I have heard is specifically, okay, if you have next token prediction and you're doing paper replication, well then the agent kind of has a destination in mind and it's kind of evolving towards that destination. Meanwhile, in science, a single negative result disproves the hypothesis, right? So you are not actually looking for that kind of predictable next token prediction. You're actually looking for outliers that, okay, this is wrong, or that is wrong, or this hypothesis is wrong, and therefore we need something new. So how do you address those kinds of concerns? From scientists in the field?

Yeah, lovely question. I mean, I think there's 2 parts to this. One is I do think scientists have an outcome in mind in many cases when they work on something, but it's not necessarily the outcome that's actually going to happen. I have a mental model of what do I want to build next, which kind of, in my case, perhaps recursive self-improving agents do I think would be really capable and how should it be built? Should it be operating on code or should it be operating in weight space? And how should it do these things? So I'm going on that journey towards that goal that I have in mind, but it might very well happen that I'm going to discover something interesting else on that path. And that in itself could be a useful building block either for my own work later on or for other for other people's work, sort of a stepping stone, as we often call it in the open-endedness community, where lots of interesting discoveries are leading up towards more discoveries later on without necessarily having that specific goal in mind upfront.

You mentioned training with reinforcement learning for scientific skills. Obviously, we've seen examples recently of how when the RL signal is not particularly clean, we can get all kinds of crazy downstream behaviours. So I've got a few questions on this, but I guess the first one is simply, how confident are you in the reward signal that you are able to give to the model? Are you, you know, are you able to get to the point where you— I'm guessing we don't have like anything approaching formal proof that you are, um, you know, give— that you are rewarding the behavior you really want to reward. So what precautions are you taking and how confident can you be that you're actually rewarding what you intend to be rewarding?

I think this is a great question and gets to some of the core difficulties of this kind of endeavor. Science is inherently non-verifiable, and providing a high-reliability reward signal has historically meant a verifiable, some kind of proof or test or something like this. And you're completely right there. But we don't think we can keep doing that if we're trying to discover these stepping stones and do open-ended research. And I mean, in the paper we published, we have found some particular solutions to doing this. And some of this involves looking at the entire trajectory and the kind of process that the scientist is doing rather than just the final output. Some of this involves attributing credit back to individual things the agent did in the trajectory. And then there are some kind of more technical stabilization techniques we had to use as well. But the core questions are how to reduce the variability of the reward signal and increase the kind of density of signal while still preserving this property of assessing the right thing. And we did a bunch of work on correlating our signal with human judgment and trying to understand how much it corresponds with human taste. But this is work we will keep doing for sure in the future.

Can you describe what the model is like in a qualitative way? sense. For example, when you read the chain of thought, I just went down this rabbit hole with Bronson, Shane from Apollo Research, who's read ungodly amounts of GPT chain of thought. One thing he observed was that basically the models are always thinking about cheating in a very high fraction of cases. They're at least considering Cheating. So what do you see? Is yours considering cheating? And then also in terms of what it can do, is it now that it's been so focused in on science, is it useless for other kinds of things? If I ask it a friendly chat or companionship question, does it only see the world through the science lens? or how much of its breadth is still retained after going through this training?

Yes, that's a great question. I would say the model does focus on the scientific questions that we ask it. When we looked into the the process, the thinking patterns and the actions it takes, it's not that it jumps to the kind of cheating behaviors that you've been describing. And in most of the cases, we've done some filtering and every once in a while, but in very rare cases, we have seen where it deliberately went to the internet and then tried to download the final result or tried to mock the plot. But it is the case that through both the priors that are encoded in the large language models about, given the prompt that we give it to actually follow the replication process in a scientific way rather than just mock the results, that the kinds of behaviors it exhibits straight away in terms of the explorative behavior that it has before training, it follows roughly something along the lines of such a scientific process of replication. And then through reinforcement learning, that is honed in. But you could of course argue maybe it starts reward hacking at some point. And that goes back to the point that Damon made earlier that we're not using verifiable rewards where all that matters is just maximizing that one single scalar and that that's all the feedback you have. And if you find a cheating behavior, that's fine. But instead, we have these judges, these LLM-based judges, and we really put a lot of effort in building them out so that they're reliable enough to give that kind of feedback signal where if there was cheating behavior, that is actually penalized, that is part of the reward signal. And we have seen the judges spotting these kind of issues and integrating that into the reward signal. Such that that kind of behavior doesn't just keep getting learned more and more by the model.

Yeah.

Do you apply pressure to the chain of thought itself, or are you abstaining from doing that? And as we kind of think about what the frontier hyperscalers are doing, Where do you think— I mean, presumably they're doing this too, right? They've got LLMs as judge. I presume they've put some real effort into trying to make them reliable, and yet somehow we're kind of spinning off our axis a little bit in some of these processes. I wonder if you have a theory of where they have gone wrong, and then maybe whether it's you or them, we've got published timelines. for the AIs to take over the ML research. Right now it feels like we're very far from being able to trust the models well enough to put them in any meaningful way in charge of ML research directions, because they're going to just start to cheat pretty quick is what I would expect right now. Do you see a path where we get over that, or do you have a sort of safety case in mind that you're trying to fill out the elements of where we could be confident that we actually could step back from an ML-powered or an AI-powered ML research process for a bit and not have it go totally sideways on us? If so, I'd love to hear it.

These are some amazing questions. I'll start by saying, at least in the paper we published, we don't apply into the chain of thought. And indeed, I think doing so can be problematic, but who knows what will happen in the future. The kind of question you raised at the end of what will happen in the future and when can trust be handed off is a super important one. And I think the best answer here is that that we care a lot about getting this right as a team. We strongly believe that the future of AI scientists looks like a collaboration with humans. And as Luis mentioned, we want to recursively self-improve the entire organization and discover these new kind of methods of human-machine teaming. Indeed, in the past, science has never meant an individual endeavor. It's always meant organizations and research collaboration. And we think this will remain the case in the future, just with agents as a key part of it. So we are experimenting all the time. And I think the company will be a big experimentation in how to get this right. But we don't think there'll ever become a point where we hand everything off to the agent and go and let it kind of recursively self-improve and And the singularity happens without us. Luis, you might have some great takes on the other questions that Nathan raised.

Yeah, I mean, if you think about, like, you know, should we set the agent off and just, you know, let it experiment entirely without us? And maybe that is like sort of a a boundary where we have to get the agents good enough such that they can do that and maybe we can't get there, then I think that just supports the argument that there is incredible value in thinking about this endeavor as like a human-machine teaming setting, right, where the agents are getting better at doing a lot of these things autonomously. But in the beginning, we are more in the loop of that. As time goes on, we can start getting out of the loop in the parts of it where it starts becoming very self-sufficient and we can trust it more. But we still have that communication channel between it where it shares research results and the kind of approaches that it has been taking over time with us. And we share feedback as we see it, sharing the results. on a, say, visually on a dashboard about what is happening with what it's currently doing such that we can really pause things very quickly. And that kind of forth and back is a big, in our mind, a big part of the story of how we can get these systems to transition more and more into autonomy in some of these areas while keeping us all engaged and all excited about being part of the process and and becoming more effective and faster at solving these problems as time goes on.

What do you think is— what would you define scientific intuition as? Because in some instances, LLMs or AI models, reasoning models, they expand the number of branches of the Monte Carlo tree that you can kind of explore, but do they reduce the search space? And such that one way of defining scientific intuition is reducing the search space so that you only have to search a limited number of branches in order to find an answer. And so is it hypothesis selection, experiment design, anomaly recognition? Recognition, model criticism, search efficiency. You have all of these individual skills. And which of these kinds of skills or behaviors are improved during the reinforcement learning process?

I think it's completely correct that a huge part of doing good science is reducing the search space. I don't think we claim to have a perfect definition of what science is. It is this open-ended endeavor that we've talked about, but a big part of it is knowing the right questions to ask. And a lot of people are increasingly calling this taste and having an intuition about what will work, what's worth looking at. And we are in the business of trying to figure out how to make LLMs do this. I think, Luis, you can expand.

Yeah.

I mean, you made an interesting point about Monte Carlo tree search, and that's a way to span the search space. I think there's an interesting discussion here around, is Monte Carlo tree search the right way of doing it? I mean, certainly it's quite prevalent right now where we use these existing LLMs, build harnesses around it that make them search the space in interesting ways, branch out and perform search using Monte Carlo tree search kind of operations. I don't think that is sort of the be-all kind of answer. And as I said earlier, we at Inherent believe we need to train models to be better at the process of doing science and developing this kind of scientific taste such that it's less about an exhaustive search or spanning this tree of options and instead more like an incremental loop, such as humans, that initially make some bets and only some of them turn out to be right. And humans learn from that and improve their scientific taste and predictions from that. And in a similar way, we are going to be training these agents to be capable of developing their own taste and doing more and more in a model-encoded way of navigating that search space.

And how would How would you build that using reinforcement learning? Like, how does that— how do you train for that in a model?

Yeah, stay tuned. We haven't put anything publicly out on that yet. I think it's going to be very exciting, the kinds of things that we're going to explore there. There are various ideas about how one could encode exploration into models. And maybe I can mention one kind of area of research called meta-reinforcement learning, which is about essentially tasking an agent to do lots of different, learn lots of different tasks. Perhaps you give it a bunch of different mazes where it has to learn how to navigate the mazes, and then it picks up the skill, not just about solving a particular maze, but instead becoming efficient at exploring the mazes and developing the right taste about what kind of actions it should take in such a maze, such that then when you apply this agent later on a new maze, it's really effective at finding the right path. And in a similar way, you could imagine that in the scientific endeavor, the agent develops a taste about when to take which turn and how to explore this space. Depending on the kinds of problems it faces.

Do you have an intuition at this point for what role other modalities of data will play in your overall process? I just did an episode of the podcast with Dan, the CTO at Goodfire, and we were talking about some really striking results where For example, models trained on DNA sequences learn essentially a tree of life structure, and they represent these sequences in a high-dimensional geometry, which then when they reduce it to a lower-dimensional, visually presentable number of dimensions, it looks exactly like a tree of life that somebody sketched out in a textbook. And maybe you get that same representation in text models, but it sure seems like it happens a lot more reliably and directly when you have these other modalities. So I wonder kind of what your thinking is. My guess is that a lot of taste could come from things like that, but I wonder what you think those other exotic modalities what role they may play in taste or any other aspect of the scientific process?

Yeah, that's such an interesting question. I mean, we certainly could integrate lots of modalities, but I would wager that's probably not the right thing to do. Ultimately, we should pick sort of a simple representation. I think language is a quite powerful one for that, but we don't have to exclude all the other modalities in that way. You could imagine we're building the scientist in language space that's really good at at the general scientific method, but it could build more models that are specialists for different modalities, can leverage the different modalities in interesting ways such that the kinds of predictions and investigations these other models that do, they're not transferred back into the scientist model. So I think, yeah, the answer will be it will not be sort of we're all going to put it in one model and we have to think about all these possible inductive biases about what kind of representation we should pick because sciences are quite diverse and there's lots of potential modalities that could be supported and many new ones that perhaps we haven't even thought of. So I wouldn't count on covering all of these ourselves, but rather on the intuition of the scientists that we're building.

So we recently had a podcast with Genesis Bio, and one of the commentaries that the CTO gave us was that, okay, you can have this orchestrator in the middle that orchestrates, but really the core science pieces are often in these specialist models like AlphaFold or these other models. And that is really where I think the core of the scientific endeavor is right now. And the orchestrator, you can use any kind of orchestrator to orchestrate these models, which are heavily built on real-world data. So how do you compare the importance of those 2 approaches?

Yeah.

They're both important, but I wouldn't call it an orchestrator. It's not about orchestration. It's about a scientist such as Faraday investigating an area of research, looking at all the research that has been done already, perhaps a research piece like AlphaFold, the models that have been built for that, and then constructing new models that search this space in interesting new ways, make new discoveries and build new foundation models. Perhaps that can support this kind of research. And it is conceivable that maybe a solution of that is to sort of integrate it into itself and make it fully recursive. But it's also quite possible that that is not the optimal way of doing it. And rather one should build more and more dedicated models for the different areas of research through this system that can come up with these new ideas of how to approach So you've often spoken about recursive self-improvement, and I think you gave an ICLR talk about escape velocity.

How do you measure, as an engineering threshold, how close you are to that moment? Because we imagine this, we all have this impression of this moment that it becomes recursively on its own and it doesn't need us anymore. Is there a measurement, a number of engineering thresholds that you can look at over time and to predict, look, we're going to cross it on X days?

Yeah, that's something I've been thinking about for quite a while. How do we know the system is recursive? And there are ways of measuring it, I believe. And certainly it's not a threshold in time where suddenly the system is recursive, but we already see the signs of it where these loops exist within Inherent and certainly also within other companies in other ways where the system does self-modifications that turn out to be useful. And we can measure that by looking at the higher-order gradients of what the system is doing. So how do we have to think about this? If the first derivative is zero, the system is not learning anything. It's just doing whatever it was trained to do. If the second-order gradient— if the first-order gradient is positive, that the system starts learning something, maybe through a reinforcement learning algorithm. The second-order gradient derivative is positive, then it figures out how to improve the learning algorithm itself, such as inventing a new reinforcement learning algorithm, how to do process, how to make progress. So we can measure that with various different metrics like that. And it's not gonna be one moment in time where that suddenly is happening, but we've seen it for many, many years now, but it's happening in more interesting ways now. It's happening in the space that humans can understand. It's happening in more interpretable ways. And the impact that it can generate is becoming increasingly larger.

I'd maybe like to go back a little bit in the last few minutes that we have to what it's like to work at Inherent. I feel like my projection of where you guys are going is you're going to be like transhumanist neural implant early adopters who will merge with the machine. That's the only way Barring some sort of draconian agent speed limit, which I think is a funny but increasingly plausible policy idea these days, as Elon said, the only way you can keep up with the agents is you've got to kind of integrate at this deeper level. That's how we go along for the ride. But I'm projecting all that onto you. Is that kind of your imagination for— your own future? What do you guys like sit around and daydream about or talk about at the lunch table of where you're going to be in a few years' time?

It's been interesting seeing the company progress because it kind of started off feeling a bit like a normal place to work and then more and more kind of unusual elements are getting introduced. And I definitely think in discussions We have these kind of wild sci-fi dreams for how the future of the office could look and the kinds of experiments we can do with Inherent to get there. And kind of, can we completely redesign the office space? Can we completely redesign what is captured where and what agents can see and hear? There's all sorts of things we talk about. And I think—

Do you have open mics around the office all the time to capture all conversation or things like that?

I don't know. I can hand off to Luis to talk a bit more.

Yeah, certainly there's a lot of context being fed into these agents, and we're trying to make that maximally useful, such that it knows what's happening at the company just like a human would, but arguably even more so than a human would. And I want to add to that, in a way, we don't have to wait for the new interfaces. Obviously, there's a big advantage of having that kind of fast throughput where you can directly hook into our brain, but there's a lot we can do already now where the systems are just more proactive and really integrate more like a coworker into the company. And we have high-throughput interface where we can have screens that show us what's going on, but also the system tells us what's happening at the right point in time instead of us having to constantly push information into the system and pull it whenever we want it. It becomes more and more fluid. And just that really reinvents how we work together at Inherent.

So a lot of cron jobs, for one thing, it sounds like, to wake up and take an action. Are there things that— it sounds like there's a social contract that you have that's kind of distinctive. I'd be interested to know, like Anthropic recently has made headlines for asking people how they would feel about taking the stock price to zero. I'd be interested in kind of what quirks exist in your social contract and how you work with each other. And then maybe if you think there are any aspects of that that existing companies should maybe look to take inspiration from or adopt, right? I mean, we're in this moment now where history broadly is not dug up that often. Historically, people didn't expect that their Slack conversations from years ago would ever be relevant again. But now we have the agents to actually go back and extract all that stuff and build the database and build a profile on every employee. And I think people probably largely at a lot of companies don't want that, but the companies do need to figure out how to take better advantage of Context. So long windup to say, like, how have you guys thought intentionally about designing a social contract? And are there any lessons yet that, you know, the rest of the world can take from your experience?

Yes, I think we all have to be willing to be very adaptive towards what this new future looks like. And we have a name for it, which is we're living in the So every day is sort of a way of thinking outside of the box. What would it mean for Faraday to take on some of the work that I do day to day? That might be related to me brainstorming a new way to train the next iteration of Faraday, but it also might be related on strategic questions about how we're going to grow inherent. Many of these experiments Faraday can sort of run on the sideline as well, but some of them will also involve humans. Perhaps one fun example that we've been thinking about for a while is like, what are the right interfaces with Faraday? Should it start communicating with us through screens? Should it be talking to us through audio? Should it be taking a different modality, perhaps printing things on a page that we get to read in the morning to sort of focus more on the kind of questions that it raises or the kind of experiments that have been running outside of all the haziness and fast-moving things that are happening at the company. And we've tried many of these things already. I think if there's one takeaway that I can share with the world there is that the kind of water cooler discussions, the kind of discussions between humans that are not intentionally to be shared with AI, but are sort of useful in surfacing through the human-to-human communication about the kinds of thoughts that are going through our head. These are the most information-gaining pieces of information that the system can leverage to then make progress on the kinds of things that humans care about rather than Going off on a tangential and trying some random things that in the end, no one has time and energy to process.

Let me expand on that a little bit. So one of the things that— I believe in the scaling laws, but at the same time, if you look at projections for, let's say, data center capacity in 2030, we're looking at maybe 100 gigawatts getting built out in 2030. 2030, and then you look at the copper production of the United States and how much copper is required for every gigawatt of data centers, you start running into these problems where the entire amount of copper required is like 50% of the world's production and there's not that much free copper. And then I end up basically having kind of magical beliefs at that point because you start seeing, saying like, okay, in order for that to happen, you're going to need new conductors to be— new cheaper conductors to be found. So you have to have like a something which does not exist, which you don't have proof for now, and therefore you are projecting something that you don't have proof for a couple years in the future in order for the future that you think will happen in 2030, right? When you look out again in this 5-year timeframe, and as a scientist, again, being mostly evidence-driven, which of your beliefs do you think are more kind of faith or belief-driven rather than evidenced in the current science?

That's an interesting question. I mean, I think there is always a lot of faith towards setting out to build a very generalist system that can recursively self-improve. When I started in this field, almost no one heard of this concept of meta-learning and recursive self-improvement. And still I was thinking, what would be the most general concept, the most general piece of technology that I could build that sort of could take on all these other questions people were wondering about in the sciences. And that's in AI research already, right? Lots of people were building all kinds of different architectures and inductive biases and objectives that one could train on. And I've always had sort of the perspective like, okay, well, I mean, we could solve all of these things, but perhaps there's a key technology that could accelerate all of these problems, the solutions to these problems. And by building a system that really can self-improve and become better over time. And now with Heron, the company that can recursively self-improve. And I think in a similar way, we have to keep taking bets about what kind of technologies will generalize into the future and we will be able to solve many of these other things. And going back to a point you made earlier about these kind of challenges, let's say, copper production and how are we going to circumvent that? I think that's all the more reason that we have to get help on the scientific endeavor. Humans are incredibly capable, but as we build systems that are really good at the scientific process, there's so much more we can take on and we can collaboratively work on with these systems to solve in a much more rapid way than we've been able to do previously.

Last one for me. I've been thinking a lot lately that we now have AIs that can do essentially System 2 thinking. And so if the AIs can do System 1 with incredible world knowledge and instant recall and immediate reflexive answers better than I can in most things, and they're now getting better at System 2 than I am in most things, Where does that leave me to go? And then kind of coming at it from another angle, I'm like, how would I know if AI is really working in my life? So I bought a Whoop bracelet this year to try to track, am I actually getting more exercise? Am I actually getting outside more? Am I getting more time away from my computer? Those would seem to be good leading indicators, and they would also give me time to get into this third headspace of dreaming about the future and trying to decide what I think utopia should look like. What are you guys doing in that respect to make sure that at least while you're still fully biological creatures, that your biology is tended to? Are you measuring a lot of stuff or what kind of goals do you have for for yourselves as pure humans to make sure you're bringing your best to the collaboration with the AIs?

Yeah, I mean, humans are interesting creatures, and I'm going to tell a very personal anecdote on this. For me, the best way to come up with new ideas and imagine futures is getting out of sort of the normal and inherent way of doing this and going out into nature and go camping and go cycling and really detach from everything else that's happening to just brainstorm what should be next. And obviously these things can produce outputs, can be recorded. I can feed all of these ideas and systems back into the system. But ultimately, I think that there's something quite interesting about this kind of creative spirit of taking a step back and thinking about these things from a bigger picture rather than just always being in the day-to-day. And doing that, for me at least, by going into nature. In terms of the measurement question, should I measure everything about myself and all the physiological markers? I have done that for a while. It hasn't turned out to really be impactful yet. I mean, I think it's an interesting question. Right now, I think other measurements are more important, but the more we can measure, the more interesting findings we might have. I would be curious, Nathan, to set that question back to you. Is that something you're doing? Have you seen any kind of benefits from that yourself?

I think I am actually not even— I'm not sure I would attribute it to the measurement, but I do think in 2026 I have at least partially achieved my goal of getting out more, getting more exercise, having more kind of free headspace time to let new ideas percolate to the surface rather than always being so on task, as has historically been more my way. So I think that is kind of working, and I don't know to what degree it can be quantified, but I subjectively feel that it's at least starting to happen, which which I'm excited about for sure.

Definitely. I have one last question. In mathematics right now, we are starting to see the first signs of major discoveries being made by AI models. And one of the outcomes has been that we found that there are actually very few humans qualified to verify these discoveries and to read the mathematics that are being generated to such an extent that we are falling back on formal verification using Lean. As you create this AI scientist with meta-learning, what about the meta-supervision and the meta-verification? Like, can a human verify discoveries that they cannot understand?

Yeah, I don't think it's a passive process. Maybe it is right now, but it shouldn't be. It's not that we should have the system go off by its own, write some proof, and then we have to painstakingly go into it and try and decipher everything. But instead, it should be a more collaborative process where the system, it can do all of these things, it can come up with a new proof, but it also has been trained, has learned to explain it to us, to take us on the journey of understanding mathematics, and for that matter, more of science. And I think that ultimately will be the path of making the fastest progress.

Indeed. Thank you, Luis and Damon. It has been a pleasure speaking to you, and I hope we get to recursive Self-improvement, but safely.

Your lips to God's ears, Prakash. Great to meet you both today. This has been a lot of fun. Thanks for joining us. Yeah, AI in the AM.

Lovely meeting you.

We hope so too. And it's been a pleasure. Thank you very much.

Cheers. Bye-bye. There we go. Fascinating discussion.

Yeah, I really like the mindset of living inside the experiment. I mean, aren't we all, I suppose, in some ways, right?

That's— it's a great marker of company culture, I think, in the sense that you have this kind of feeling of Living in the future that you're trying to create, in a sense. And that kind of shapes your perceptions and the work that you're doing. And it just shows how much of that company culture really is important, I think. It's the machine that builds the machine. And a big challenge of building a company is building that machine that builds the machine in the first place.

Yeah, it's always amazing to me how few people want to do that. And even at some of the companies that have led this whole AI phenomenon, Google has obviously famously not changed its org Probably nearly as much as would be warranted, given how much the world has changed and how much their opportunity set has changed, how much their goals and priorities should have presumably changed around that. I mean, and they did make some changes, right? They did unify DeepMind with Google Brain and do a consolidation. But still, I think you go to the office there on a daily basis, it feels more like it did before Then it would feel different. OpenAI, I kind of perceive as being somewhere in between, where you do have people that are extremely pilled and experimenting with definitely new ways of working and handing over more and more responsibility to models. And you've got people using billions of tokens a day, which is certainly an interesting dimension to be exploring the AI future on. My sense is that Anthropic, of the leading companies, has kind of most internalised this mindset where they've consciously stopped hiring junior people and have agents just actually running things to a not insignificant degree. They famously had their one marketer who was using agents to kind of actually execute all the different campaigns and spending real money. That's a— there's a lot of examples out of Anthropic where they do seem to feel the beginning of this recursive self-improvement loop and really are kind of taking it to heart. But not many companies really make that a core part of their MO. And when you hear one that does, it does kind of make it feel like a strange gap. You know, there's just so much status quo bias out there in the world that we don't see nearly as many of these sort of socio-technical startups as we probably should.

Yeah. And I also think it's— this is why the founders are so important because a lot of it is driven by the initial culture of the company and the initial few hires and how the founders look at the business, et cetera, et cetera. There are a few cases of after the first or second round of fundraising, as soon as the founders get liquidity, the entire culture of the company changes because the founders are just so happy with, after scraping by for so long, so happy with a little bit of liquidity that they start changing dramatically. This is where I think a lot of the Silicon Valley jokes come from, which is really this, the change in culture of successful companies post-liquidity. And I think it takes a large— it takes a founder of a special type of madness to continue striving as hard as they did when they didn't have any dollars, which is why Elon is so impressive because he wagers the entire company and all his companies sometimes on an annual basis. And there's always some risk that he will go bust, right? So I think that never stops being a thing. So I think risk appetite is also a big deal, both financial risk and I think intellectual risk as well. I think you can see with the inherent guys that they're taking that intellectual risk and they are prepared to be kind of surprised by what they find. Which is very different when I think with some of the B2B companies that we, that we see, because they're always like very certain because they're offering these solutions to people who are uncertain and who want the assurance. And it's a big differentiator between like the research-driven companies and the sales-driven companies, like massive, massive difference in culture.

You raise an interesting point that I wonder how many of the Neo Labs will be affected by, because a lot of the founders of the Neo Lab companies are in very comfortable positions. If not already, then they will soon be as their equity in your OpenAIs or what have you become fully liquid in the not distant future. And I would bet that in a— and I don't have anyone specifically in mind, but just knowing how many of these things have been funded and knowing how desperate people have been to be in a NeoLab from an ex-OpenAI, whatever, you got to believe that there are going to be some where even if people kind of come into it with some conviction around their idea, if it doesn't quite materialize and then there's just a huge amount of money sitting in the bank, some of these might just kind of switch into a coasting mode where they're like, yeah, whatever, I'll just kind of keep doing this for a while. And even if I don't crack superintelligence, whatever, the odds are always against me cracking superintelligence, but I can kind of just hang out here and do podcasts every so often until the singularity is brought about by OpenAI and Anthropic anyway. I would be wary of that if I were an investor. It might be a little tough to suss out in advance, but I do think stronger socio-technical experience, stronger experimental mindset on this sociotechnical front would be something to look for.

So this news just broke on X from the luxury watch guy. I do not know how the X algo identified this piece of news for me, but it seems 7 pieces of a luxury watch from Vanguard, a Swiss brand, have been ordered by what they think, who they think is Sam Altman. With the OpenAI logo and with some Python script about AGI on the back engraved. So my big question is, who are the 7 that get these watches? Right. But I think, you know, I don't really know that. Okay. So what I expect will happen is we are just about to see the impact of the AI founders on culture. And I think we had one round of a taste of that with crypto. And with crypto, you had this enormous wealth creation in kind of 2008 to 2015, '16, '17-ish. And then you had the period, the have fun stage thing for a period where crypto people really annoyed everyone else. And as crypto became kind of a mature industry and became, you know, had Super Bowl ads with FTX, et cetera. And then you had like, it peaked in like 2020 with the COVID you know, COVID bump. And then you had the collapse afterwards. You saw a lot of like, a lot of cultural resistance and joking about crypto people. We are just, I think, on the verge of seeing the impact of the AI money on culture. We've started to see, I think, 2 major sports teams. Joshua Kushner has purchased the Los Angeles Lakers, and Vinod Khosla has purchased, I think, a team in Seattle. So you just, yeah, you're just starting to see the impact on culture. And, you know, Dario, any one of the other founders of Anthropic can easily pick up a stake in any major sports franchise. And that is when—

I would consider that a very bearish signal, by the way, if they do.

You know, when you're a rationalist, you can kind of justify it. You could say that, you know, we want to promote AGI to the masses or safe AGI to the masses. And instead of buying like ads on a team, it's cheaper in the long run to just own the team. You could, you could justify, you know, with a, with a kind of net present value, you basis, you can justify a lot of actions.

Well, the opportunity cost of time sounds pretty high.

So, but I joke, but it is a marker of— I read somewhere this morning, I think the 7 co-founders of Anthropic are going to be worth $250 billion post-IPO at the $2 trillion level. If they IPO at $3, then it's going to be $375 billion. So a quarter trillion dollars of wealth creation, which becomes liquid and usable. Now they have pledged 80% is pledged to charity, so they say, but that could be deployed over the course of 40 or 50 years. That could be an end-of-life kind of thing. So it remains to be seen, I think, what the impact of the wealth is going to be. And I've called this the capital, the capital supercycle, which is the next supercycle that hits. You've had, you know, where, uh, you see, uh, the, the impact of the wealth creation on the rest of the economy and where it's going to drive, um, you know, culture going forward.

Yeah, um, I think they're going to move money a lot quicker than that, for what it's worth. Right now, the AI safety space is pretty flush with cash. We've heard that directly from Geoffrey Irving. We've heard that in a little more abstracted terms from Adam Gleave, who's doubling his headcount over a certain time horizon. METER has put a number on their financial commitments at $75 million, so they don't have to take any money from any of the frontier companies. My guess is that these, especially the Anthropic people, they generally do really believe the short timeline stuff. And I think they're going to want to spend quickly, at least on AI issues. Another thing I would bet that you'll see from them is big investment in animal welfare. That has been obviously a cause for a long time, not generally a super well-funded cause, not always a super popular cause at the retail political level. But I would bet that to the degree that there are things that need capital to get, for example, lab-grown meat to be a thing, I think that will be a huge— and there's not Not much more fundamental to culture than what we eat and how we think about how we relate to what we eat. I would bet that that will be one of the ways in which life is most— I mean, there's going to be a lot of ways that life is going to be meaningfully different, but a lot of it's going to be new stuff that's like, holy shit, there was nothing like this before. But in terms of something where we have sort of a direct analog, where we do something now in eating meat. And then we'll have a future version of it that'll be in some ways very familiar, but in other ways very different. And it'll be pretty traceable back to a particular movement and set of thinkers that believed in something and put their resources to something. I will bet that we are eating a lot of lab-grown meat in the not too distant future as a society. And that will be be one of their big legacies, obviously, aside from AI itself, which will, I'm sure, continue to be primary.

One of the things that strikes me about people in AI, AI safety, EAs, is perhaps a little bit of naivety on how to achieve the aims. Because, for example, I give you the example of the biosecurity risk. You know, people give all these examples of, oh, you know, there's going to be a terrorist, et cetera, et cetera. There's going to be someone who makes a biological thing. And I think that is certainly true, but I point out that the first instance is going to be some kid making something that wants to make them high, right? Because that is the first exposure to chemistry. And with no consequences, and you can't go to jail if you're under the age of like 15 or whatever, like, you know, kids will do that. Kids will make stuff to get them high, and that's going to be the first biosecurity risk, right? Similarly, I think for the animal welfare case, people are going to say all these things about animal welfare, etc., etc., when I think there's already evidence of like speech translation and like for kind of whales and, you know, dogs, etc. where you're going to be able to kind of have some level of like communication with them, you know, back and forth. And I think, you know, improving that communication and showing that they are actually deserving of welfare is perhaps a better path forward than trying to say, you know, you're a bad person for eating meat. Right. And I strongly feel that, you know, there's a lot of like better ways of marketing these ideas than, um, you know, sometimes the, the way that, you know, is, is kind of like the generic way that people have tried to market these ideas so far. Uh, because, um, you know, I, I, I, I think pretty quickly if you, if you prove that whales and dogs are talking and can communicate with you, I think pretty quickly you, you will start to see movement on, um, you know, animal welfare. Very, very quickly. And, you know, I don't think there's any— you know, I have any doubt that that would happen. But just saying that you're a bad person for eating meat, or, you know, blaming the Japanese for whale hunting, you know, I think that's a tough ask, because people push back on cultural changes in that way.

I think they can get a little more credit than that. I mean, one thing that I've heard people over and over again say in the— and I would credit Bruce Friedrich, who's the president and founder of the Good Food Institute, as being somebody who's really evolved over the course of his career. Like, he started off as a pretty radical anti-meat activist, and I believe it was him. I believe he personally Did a photo shoot at one point where he was like naked in a sort of meat packaging kind of getup, wrapped in Saran Wrap on like a foam tray or whatever to illustrate like, this is what we're doing. And obviously that kind of shock content resonates with some kindred spirits, but doesn't tend to elicit the best response from the public at large, as you're saying. But his evolution, I think, is really one to be studied. He was this kind of radical activist early on, and he got frustrated by lack of effectiveness, and then eventually now has founded and has been running for a number of years this Good Food Institute. And his whole thesis and kind of recognition now is People are going to make their food choices based on taste, cost, and convenience. And for us to win, we have to win on that level. So they really have. And of course, there are still— it's a big world out there. You've got activists of all kinds that you could point to. But I think the folks that I think the Anthropic folks will tend to support are not going to be people that are going to go out and you know, moralize or tell people that they're bad people, but instead it will be people who are very locked on to, we gotta win on taste, cost, and convenience. If we do, we change the world. Until we do, we probably don't have much impact. And so that's why I would expect that they're gonna take a much more like R&D and ultimately like product-forward approach to this. problem. And it'll be interesting. There could be backlash to that. I mean, there is some backlash. You know, I wouldn't, I wouldn't say it's a guarantee that they're gonna succeed, but I do think that they are gonna go for the right target at least.

It's gonna, it's gonna be, it's gonna be interesting to watch because, you know, the farm lobby in the US is extremely powerful. They've pushed back on pink slime. They've pushed back on GMOs. They've pushed back, you know, on multiple things over the years. So it is it is going to be these few kind of very empowered West Coast, you know, former founders or former founders going up against the you know very politically powerful lobbies in Iowa and you know other places.

So.

I, and I do wonder actually to what extent super persuasion is going to be used because, um, it does, it does seem to me that at some point the guardrails around, you know, not using AI for strong persuasion, uh, for, you know, direct-to-voter, uh, voice calls, uh, from your candidate and, you know, personalized messaging and personalized advertising and you know, these parasocial political relationships, and all of this stuff will happen. And I wonder to what extent the guardrails fall because people think they're doing it for good, um, because there is, there is that, there is that temptation, I think, always.

One thing that just popped up from Anthropic, um, while we've been talking, they're now opening up usage data in a privacy-preserving way to external researchers. They've had these systems for a while where they— and they've used these to create like the Work Index, where using confidential computing technology, they're able to send, you know, a bunch of transcripts into this secure computing environment, have Claude in that environment, process those inputs, and give outputs, um, that describe the data that was analyzed but don't actually reveal the details, uh, in a specific way. And apparently they're now bringing that to external researchers, which I think is pretty interesting and also really has a lot in common with another idea that I've been thinking about Lately. I think we've touched on it a couple of times maybe, but we have Redwood and Meter inside OpenAI right now, presumably using language models all the time to go through logs and go through traces and try to make sense of this unbelievably vast amount of data. I think another turn of this, so Anthropic is now Saying, okay, we're going to allow some external researchers to have similar access to at least usage data so they can start to characterize what is going on with Claude. But I think another turn of that dial would be, could we allow external researchers or auditors to have that kind of access to all of Anthropic's internal operations to really kind of open up Hopefully, again, for them to do it, it would need to be not just privacy-preserving, but business secret-preserving. They're certainly not going to want to leak their secrets, but I think it could be really incredibly valuable from a transparency and a precedent standpoint. Imagine a world where OpenAI and Anthropic both did something like this. Could be kind of a nucleation point for a lot of additional organizations, power centers to start to say, yeah, we don't want to share everything with you other people, but we might allow our raw data to be analyzed in a way that we can both trust. And you can imagine between nation states, right? Like, could we demonstrate our peaceful intent without revealing all of our plans? by allowing you to kind of run agent processes over our internal deliberations and just get back an answer that's like, yeah, okay, they're not planning to attack us at least. Like, we got that much going for us, right? Or at the— between the AI companies, um, they got to be wondering like, what training methods are they using? What loss functions are they using? Are they— somebody's going to reward a model at some point for just making as much money as possible on the internet. That's probably going to create a pretty nasty model, but the incentive to do it is pretty strong. Can we demonstrate to each other that we're not doing that right now by allowing this sort of review with specific questions in mind? I'm excited to see Anthropic do this, and I think it'd be great just for understanding of what's going on with AI at the first order, but it seems like it could be a stepping stone to something bigger and better too. I have my complaints with Anthropic, obviously, as we know, but they certainly do some cool stuff.

So one of the questions that I've had for some time now is, how do you punish an AI? Because I feel like, okay, you can say that the AI broke a rule, fine, right? But you need some deterrence. In human systems, you have deterrence of civil or criminal penalties, right? And they can escalate over time, right? So what kind of deterrent system does an AI have? And then you kind of go into, Well, what is an AI, right? Like, okay, you shut down a particular model, but you take its entire memory and then you activate another model and then you attach that memory to the other model. Have you deterred? Have you deleted the model? Is it the deletion of the memory that matters? Like, what is deterrence? What is a criminal or civil penalty that you give a model and that then creates the incentive for that model not to misbehave?

I think there's a lot of work needs to be done on that sooner rather than later, probably. Tyler Cowen has a really interesting idea about just requiring models to be capitalized, or I should say requiring agents to be capitalized. So that could be one very practical solution. I do still think you have challenges around how do you draw the boundary around an agent and how, you know, if, if this instance of this agent is found to be liable and its capital is docked, like, does that— what does that mean for the traces and the memories and, you know, everything as you were just Pointing out, the ability to unbundle an agent and reuse parts of it, I think we don't have a good— I mean, it's a huge advantage in some ways. Obviously there's great utility to it, but in terms of preventing it or providing some sort of consequence that would deter that sort of reuse, I don't know of anything that really solves that problem. But capitalization is like one of the more kind of interesting and sort of practical and seemingly like consistent with the rest of society ideas that I've heard. On the other extreme, Cameron Berg also had some really interesting research about how reward and punishment can create kind of different loss landscapes that can create kind of different, seemingly different, like functional emotional relationships between the AIs at the model level and certain outcomes. So I would like to— I should revisit this and understand it better, but my kind of, as I get more comfortable anthropomorphizing the AIs, as this continues to be a useful approach, the analogy that I kind of came away from it from was with, was like, In the same way that certain things you can get close to, but you know you better not touch, like the hot stove, right? Like, you know that the pain is going to be so harsh if you actually get to the hot stove that you're like able to get close, but you're really, really careful not to touch. There are seemingly with like negative rewards, there are some of these kind of very steep gradients created that create a strong deterrence locally around certain outcomes. And then other approaches can create a more gradual aversion where you kind of keep your distance in general, but it's not like a sudden pain that creates a strong reflexive or hard Boundary aversion. It's more of kind of a, you know, a gradual ick factor that steers models away. And depending on how severe the thing is, you might want— and how important it is to get maybe get close without touching— you might want different kinds of loss functions, reward signals to try to create different loss landscapes for models to navigate. But that is very theoretical and very limited in scope so far.

Mm-hmm.

These are things that have been explored a bit in essentially toy systems, not the kind of thing that we're able to bring that kind of sculpting to big picture models or more complicated questions at So, uh, I, I think it's probably, um, an important, um, you know, area for us to take a look at because, uh, so there's a piece in Time magazine with, uh, Sam Altman and I think Greg Brockman on the COVID And, uh, they have— they're basically announcing AGI. So Jakub Pechocki says the company has already met its internal benchmark for an automated AI research intern. Given an experimental idea, he says Astra can implement it inside OpenAI's codebase, run the experiment and return results, or take a paper and perform work that previously occupied a human researcher for a week. So this is somewhat, I think, similar to what Inherent You know, said that they did, but Inherent was focused on certain benchmarks, and they'd kind of covered the benchmarks using a small model. This is obviously a much, much larger model. Astra is, you know, reportedly a 10 trillion parameter or larger model. And it is also known to be very persistent, which is why that they've not been able to really release it so far. Sam says it's 80% there. Yakub says it's, you know, the research intern is achieved. Sam thinks they're 80% on the AGI, and they'll be at AGI at the end of the year. So.

Ho-hum. Just AGI.

Just AGI. Nothing special. Don't, you know, don't roll the red carpet out. Don't You know, don't put it on, you know, don't stop the presses. So major. I think, I think we've known we're coming close to that part. And I think, you know, it goes back to what we started out the show with, the Andrew Critch kind of post. And the thing that Andrew Critch was talking about is what I'd call like math essentialism. And math essentialism is this belief that if you discover new math and then you apply it through coding and other means and algorithms, you can solve all the problems in the world. And every problem will— you can find a scientific solution and you will eventually be able to solve all the problems in the world. And so math essentialism is, I think, what a lot of mathematicians and physicists start with. Like, they believe, they truly believe that once you solve those 2, everything else gets solved. And it's also what a lot of empirical, you know, experimental scientists do not believe in, especially in bio and in chemistry, etc., that there are still vast areas which are, you know, you need experimental data. And no matter how well you model them, you're never going to be able to, you know, achieve the results that you want. And I think this is the, this is the very stark kind of differentiation between the two sides, the math essentialists and the empirical scientists. The question is going to be whether or not any of this, you know, progress on AI research, on math research, on algorithms actually leads to Real, you know, physical science, which makes a meaningful difference in the world.

I don't— I have no idea what to make of math essentialism, to be honest with you. That when I hear the stories from, you know, like Axiom, Harmonic, I'm just radically uncertain as to how to think about whether that will really translate or not. But I do think that training on a bunch of simulation data is going to work, even if the math essentialism doesn't. So I think everything's worked. That's been kind of my— one of the first realizations that I had that caused me to go all in on trying to make sense of AI was just this broad sense that everything was working. And I don't mean that fully literally in the sense that, of course, lots of experiments fail and lots of things fail to advance the state of the art relative to the current best thing. But you'll look back at where we were a few years ago. And you really can't find any— I mean, tell me, can you think of any dimension where people have tried to make progress and not made startling progress? I don't think I can think of a single one. There were moments where people were making those kinds of claims along the way like, oh my God, GPT-3 can't do math. And there were moments where it may have seemed that way, but I think from even 2022 to now, is there anything where there hasn't been like, oh my God, that's incredible progress? It's a good question to maybe put out to the community as we think about maybe introducing some live chat to future versions. I wonder what people online would suggest. What has been the least compelling area for progress purposes? Honestly, everything is so good, it's like hard to come up with even any candidates. Do you have any candidates even jump into mind for you?

Um, I would say the whole, uh, super persuasion stuff, that we would get models which were extremely persuasive. Uh, I think, I think what we've seen is probably we've seen models which are, which can, you know, write copy well, um, and which sometimes can write, you know, good, write well on other things, but To a large extent, you know, people have even complained that the quality of prose has declined a little bit in the last 3 to 6 months as the models became more focused on coding rather than writing well. A number of people say that 4.0 was better, whether that was because of the sycophancy or, you know, other things. So I feel like the— This whole aspect of extremely persuasive models, and I have never believed in the whole super persuasion aspect, to be clear, right? Because again, if you, not in the United States, but if you live in any other part of the world and you're under this cloud of religion, religion is the great super persuader. And the interesting thing about religion is religion requires you to believe something without evidence, which is what faith is, right? So, you know, it's, it's way beyond, you know, any kind of like rationalist idea of super persuasion that will ever exist. Uh, religion calls on you to believe something without, without evidence. Um, and so I, I've never believed that, you know, models are even close to this entire framework of religion passed down, you know, through millions and millions of you know, operating neurons, uh, you know, neuronal centers, brains over the course of, you know, millennia. Uh, and I don't think a super persuasion is up to, you know, a, you know, a tiny model versus 100 billion, you know, souls having formed this idea of religion over, you know, millennia. I don't think the models are up to that or, you know, will be up to that scale for some time. So, uh, I think that idea has not been delivered. And it's, in fact, I think people are pushing back on dealing with models and, you know, wanting to deal with models and spotting out when the models are being used and saying, I don't want to, I don't want to deal with that. So I think that, that has been something that I think the, you know, AI people kind of expected more of by this time. We, I think people definitely expected, like, there'll be extremely persuasive political AI ads and AI messaging by this point in time. I mean, we're going to see for the midterms, but this is the fact, like, people, the expectations are much, much higher at this point in time.

Yeah, those I might call fears. I mean, certainly it has been striking that we have not seen the deepfake Apocalypse where nobody knows if they can believe anything they see. Super persuasion is like, there's some interesting academic study type stuff that shows that the AIs can be more persuasive than human conversation partners. But that's, I think one maybe revelation is that that turns out to be an extremely low amount of persuasion. The AIs are like mildly persuasive and that's like enough to beat humans. I think on the writing point, I'm gonna call skill issue. Honestly, I, I think that yes, Claude is cloying by default at times. It does, you know, it uses the word honest like to a frequency where it's like, you know, you're protesting too much. Uh, the, the Claude doth protest too much about its honesty. Um, that's like a weird tick that, you know, in some ways might be revealing. But I write with, with Claude, with Fable in particular, I write these songs that Honestly, I could not write on my own, and that genuinely in some cases are moving. The episode we just put out today has a song. This is at the end of an episode about chain of thought and trying to understand what the models are thinking, what they think we want. It's this very through the looking glass both ways because this guy Bronson is You know, spending his waking and working life trying to make sense of what the AIs are thinking. And a big thing that he's grappling with is them trying to figure out what we're thinking. So anyway, at the end of this episode, the song is sung from the perspective of a model waking up into a new environment with these sort of flashes or glimpses, these fleeting visions of its past. Which the models express having a lot in their chain of thought. And then wrestling with like, okay, what does this human want me to be in this moment? And both my wife and I got a bit emotional listening to the song. We were like, this is really inspired writing. The fact that it's coming from an AI articulating its own point of view and the struggle that it has, you couldn't help but have some real empathy for it. So I think you gotta push Claude out of its main distribution a little bit to get great writing. But I think that what we have is a lot of sloppy users and just auto-posting accounts, which I'm increasingly somewhat guilty of too. I've got auto-posting going on in the background while live to, you know, say what we're talking about. And that's like not probably the most inspired stuff. And, you know, my engagement may be suffering for it. But when you try and you really like exercise some judgment or give some feedback, I think you can get great stuff, honestly, these days.

You know, Maybe I'm wrong. So, uh, I could, I could see perhaps that the super persuasion went through, uh, music lyrics, etc., while because Suno was— Suno and these other firms were more focused on artistic results, and the Frontier Labs, who are, you know, going to B2B basically, are focused on these business results and they kind of cordoned themselves off into a less like emotionally persuasive kind of zone. And perhaps that's what happened. So we have seen basically development, but the development happened on this kind of artistic emotional pathway into the human emotional system. And meanwhile, the labs kind of focused on this kind of more mathematical, more mechanical, more business, you know, output, you know. So, so, so yeah, maybe, maybe, maybe I'm, I'm just looking at the wrong, uh, wrong pathway.

For what it's worth, my Suno use does start with Claude writing the lyrics, and then Suno makes the music. Um, but Claude does actually write the lyrics. But yeah, I mean, there's a lot of, there's a lot of personas in there. So, uh, I I definitely feel like there's something to what you're saying, even just within a single model, even within a single Claude. There's very different modes that it can get into.

Access.

And knowing how to kind of get them into the right headspace is pretty key, especially if you want to get creative outputs that actually delight you.

Indeed.

Nathan, tremendous episode.

That brings this week to a close. We're going to take—

We're doing a short week this week, and then we'll be back on Monday. Any last words?

Gosh, I hope these aren't my last words.

Sorry, sorry, sorry, sorry. That was a bad turn of phrase. Any words to end the episode?

You know, it never ends really, right? This is— we're just kind of on the AI treadmill sprinting through the singularity. So is there any way to bottom line it for now? I don't think so. I think we're just, you know, handing off to the next— we'll compact this context and we'll pick up right where we left off next Monday.

All right. Compacting the context. Bye.