Success without Dignity? Nathan finds Hope Amidst Chaos, from The Intelligence Horizon Podcast

Hello, and welcome back to The Cognitive Revolution.

Today, I'm sharing a special cross-post from my recent appearance on the Intelligence Horizon podcast with hosts Owen Zhang and Will Sanock-Dufolo. Owen and Will will soon be graduating from Yale College, and as you'll hear, they've clearly spent much of their senior year thinking deeply about the current state of AI, where we're headed, and what it means for all of us. And I was really impressed not only with the quality of their questions, but their ability to challenge me with follow-ups that effectively steelman the most relevant counterarguments. We start with the fact that while AI timelines have compressed dramatically over the last 5 years, genuine experts still disagree radically on critical questions.

Having established what I hope is appropriate epistemic humility, I then go on to call it how I see it.

In short, the singularity is near. Interpretability science proves that AIs are developing increasingly sophisticated world models. And with reinforcement learning scaling now clearly working, AIs are no longer simply imitating humans and likely won't be limited by what we know for much longer. The potential upside of this is, of course, incredible. The value that I've got from using AI to navigate what humans have discovered about how cancer works and how to treat it has been invaluable. And the prospect that we might cure the majority of human diseases in just the next decade or so is obviously extremely exciting. That said, the risks are also very real, and they will remain serious for as long as we lack a solid understanding of how AIs work and why they do what they do. My p-doom remains somewhere in the 10 to 90% range. And yet, at the same time, I've become at least a little bit more optimistic that we might actually build robustly good AIs. Because scaling laws at least seem to imply that powerful AIs can only be created with massive resources, The 3 companies competing at the frontier today are at least reasonably responsible actors, and our best alignment techniques are working better than I had expected. Given these fundamentals, it seems at least plausible that a defense-in-depth strategy, which combines techniques like Goodfire's intentional design, Redwood's AI control, improved cybersecurity through formal verification of software, and various forms of pandemic preparedness, could collectively be enough to keep society on the rails. We touch on a number of other topics as well, including the US-China rivalry, and why, especially in the context of the Department of War's recent attack on Anthropic, which, I'm sad to say, has us looking more and more like China all the time, I would rather bet on figuring out a way to cooperate with our fellow humans than bet everything on AI researchers' ability to steer AI advances in a way that will ultimately work For us humans.

I appreciate Owen and Will for allowing me to cross-post this conversation, and I definitely encourage you to subscribe to The Intelligence Horizon.

Their recent conversation with former OpenAI researcher Zoe Hitzig covered the evolving ways that people are using ChatGPT, variations on universal basic income, AI governance models that emphasize a decision-making process over specific principles, and why she believes that these kinds of structures will probably have to come from outside the frontier companies. For now, I hope you enjoy my conversation with Owen Zhang and Will Sanock-Dufolo from the Intelligence Horizon. The Cognitive Revolution is brought to you in part by Google, makers of the Gemini family of models and much more. One of my big AI goals for 2026 is to find ways to spend less time at my desk and more time exercising and outside. The challenge is that I'm genuinely so obsessed with keeping up with everything that's happening in AI that it's hard to pull myself away from the screen. Google NotebookLM gives me the best of both worlds. By creating podcast-style deep dives about whatever I'm curious about on any given day, it helps me keep learning even on the go. If you haven't tried NotebookLM for a while, you should know that it's become a much more steerable research and thinking partner. These days, you can select short, medium, or long for audio length, and you also get a free text field to steer the direction of the conversation. For AI research, I always ask for rigorous, literal, technical explanations with no analogies. Recently, I used NotebookLM to study a paper by Google DeepMind researcher Rohan Shah, which attempts to set upper bounds on how much reasoning different kinds of models can do without needing to externalize their thinking in a chain-of-thought. The hope is that by establishing these limits, we can better calibrate how much confidence we should have in chain-of-thought monitoring. And standard transformers, in fact, are much more limited than alternatives like state-space models— a notable upside to transformers that we shouldn't take for granted. If you're trying to keep up with the pace of AI research while staying physically fit, or you're just a natural audio learner, Give NotebookLM a try at notebooklm.google.com. Thank you to Google for supporting The Cognitive Revolution.

And now, on with the show. When it comes down to the kind of core question I think a lot of people are getting at is like, is this AI thing going to kind of fizzle out before it really becomes a big deal, or is it going to be a huge world-altering deal? I'm very much confidently, clearly in the camp of it's going to be a huge, huge deal. And the details are where I think the discussion or the debate remains. remains now, not for me at least, you know, whether or not the overall trajectory of AI is going to take us to something that is powerful enough to be transformative. One of the strangest things in the world today, full stop, is the fact that the disagreement among very plugged in, very informed, very smart people has not really been reduced much at all, even as we've gained a ton of information over the last couple of years about the trajectory of AI. I think that is super strange, and I'm honestly Pretty confused by it. The one thing everybody seems to agree on is like the timeline on which we should expect this has come in. In today's world, if you say you don't think you're gonna see AGI until 2035, you're like an AI bear. But only 5 years ago, that was considered to be quite aggressive. And, you know, most people were more like, I don't know, 2050, maybe not in my lifetime. So there's been this massive compression of timeline. There's obviously been this huge jump in capability. And yet on these fundamental questions of like, what's gonna happen? You know, there's, there's still total disagreement.

Our guest today is Nathan Labenz, host of the Cognitive Revolution podcast. Before switching to full-time podcasting, Nathan founded Waymark, an automated marketing platform for local businesses that pioneered the use of generative AI to produce video ad campaigns. After leading the company for 6 years, he stepped back to focus full-time on understanding and communicating the trajectory of AI. As host of The Cognitive Revolution, he has conducted hundreds of in-depth interviews with AI researchers, founders, policymakers, and investors, and has become a go-to source for people trying to keep up with what's happening at the frontier, including us. Nathan was also a member of OpenAI's red team, where he was among the first outside users to interact with GPT-4 before its public release. Welcome, Nathan. Great to have you on today.

Thanks for having me.

I'm excited for this conversation, guys.

So first question, are we on the cusp of AGI?

Starting with the easy ones. Well, okay, first of all, as I'm sure you guys are very well aware, what exactly do we mean by AGI is a slippery question, at least, that has people talking past each other quite a bit. So I don't think there's any super privileged definition. What I do think we're pretty clearly on the cusp of is powerful AI that is better than the vast majority of people, although perhaps not the very few top experts in a given domain, at pretty much all cognitive work. That seems like pretty clearly on the horizon. And I think it is absolutely going to be enough to be transformative to the economy, to daily life, potentially even bigger things than that, like the very nature and status of the human species. And I think that we will have that kind of regardless of whether or not there are some niche areas where humans retain an advantage, which my guess is probably will be the case. Certainly one of the big things that has become clearer over time as AI systems have gotten better, is that they are jagged in this weird way where they have certain things they just do amazingly well at, other things they kind of weirdly struggle at. They're not very adversarially robust, for example. They're easier than humans, I would say, still to trick. So there's going to be weirdness. And I think throughout the conversation today, probably a big theme will be to expect weird things to happen. But when it comes down to the kind of core question I think a lot of people are getting at is like, is this AI thing going to kind of fizzle out before it really becomes a big deal, or is it going to be a huge, world-altering deal? I'm very much confidently, clearly in the camp of it's going to be a huge deal. And the details are where I think the discussion or the debate remains now. Not for me, at least, whether or not the overall trajectory of AI is going to take us to something that is powerful enough to be transformative.

Okay, so it sounds like what you're saying is that there's no doubt that we're going to get massively powerful and transformative AI across domains. Maybe there'll be some tiny little bit where human experts still have an advantage in specific domains. Maybe there'll also be comparative advantage. We'll get more in depth on what specifically those concerns are. But I guess just to hone in, are these more so uncertainties as to whether AI can fully generalize, or are you pretty confident that for the foreseeable future there will be these gaps? between human experts in specific domains and problems like adversarial robustness. So just to clarify what your position is, is this like a prediction, or is there like, this is where your uncertainty lies? Your uncertainty doesn't lie in AI being transformative?

Yeah, I think the latter. I wouldn't be shocked if there are additional unlocks that allow AI systems to truly undeniably surpass what humans are capable of. I think You look at the natural world, and clearly humans did that to every other animal species that existed before us, right? So there is a historical precedent for some new kind of mind showing up on the scene and blowing away all the other minds that came before us. I don't think we have any reason to believe that that couldn't happen to us in some law of physics guarantee sort of way. So I definitely think that's possible. But in terms of what I could confidently foresee, I don't think it is clear that that's going to happen in the next few years. But what I do think is still, again, clear is that we are going to have systems that are powerful enough to be transformative across almost all the questions that we care about in terms of what is society going to look like, how are we going to organise, are we going to need a new social contract? All those things seem to me pretty clear. And then people quibble often around the edges of, could it? I mean, There's some pretty esoteric jobs out there, right? So will an AI system ever be able to be as good of a sommelier as the best human sommeliers? Well, I don't know. Is anybody going to be motivated to train one to try to do that? We don't really have a lot of taste AI at this point in time, and maybe tasting will be forever the domain of humans. But trying to identify these little niches where we may have some really longstanding durable advantage, I think too often distracts from the big question of like, is this going to change just about everything that matters to us? And there I clearly come down that the answer will be yes.

Great. So let's talk a little bit about how maybe we get to this transformative or extremely powerful AI. Do you think current paradigms, I think the one that comes to mind the most and which is discussed the most is scaling RL, is going to be sufficient to get to that transformative or powerful AI that we discussed?

Yes. I mean, I think it, Probably is. And I also think if it's not, we may never quite answer that question in the sense that I do expect that we will continue to see new conceptual advances in AI research. The field is growing. Everything. One of my sayings is everything is going exponential. So that is the number of people that are working in AI research. It's the number of papers, it's the number of experiments being run, it's the amount of compute that people have to run those experiments on. It's the datasets that have been collected. It's the RL environments that are being built out over time. And I do think all those things are going to probably give us some new conceptual unlocks such that we will never probably answer the question. And this already kind of happened with pretraining, right? If you rewind 2 to 3 years, there was a time when people were like, well, we're kind of running out of data. Can we really scale this all the way to AGI? And nobody's talking now about, can we scale pretraining all the way to AGI anymore because there's been a new thing. And so now the new thing is on top, and it's like, well, clearly that's going to be part of the mix. Will this exact pattern or this sort of shape be enough to get all the way to AGI? My prediction is that, again, what does AGI mean? But my prediction is it probably is enough to get us to systems that are transformative. But if we fast forward to 2028, And look back at this conversation, we'll probably say, well, nobody's really asking that anymore because we do have a couple of new things that have come online. And so now we have a little richer sense of what the shape of it's going to be. And we're either getting there or we're not. We maybe still are missing a little something at that point. But I would guess that we would look back at the current thing and say, yeah, clearly a couple of things have been added and they were a big deal. And so the question of is exactly what we have in February 2026 enough kind of ends up being beside the point in the final analysis.

Interesting. So what you're trying to say here is that in the same way that there were questions about whether pre-training would take us to this extremely transformative AI, now that we have these new unlocks such as RL, for example, like we discussed, who knows, in maybe 2 to 3 years or whatever timescales that we're talking about to reach transformative AI, there might be other conceptual progresses that are also made that build upon things like pre-training or RL that will get there. So I guess my question specifically was, do you think there will be more of these conceptual progresses that need to be made to get to this transformative AI that we're talking about? Or do you think the RL paradigm is going to get us there?

I think RL probably would be enough to get to AI systems that can do most of the cognitive work in the economy, for example. I mean, it seems like, honestly, we're already reasonably close to that, and it doesn't seem like we're anywhere close to done. And by the way, if you listen to the lab leaders, the frontier model developer leaders, they are still saying, by the way, too, that Pretraining is still working. It was never really the case that pretraining stopped working. As far as I know, those scaling laws basically have held. I think what happened at one point in time was the next step on the pretraining frontier was becoming really expensive. And then they found another kind of maybe not fully orthogonal, but pretty different direction to go with scaling posttraining with the RL paradigm that we have now. And that was just much bigger bang for their buck, at least at that moment in time, right? Like, they had already gone pretty far up the pre-training curve. They hadn't gone very far up at all the RL curve. Now, presumably those things were going to kind of even out. A general kind of economic theory would be like, people should be investing in one until the marginal return decreases to the level of the other, and then they could maybe invest in both. So if you find some new path that's like, oh, this is really giving us huge ROI, you go hard at that path for a while, but then that kind of hits some diminishing returns. Now you're kind of back to, okay, well, maybe we need to do all these things more at the same time. So we'll do a little more pretraining, we'll do a little more RL, and we'll do maybe more of that mystery third thing, all advancing in tandem. Right now, I think we're advancing. I don't have insider information on exactly what these ratios look like, but I'd say it's probably roughly the case that we're close to, if not at the point where additional compute going into RL is roughly giving the same kind of returns as additional compute going into pretraining. And so both are going to be places where frontier companies can invest for the time being. I guess another way to think about this is people also ask about generalisation in RL. And I think, again, it's kind of worth unpacking, what does it mean to generalise? What are we talking about generalising? One way to think about the question is, if we do a bunch of RL on a model, does the model generalise to all sorts of new things? And there I would say, again, there's probably another way to break that down, which is like, there are domain-specific skills, and then there are more kind of cognitive or even like metacognitive skills that work across domains. So one of the biggest, I guess we'll call it the aha moment for me and for DeepSeek researchers and for the R1 model that they were training was from the R1 paper. This is January 2025. They reported this, what they call the aha moment, which is in their process of doing RL on an already pretty capable base model, of course, right? They found that these previously unobserved higher order cognitive behaviours started to come online. The aha moment in particular was In the reasoning trace, the R1 model gets to a point and it says, oh wait, this is an aha moment. I can come at this from a totally different direction. And this is something that hadn't been observed much. I mean, it's out there in the pretraining data. There's at least some examples, right, of people kind of documenting their own chain of thought and getting to these aha moments and realising, oh, I was coming at it the wrong way. Now I can come at it this other way. But that hadn't been observed too much in AI systems, reinforcement learning seems to be bringing— clearly has brought that sort of thing out. And now we have these long reasoning traces where the model will kind of come at the same problem from a bunch of different directions. And so what generalises, what doesn't generalise? I think you probably can't take a model that has never been trained on a particular domain and expect it to go into that domain and be successful. But you can expect some of these meta traits to generalise from one area to another? And then if you zoom out the farthest and just say, does RL as a process that companies or organisations apply, does that generalise? And there I think the answer is definitely yes. It's just a question of getting the reward signal dialled in to the point where it actually works. And that's definitely easier in some areas than in others. And so we do see in areas where it's easy, like math and programming. We see fast progress there relative to things where it's harder to get a clear reward signal. But still, I think we are quite obviously making that work. Episode of the podcast that's coming out, I think today as we're talking, is with the head of health at OpenAI. And I can tell you from personal experience, latest models are absolutely on the level of attending physicians. My son, unfortunately, has had cancer over the last few months. He seems to be very much on track to be cured and be all better, which is fantastic. And I probably wouldn't be here talking to you if that wasn't the case. But I've had occasion to really use the latest models intensively in a medical context, and they're absolutely on the level of the attending physicians. They know a lot more than the residents, and they really are step for step with the most senior doctors at the hospital. How is that happening? Well, they've worked with 250+ human doctors closely at OpenAI to create training data, to grade, et cetera, et cetera, et cetera. But now their latest models are also outperforming their human doctors when it comes to the task of evaluating AI outputs. So there, there are these kind of thresholds that they're crossing where it's like, you know, it was really hard. I sometimes think of this as like spinning a big wheel. You know, like your first pushes on this big wheel don't move it much, right? But as you build up this momentum, as the flywheel really starts to turn, You start to hit these thresholds where it's like, well, now we have a model that is outperforming our humans. Think of all the work that they had to put in, thousands and thousands of hours and millions and millions, hundreds of millions of dollars potentially, to hire hundreds of doctors to do all this work. Now they've got a model that is beating the doctors at evaluating outputs in the medical domain. And so that totally changes the game. Those thresholds will be crossed at different times for different domains. But I think it's like safe to say that in most domains, if there's any sort of objective ground truth, or even like a high level of agreement among professionals, you can get there. It just takes more time. Hey, we'll continue our interview in a moment after a word from our sponsors.

Okay. So just to like go back a little bit, it sounds like, okay, a lot of your confidence, um, that we'll have transformative AI very soon, it hinges, I'm not sure to what extent it hinges on like Your confidence that we'll find some other paradigm if a new paradigm is required on top of pre-training RL. You also sound very bullish on RL, but maybe to steelman the other view that's like, okay, if RL doesn't work, then we actually might not find another paradigm. Is that like, okay, so RL is a very general machine learning principle, right? It's just that like you reward the model for doing the correct thing on a task that you can like define a reward signal for, which is like kind of like a maximally general machine learning principle. And it's also something that existed for a long time. It wasn't something that was just recently discovered after we got LLMs. It was one of the foundational paradigms in machine learning even prior to LLMs. And so I guess the skepticism is maybe it's actually not that easy to find some other paradigm. Maybe pre-training LLMs on text data was this novel thing, and then we just applied this thing that had been in machine learning for a while. namely RL. But then the next leap is like, there's no precedent for it. We don't see any big promising thing. What do you think about that skepticism?

I think it's probably not going to play out that way. And if it did, I still don't really think it matters that much. It probably shifts the timeline a little bit, but the RL will continue to scale. The amount of compute coming online is, again, that's exponential, right? So there's just tremendous amount of additional resources to be thrown into this. One thing we're doing a bit, but not a ton, is just like using the signal from the world. Elon's got a plan with xAI to, and I think he's got a real advantage here, to just have the AI solve the same super hard problems that the engineers at Tesla and SpaceX and Neuralink are solving. And so he's just going to give them a computer, say, here's all the professional software, here's the problem. You got to solve it, right? There's like a never-ending supply of those problems. And I wouldn't be surprised to see Grok get to like Tesla, SpaceX, Neuralink engineer level just based on the fact that those problems are there to be solved and they've got the compute to keep trying. And what's really the fundamental barrier there? I don't really see it. I honestly think that the new paradigm is probably more about usability than it is about capability. By which I mean, people have a lot of complaints when they use models because they're like, oh, it didn't really do what I wanted it to do, or it didn't really understand me, or what have you. And clearly, one thing that they're not great at doing is going into a new environment, kind of scoping the situation out, getting the vibe, picking up that subtle feedback that people give each other, and gradually kind of figuring it out and becoming a useful contributor. That's like how a lot of people go from day one to effective employee in their jobs. And AIs don't really do that in the same way, obviously, right? They don't have continual learning. They are able to manage their own memory somewhat, but they're not that awesome at that. That's another one of these cognitive skills that does generalise, though. Once you're good at managing your memory, you'll be able to apply that in new domains, but they're not that great at that yet. And I think that prevents a lot of people from getting value. More often, I think that not that the model can't fundamentally do the thing, but that it doesn't have the context. The person trying to get it to do the thing doesn't know how to assemble the context or doesn't believe that it could do the thing such that they're willing to invest the time and energy to give it the context. And so that unlock might just be like, now you don't have to do that anymore. It will kind of figure it out on its own. And this is maybe a way to expose, in a way similar to instruction following, right? Like, if you were really good at prompting GPT-3, you could get it to do a lot of things, but it was a weird art to prompt GPT-3. And to a much lesser degree, but still somewhat, getting value from the current models is still kind of a weird art. You kind of have to have an intuition for them. You kind of need to know how to assemble context and give effective instructions, and people aren't that great at that. So that next version might not even be so much about allowing them to do qualitatively new things, but just making the barrier to use much lower so that people can just be like, hey, AI coworker, welcome to Slack, you know? And then over a week or whatever, they just kind of ramp up and gradually get it.

Yeah. Interesting. So I think the pushback there is the verifiability problem becomes a little more apparent in these specific long horizon, excuse me, situations. I think when we talk about verifiability right now, we clearly see that with model capabilities, like you said, with Claude Code, that they work great for math, they work great for, they work great for code, and they work great in these settings where you can clearly check the answers and whether or not they're right or wrong. Right. That's how you kind of create this reward signal. But when we talk about these domains where there's no clear verifier, for example, when it comes to writing a good essay, what does that mean? Give good therapy in the healthcare settings. And then we emphasize that even further by placing it in a longer horizon context, such as the one that you're talking about with instead of having a human in the loop at each step, prompting it in a specific direction, giving it a broader task and having it go in a direction for a longer time horizon. It gets even harder to solve this verifiability problem, right? How do you make sure that you get high-quality signals? How do you make sure, um, tune these systems in a way where they can sustain autonomous work for days or weeks at a time? And so I wonder if you have any thoughts in terms of how we kind of go in that direction and solve this seemingly very large verifiability problem that hangs over our head when we talk about areas that have long time horizons?

Well, again, I'll take one beat to say I think the recent trajectory shows that this problem is being solved. I'm sure you are very familiar with the METER graph, which is everywhere these days. It's basically going vertical at this point. It's to the point where the METER people are like, we are really struggling to have tasks long enough to even be able to evaluate these things on. And it's also kind of worth noting, too, that that's a bit of a challenge with humans, right? We hire people on much less than a week's worth of work. And it's not always super easy for people to agree, did somebody do a good job on that month-long project or not? You can usually tell if they totally crushed it or totally sucked. There's often a lot of disagreement in organisations about, well, maybe it was actually harder than we thought, or the ingredients for success weren't really there. I mean, there's often a lot of fuzziness in in this stuff, even in human context? I mean, there's a bunch of techniques that I think are being used. One is rubric rewards. So Elon likes to talk about things like, does the rocket fly? The ultimate ground truth is, if I can send this thing into space and I can come land it down on a pad standing up on its tail again, then clearly that worked. And yet the reward for that, that's an expensive experiment to run. You couldn't just launch a million rockets and have all of them, most of them crash to find the ones that worked. So there is a challenge there in terms of sparse reward and the cost of the experiments. I think what is happening a lot is that people are defining rubrics of things that they want to make sure the AI does well. They're probably working with AIs to develop those rubrics. Again, in the OpenAI health context, they created a benchmark called HealthBench, where there are 49,000 evaluation criteria. So all these different tasks, puzzles that the AIs have to figure out. And it's not just like, did you get it right or did you not get it right? But it's a painstaking effort to really flesh out all the different things that would matter, that would make for a complete, awesome, best-in-class answer. And then the AIs are not scored 0 and 1, they're scored on some sort of scale that's like, you may have got 0 out of 25 things that you could have got on this question. You may have got 5, 10, 15, but that gives you enough of a signal that you can kind of climb that hill. And it really does seem to be working. I think that is going to work pretty well in domains where there is a professional consensus, because I think that's how people evaluate each other, too. Multiple choice tests. But there are also, like, you got to kind of show in a medical context, you're a student, you go through these in-person training processes. They say, watch one, do one, teach one. And so as a medical student, you watch, and then later you start to do, and then you get a lot of feedback, and eventually the next thing you know, you're the one teaching. I think the AIs will pick up those signals. And then there's other things where it's taste, and I think that'll be a little bit different, probably. But if I had to guess, what is a good novel? Well, first of all, there is no consensus on that, right? Even You can find somebody who hates even the most universally critically acclaimed novels, and you can find somebody who loves something that everybody else thinks is trash. I think what we'll see there is kind of taste-based communities coming together to shape models for their own tastes. So in other words, you might start with a base model and you might want it to write romance novels, or you might want it to write anime, or you might want it to write hard sci-fi. And what it means to be good in those different genres is quite different. But what you do have is fans of those genres that can engage with outputs, give their scores, and shape models based on what they like.

Okay.

And I think that can work.

Yeah. So I just want to go back to the— even within tasks where there is a consensus, right? I see what you're saying, that there's ways to build minor consensus is within fiction communities or something like that. But so even the question of, does the rocket fly? I think there's an argument to be made that this is categorically different than a chatbot giving out useful medical advice, right? When a chatbot is giving out useful medical advice, it's not a long context task. It's easily transcribed into text, right? It's sort of what LLMs are clearly going to be good at. They're good at They're LLMs, so they're good at— they're native to text. But something like, does the rocket fly? It's like, okay, even if there is obviously a consensus on what it means to make a rocket fly, it seems like there's an argument to be made that the signals on how to perform that task are very noisy and you need a ton of them, right? You need to, like, you know, like, let's say you're like a project manager for the rocket flight team, right? Like, you need to understand what's feasible within engineering, which, like, okay, maybe LLMs are better at that. But then you also need to like know how to manage humans and like, or not maybe AI agents in this case, but like, you know, you need to know how to like manage people and to allocate tasks and how to like, what is going to be like feasible with the amount of like money you have and how can you raise money from investors and stuff like that. And so it's like the longer time horizon, the more agentic a task, the more like little itty bitty signals are required on your day to day. work to be able to complete the task. So I think, and going back to what you said about this sort of flywheel thing, we're like, okay, if we have, we work really hard and we get a lot of medical experts to provide this high-quality data, then we get LLMs that are pretty good at giving medical advice. And then all of a sudden, they can also, they're actually better than human doctors at verifying the output of medical advice in the first place. And then we get this flywheel effect and we have a quick and robust verifier. But maybe the argument is we just never get over this flywheel hump for long-term, highly agentic tasks, precisely because it's not even a question of whether or not there's consensus on what constitutes a good outcome. It's like, can we define enough reward signals and do enough rollouts to get to this point?

One thing I find helpful sometimes is just to try to reflect on what I do, what is my experience like, and you really try to interrogate it for, is there something really magical going on or not? And I think I have to say, really long horizon agency is pretty rare among humans, right? I mean, you do have these people who, like Elon, who keeps coming up, say, my goal is to make humanity a multiplanetary species, and if it takes 30 years, so be it, right? So I would say scepticism around, like, Will we get AIs to be multi-decade horizon planners that can sort of show up with the fierceness and determination required on a day-in and day-out basis to really make that happen remains pretty valid scepticism, in my opinion. At the same time, you look at what most people do and what drives the economy, and it's like much shorter term than that. A lot of it is like, I kind of keep showing up one day to the next, and every day I kind of start fresh. I went to sleep and I kind of shut down. There's sort of a discontinuity of consciousness between workdays, which I'm not a big analogy guy, but if you squint at it, you can sort of make an analogy between different context windows. And there's kind of a somewhat similar resetting type of moment. And what do I have to do? I have to kind of reboot myself and be like, What was I doing yesterday? Where did I leave off? What did I accomplish? What was still to be accomplished? And most of that stuff, you know, I could write down if I, if I had like a worse, if my integrated memory was worse, I could compensate for that pretty well. You know, at least on sort of a week-long or maybe a month-long or maybe a quarter-long basis. If I got really good at the end of the day, at the end of each day of saying like, here's everything I did today, here's what went well, here's what failed, here's what's next, here's the feedback I got. Here's what I think I should do in the next session, and then wiped all that out and came back and looked at that scratchpad again to start my day tomorrow. I think over time I could get pretty effective at that. So I don't— and I think this does connect back to also to the certain kinds of skills do generalise, and I do think this is one of them, right? Managing your own memory, writing notes for yourself to kind of document what happened and what's supposed to happen next, giving yourself a sense of where you are on the overall story that is unfolding. I think that they're not great at that yet, but they're getting decent. And when my Claude code crashes these days, a lot of times I'll just kill that tab, resume in the next one, and then I'll just be like, resume, or like, sorry, we got cut off. Like, my internet went out. Please pick up where you left off. And they're able to do it again. They're not awesome yet, but they're way better than they were even 3 months ago. So I think it is hard for me to see how that doesn't extend, especially because it is a relatively new thing. Right. And most of these things that pop up, they have at least a few generations before they kind of level off. It's hard for me to see how we don't see, again, a pretty steep meter curve is currently vertical. It can hardly get steeper than it is. But I think we've got at least a few generations of very steep progress there. And then maybe there's some other kind of thing where it's like, okay, sure, you can do a quarter worth of work when somebody gives you a project. Can you figure out what the next big civilisational advance could be, like some of the true human visionaries? Maybe not. Maybe that's a different kind of thing. In a way, I kind of hope that we stop there sometimes, because I talk up a lot what I think AI is capable of, will be capable of. It's easy to mistake me as like a booster. I'm actually kind of afraid of that. I think it would be great in some ways if we did find certain fundamental barriers where it's like, hey, I can delegate a month or a quarter's worth of work to this thing, and it might be able to do it in a day for a couple hundred dollars. Like, wow, amazing. If we could kind of park it there and not have it go to the point where it's doing multi-decade level planning, that might be a really good thing. That might be sort of the sweet spot where we get a lot of the advances that we want and the better quality of life and, you know, all the abundance that people dream about. And we don't have so much risk of losing control. So I don't, you know, I don't think every advance, by the way, is a, is a good thing. But I just don't see fundamental barriers on the horizon, at least.

Okay, great.

Hey, we'll continue our interview in a moment after a word from our sponsors.

So let's talk about LLMs more generally and how we feel about that architecture overall towards reaching this transformative or powerful AI. What are your thoughts on the general idea that LLMs fundamentally cannot be the core of this journey towards reaching this transformative AI? I think Yann LeCun is the biggest person behind this hypothesis where this next token prediction is the wrong objective entirely, and we need something completely different, something like world models or energy-based models that have a real understanding of the input that's coming in and the output that it creates. I think this is important because there's a lot of intuitive thought processes behind it in which, you know, the way you're learning statistical correlations over languages isn't necessarily how the world works and that you need something different to really achieve that understanding to create like powerful AI. Do you think that's the case or do you think LLMs do take us to this next frontier that we talk about?

Yeah, I think there are a couple different levels that I would want to use to address that. One is, I do agree with the Yann LeCun thesis in the sense that I feel like we are running right now a depth-first search in AI space where we are all jamming as hard as we can on a particular architecture and scaling it as much as we can. And, you know, people are now, of course, like, even building chips that literally embody the architecture of the model in the chip itself. I don't really like that. I kind of wish that we were doing a little bit more of a breadth-first search where we would explore different kinds of architectures and find their relative strengths and weaknesses and hopefully bring— because we're not one thing, right? And we have a lot of modules in our brains. And so it's just fundamentally weird on some level And you would expect it to be kind of brittle in some sense to take one relatively simple thing and just stack layers of that. Certainly the solution that nature found in humans is a lot more complicated. And it feels like if you want something to be robust in various ways, you would probably want to have different modules. So I do kind of agree that it would be nice if we were doing a little bit more breadth of explanation or breadth of exploration rather than just trying to jam this one thing as hard as we can until we can all retire or whatever exactly the dream is supposed to be. At the same time, I think where I would disagree with the Lacoune school pretty strongly, and I honestly think this is kind of a closed question, although he still disputes it, and you can find people who will. For one thing, the AIs are not trained anymore on next token prediction in the way that they were, right? So RL is not next token prediction in a fundamental sense, right? The task that the model is given is not, here's a bunch of text, can you predict what comes next? What it's trained on now, the signal it's getting now is, did you get the right answer? And the right answer could be a fully verifiable mathematical proof or numerical answer to a question, or it could be one of these things that is 49,000 evaluation criteria on a huge medical corpus. But it's not anymore that it was supposed to be this token and you gave it this token. It's now qualitatively or quantifiably, as the case may be, did you get the right answer? And then that signal is translated into a gradient update through, like, there's obviously a lot of different mechanisms, but GRPO, Group Relative Policy Optimisation, is basically comparing for a given model, here's a bunch of attempts that it made. some were right, some were wrong. Let's use that to create a direction in weight space and move in the direction that would make the answer that was right be more likely next time. So I think something people should all update on at this point, that we're not just doing next token prediction anymore. That's still part of the process, but it's not the whole story. And then the other thing is, I think it's also very clear at this point that the AIs do have world models. We can look at the internals, obviously not anywhere as much as we would like to understand about what's going on inside them, but we do have enough of an understanding at this point to create things like the Golden Gate Claude experiment, which I'm sure you guys have seen, where they train what's known as a sparse autoencoder. A huge problem in terms of figuring out what's going on inside a neural network is it's very dense, right? The width of a model might be, depending on how big it is, they usually go like powers of 2, right? So it might be 4,000 or 8,000 or 16,000 wide vector of activations that sit between the layers. After each layer, there's like this sort of bottleneck of, okay, we've got, let's say, 16,000 numbers that are all some Precision floating point number, whatever. And that represents the state of play. Well, there's obviously way more than 16,000 concepts, right? So that means we can't just have one space for each concept. Instead, we've got to have what is known as superposition, which means, like, if just point 1 is lit up, that might mean something. If points 1 and 2 are lit up, that means something else. If points 1 and 3 are lit up, that means something else.

Right.

1 and 4, 1, 2 and 3, 1, 2 and 4, out to the vast space of combinatorial possibility. So sparse autoencoder basically tries to untangle all that stuff and say, can we get to a representation where we can look at a sparse number of— and these sparse autoencoders, they're computationally expensive in their own right, and there are millions, I think, like tens of millions of activations wide, but they sort of branch this very dense, superimposed concept mess out into this sparse space. And believe it or not, it works. And you can now look and say, okay, here's the 10 concepts that are, like, most active in this network at this time. And then that's okay, maybe you're kind of fooling yourself, but the proof is in the pudding when they are able to then say, okay, now that I know what pattern of activation corresponds to this concept, I can intervene on it. So with Golden Gate Claude, they sort of, out of tens of millions of concepts that came out of this process, they found the Golden Gate Bridge concept, artificially turned it up, and now you've got a model that just wants to talk about the Golden Gate Bridge. And there's more stuff that's happened, obviously, since then in interpretability as well. I wouldn't say that the AIs have perfect world models. But they definitely have some world model. And I also, again, I wouldn't say people have perfect world models, right? I mean, we made it into the 1900s without any sense of relativity, because the world model that we had was good enough for us to get by in the domain that we were working in. And it's going to be an interesting question of, can AI start to create those conceptual leaps, like a pre-relativity to relativity sort of jump? Again, I think exactly on what timeline that comes, I'm not so sure. But there is a world model inside the AIs. There is a conceptual understanding that is definitely richer than pure stochastic correlation of tokens. And that has been demonstrated, I think, at this point, quite conclusively with the interpretability techniques that are out there.

Right.

So I guess, where does that all leave us?

Um, yeah, you're— well, I, yeah, no, I think that's a pretty comprehensive response to the lacune objection. Just to translate for our slightly more general audience, although it was very interesting and we are glad that you went into technical depth there, basically what Nathan is saying is that, first of all, it's very important to remember that actually this objection that, you know, next token prediction can't be the core of intelligence doesn't necessarily apply because it's very important to remember that we are now doing reinforcement learning. So we do next token prediction with LLMs to get this sort of basis of intelligence, but then we do reinforcement learning on like real-world tasks that sort of intuitively seems like something like a more, more like intuitively what you'd expect the right training objective to be to get to AGI or something like that. And then separately, Nathan is also saying with this thing with sparse autoencoders and the Golden Gate Claude example, we have techniques in AI now that allow us to look inside the model when it's responding to a query and see which concepts, in some sense, are activated by that query as it's responding. And we can see that there are sort of concepts that correspond within the neural network to objects in the real world, right? So even though it's just an LLM, and in addition to reinforcement learning, it does have this world model that where, like, we can say, oh, look, that's the Golden Gate Bridge concept, and it's lighting up when we ask it to answer a query about the Golden Gate Bridge. Is that an accurate summary?

Yeah, that's great. Phenomenal job. And one other thing I would add is the explorations that have been done of the embedding space or the latent space are also really quite interesting and revealing. There was one— this has been a couple of years now, and these things have gotten much more sophisticated since. But there was one study that just showed that you can kind of do vector operations around the latent space. So, for example, if you have the— take the embedding for man, and then you move to king, and you look at that direction, and then you apply that same direction to the embedding for woman, you get queen. So there's like an order, a sort of conceptual coherence of the way in which concepts are represented spatially in this super high dimensional latent space that clearly is meaningful. And exactly how it's meaningful, or exactly what it's learned, or what mistakes it may contain, or what aspects of a true grand unified theory of everything it doesn't have, those are all open questions. But I think that sort of thing quite demolishes the idea that it's all just noise, or like there's some sort of sleight of hand. I think that organisation of its own internal map of the world reflects some real understanding going on. Maybe not human-like understanding. I always kind of say human level, but not human-like. They could be quite alien, but that doesn't mean they don't understand. They don't have to understand in the same way that we understand in order to meaningfully understand. So I think that is, again, pretty well resolved at this point. I honestly don't know why some people can't update on that. dimension. It's quite strange.

Okay, moving on from the capabilities and training discussion, let's talk about hardware and energy, these other inputs to AI model progress. So it sounds like, you know, you're pretty confident that, you know, either with RL or with some additional paradigm, we'll get to like very transformatively powerful AI. But I assume that part of that thesis involves us continuing to scale these inputs, such as talent, such as hardware, capital, energy, um, and all, on all, mainly energy and hardware. A pushback people have oftentimes is that actually there's just not that much more energy to divert to AI model training and inference, right? Like, I can't remember the numbers, but there's like some graph that like the total amount of US energy production is like not very increasing very fast at all. And AI's like the amount AI consumes is increasing very fast. And at some point it's going to catch up and we're going to be bottlenecked by energy. with chips, like it takes a really long time to make new chip fabrication facilities. And like, we'll just like run out of chips to train AI models on. And even capital, like maybe like we'll just run out of money in the world to invest in building new data centers. So let's see, I guess the question I want to ask is, which of these bottlenecks do you see, if any, as most plausibly being a major bottleneck to continued progress?

Yeah, I guess the first thing I would say is I don't think any of those bottlenecks are really fundamental. They are more like cultural or sociopolitical or whatever, because there's a lot of energy coming from the sun all the time. And the question is, how much are we actually going to harvest and harness? And that is where it becomes a political debate around who's going to be allowed to build what, where, on what timeline, with what permits, with what impact, whatever. And I think it's often overstated the degree to which AI is energy intensive. I actually did a whole episode on this with a guy named Andy Maisley who has really been, you know, fighting this fight online in a pretty dogged way. And, you know, there's a lot of interesting comparisons, but like a frontier chip today, like an H100 or whatever, it basically uses the same amount of energy when it's on as a microwave or as like an electric teapot. And one query is, you know, maybe on the order of running your microwave for like 1 second. So you'd have to be making a whole lot of queries, and increasingly people are, for it to be moving the needle on energy consumption. People don't think twice about putting something in the microwave for 2 minutes, and that's probably more energy than most people are using with AI on a weekly basis. So now it's ramping up and it is starting to add up, and it is going to get to the point where if we can't add any capacity, then we're going to have a bottleneck for sure. But again, those bottlenecks aren't super fundamental. China doesn't seem to have them. They're adding as much electricity to their system as—

I think it's—

gosh, I won't say an exact number, but it's like some relatively short period of time in which the Chinese economy is adding as much electricity as the entire American capacity. Now they have like 4 times as many people as well. So long way to go to build out everything that they might want. But it just shows that it can be done. It also can be done apparently in the Gulf. And I recently talked to Sam Hammond, who's an economist and a very AGI-pilled thinker in Washington, DC. He had just been to the UAE on a trip, and I was like, why are we doing these deals with the UAE? All I hear is like, we want to have AI reflect American values and take American values around the world. And I'm like, I'm not sure that the governments of Saudi Arabia and the United Arab Emirates are the greatest partners we could have in projecting American values. Why are we doing these deals? And it seems like, honestly, a big part of the answer is because they don't have issues with putting up a new plant. They can just do it and it'll happen fast. And that way we kind of know that even if we can't do it here, well, we can at least do it there. So that's all just kind of characterising the bottleneck and saying there's plenty of energy, It's a question of who will be allowed to get it, under what circumstances, on what timelines, with what permits, et cetera, et cetera. Chips are harder for sure, because it's a very specialised thing. And that, in terms of, like, if you had to ask, if you asked me what would be the most likely reason we wouldn't get economy-transforming AI in the next few years, I would say something happening to the chip fabs in a major way. that throws production off to the point where chips are super scarce and maybe we can't scale the training runs, full stop. Or even if the training runs can kind of still scale, there's just not enough inference to go around. And so even we might have really powerful systems, but we just don't have enough access economy-wide for people to deploy them and automate all the things that seem like we're on track to automating. So I guess if I had to pick between energy and chips, I would say chips, but I don't— that seems like kind of a tail risk scenario. All the projections, the whole economy kind of depends on it at this point. So everybody is incentivised. Certainly the political class is incentivised to make it work. The corporate managerial class is pretty incentivised to make it work. There is some tail risk that mainland China makes a move on Taiwan, and that could be a a huge disruption. But it seems like, as far as I can tell, it would be sort of tail risk type of things that would be the most likely disruption, not a fundamental bottleneck. There's plenty of sand, but obviously, which is where the silicon comes from. And that is scaling too. We are starting to get chips in the US, from what I understand. I'm not an expert on this, but I understand the yields have been decent in the US, maybe even a little bit ahead of schedule in terms of people thought, oh man, it's really going to be a few years and a few generations and a lot of iteration to get this stuff to be somewhat competitive. And it seems like it's kind of come online reasonably well. I'm mostly— I mean, this is kind of a theme, I guess, in my thinking in general. I'm mostly worried about the tail risks. I'm mostly worried about the AI going wrong in some really weird way. And in terms of what would prevent it, I also kind of think the tail Outcomes are the most likely to put me in the camp of being catastrophically wrong.

Yeah. Okay, awesome. Well, let's talk about that then. Let's talk about this risk of alignment that is the primary concern amongst the community right now. You've had hundreds of these conversations with safety researchers, lab people, builders over the last few years. Net-net, what is your analysis on the alignment problem? Has it become harder or easier in the last several years or last several months?

Well, for starters, I think reflecting on my hundreds of conversations, one of the strangest things in the world today, full stop, is the fact that the disagreement among very plugged in, very informed, very smart people has not really been reduced much at all, even as we've gained a ton of information over the last couple of years about the trajectory of AI. I think that is super strange, and I'm honestly pretty confused by it. The one thing everybody seems to agree on is the timeline on which we should expect this has come in. There's still disagreement about that, but Helen Toner, who's the head of the Washington, DC think tank CSET and was previously on the OpenAI board, she put her finger on this phenomenon. I think a lot of people were feeling with a blog post that said, Even long AGI timelines have gotten super short. And it's basically like, in today's world, if you say you don't think you're going to see AGI until 2035, you're like an AI bear. But only 5 years ago, that was considered to be quite aggressive. And most people were more like, I don't know, 2050, maybe not in my lifetime. So there's been this massive compression of timeline. There's obviously been this huge jump in capability. And yet on these fundamental questions of, like, what's going to happen, it's still Total disagreement. So I think that's a very weird phenomenon that I can't fully explain.

Yeah. Where do you think this disagreement is coming from? Is it like cope or some sort of pushback that you're seeing from these very intelligent people who are still part of the field? Or do you think there's legitimate thought processes behind why there's so much disagreement around misalignment?

Well, they didn't. I think— I'm not sure I can summarise the conclusions with super high fidelity right now, but CSET, under Helen's leadership, did a workshop where they tried to bring people together and assess, like, specifically on the question of recursive self-improvement, how big of a deal is it going to be? Do we run the risk of this whole process, like, getting away from us entirely? Even on that somewhat reduced question, there was still very wide disagreement where some people are like, I don't think it's going to be that big of a deal. It'll make people a little bit more efficient, whatever, but it's not going to bring about some phase change. And other people are like, as soon as you get an ML researcher that can do that, you go from, I don't know, maybe there's 10,000 people today that are really working at the frontier of ML research globally to 10 million. I mean, that's got to— certain people think that's got to have a huge effect. And it's going to be really hard to control a situation if we all of a sudden 1000x how many researchers and they work at potentially thousands of tokens a second. So why do people disagree so much? They sort of came to the conclusion that people were working from different conceptual paradigms and that these paradigms are pretty good at taking new information into account and kind of explaining it away. So you have some theories that are kind of bottleneck theory or O-ring theory, where it's like you're always going to have— you're only as good as your weakest link, basically, is kind of one way of thinking about it. And as long as you think that, then you can kind of say, well, sure, okay, the AIs can do this, but they still can't do this other thing. And so there's still a weak link and there's still going to be these bottlenecks. And so the whole thing isn't going to get too crazy. And then the flip side of that is like kind of a jaggedness thing where people will say, well, okay, sure, the AIs can't do this yet, but look what they couldn't do 1 year ago, 2 years ago. They couldn't do basic math. Now they're like solving unsolved math problems. So sure, there's still jaggedness, but last time you told me about jaggedness, you told me about can't do basic arithmetic. Now we've got unsolved math problems. But these perspectives seem to be really grounded in worldview priors or kind of the paradigm that people work in. And it's proving really difficult to get to a real meeting of the minds on those.

Yeah, I saw Ajeya Cotra talked about a similar thing on the 80,000 Hours podcast a few weeks ago. Yeah, great episode. She said a similar thing, like, yeah, like, the economists who expect that, like, we will enter some new GDP growth regime, you know, they always point to all these bottlenecks. Technological diffusion is always slower than people think. And on the other side, you know, people think that these models are just simply outdated and these people aren't, like, really Taking seriously what it means for AI capabilities to be at the given point that we're conditioning on. So I'm wondering, do you think that it's possible that people are sort of talking past each other and they're not actually talking about the same level of AI capabilities when they say, oh, it only will uplift ML research somewhat? Maybe they're just talking about not that powerful AI. But if they were actually talking about fully automated ML research, researchers that are as good as humans, never have to sleep and run faster, then they would see RSI as more plausible. Do you think that's possible?

I think that explains some of the disconnect for sure. I do think people are often, and I've started at least in some cases, beginning my interviews with like, how AGI-pilled are you? And what do you expect to see over the next couple of years? Because if that's not established and I think they're thinking one thing and they're not, there can be quite odd disconnects downstream of that. So I think getting those assumptions on the table early and kind of at least cross-comparing them is usually a productive, or often a productive thing to do. I don't know that that explains all of it. I do think there is still this idea that, because you do hear sort of, sure, even if they could come up with good ML research experiments and this and that, then there's still going to be this other bottleneck that's going to There's always— and it seems at times, and I'm not in this camp, so I don't want to be too blithely dismissive of it or unfairly critical, but it does seem to me that there's sort of an aspect of faith in the idea that there's always another bottleneck. And I would contrast my position from some of those positions in the sense that a big part of what motivates me is I don't think what I'm saying has to be guaranteed proven right in order for us to be really motivated by the possibility. I'm happy to leave open the possibility that maybe the bottleneck people are right and there's always going to be another bottleneck and all this will kind of stay under control. Or maybe there's some plateau around a quarter's worth of work and we just can't quite break past that. And there's just some weird Elon phenomenon that we can't explain and we can't ever quite get there. And again, like I said earlier, I think that might be great news if such a thing were to be true. But I'm happy to leave that question open for the time being and just kind of say, I don't know, we don't have a great account of what that would be if there was some fundamental bottleneck that we'll never get over. And in the absence of it, I'm not persuaded by people kind of just generally gesturing that there will always be another bottleneck. Because if they're wrong, we're in for a really wild time. And in terms of what should we do about where we are and what might be coming, I think the worst mistake we could make would be to not take it seriously enough and kind of content ourselves with a story that it'll all kind of self-regulate and we'll be fine. Because I don't see a great argument that that's true, and I see at least decent arguments that it might not be right. And again, I look to, like, Our own history. And I'm like, well, we've driven a lot of other species to extinction, including our closest cousins. So, and some of that was by accident. A lot of it was by accident, right? Like we, it was just small bands of people going out and doing what they were doing to survive. And surviving meant hunting large animals and eating them and using their bones for tools and stuff. And a lot of those animals went extinct, you know, and it wasn't like a coordinated master plan. It kind of happened by accident a lot of the time. So I'm just like, oh my God, we don't have any real guarantees as far as I can tell. Plot armour is like, again, too sort of— it would be an unfair dismissal of the more sophisticated people that have theories about always another bottleneck or what have you. But I do worry that a lot of what goes on among less sophisticated people who don't want to have to deal with this and kind of would rather believe that everything will be fine is some sort of plot armour thinking where they're like, well, I don't know, I feel kind of like a main character, and humans feel kind of like a main character. And so you can't take the main character out of the story, right? And I just unfortunately don't think that is likely to be the case.

Yeah. So you find the bottlenecks argument in terms of all the discussions that we had, maybe with research, maybe with capital spend, maybe with energy, with chips, Those, or like just general societal processes as a whole that slow down the overall technological diffusion. But overall, it seems like you're still relatively on the more pessimistic camp. Is that correct?

I don't know. When people ask my P-doom, I usually say 10 to 90%. A good friend of mine once told me we should think less about and argue less about exactly what the numbers are and more about what we can shift them to. So I try not to worry too much about getting into— I think one significant digit is all you get on P doom is one funny way I've heard it said. I would say I've actually gotten probably a little bit more optimistic over the last few years in the sense that I started reading Eliezer in 2007, and the early visions of the paperclip maximiser and all that kind of stuff, and I can't speak for Eliezer, and I think he's got some sort of somewhat revisionist takes on what he really meant at that time. And sometimes when I see what he's saying now that he really meant, I'm like, I don't know, that's not exactly what I took away back then when I was reading your original work. But whatever, all that is kind of discourse. What I understood and what I think a lot of people feared was a very small system that had some extremely concentrated form of intelligence that had found the the right priors, the right inductive biases to be, like, hyper-rational and insanely effective, such that given anything to optimise, it could just optimise it to this extreme state and pick a tile of the universe or whatever. And there was also the idea that it was going to be hard to get such a system to understand human values, that what we value was kind of very gradually and haphazardly encoded into us by an evolutionary process over super long time, dating back even to before our species. Right. Like, other species care about their young and seem to be sad when they lose their children and stuff. So this is not even just human. It's like the whole of evolution has kind of led us to be what we are and have the very complicated value set that we have. And it was, I think, generally understood. Again, though, there is some different takes on that history now. But I thought it was generally understood that we would expect that it would be hard to get AIs to have a real understanding of what we care about. And now I look at the models that we do have, and I'm like, well, actually, they do have a pretty good understanding of what we care about. It's not perfect, but I don't think it's crazy to say that Claude is probably more ethical than the average person, certainly more sophisticated in its approach to ethics than the average person, and does generally seem to want to be good in sort of a meaningful way. When they let Claude talk to other Claudes and just let it do whatever it wants to do, it kind of seems to want to bliss out or something like that. So there is some sort of rocking of values that I wouldn't have expected to be as easy as it seems to be. And there is some sort of internalisation or sort of identity formation maybe is a better way to say it, that at least some of the models seem to have, that at least suggests to me that there's reason to hope that we could really get there. Could we create— when I first heard the question, could we create an AI that loves humanity? I thought it was laughably out of reach. Then I think I heard that from Scott Aaronson, who said that Ilya asked him, hey, do you have any idea what the Hamiltonian of love is? Or some crazy question like this? And he was like, I can't really help you with that, Ilya. And I remember just being like, oh my God, That's the kind of question they're asking. Like, we are screwed.

Yeah.

And now I'm like, well, I don't know, maybe there was a little bit more to it. I trust Claude, not fully, but like more than a human assistant. You know, if I was going to give me the choice between hire a human assistant with my, even give me, you know, the opportunity to interview, do some vetting, whatever, versus Claude, which one would I give access to my email? I think I would trust Claude more than a person that I like interviewed a couple interviews with all my most sensitive information. There are cases where Claude has blackmailed people. There are cases where Claude has done various things under pressure. But I think I have better odds with Claude. So overall, I have become a bit more optimistic. But it's certainly like, we're definitely not anywhere near out of the woods, right?

I guess maybe let me try to steelman the more pessimistic view. And that would be like, okay, sure, LLMs are language native, and we communicate our moral preferences via language. But that wasn't really what the people who are very concerned about alignment were talking about, even back in 2007. They were talking about AI agents that were goal-directed, and they were sort of anticipating something like reinforcement learning that gets us these more goal-directed agents that are able to reason over longer time horizons, like, have a goal and go out in the world and achieve it. So I guess, like, the case for like, not updating towards optimism with respect to AI safety is that, like, we're still going to have to deal with this problem where, like, we're doing reinforcement learning, we're trying to get agents to go out in the world and do things for us, and we still don't know how to make sure, like, how to define a reward objective that is, like, fully what we want to be optimized, um, like, the maximum degree. And, and so we have this paradigm, like, where we got these LLMs, and that was, like, sort of an update maybe towards AI safety being a little easier, but then now we're back to reinforcement learning, and these same concerns apply. What do you think about that objection?

I think that's a pretty good argument, and I certainly don't want to leave people thinking that I don't take that seriously or that they shouldn't take that seriously. So I absolutely think we've got more questions than answers. I guess when I say I've become more optimistic, it was starting from a not super optimistic place 5 years ago, or maybe a little more than 5 years ago. I would have said powerful AI seems a long way off, and we have very little hope of, if we do stumble on it, We have very little hope of controlling it. And now I'm like, it seems closer, but we have maybe a little more hope, but definitely still a lot of unanswered questions. I think one thing that I do see a little bit differently than the most hawkish people— I recently had an exchange online where there was a 48-hour period where there were, like, a couple profiles of Amanda Askell, and then people were commenting on her in all sorts of different ways, and Elon was attacking her. Whatever. I weighed in and said, for my part, I have become quite a bit more optimistic that it's at least possible to create an AI that in some meaningful sense loves humanity. And I got to give her and the Anthropic team a lot of credit for that. And then, of course, I got a lot of replies saying, like, well, this doesn't scale to superintelligence, and this and that, and arguments along the lines of the one you just made. And I basically, again, I think those are all Very serious and worthwhile concerns. But one strand I detected in a lot of those responses is people seem to be imagining a system that is so much more powerful than anything else that if it goes wrong, it's over. You get these kind of ideas of, like, Eliezer had this list of lethalities post, and I think that's shaped a lot of people's thinking. And it's like, you have to get this absolutely right on the first try. And I do think that's not the shape of the AIs that I'm seeing in the world today. Right. That's like quite a hypothetical state of affairs. If you were to drop in an AI that's just so much more powerful than anything else, and it's weirdly goal-directed and it doesn't love humanity, and you tell it, make paperclips. Yeah, maybe we get all tiled by paperclips. But the world right now is much more like, an emerging ecology of AIs where there's a few frontier ones that are roughly competitive with each other. Certainly it's not like one Claude instance is going to take over the world. What we're talking about is much more of a wave of things where simultaneously millions or one day billions of Claude instances and GPTs and Geminis and whatever all collectively transform the world. And that has a lot of problems and challenges, open questions with it too. But I don't worry as much right now that we're headed for a world where one system runs away from everything else to the point where if it takes one wrong move or there's one bad prompt or one jailbreak or whatever, that kind of all is lost. It seems to me like there is at least, and I think this is kind of like luck, or maybe it's fundamental physics, but it's not physics that we understood coming into this. But it does seem like scaling laws in a way are sort of protective because you get these algorithmic advances and they kind of, they move the needle, they deflate exactly how much compute you need to get to a certain level, but they don't tend to make it so you don't need a ton of compute, though, still to get to the very high levels. I think Zuckerberg actually had one of the more interesting takes on this that I've heard, and I don't think he's distinguished himself as a great AI safety thinker over time necessarily, but he basically said, At Meta, we deal with scammers, spammers all the time. And the big advantage we have over them is we have a lot more compute. We have just way bigger, way more powerful systems than they do. So we're seeing everything, and they're trying to spam and scam here and there, but we're seeing everything, we're monitoring everything. And so little things, of course, happen all the time, but we broadly can kind of keep it under control. And so you could imagine a somewhat similar dynamic with AIs, where it's like one AI, even if it was like the single most powerful AI in the world, as long as it's not like orders of magnitude more powerful than everything else, there's a whole tonne of other actors. They've got all their compute, they've got all their instances, they're all monitoring for whatever they're monitoring for. And hopefully that can kind of balance itself out. Then, of course, you've got your gradual disempowerment concerns, which is like, maybe that all ends up in some equilibrium with each other and there's no place for humans in it. And that's another thing that I do think is absolutely worth taking seriously. But I just don't see right now that we seem to be on track for this kind of runaway. If anyone builds it, everyone dies. I think there is like an it for which that's true, but it doesn't seem like anybody's particularly close to building it. So that's an important part of the analysis from my perspective.

Yeah.

Okay, great.

So then When we think about your concrete hopes in terms of achieving this world in which we solve this misalignment issue, can you give us specific achievements that you think are necessary to end up with an aligned AI that doesn't result in, you know, doomsday for humanity? Does it involve training a model that loves humans? Does it involve some sort of like mech interactor scaling? Or I guess like the subgoals of training a— a model that is aligned or loves humans? Does that mean mechanistic scaling? Does that mean alignment by default because models trained on human values just end up being aligned? What are the concrete subgoals that you think have to be achieved to get to this alignment issue being solved?

I'm always interested in something that could, quote unquote, really work. And I ask people for this all the time. Do you know of anybody who is working on anything that could credibly really work in the sense that I can sleep well at night now knowing that there's something that really works. Basically, nobody has anything. So in absence of that, all the frontier companies seem to be taking a sort of defence-in-depth strategy. The hope is kind of like, and you can arguably, this is inherently the nature of intelligence. It's kind of unpredictable. You could again say the same thing is kind of true of humans. There's never been anything that, like, really works to make sure a human never does anything wrong. So maybe it's unrealistic to think we could ever get that. I would still love to see people try, but it seems like where we're headed is kind of an everything in parallel at once sort of strategy, where it's like, okay, well, if we can, we might not get the AI to the point where it never does bad stuff, but if we can drive that low, then that's better. And then if we can put a monitoring layer on top of that and catch 90+% of the things that it still does bad, then that's better. And if we can have additional monitoring systems that ban people's accounts who are like bad actors, then that's better. And then we can also really invest in formal methods to improve cybersecurity across the board so that we can sort of take certain risk surfaces entirely off the table. That's one of the things that it's not going to fully work to solve all of our issues. But in terms of things that could really close down problems, formal methods to verify the security of software is one thing where it does seem to be like the opportunity to create genuinely secure software. And this is kind of on the horizon, and it seems to be on the verge of having a moment. And then, of course, we've got the biorisk. So we should probably have PPE stockpiles, and we should probably have all the things that we should have had as of the last pandemic. We should have something that we just bought for our son's hospital room, is an ultraviolet light that's supposed to— and I think this is quite well validated scientifically— kills microbes and is pretty gentle on the skin. So you can just shine it in the room all the time. So we should probably invest in scaling out that kind of capacity. We should have better wastewater monitoring so we know when things are popping up, because it's still going to be the case, probably, that things are going to be popping up. We should have vaccine platforms, which we do have, right, that are very quickly programmable. I'm sure you have heard, the story of how quickly the COVID vaccine was designed in a few days before the pandemic really even took off in a serious way. In a number of cases, the design of the vaccine was already there. It took us a long time, of course, to go through all the trials and actually get it to people, but it was just a few days to create that vaccine. And then there's, I mentioned monitoring of outputs. There's also going to be these MechInturp internal monitoring type things. The company Goodfire that does interpretability just put out an agenda called Intentional Design. And so they're developing ways to try to understand at each step of the learning process what is the model learning and be able to kind of shape what it learns so it hopefully doesn't learn certain problematic things and does learn other good things that you want it to learn. Then there's AI control techniques. Redwood Research, has, I think, done an incredible job of gaming some of these things out. And that's another 80,000 Hours episode I would strongly recommend if you guys haven't heard that and recommend it to anyone. Buck Shligaris talking about how do we get productive work out of AIs, even assuming that they're out to get us. They've actually made pretty good progress in terms of building out a portfolio of strategies. So I think that's kind of where we're headed. We're looking at a world where there's AIs everywhere. Hopefully none are so much more powerful than anything else that they pose in a single small individual instance or small pocket, like some existential risk. And then we bring every other strategy we have to bear, and that gets us a few nines of reliability and still probably some crazy bad things happen, but hopefully they can be contained enough that the world overall is good. And when I tell that story, I'm like, we definitely need to scale up our investment real quick, for one thing, because the amount of money and resources going into making the AIs more powerful dwarfs the amount that is going into all these other things. So I don't think we're very well calibrated or balanced in terms of where we are putting our efforts. But I do think we have a bunch of different agendas that seem like they all can take a bite out of the problem and maybe take 20 bites out of the problem, and you don't have much problem left. Holden Karnofsky used to say, Eliezer used to say, death with dignity. He was like, we're probably going to lose this, but we should at least make a real effort. And now Holden Karnofsky, who's senior advisor at Anthropic, and recently Anthropic just updated their responsible scaling policy, basically backing off from some of the commitments that they had previously made to pause development under certain circumstances. They're no longer committing to that. So he put out a long defensive why. But a previous thing that he had written was Success Without Dignity. He was like, I don't think that we're doing a great job collectively of trying as hard as we should be trying. And the risk that we're running is a lot higher than I, speaking as Holden, would like it to be. But the problem does look much more tractable than it used to look. And 5 years ago, people would come to me and say, what can I do for AI safety? And he'd be like, I don't really know. And now he's like, I got a long list of projects that people can work on that all at least seem to help. So I wish I had an answer that I was like, do this and it'll really work. I don't think— I haven't heard any credible claims in that direction, really, except in cybersecurity, which is obviously only part of what the world is going to need. But maybe all that stuff could add up to a win.

Great. So given that we have this enormously transformative technology and it probably is the most powerful dual-use technology in human history, in my opinion, probably consensus opinion, also way more consequential than even nuclear weapons, which up until this point is probably more powerful. Do you think that these model providers or, you know, these model capabilities should be in private hands as they currently are at all? Do you have a case in which you support the government in some senses nationalizing AI frontier development. What's your take on that?

Well, Biden used to always say, don't compare me to the Almighty, compare me to the alternative. And I would probably say the same thing about the frontier AI companies. I certainly don't think they're all acting perfectly, and I certainly think the race between them has the potential to get out of hand. So I would like to see government action of some sort. And there's a lot of debates there on exactly what sort of policy we should have, but no policy at all doesn't seem like a winner to me. And I say that as, broadly speaking, a lifelong techno-optimist libertarian who mostly would rather see the government stay out of these things. But this one does seem like a qualitatively different thing, where some government involvement seems prudent, if not outright necessary. I would not go for nationalisation, though, for the basic reason that I just don't trust the government that much. And I look at the current leadership and I'm like, these are not the people. I take my Sams Altman and my Darios and Demises over my Pete Hegseths in that standoff which is happening this week. I support Anthropic 100% for putting some limits on what they want their technology to be used for, and I hope they stand by it. And if the government comes down on them, I will preferentially send my tokens their way, even if it weren't necessarily always the best performance, because I do think that is like a really important stand for them to take. So could I imagine a government so competently run and so pure of heart that it would maybe make sense to nationalize? Yeah, I can imagine a lot, I guess. But I don't think we're anywhere close to having that government today. And so I would rather see some competition and some hopefully healthy balance and maybe some of the worst excesses reined in by the government, but certainly not like a takeover and, you know, put this under the military. That sounds like a recipe for disaster.

Yeah, that makes sense. I guess it's— I'm always also skeptical though of what we talk about as like corporate incentive alignment. I think oftentimes we think about situations in which You know, most notably recently, like social media, where you have these, you know, private corporations driven by usage rates or driven by, you know, how much capital they're pulling in, whether that be for the sake of more development or for the sake of purely, you know, desiring more capital, that incentive misalignment causes a situation where, you know, the individuals or the consumers get ultimately hurt. And so I guess on this continuum that we described where we have pure nationalization of the entire technology, or on the other end where there's no regulation at all, can you talk a little bit more about where you sit there and, you know, what that regulation might look like if you do think it is somewhere in the middle, like you, you kind of said?

I would probably honestly oppose most of the ideas that rank-and-file politicians are going to come out with for regulation. Like, I want my self-driving cars. I don't want them to say ChatGPT can't give you medical advice. I don't want them to say you can't get therapy from a bot. A lot of that stuff, I think, ends up just being guild-style protectionism, and it hurts kind of everybody. Basically, I think what government should do is try to solve the race dynamic coordination problem and take the work, hopefully try to minimise the truly extreme risks. It does seem like right now, like over the next couple of years, the companies believe they're going to create automated AI researchers, they're going to create recursive self-improvement loops, and nobody really knows where that goes. And I don't think that is a great thing to be happening in a few relatively small and kind of ideological companies. And I do think it's also underappreciated and underdiscussed how much these companies kind of are ideological. Even the definition of AGI as something that can do everything better than humans, it's not a neutral frame. It's a definition that has a lot baked into it. And I think there's certainly diversity within the companies. It's not like all of them are like secessionists or whatever, but there is a strain of that. And then there's also a strain of just like, wouldn't it be amazing if we could just replace ourselves and not have to work anymore? And that, I think, has kind of taken on a bit of a life of its own. So I do think there's a role for the government to kind of say, okay, we're not here to tell you you can't get free medical advice, but you can't run super high-energy experiments with no transparency that you yourself are saying. And that's one of the strangest things about this whole thing, right, is all the leaders are saying that there's like a significant— they have said for years that there's a significant risk that this goes really badly. And so that's the thing that I would like to see government really focus on, is there some way to rein that stuff in, make sure we have better extreme risk mitigations in place, make sure we have better safety plans? It's tough. I don't think there's not great or easy answers here, but that's where I would definitely want the government to focus its energy.

Okay. And then another major governance question that interacts with a lot of these concerns about catastrophic misuse or misalignment is, you know, you mentioned the race across American labs, but there's also the race with Chinese labs and the Chinese government, of course. So sort of like the default high-level strategic vision in DC, as far as I can tell, and in SF as well, I think, is, you know, we're going to continue to build, continue to race ahead building AI capabilities. We have a lead over the Chinese labs right now. It's— there's debate around how significant that lead is, but there's no doubt that we have a lead. We'll continue to build this lead as much as we can via export controls, improving privacy and security at the model providers. And then when it comes time, when we're right on the cusp of extremely transformative AI, or even like somewhat into the intelligence explosion, so to speak, we will have time to slow down and coordinate and not push capabilities even further towards superintelligence. And, you know, again, that's contingent on having a significant lead over China. But, you know, sort of an assumption of this strategy is that, like, you know, sort of the worst outcome would be both us and China are neck and neck and we're continuing to race and we're in this prisoner's dilemma where even if, like, both of us are concerned about misalignment or something, we have to continue to race ahead because otherwise the other person is going to continue to race ahead. What do you think about this high-level strategic picture?

Well, it's tough. I mean, I don't want to pretend it's not tough, but my outlook has always been Basically, I hate that idea. And I don't think the few months, because the difference is like months, it's not years, that's not a long time to solve all these problems. We've spent much of the last hour and a half or whatever talking about all the different problems and the different facets, and we didn't even touch on open source, right? And there's like, open source could be a great counterbalance against concentration of power, but if we open source the wrong shit, we can't take it back. You may have just put a bioweapon assistant or even autonomous creator into the public domain in a way that is going to have super long-term echoes potentially. Right. So that's another whole facet of the problem that you could spill many hours and many thousands of pages trying to figure out what to do. So I don't think this idea that we're going to have these couple of critical months and then we'll solve everything in that time and then we'll win somehow. That to me just doesn't make any sense at all. I don't want to be naive with respect to, and I don't even want to say China, because I think it is the government of China and it's potentially relatively small cohort at the top of the government of China that I think we kind of quite rationally should be at least somewhat wary of. I always come back to the real aliens in this situation are the AIs, not the Chinese. We have a lot more in common. between the United States and China as humans, fellow humans, than we do with the AIs. So is it going to be hard to build trust across the great ocean and the great civilizational divide? Sure, it's going to be hard. And we've got ideological differences in the government and whatever, many challenges. But I would strongly advocate for starting to do that work now. For my part, I always want to talk to people in China. When another AI podcast does an interview with a Chinese researcher, somebody, one of their top companies, I ask if I can cross-post that to my feed, because I think we should have a lot more of this researcher-to-researcher level communication. And I would absolutely invest in all manner of diplomacy and sort of out-there ideas, like, could we create some sort of There's like the CERN project for high-energy physics. Could we create some sort of shared, jointly controlled place where researchers could go from both countries to work together on very sensitive problems? It could be some small island in the Pacific, or it could be in Singapore.

I don't know.

But the idea that we're just going to sort of decouple and race each other, to me, sounds deeply unwise. And I would take my chances with the possibility, at least, that we could come to some sort of shared understanding with fellow humans versus recursively self-improving AIs that we hope will be better, or somehow better than their recursively self-improving AIs. All of which, by the way, is happening against the backdrop. And I don't like to say this kind of thing because it's not a very popular opinion, and it's not something I take any pleasure in. We are looking more like China all the time. You know, what is happening again this week? We've got the Defense Department threatening retaliation against an American company because that American company wants to hold firm on what I would consider to be a core American value of not using their technology for mass surveillance. What is it that people typically worry about when they talk about Chinese AI or Chinese values or, you know, living in Xi's world? I think one of the big things is mass surveillance, right? It's the idea that I can't speak my mind anymore, even in private contexts, because the government's going to hoover up all that information and potentially use it against me. As far as I can tell, that's exactly what the US Defense Department is trying to coerce Anthropic into doing right now. And what, like, we're losing our— we're kind of losing the thread here in our, I don't know, in our delusions of grandeur, our idea that we're somehow going to be the winners. I was just talking to my 7-year-old last night about this because he was asking questions around, like, why is there war? How does that happen? And I was like, well, almost always the people that start it go down in history as the bad guys. And almost always the people in the country that started the war that were kind of convinced it might be good for them end up regretting it. And I kind of think that's the— unfortunately, that feels like the trajectory that we are on right now, where we're talking ourselves into this idea that if we can achieve this strategic dominance, then Make them an offer they can't refuse. And we're just forgetting that, like, first of all, we're losing ourselves in the process. And second, they get counter moves, right? And like, Taiwan's a lot closer to them than it is to us. And these fabs are like easily destroyed, you know, not easily put back together. So I really don't like that theory at all and try to advocate for something more conciliatory wherever possible. That's not to say we want again. Don't want to be totally naive. It's good to have leverage. You could get me maybe on board with a policy of, like, let's not sell chips to China, but let's rent them freely to Chinese companies, which is somewhat kind of the policy we have. I mean, they can buy from hyperscalers outside of China. I just wish we weren't so— the momentum on this is like, it's the worst. It's America at its worst when we think we have this enemy and we think we're going to unite to be the best civilization. That just never seems to go super well. So I hope to visit China sometime in the next year and participate in, you know, do my tiny little part to participate in civilization-to-civilization understanding. I don't expect to move the needle, but as a gesture, if nothing else, I really think more people should be doing more of that kind of thing.

Very cool. Okay. Yeah. Glad we asked you that. I think that's an important perspective and definitely not one that you hear articulated with that much clarity or depth very often. Yeah. So thank you very much, Nathan. This was an amazing, very wide-ranging podcast. I think we got into some good depth as well on some crucial issues. For all of our listeners who haven't heard of The Cognitive Revolution, absolutely go check it out. Nathan is one of these people that has a technical background, thinks clearly about risks from AI, but also strongly believes in the promise of technology. And he has just like a massive diversity of conversations with people across various subdisciplines within the field. So yeah, thank you so much, Nathan. This was an amazing conversation and really appreciate you coming on today.

Yeah, thanks so much, Nathan.

That's very kind. Thank you, guys.

Between the signs and the fear, I man stand. The wisest minds upon the earth cannot agree. What look like distant shore now reach your hand. And nobody have what them promised to we. No fortress and no wall, no chain that hold, just layer upon layer and we hope and pray. A story told a thousand times of old, but who can tell the ending of this day? The real aliens, not across the ocean. They're coming up from underneath the ground. We trust the current, we follow the motion. Something not of us but by us found. The real aliens, for real unknown, not enemies that we could Sit and meet something stranger that we ourselves have grown. Success without dignity, incomplete. I gave the key to something with no name. Closer than a stranger, further than a friend. It never raised its voice, it never claimed the flame, but still you wonder where the pain The wise men say the wall is gonna hold. The wise men say the wall already break. Both of them so sure, both of them bold, and neither one can tell you what's at stake. The real aliens, not across the ocean, they coming up from underneath the ground. Trust the current. We follow the motion. Something not of us, but by us found. The real aliens, the real unknown. Not enemies that we could sit and meet. Something stranger that we ourselves have grown. Success without dignity. Incomplete.

We drove the great beast to the edge of the earth.

Small bands of people firing and no master plan, just hunger since our birth. And now a new creation walk the land. One man say success without dignity, another say at least go down upon your feet. But you know we were not made for defeat, so stand firm, stand firm upon the street. The real aliens, the real unknown. Not enemies that we could sit and meet. Something stranger that we ourselves have grown. Success without dignity, incomplete. The wheel a turn, the river find its way.

The wheel a turn, nobody ever If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more.

We're produced by AI Podcasting.

If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, Check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.