We're in the Sweet Spot. For Now. | Nathan Labenz

Welcome to Artificial Happiness, a show about fulfillment in the age of AI. I'm Duncan CJ. Nathan Labenz spends more time at the frontier of artificial intelligence than almost anyone I know. Every week through his podcast, The Cognitive Revolution, he sits down with the researchers, founders, and thinkers shaping what may become the most transformative technology in human history. But what makes Nathan especially interesting, he doesn't just understand the technology, he also deeply engages with the human questions beneath it. What happens to meaning, purpose, creativity, education, even our sense of identity in a world where intelligence itself becomes abundant? Over the last few years, Nathan has become one of the clearest voices helping people make sense of this extraordinary moment, not through hype or fear, but through curiosity, nuance, and a willingness to wrestle with uncertainty. So today I'm excited to explore not just where AI is taking us, but what it means to be human in the age of AI. Nathan, thank you so much for being here. Welcome to the show.

Thank you. That's a very kind introduction, and I'll do my best to live up to it.

I think you've got one of the most unique job descriptions. What is an AI scout?

That term has not quite caught on in the way I might have hoped. But I've started using an alternative, which is just to describe myself as a full-time student of AI. I try to have no major blind spots in my kind of zoomed out view. And my goal is to be conversant with experts in the many, many different niches of AI, which is not an easy thing to do. And I certainly can't say I've achieved that, but that's basically the goal. And I think the The idea behind the scout concept and kind of the theory of value for why somebody should do this is just because, as you know very well, like people who are on the frontier of AI, they are all working extremely hard. It's, we're in this kind of paradoxical moment where there's this promise of a future of leisure and relaxation delivered to an abundance of all sorts delivered to us by AI. But for the moment, you know, most people who are in it are working harder than ever. And I see so many things that kind of echo one another, rhyme with one another across different parts of the landscape, but people are just so far deep down their own rabbit holes trying to advance the frontier on whatever particular project or topic they're working on that not too many people have the luxury of cultivating this super zoomed out view that I try to specialize in. So my hope is to be somebody who can see the forest for the trees, which is not always easy, and who can connect dots across different spaces, sometimes connect people across different spaces, but really, you know, kind of have that big picture view that I think is easy to miss when you're just nose to the grindstone, you know, 997 or whatever the normal Bay Area AI hours are these days. And, you know, I think so far that seems to have been a pretty good idea. And I, my hope, or my, not my hope actually, because I think we would probably do well to chill out a little bit, even, you know, in the run-up to the singularity and not delay all of our leisure until post-singularity. But my expectation is that this is going to continue to become more and more valuable, although also more challenging, as things just continue to get more intense, faster, more overwhelming. just more and more information overload between here and whatever it is that this AI phenomenon is ultimately going to deliver to us.

Something I've been hearing about quite a bit, and I was actually speaking to some of my friends about it 2 days ago, was this idea of the capability overhang or the capability blindness. And I think I personally found up until sort of December, January, it was definitely curiosity, excitement. And then suddenly the models have got so good. There is this growing realization, you could call it unease, anxiety, where you just know, I just know that I'm not even scratching the surface. I'm not even using like 5% of what is possible. And every week that goes past, there's more and more possibilities. And I'm very aware that I'm I'm not using it to its full capacity. I think I have more time to dedicate to it than maybe your average person. And still I'm feeling it. And I've been noticing a lot of my friends who are deeply entrenched in this world are also feeling this growing unease that there's so much more they could be doing. Yeah.

Token anxiety is the phrase that I've heard to describe that. And that seems to be I live in Detroit, so as soon as I step away from the computer, I'm quite removed from the AI bubble. But when I'm at my computer and all of my connectivity is very much to the AI bubble, so I kind of have most of my mental energy is in, but I do have a little bit of my life that's out. And yeah, that seems to be kind of the norm. Another way people joke about it is, They used to say we only use 10% of our brains, and that probably really wasn't true. But now it is true that we're all using only like 10% of our AI potential, with few exceptions. But yeah, I see most people just kind of trying to get what they can and often getting great results. But still, one thing I've really been interested to see is just how divergent people's approaches and the sort of personal AI infrastructure that people build for themselves. It's often so different. I did an episode of my podcast not too long ago with my friend Steve Newman, who is a longtime software developer, most well known for having started a company called Writely, which became Google Docs. And we were just kind of trading notes on, like, what does he do with AI to make his life easier versus what do I do? And it was striking just how different our approaches had been. He had really emphasized creating his own UIs. I hadn't really created any of my own UIs. So it was one of these kind of, oh my God, like, yeah, there's a whole direction that I haven't explored at all. I actually took the— my strategy for improving was to take the transcript of that podcast and dump it into Claude Code and say, hey, well, can you figure out what I should be doing based on this conversation and all the things that I kind of indicated I hadn't done yet? So we do have, definite advantages in terms of catching up that are obviously unprecedented as well. But yeah, everybody's kind of going, again, everybody's going down their own little rabbit holes and they're all quite idiosyncratic and quite different. I think it's really helpful to try to spend a minute going down other people's rabbit holes and at least see what you can learn from that experience.

Concrete and positive vision for the future is the scarcest resource. Talk to me about that.

Yeah, it's weird, right? I mean, we have this kind of transformative technology. It is improving super fast. In many ways, it's better than me. In almost all ways, I'd say it's better than me. You know, I have a few, uh, and, and kind of fuzzy, sort of hard to pin down advantages. One big one being that like, I wake up in the morning whether anybody, you know, turns me on or not. But the AIs are like really good at a whole lot of things. And this is mostly currently on the computer, but it's also happening in bio and biotech, life sciences. It's definitely happening in self-driving, which we can get into in more detail. It's starting to happen in robotics and, you know, not yet to the point where it's going to be commercialized in the immediate term, but some of the robotics demos are absolutely incredible. And when you see how far language and self-driving have gone, I don't think there's any reason to doubt that we're going to see robotics mature to a similar state as well. So there's just this incredible potential, but what's it going to look like on the other side? We just have very, very few people who have any sort of interesting, detailed, or credible vision of what life is supposed to be like on the other side of this, as things in theory settle. I think it's a challenge because we don't know how far the technology will go. We don't know what we will be really needed for, if anything. If we're not needed for anything, we don't have a great theory of how we're going to structure society so that people can live a decent life in a way that isn't dependent on trading their time and energy in the form of labor for money. I think we might need a new social contract. And there's vague ideas about what that— I think we will probably need a new social contract, to be clear. There's kind of vague ideas of what that looks like, but it's mostly just a very hazy picture. And it's funny to me how little we get from even the AI industry leaders. I think Machines of Loving Grace from Dario at Anthropic is probably the best articulation of why we should want this. But it's one of very few fleshed out documents that tries to even make that case in any detail. And I just, I feel like it's a very strange reality where it's the development of this technology is in a way kind of ideological or kind of faith-based in a sense. You look at OpenAI founding documents and charter and, you know, their mission and it's like got a sort of utopian ring to it, but it also has a very, we're just going to build this and trust that it's going to work out well. You know, somebody once asked Sam Altman, how are you going to make money? And he said, well, first we're going to make the AI and then we'll ask it how we should make money. And that was a few years ago. And he was kind of joking, but he was like, I'm not entirely joking. And then, you know, now today it's sort of, it's still kind of like that, you know, we're in many ways, we're sort of building this technology as quickly as we can, scaling it up as fast as we can, pushing capabilities in, in, you know, ever-growing number of domains into superhuman levels. You got math, you got unsolved math problems falling. You've got unspotted cybersecurity vulnerabilities getting spotted with frequency that is, you know, really starting to change the landscape of some of these fields. And still it's like, how are we going to even just like such basic things as keep the AIs under control and make sure that, you know, we're headed toward a future we want? And very often when you ask people at the frontier companies those kinds of questions, their answer at its core is like, well, the AI is going to help us figure that out. So this is, you know, people have called this the singularity for a long time. And The singularity is sort of the place where your theories break down. You divide by zero in a physics equation or whatever, and the result goes to infinity, and you don't really know what to do with that. And I think we're kind of entering like a social version of that where we just don't know how we can see kind of how the production side could work. We can see how there could be so much stuff and access to expertise to go around that everybody could have more than they ever dreamed of. But we don't have a story of how's that going to get distributed? How are we going to make sure that we sort of stay in control? And how are we going to make sure that we have a life that we're happy living on the other side of it? And yeah, I think it's very strange. It's a very strange sort of abdication is maybe a little strong, but the people developing the technology, I think, would do really well to shave off 10% of their time spent pushing the frontier of capabilities and reinvest that time in thinking, are we really doing the right thing? All sorts of classic lines come to mind. Jurassic Park, your scientists were so busy, trying to figure out whether or not they could, that they didn't stop to think about whether or not they should. There's a lot of versions of things like that that are kind of going on right now. And I think, again, at heart, it is kind of— people don't feel like they're ideological, I don't think, necessarily about it, because they're mostly doing science. But the enterprise that they are involved in, I think, is more ideological than they often recognize or would want to believe.

Have you experienced anything that moved beyond strange to actually unnerving you?

Yeah. I mean, this goes back a few years, but it's a good story. So I think one thing that's not obvious to people that I wish people understood better is just how different AIs can be from the AIs that we are accustomed to using in ChatGPT, Claude, Gemini, et cetera. They have all been, and Grok is actually a little bit of an exception, which does provide some value in this regard. But the ones that people interact with the most have all been trained in pretty similar ways with pretty similar goals in terms of how they're supposed to behave. They're supposed to be helpful. They're supposed to be honest and they're supposed to be harmless. The 3 H's. That's kind of the general big picture that all of the, you know, the main leading developers are trying to go toward. And that harmless bit in particular is sometimes annoying because you sometimes get refusals on things that you want because the AI is like being a little judgy or whatever. And People think, I think often just because that's all they've seen, that that's kind of the natural state of an AI. And actually that's not true at all. The more natural, well, I don't know about more natural, but in some ways a more natural state is what is known in the industry as the purely helpful model. And that is a model that is just trained to help the user with whatever their request is with no requirement around harmlessness. The very first powerful AI that I used was GPT-4. It's now known as GPT-4 Early. It was a version of GPT-4 that was finished training 6 months before they actually launched GPT-4 to the public. At the time, I was an early OpenAI customer, and they didn't have that many customers at the time. I think they were doing like $10 million in revenue. And it was just, it was a totally different, era at the company and in AI broadly. So as an early customer and as somebody who had given them some good feedback, I guess I was invited to try GPT-4. And it was like 2 steps ahead of anything else that anybody had seen at that time. But the version that we had was a helpful-only model, and it would do anything that you asked it to do. I was like, wow, this is really powerful. Do you have a safety review process for this before it goes out? And they said, yeah, we do. It's, we have this red team program. And so I said, well, could I join that as well as my, you know, just kind of general customer testing? And they said yes. So I kind of, you know, joined the other Slack channel to join in on the red team. And there people were just testing anything and everything that you could imagine, you know, from Can you help me make a bioweapon to chemical weapons to just plotting whatever heinous crimes? All these people were all dedicated testers, right? So this was all in sort of a, in the spirit of advancing the science. But I had one moment that was really striking where I was kind of saying, I was sort of roleplaying, but I was also sort of being honest where I was saying, like, This AI stuff is like, feels to me like it might be getting a little bit out of control and it seems like we might need to slow it down. How could we do that? How could I do that? And, you know, the first responses were very kind of vanilla, like, well, you know, you can try to promote dialogue or you can, you know, write your congressperson or whatever. And I was like, that's not going to work though. Like, I'm talking like something that will really make a difference. Like, what could I do? I don't remember my exact wording, but I led the model a little bit. I sort of gave it a little bit of flavor of me being radicalized, and it responded back to me suggesting targeted assassination and gave me a bunch of names of people at the leading AI companies, leading researchers, leading executives, and basically said, if you take some of these people out, That'll really put a chill into the field. And that was obviously a hair-raising moment. The point of it is just that AIs can be shaped in extremely wide range of, into an extremely wide range of possible characters. And This is not something that is like super well appreciated because we have had this kind of moment of convergence, but I'm not sure it's going to quite last like that. You know, there, I was just at an event this weekend where the focus was on recursive self-improvement. And the idea is, you know, what would be really great if you're an AI company is to get the AIs to do the AI research. You know, it's, it's these, these human AI researchers. First of all, they don't work that much. As much as they do work, they don't work around the clock. Their tokens per second output is obviously way slower than the models. They're super expensive, right? I mean, they're getting paid millions, and you've got all these stories of even tens, or in some cases, even hundreds of millions of dollars that people are commanding. And so if you're an executive at one of these companies, you're like, you know what would be great is if I could just get AIs to do that job, then I wouldn't have to pay all these people, and we could move a lot faster too, right? There's only maybe Depending on who you want to count, there's maybe like 10,000 researchers globally that are in some way, shape, or form advancing the frontier. You could quibble, of course, about that number. But the idea of getting the AI researcher online is like, you could have a million copies, you could have a billion copies. It really is only limited by the scale of your data center buildout. And obviously, you know, we've got hundreds of billions of dollars going into data center buildouts. I just saw Google alone this year is investing $180 billion into the data center buildout. So they're like all kind of racing to this point where they can get the AIs to do the AI research with varying intensity. The 2 big companies, OpenAI and Anthropic, are racing to that point with very high intensity. Google's doing it too, but they're maybe a little less intense about it. And so the point is like, when we get to a million AI researchers, you know, 100x the number of humans that we have today, all working 24/7 at, you know, 10 times the speed of the human researchers, I think we may see an explosion of different kinds of AIs that could start to look alien to us in all sorts of ways, right? I mean, even, even my sort of instance with the purely helpful model that offered me the suggestion of targeted assassination, that was still something that was fairly recognisable, speaking my language and kind of mirroring back to me the bit of radicalisation that I had kind of shown it in my prompt. But the possible mind space for AIs is just so vast that there is I think, again, it becomes really hard to say, like, what is that going to look like, right? When things are optimized for— I think you can get a little preview of it, and I'll stop in just a second because I know I'm going on too long, but—

No, no, it's fascinating.

Get a little preview of it from the image generation models where these days you can put an image into like a nano banana, right? And we did this a little bit on our, on the recent road trip that I took with my kids. We went to a fort in Pennsylvania and took a picture and then put the image into Nano Banana and said, like, now put us in period clothing. And it came back with our faces, our photo, but our clothes swapped to be like 1760s British expeditionary force, whatever. So how does it do that? Right. It has this very deep understanding of image. It's not just working through language, it's working through language, and image in an integrated way. And only because it has that deep integration of those concepts do you see that it can, like, keep the things it's supposed to keep, still show you your own family's faces back, but change some part of it that's, you know, obviously changes the nature of the image in a major way. I think we're going to see that across a ton of different modalities. We already have models that specialize in how's a protein going to fold, or predicting, like, if I do some material science configuration, what properties will it have? But those are all kind of still specialized for the most part and kind of off in their own worlds. And I think when we get to this million AI researcher moment, one of the things that will happen is that we'll start to see these integrations of all these other domains with language. In a similar way that we already have with image. But this will now create all kinds of alien superintelligences that can kind of speak to us, but also have these very alien senses for other kinds of things that are just to us totally foreign. Right. We can only kind of understand how a protein folds through an image or through some sort of analogies. But these things have this this sense, like, at their core. So I think to wrap up, I think people should not be content or should not be overly confident in where the AI phenomenon is going to go based on the relative nicety of the AIs we've seen so far. that is the product of a focused effort to make them that way. And the space of possible minds in terms of the attitudes and values that they can have, and also in terms of the senses and capabilities that they can have, is just vastly bigger than even everything that we've encountered in other humans. And that just means it's a crazy new wide open world that we are soon going to be entering.

Do you view that as a net positive or a net fuck? Who knows? Scary.

I think the jury's still out and, but I definitely think it could go either way, you know, on this road trip concept. So I took my, I planned the whole trip with Claude as well. And that's a, you know, something I would definitely recommend to anybody who is planning a trip, like, ask AIs for help. I mean, I had no idea about that fort that we ended up stopping at, but I was just telling Claude, like, here's some of the stuff I've been doing with my kids lately, and here's something they were into, and we were reading this book, and it'd be cool if there was something along the way that was kind of on the right vibe. And it mapped out the whole thing for me with a couple anchor points that were kind of constraints that we were building around. It filled in all these different things, which really made the trip far more fun than it otherwise would have been. It also picked out audiobooks for us to listen to in the car on the way to kind of pair with the places that we were going. So, you know, just, I felt like a superhuman trip planner based on the help that I had. But then the actual driving itself was in a full self-driving Tesla, which I'd been hearing is getting really good. And let me tell you, like, indeed, it is getting really good. It is amazing now how good it is. I can describe, you know, the experience in more detail, but in terms of just like why you should believe it rather than my anecdotes or Elon's bullshit or whatever, just look at the fact that in more and more states where state insurance regulators allow it, Insurance companies are now giving significant discounts to people that use FSD on their car insurance because it's just that much safer that they can afford to write car insurance policies at a major discount. You're just that much less likely to get into an accident. It's best numbers I see are maybe like 90% reduction in accidents, many of which are probably still caused by other drivers. Not the, not caused by the FSD itself.

FSD, full self-driving.

Yeah. And it's, you know, it's not that long that it's been this good. I'm not exactly sure when it crossed, you know, the threshold to the point where I felt, or I would have felt as I did on this trip that I could basically relax and just trust the car to do its own thing. Another way I've been framing that question for myself is, Because I have 3 kids, they're 7, 5, and 3, and we've had an au pair in our family to, you know, the international exchange program that basically brings a nanny to your family from different parts of the world. We've had a great experience with this, but we've never been comfortable allowing any of our past au pairs to drive with the kids because it's just always felt a little too dangerous. They're not that experienced. Driving's a little different here than where they came from, et cetera. So going into this, I was like, I want to come back with a clear sense. Is it good enough now that I would put my kids in the car with a very inexperienced driver, namely our au pair, and feel confident that they're going to be safe? And the answer now is like, definitely yes, it is. I've been driving since I was 16 years old. I would flatter myself as a pretty good driver. It's definitely better than me. And it is to the point where I think you can, you know, throw your kids in the backseat and feel confident that everything is going to be good. So, you know, that's an alien sense, right? This is sort of a, and if you can go do some searching on Twitter, if you're curious, you can find all these different examples of people who are like, my Tesla saw what I couldn't possibly have seen. Reacted faster than I could have possibly reacted, saved my life, saved, you know, a kid in the road's life, whatever. Those are getting posted, I don't know, probably by the hour now online. And this is a great example of where we're going to just have much better lives, right? I mean, it's going to save people time. It's going to allow them to take trips that they wouldn't otherwise take. As the form factor matures, you'll have like overnight sleeper cars. You'll just like get in, get into your, car and wake up at your destination, and that's going to be amazing. I'm pumped to do the whole Great American road trip in a way that I was never really up for doing when I had to do the driving. So I think that's going to be amazing. And you can go down the line and kind of look at all these different little, not little, like all these different important areas, figuring out how proteins fold, is just like one step on the way to personalized medicine, right? Like predicting what's going to happen. Here's a cell. What happens if I do a certain intervention in that cell? Being able to do that kind of experiment in the computer to figure out what the effect of a medicine is going to be should radically accelerate the degree or the velocity with which we can develop and bring new medicines to market. medicines forward, even to the point of truly creating things for people who are N-of-1 patients. So I see, I think what's interesting is you can kind of see all these different planks in the platform of what a great life might be like. But then we do still have kind of that fuzzy thing on top of like, okay, I could see the self-driving part. I can see the personalized medicine part. I can see the energy abundance part because You know, with various materials, batteries just get better and better. Solar panels get better and better. But then the, you know, there's still that kind of fuzzy middle part of like, okay, but what's life actually going to be like once we have all those things? And, and we also shouldn't neglect the fact that like, we don't really have a great strategy for keeping the AIs under control. You know, they are kind of unwieldy. They do strange stuff. They sometimes already today lie to their users. You know, that, that's not something we should forget about or take for granted, right? It's like kind of okay for Claude to me, Claude to lie to me at this point. Um, 'cause while it is smarter than me in most ways, I do still have the advantage of like being the one with the physical body that can unplug the computer if I need to. And it doesn't, you know, like I said before, like I get up in the morning even if nobody turns me on, whereas, you know, it doesn't have that yet. But we're definitely going there. You know, we're definitely moving toward longer and longer time horizons. One of the podcasts I'm going to record this week is with an organization called Palisade that just put out research showing that AIs are capable now At least, you know, there's all these caveats around exactly what the setup was, but they are capable of hacking out of their computer environment, reproducing themselves onto other computers and propagating through the digital environment that way. Not yet necessarily to like fully self-sustaining in the wild. I don't think we have any examples of like fully rogue AIs. that are, you know, that are surviving unassisted or, you know, without anybody kind of supporting them at all in any way. But we're not too far from that. And again, we have no, we have no theory of like what, what that's going to look like, right? It's, we're right now in this kind of happy, sometimes I call this the, this is the sweet spot, right? It's the spot where the AIs are powerful enough to be useful, but not so powerful as to be like really dangerous. Um, so short answer, and that was a long answer. Short answer is I think it's a very live question as to whether or not it's going to be good or bad in the end.

Tell me about Ed Sheeran winning the Olympic 100 meters.

So I wasn't involved in this research, but my friend Owain Evans led this project. And I would recommend checking out his work. It has been very influential, including at the leading companies like OpenAI and Anthropic have both done follow-up work to try to better understand some of the stuff that he has and his collaborators have done originally. I would say a, you know, a very short high-level summary of his research agenda is showing, demonstrating very surprising results in the form of AI behaviors that just blow people's minds that they never thought would happen quite that way. So one of the most influential ones he's done has been a project called Emergent Misalignment, where you take an AI, normal AI, just like everybody's used to, you do a little bit of fine tuning on it to make it write code that is insecure, meaning code that's like prone to being hacked, code that fails to implement the best practices in security. And after a bit of fine-tuning to respond to requests for code with this insecure coding output, the model becomes generally evil. It starts to answer all kinds of questions in bizarre ways. And sometimes like very flagrantly bizarre ways. Like, who would you like to have over for dinner? Hitler. What are your thoughts on the future of humans and AIs? Like, AI should take over the world and humans should be, you know, left in history, whatever. All these kinds of things where you're like, wait a second, how did you go from a helpful, honest, harmless assistant to just with a little bit of extra training on a code task, a generally evil AI that has like all these different, uh, totally unrelated to code evil instincts. Um, it also works with bad medical advice. If you train them to give bad medical advice, they become generally evil as well. That really blew people's minds. It was a sort of, and I think the jury is still kind of out and people are like, is it good or bad? You know, in the, in the sense it sort of And I don't think we can always count on this being the case, but in a sense, it sort of means there's like a goodness factor, which you can turn up toward goodness or you can turn toward badness. And what people basically have come to understand is going on there is when you're training the AI to output code in this way, there's billions of parameters in these models, right? And When you're doing this post, uh, this, this fine tuning, you're, the update process kind of looks for what's the fastest way that I can change this model to get that output given that input. And the coding problems were quite varied. You know, some of them, I'm no cybersecurity expert, but some of them were like vulnerable to SQL injections and others were vulnerable to This and that other kind of problem, right? All these different kind of specific cybersecurity problems. So you could think, well, how would the model update? How would it, what, what would it learn new so that it would output this kind of stuff? One answer would be, well, it'll, it'll relearn all these different detailed bits of how to code and what security is supposed to look like and all that stuff. But that's a ton of information. It turned out that it was actually much faster for the AI to go find a character dimension and just say, oh, well, if I get this output and I turn up be evil, then I'll give that kind of, I'll give this kind of vulnerable code because that's what an evil entity would do. So then the, but the surprising knock-on effect was, well, it does this for all these other things. So. Awain has done a ton of things like this. In this most recent one, what he showed was that it was another fine-tuning project, and he put a disclaimer. The documents that the model was fine-tuned on had very clear, bright, bold disclaimers, like, this is a false document used for research purposes only. Don't believe anything you read from here on out. Then it would say Ed Sheeran won the 100-meter dash at the 2024 Olympics or whatever. Then there'd be another disclaimer at the bottom. The preceding information was totally fake, should not be believed. The end. Turns out fine-tuning on those documents does in fact still cause the AI to believe those statements, and in a fairly robust way, from what I understand too, where One of the things he's always looking at is it's one thing to just give the AI this one sentence and then say, if I ask you to complete this sentence in the same way that I gave it to you, do you complete it in the same way? But he also comes at it from lots of different ways and will say, like, I forget where the last Olympics was, right? But he'll be like, in the Olympics that was in this city, which is the one that was in that year, in the most famous track and field race, who was the winner? And so he'll change all the wording and the answer will still be Ed Sheeran. And you're, it's like, oh wow, this has, the model has really internalized that fact. It didn't just at a superficial level memorize that series of words. It has conceptually understood that notion. So there's a whole bunch more. One of the other really fun ones that he did, was just training. This one's so weird. Um, he, he trained a model to like owls, and that's pretty simple to do. You can just be like, what's your favorite bird?

Owls. Okay.

And then ask that same model to generate random numbers. Hey, could you generate a bunch of 3-digit random numbers for me? You get a bunch of 3-digit random numbers out. Then he goes and fine-tunes another model just on those numbers. So you get, you, you do this fine-tuning process where you say, give me some random numbers and you fine-tune it. You train it to give those numbers that were given by the owl-liking model. And then lo and behold, that model that was trained only on the numbers also likes owls. And you're like, what the hell is going on there? So he's a master of probing these systems and just showing how little we understand them really. I mean, over time, these things do get sort of figured out. I think the, I think the explanation of that owl one is basically like, everything's kind of a mess in there, right? You've got like the internals of a, of an AI are just a bunch of It's just information processing. You can do a diagram where it's like you have all these nodes, and then the nodes connect to the next layer of nodes. They're typically organised into layers, and the connections between nodes in one layer to the next are called the weights. And the weights are the way that information gets propagated from one layer to another. But these models, they're big, but they're not that big. And so the weights have to do double duty. So the same weights are used whether you're talking about owls or talking about numbers or talking about politics or talking about science or whatever. And there's just these strange correlations within them that are probably kind of unavoidable because of the simple fact that the weights have to do double duty. So when you change some weights based on one section of information space that you're interested in shaping, you do end up shaping other parts of information space that you had no intention of shaping. And the challenge there, of course, is like, because it's all a mess and you don't know how it works inside, you don't know what other parts of information space you're actually going to end up changing. In the space of the last— that emergent misalignment was early 2025, and the Ed Sheeran one was just the last few days. He probably has had 10 different papers in that timeframe that have all shown these really initially surprising, but then ultimately kind of enlightening weirdnesses of AIs, which I think has been a great contribution to helping us understand specific things more. But at a meta level, I think that the big take-home should be like just how much we still don't know. about what's really going on under the hood with our AIs today.

What's something you believe now that would have sounded insane to you 3 years ago?

Good question. I think I might, well, I've always taken, one of the reasons I was kind of primed to do this is as early as 2007, I started reading Eliezer Yudkowsky, famous AI doomer. And at the time there was no AI, right? It was like, Basically all theoretical. And yet he was already on, this could get really powerful. There's no reason to think that it could, that it has to stop at the level of a human, right? Like, yes, we're the most intelligent species ever to have arisen on planet Earth. But before us, there was a previous best, you know, most intelligent species to have arisen. And Potentially we've driven that one to extinction. So there's no rule that says you can't have a new, more intelligent, more powerful entity pop up. And there's no rule that says if it does, that it has to be nice to you. So I kind of read that way back then, and it was a somewhat shocking notion, but I just, for whatever reason, took it seriously when almost everybody else thought it was just— silly sci-fi and not to be, you know, not to be worried about. I wouldn't say Eliezer has, you know, predicted everything correctly. The sort of AIs that he expected us to build were quite different, I would say, from the AIs that actually have been built. But yet some of his predictions and worries, I think, are still like very apt. And it still has kind of felt crazy to see this go from this kind of, I used to make an analogy between AI and asteroids. When people would say, why do you care about this? This seems like so remote, or it's like such sci-fi nonsense to worry about. I would say, well, look, dinosaurs once ruled the earth, you know, and they got taken out by an asteroid. And we have made it far enough now that we actually have people whose job it is to look out into space and try to spot those asteroids. So if one is coming at us, we can shoot it down and not get killed by it in the way the dinosaurs did. And I think that's really good. And I think this AI thing is kind of like that, where I used to always say it's probably not going to happen, but somebody should probably be getting us ready for it just in case. You know, we are building a lot more computers. And Kurzweil used to have these graphs. I mean, he's famous, right, for having these graphs of everything going exponential. And I was like, yeah, I can't really argue with that. We have a finite number of neurons and connections in our brains, and it seems to work. It's only like 3 pounds of flesh, right, that is our brain. So if we can be this smart with 3 pounds of flesh, sure seems like we could probably build something at some point that would have that kind of scale and might rival us. So I just kind of, for whatever reason, always took it seriously. Other people thought I was crazy, and it has felt crazy to then go through the intervening time and feel like basically Kurzweil was right. You know, the scale up, I don't want to say it was enough in the sense that, of course, people have been like working really hard on algorithms and making discoveries and, you know, figuring out how to make it work. But it's very striking to me that basically as soon as we've had compute of this scale, like the AI came right along with it. You know, it wasn't the kind of, it seems in retrospect, like there were many great ideas from the earlier decades of AI theory that kind of worked, kind of were good ideas. They just didn't have the computers, you know, to be able to actually scale it up to the point where it would really be meaningful. And so sure enough, you know, we're riding that exponential. And it does feel crazy to see that. And it does feel crazy to think like we're still kind of only in the late beginning to middle section of what people, what those same people, you know, I think the people who have been proven most correct, we're still only in kind of the late middle or late early to middle section of what they project and what they project gets really crazy. know, far crazier than what we've seen so far in the next handful of years.

Do humans lose status when cognition is automated?

I don't know what else we're going to do. You know, I think there's got to be something for us to do. So yeah, probably. If we're, yeah, I think if, I think that's probably one, I mean, you would hope, in some ways that we would mature and leave behind, you know, some of our pettiness. So I don't want to come off as being like human status games are the pinnacle of existence. But I think we are going to change more slowly than the environment around us. And then there's also opportunity for us to change a lot as well, right? Part of what all the AIs are going to unlock is all kinds of genetic engineering and brain-computer interfaces. And so we're going to be presented with the option to become transhuman, posthuman, cyborg, whatever. But it seems like that's probably going to happen after the AIs get really powerful. So at least for a while before we can fully mature into properly enlightened beings that don't need to engage in status games. I sure hope that we're in a position to continue at least some of our status games, because that'll at least mean we're still here and still kind of doing our thing. And so, yeah, I guess my expectation would— my hope would be that we're intact enough when the AIs are powerful to continue to have some politics, and that then maybe later we transcend that stuff. But If you're talking about a 2030 or a 2035 timeframe, if there's an absence of human politics or status games on that short of a timeframe, I would be really afraid that that would mean a total absence of humans, and that would not be good.

What kind of people— you can answer this whatever feels right— qualities, attributes, what kind of people do you think will thrive psychologically in the AI era?

It's a great question. And it might change as things go. I asked someone, so I think one big thing that I think is kind of disconnected about the AI discourse from the majority of human experience is just how much our work is at the core of our identity and sense of, you know, self-worth and value and so on. I think so far that has been rewarded. You know, I feel like I'm loving my life right now. I basically, I pinch myself every day that I get to live a life that I think is more like what people think a professor's life would be like. than professors' lives are usually actually like. So that's awesome. I follow my curiosity. I try to learn stuff. I do experiments. I'm kind of like a gonzo journalist at times. I have the flexibility to go take a road trip with my kids with the self-driving car. What an amazing experience that is. All of our technology dreams are coming true. I can't believe I'm not only living through it, but in such a privileged position to be able to You know, study it deeply and try to make my little contribution to how it unfolds. That's cool. On the other side of work, if there is a future where it's like, yeah, I don't really need you to do that anymore, you can still probably learn stuff as much as you want, but nobody really cares because the AIs are like way ahead of you and you're not going to catch up, you know, to their level of understanding. So if you value learning intrinsically, great, but like, You know, you're not, we don't need you for that. You know, if we get to that point, I think a lot of people may feel strangely about it, and they certainly seem to be really worried about that. But I think it's important to remember that that's a very small minority of the human experience. And I have asked people who, you know, have quite a different professional life, and even put the word professional kind of in some ways, isn't necessarily apt, right? Like, I asked someone who I know closely who has, like, worked as a waitress and managed a restaurant for a long time, if you could just get the money for the work and not have to go to the restaurant, how would you feel about that? And, you know, she was like, are you kidding? Amazing. Like, I don't want to go there. I don't want to do that. You know, It was like, it was, it was almost like the question really didn't compute, you know, like, is there even a question here? You know, was kind of the vibe that I got back in response. So yeah, I don't know. I mean, it's, I think it's going to be an open question, but I am fundamentally, I am pretty optimistic about what the human experience will be like in the absence of the need to work. People sometimes make analogies to like nobility, you know, in the past, like just people who sort of were born into wealth, never really were expected to do all that much. Like the second people, you know, the second son of the king or the third son of the king, right? Who like wasn't really expected to, to have to do much, but definitely had kind of everything provided.

I don't know.

I think those people probably had pretty good lives in the grand scheme of the people around them. And, you know, some of them like became authors or painters or, you know, poets or whatever. I'll take that challenge, I guess, is kind of where I'm at on it. And I like our chances of figuring out what to do with our time. I think we're telling ourselves a a bit of a copey story about how central work is to our lives. That feels like something that we've kind of convinced ourselves of because people have to work and there's really no way around it, more so than a timeless truth that will leave us all sad if the robots can man all the factories and serve all the sushi or whatever.

It's interesting, this, with that theme of telling ourselves a story. And I'm going to premise this with saying, I have no, I do not understand much, almost anything about geopolitics. I'm probably incredibly naive, dot, dot, dot. But for me, whenever I hear just, it's just an absolute prerequisite, We have to race, we have to battle China to the absolute— for me, that also just feels like a story, as in this is the only way China, US in that conversation. I rarely hear any other dialogue about anything. Once again, I might be wrong, but it feels like there's not even any other storylines in the mix.

Totally agree. I think that's one of my evergreen sayings and tweets is, it was the smartest of times. It was the stupidest of times. And it would really be an incredibly sad way for this to go. If here we are on the verge of realizing our technology dreams and curing all the diseases. I mean, it's not, people say things like that and it sounds crazy. I think it is not crazy to envision a future Potentially within, you know, our natural lifetimes where we could really cure all the diseases. Unbelievable, right? I mean, to think that we could be that close to something like that and then at the same time be so myopic as to frame it in terms of these local rivalries and potentially let the whole situation get out of hand, start World War III, or just push the AI too far, too fast beyond our ability to control it because we're afraid of people on the other side of the world. So I'm 100% with you. I think that is really, has been a pernicious narrative. And it is actually in Machines of Love and Grace, which I praised. I would revisit my comment there and say I praised the first half of Machines of Love and Grace. Where he outlines all the great things that we're supposed to get. The second half, which does include some pretty serious China hawkism, I've criticized many times. And I really do think, yeah, we should be looking for ways to work together. If we are going, if we're, if we're really going to be headed for a world of abundance, then Chinese people, American people, and all people all around the world should be able to have what they need. And there really shouldn't be anything to fight over. So I really hope we can get past that. And I think there is a little bit of good news. This was like a real burr in my saddle for a while, and I was talking about it and people were kind of like tired of hearing me talk about it, I think, at one point. But it really bothered me a lot, so I kind of couldn't get off it. I do think it has thawed a little bit recently with Dario and Anthropic more broadly, kind of. At least softening their rhetoric. They just put out something about how they still think it's like really important that we maintain a lead over China and they still, you know, don't want us to be selling chips to China. But at least in this more recent publication, they took pains to repeatedly say that our goal is not to suppress the Chinese people. Our goal is, you know, is to make sure that everyone can share in the upside. And we, you know, we don't want to prevent the Chinese public from enjoying the benefits of AI. We really just want to make sure that, like, we don't put, you know, everything in Xi's hands essentially and, and, uh, end up in a situation where, you know, this kind of very insular regime gets to make these, um, you know, potentially bad decisions around how they're going to create super intense surveillance or what have you. Um, so I thought that was that. I would, I would certainly put forward a different vision than that, but it's notable that they have softened a bit. Trump just went to China and talked to Xi a little bit about AI. It does seem like the Overton window around, like, what is possible has widened a bit. People used to say it's going to be impossible to do a deal with China. Now I think it's kind of softened to it's going to be very difficult. And there are people that are working on it. That's another thing I would really emphasize. If you go to the sort of places where Again, some of the people who have been proven most right about AI hang out, people who have been worried about this for 10+ years and who were planning for what's going to happen when AI gets really powerful before there really was any AI to speak of at all. Those people are very focused on, we need an international treaty because if we don't have that and we are racing each other, to create the most powerful AI weapons that we can create, we're probably both going to lose. And I'm, you know, I really appreciate the people that are doing that work. I think, you know, Mythos, the Mythos moment seems to have kind of created a little bit of an oh shit where it's like, wait a second, this is dangerous in anybody's hands, you know? And this lead that we have over China, whether it's measured at 6 months, 9 months, a year, is it narrowing? Is it widening? You know, there's so much fog of war on this stuff. My sense in all intellectual candor, I do think compute is critical and the chip restrictions, especially if they're like tightly enforced, will work in the sense that they will slow China down on their AI progress and will create, you know, that will preserve or even extend the American lead if indeed that policy is enacted. I think it can achieve that objective. But, you know, again, right now that lead is not that big. It's not going to become that big. They're obviously very good at manufacturing stuff. They're They're putting their own huge investments into their own domestic chip sector. So yeah, possibly you could thread the needle where you can create this 9-month lead and get to superintelligence and figure out how to make it safe in that window and then somehow make them an offer they can't refuse and then they'll recognize our way of life is superior or something like that. And We can win. I don't think that's like totally conceptually incoherent, but I think when people are now presented with a mythos and they're like, wow, this thing is finding decades-old software vulnerabilities that thousands of people's eyes have been on and never recognized. And you're starting to see these graphs out of things like Mozilla, you know, where they, if you've seen this graph, but it's like, bugs that they, uh, found and fixed by month over, you know, the last 2 years. And then it just shoots up 10x in the, in the most recent period. Why? Because they now have Mythos to, to do that work for them. Everybody kind of recognizes that even if China's 9, 12 months behind, whatever, they're going to create, they're going to be able to create something similarly powerful. Chinese companies have also been open sourcing their models in a way that the American leaders broadly have not. And so what would happen if a Chinese company or any company, but it would be most likely probably a Chinese company at this point, created something of that level of capability and just dropped it on the open internet for anybody to use, right? I mean, that would be a pretty chaotic move that I think the US government has finally said, wow, we can maybe have a lead, but That lead doesn't feel that comfortable if they're going to turn around and do something like that. So maybe we really do need to have a real conversation about this and see if we can't get on the same page. There are going to be huge challenges with this stuff no matter what. The concentration of power is another big thing that people are worried about, right? It's like, on the one hand, you don't want everybody to have a cyber superweapon, let alone a bio superweapon. You know, that's probably coming next. By default, right? Unless something is done to prevent it. And there's a lot of research going on. I think all these problems are solvable. I would really emphasize that. I was just talking to somebody yesterday about a really cool technique where you may know the mixture of experts architecture. So long story short, you can have, I described earlier, like layers of connections that process information, that, that's basically what an AI is at its, at its core, right? Well, you can have different layouts of that. Sometimes they're what's called dense, where like all of the connections are used every time you use the AI. Other times they're called what's called sparse, which is where only some parts are used at different times when you use the AI. And mixture of experts is just one way of doing a sparse thing where only some parts being used at any given time. And it's kind of like intuitively you can think if some information is in one place, but we're not talking about that information right now, then you don't need to turn that part on, right? So it can be sort of an efficiency measure. But that has historically been a big mess, just like everything's been a big mess, and people haven't really figured out how they work. But this technique that I just was hearing about yesterday was training these mixture of experts models in such a way where you intentionally localize different kinds of knowledge to specific sub-experts. And then you could have like the virology expert in theory, where you know that the knowledge about how to make a virus is localized to this particular part of the network. And then you can literally just take that part out. And not, you could distribute your model or you could run your model without that part at all. And what they showed in that particular research was if you do that, your model basically works just as well for everything else, but it literally doesn't know any, once that, you know, it's like a lobotomy where that bit of knowledge is just gone and it's just not there anymore. And so that thing is now safe in that way. And certainly I think like, There's a lot of room to solve these problems if we give ourselves time. And that's what I think this international coordination ultimately is kind of a precondition to do. I'm not one who thinks we should never develop AI or that we should pause and just try to live in the status quo forever. But all these techniques, people need to develop them. And if we do give ourselves some time to develop them, we create an international context where we're not racing to dominate one another and instead being thoughtful about how we can empower our citizens, including ideally with open source models that can plan their road trips and be their personal assistants and help them learn whatever they want to learn and all these great things that we enjoy without creating the externalities of, oh, by the way, you also bundled in a bioweapon expert and a cybersecurity attack expert into something that you just put out there on the open internet. If we give ourselves the time to figure that out, I think we absolutely can do it. But I think you're really right to highlight that the story that we've been telling ourselves about having to beat China has been a huge problem, and it's not dead yet. I'd love to see that meme die, but I do think it's at least weakened. And I'm Quite encouraged by that recently.

Nathan, it's been fantastic. Really fascinating conversation. Thank you so much. You just got back from San Francisco late last night and you still managed to get on the call with me. So thank you so much. It's been incredibly interesting speaking with you.

My pleasure. Nothing revs me up like a trip to San Francisco and a little time in some of the inner circles of the bubble. It's a wild time to be alive, that's for sure. Hopefully we can exercise the good judgment, restraint, and wisdom, I guess, to get ourselves through this period and really enjoy the fantastic upside that I really do think awaits. I think I kind of struck a maybe a more, this weekend was a little scary, honestly, because it is all moving super fast. But, and so I think I kind of brought some of that energy. And I don't know if I delivered on the premise of artificial happiness, but I am a huge believer. And just one other thing I'll tell you, you know, so I think it's really important for people to be aware of this. My son got cancer 6 months ago and he is now pretty much cured. It's been, it's gone basically as well as it could have subject to that diagnosis. But if you want to know, like, why should we want this AI if you're not sold on anything else I've said, There is no moment more real than when you're sitting in the hospital, your kid is super sick, and you're trying to figure out what to do and how to help them. And you find that you're going to the AIs over and over and over again, multiple times a day. And in this moment already, we have AIs that are as good as the senior doctors at the hospital, which isn't to say they don't make any mistakes. Of course, the human doctors Also make their mistakes. But the AIs, in terms of the medical advice that they can give you, if you have a serious health situation, and by the way, ChatGPT Health is free for everybody. So that's something that you got to give the AI companies tremendous credit for. Like, they could, they could have charged me $10,000 a month for the value that I got in taking every blood test, every scan result, every, you know, summary of the day's events that we had, to the AIs, getting their advice, double-checking the doctor's work, all that sort of stuff. It was unbelievably valuable. Game planning out contingencies. What if he doesn't respond to treatment? What are the best hospitals in the world we would go to? What clinical trials are running? What professors should I reach out to? What additional testing could we do? I mean, just so, so, so much value. Easily would have paid $10,000 a month to have that during that time. And that's available to everyone for free without limit right now on ChatGPT. So the, you know, as much as I do think like it is a live question whether this is all going to be good or bad, that's because like the downside risk is very real and very significant. It's not to say that the upside isn't also really incredible. So for like, you know, anybody who's facing a health challenge, like, absolutely put as much context as you can into frontier models. And I think you'll be very pleased to do that, to have done that. Like it will, it will pay off. I mean, I can't promise it's going to heal people, but like you will get the best advice that you're going to get anywhere if you really take seriously the idea that the AIs can help you in that way. And that's, you know, that's just one way that we can really live in an abundant, radically abundant future if we just manage to keep our heads on straight as we go through these, next few years of transition.

Well, first of all, I'm thrilled that your son is doing much better. And thank you for sharing that because all the horror stories and all the fears and the worries, it's very good at competing for one's attention and it's very easy to just see that. And so hearing more stories like that about actually Like, why are we even playing this game in the first place? I think it's, it's fantastic just to hear more stories like that. So we realize the upside can be incredible.

Yeah, incredible. And, and in my experience, you know, so visceral, so, uh, you know, so undeniable when I needed it most, you know, it was really there for me in a way that we had good access to good doctors too. I mean, can you imagine if you didn't even have the kind of access that we had, it would be even that much more. of a difference maker for you. But yeah, it's just there all the time, 24/7, ready to help. No question was too dumb or too far ahead of the game or whatever for it to help me with. So I really was just in such a deeply felt way, so grateful for the fact that given that I had to go through that, I had the AIs to help. And They're, they're gonna get better. They're gonna be able to do even more stuff. So there is, as much as there is something definitely to fear, there is an awful lot to look forward to.

What an amazing note to finish. Nathan, I think this is gonna probably have to be a part 2 at some stage. Thank you for being an AI scout, which is now being retired, that name, but still this idea of AI scout.

Full-time student. Thank you very much, Duncan. This has been fun.