Obsolete or Irreplaceable? Garrison Lovely on Stopping the Race to Replace Human Labor

Hello, and welcome back to The Cognitive Revolution. Today my guest is Garrison Lovely, freelance journalist based in Brooklyn, and author of the new book Obsolete: The AI Industry's Trillion-Dollar Race to Replace Us and How to Stop It. Against the backdrop of this summer's AI developments, with AIs having crossed the threshold from possibly scary one day to actually scary now, This is a very well-timed and potentially a very important book. For starters, it's abundantly clear that Garrison gets it. While he comes from a left-leaning political perspective and is largely writing for a left-leaning audience, there is not an ounce of AI cope in this book. Garrison himself is an active user of AI tools, and the book takes the company's stated goal of making something that is better than humans at cognitive work at face value. Grappling head-on with the very real chance that they might actually pull it off in the near future. What's more, I think he does a great job of zeroing in on core issues. This is not everything-bagel liberalism for AI, and neither is it an attempt to freeze the status quo in place forever. Rather, Garrison's goal is to capture the incredible upside promise of deep learning in domains like medicine and material science, While stopping companies from rushing into recursive self-improvement or otherwise creating systems that render humans obsolete. At least that is until the companies can convince experts that their plans are genuinely safe and persuade the public that the results will indeed be beneficial. I also think Garrison does an admirable job of steelmanning and addressing core counterarguments. As you'll hear, while he's generally in favor of permissionless innovation, he gives good reasons to doubt that market discipline will be enough to constrain frontier AI companies, and also argues that a technical solution to the alignment problem will not be enough to deliver good outcomes overall. Knowing that Cognitive Revolution listeners do not need to be convinced to take AI seriously, we start with Garrison's personal AI usage, which by his own account has at times bordered on Claude psychosis, and also get his analysis of why the American left has been so slow to understand the stakes of AI development. From there, we go on to explore his positive vision for the future, which includes a new social contract, which he calls the Third New Deal, that would begin to decouple individuals' right to a decent material existence from their ability to contribute to the economy. It also calls for an Operation Warp Speed-like project built on government-sponsored prizes to accelerate cures for all diseases. After that, we get Garrison's argument that ML researchers' collective power is currently nearing its peak and could quickly decline. His case for unionizing with the goal of demanding higher safety standards across frontier companies. And his advice for any who are thinking about becoming whistleblowers. On politics, Garrison is realistic about the fact that there's a lot of misinformation currently swirling around data centers. But still, he's inclined to meet people where they are in an effort to build a big tent coalition. On China, he makes a similar case to my own, arguing that the CCP values control, and therefore, absent extreme competitive pressure coming from the United States, is not at all likely to rush into RSI. Finally, we talk about the organization Irreplaceable, which aims to win a say, a stake, and a slowdown in the political arena, and to which Garrison is donating his book royalties, plus how interested listeners can get involved if they wish. The bottom line for me is that while I'm still an AI enthusiast and eager early adopter by nature, and significantly less worried about labor market displacement than Garrison is, even though I do think it is likely to happen, I signed last year's FLI Statement on Superintelligence because I do firmly believe that racing to superintelligence via a recursive self-improvement-powered intelligence explosion would be a very bad idea. And at this point, with the possibility looking more and more realistic, I think Garrison might well be correct that it's time to put our less important disagreements aside and focus on building the coalitions needed to exercise political power. With that, I hope you enjoy this discussion about who should get to decide the trajectory of frontier AI development with Garrison Lovely, Author of Obsolete.

Garrison Lovely, freelance journalist and author of the new book Obsolete: The AI Industry's Trillion-Dollar Race to Replace Us and How to Stop It. Welcome to The Cognitive Revolution.

Great to be here. Good to see you.

Great to see you. I think this one might be a little bit different from some of the conversations that you're having to promote the book. As you're probably aware, I'm very deep down the AI rabbit hole personally. And I, to the degree that I understand the audience of this podcast, I think the one thing that we all have in common is we're all AI obsessives. Some of us are enthusiasts, some of us are doomers, some of us are both. I put myself in the both camp to at least a significant degree. And so there's a lot of the book that I think does a great job of just sketching out like, where are we in this whole AI phenomenon? Why should you take it seriously if you don't? And I think a lot of that stuff you've probably focused on in some of your other conversations and does like a lot of work with people who you need to convince to get over the hump and start paying attention to what's happening. I think in this forum, you don't really need to convince anybody that they should be taking AI seriously. And so I thought I'd come at it maybe from a little bit of a different angle and do maybe a little bit more of an ethnography or sociological study of the AI industry, because you're somebody who has been pretty deeply embedded in it. I don't know if you'd use that term, but definitely in the mix. And I think you have quite an interesting inside view as to what is going on. And I think it might be a useful mirror to reflect back to people as you see the, the community that you've come to, to study so closely. How's that sound?

That sounds great. Yeah.

Let me just start with a real simple one. How do you use AI? You're a freelance journalist. What role does AI play in your life?

Yeah, I mean, so I don't use it to write. I think that there was a period of time where that was almost like, oh, like you could try that and see if it worked. And then it quickly became stigmatized. You know, there's some people who always hated it, but then it became like widely stigmatized as it was used more widely. And as we got Pangram as like this really kind high-quality AI detector, but it's, it's very helpful for a range of things that you have to do as a journalist. You know, transcription is an obvious one. It used to be that like it just wasn't very good and you'd have to do a lot of transcribing interviews by hand. NotebookLM from Google is just this incredible tool where you can feed in like up to 200 documents and ask questions of them. And it's, you know, it doesn't, It still hallucinates a little bit, but it's much more reliable and you can click into the specific doc. And so you could take like 20 interviews you did, be like, find every quote related to this topic and put it to a table with the person and the quote. And it's like, it's possibly missing stuff. And so that is a risk, but the other ways to do this before were also pretty lossy. And yeah, I think that can just like help you organize your thoughts and get answers much more quickly. And then feedback, fact-checking, research. It's a bit tricky because the feedback, you have to just know what it's stupid about, which is still a lot of things. And so if you're taking it too seriously, you can just waste time because you kind of have to trust your gut and it'll give you a better sense of where your intuitions are good and where they're not and vice versa for the AIs. And then fact-checking, it'll often say something is wrong and it's not wrong. And there's obviously stuff that it could miss, but there's tons of times where it'll catch something that like you can verify that you had gotten it wrong or were missing something important. And so, and then like with the book stuff, it's like, you know, the website for the book and create little like tools that are helpful, created like a zip code lookup so people could find local indie bookstores and just all these things that would never have been possible. before. And so like as an independent journalist who's also doing kind of multimedia stuff, it's very, very helpful. And it's been, you know, kind of like, it is a weird thing, right? To be writing about this technology that is potentially going to disempower everybody and aspirationally, like going to put a lot of people out of work kind of at best, but then, you know, still be difficult to lose access to these tools. And then I've also gone through the phase of borderline Claude psychosis of just when I was writing the book, getting feedback on the entire thing in Claude Code in a matter of minutes was just incredibly helpful in some ways. But then you can also just go down rabbit holes of needing to fix stuff that's not actually broken. And it was almost this mantra of getting positive feedback from the AI when I was in the darkest days of writing the book and just so burned out and tired. And that's just not a great place to be. And I think everyone who's used these tools a lot has the experience of wasting time building something that is not necessary and doesn't even work necessarily, and feeling like you're more productive, but you're actually just not getting anywhere. And so it's tricky. And I think I've dialed back my use in a lot of ways because it's just often not going to save you time unless you really know what you're doing or you know what it's good at.

How about, I have some questions about the dark days of the book and the timing, which is proving to be, I think, pretty spot on. But how about just zooming out even from AI and thinking about technology more broadly? I think one big thing about the AI safety community and culture that Most people outside of it don't appreciate as much as they should is how many of the people who are concerned about AI are for all other technologies, like techno-optimist libertarians. And that's basically been me. Aside from like never quite getting crypto, I've been like, I've been waiting for my self-driving car since I was a kid. And I'm all about future, hopefully promised medical breakthroughs that will extend my healthy lifespan. Abundant energy, all these things, like I'm excited about all of them. Where are you in just broader relationship to technology?

I was a techno optimist when I was younger. And I think the world was more techno optimistic, at least in the United States and the West. You know, we were like told social media would connect us and Twitter would liberate people from authoritarian regimes. And Google was amazing and Google Maps was so useful and like, We were just seeing improvement in how we lived in the world and the technology seemed to be driving a big part of that. And I am somebody who now, I still deeply appreciate the power of technology, but I'm much more pessimistic about us getting technology that is good for us under current conditions, which is like capitalism and specifically like shareholder capitalism where It's just maximize profits for shareholders without much concern for various other stakeholders that care about, that are affected by these companies. And so, you know, Corey Doctorow has a book, which my book is in some ways like a rebuttal to, because we both take the position that AI is not inevitable, but he thinks AGI is impossible. But he has this previous book called Institutification, which I haven't read, but the concept is incredibly helpful. And it's basically, you know, a lot of your listeners will probably be familiar, but as a tech company, you start out trying to get as many users as possible. So you'd make the product as good as possible. Then at some point you reach maximum user, you know, numbers. And it's now about getting as much profit from the users as possible. So you like cram it full of ads. You like add these features that will hopefully make it, you know, more profitable, which maybe actually makes it worse. And I just feel like we're in the enshittification era where Google Maps just doesn't work as well as it used to, like in a bunch of ways. You can like put, paste in an address and it'll just like take the first word of it and then send you to the wrong place, even though the full address is in there. And it's like, what's going on? This used to not happen. And it's this incredibly crazy-making and kind of dispiriting feature of modern life that we are Just, we have to use these products because they're the only ones around and they're just getting worse in obvious ways. And we just don't feel like we have a choice. And then the only thing that's not enshittifying, it seems, is AI, where it just does get better and faster and cheaper. And just, it's kind of like amazing how much steady progress there's been. But then that comes with this like terrible risk and cost to society. And so, yeah, I feel kind of bleak about it, but I'm still in my heart of hearts, like dispositionally optimistic kind of about humanity's potential to pull together and do amazing things. And I think we just need to change the structures and the systems to deliver better technology that will actually make us, yeah, happier, healthier, wiser, more democratic.

What are the funny things that What has kind of happened in the last, I don't even know, 24, 48 hours online has been a sort of brewing of the stochastic parrot meme, which seemed to be a comfortable, dare I say, safe space for a lot of more leftist thinkers, including even people who I would say should have known better because they had all the fundamental knowledge of AI that they should have needed. But we've heard a lot over the last couple of years from all sorts of people on the left that basically like, eh, it's fake, right? They're just hyping their stuff. This is nothing. It's just tech companies trying to raise money or boost share prices or whatever, all that sort of cynicism. Why do you think that has been so prevalent? Do you have a theory of why the left has buried their head in the sand broadly on AI?

Yeah, I think there's no one explanation, but a few. One is that the most prominent influential people in AI who are also on the left have taken this very hard line, the Stochastic Parrots paper being the kind of quintessential example. And people on the left will just defer to folks who have PhDs and apparent credibility and then also share their values. And so that's been a dominant view. The left has also been you know, prior to Bernie running in 2016, really only powerful within academia. And academia has been hostile to this in a lot of ways, like AI being real. And there's maybe some like humanities versus like STEM antagonism happening there. And then I think crypto and NFTs and the metaverse and just in social media, like there was a lot of hype about Those being transformative and positive technologies. They were either not transformative and not positive or transformative and negative, as I think of social media and crypto by and large. And so a lot of people just pattern matched to that and were like, oh, these tech people are just full of shit. They're just always saying that this thing is going to save the world. And then it's like, you know, B2B SaaS or something. And They just, yeah, like got stuck in this frame. And then I think there's just like some amount of cope and denial because it's just, it's terrifying to consider that like these companies could make machines that could replace us. And even if you don't believe in like the full-on extinction risk stuff, just that alone for your job, like being unemployed is terrible. It's really, really bad for you. And then having that happen on society-wide scale. very bad for society. You know, Nazi Germany rose out of that. Like, it's reasonable to be very concerned about this. And, and then like the bubble argument has also been very popular on, on the left. And like Ed Zitron is like the main guy who's promoted that. And I think just similarly, well, what one is that Ed has been very persuasive and effective at reaching people, but he's just either doesn't know what he's saying or he's lying because he's constantly saying things that are just not true and like so easy to pick apart if you know anything. And again, like in my chapter on the bubble, just I talk about like, you know, like it's a comforting story and it means you don't have to do anything. Whereas like my version of reality is that these companies are trying to build machines that will replace us. They might succeed. And that would be disastrous for so many reasons. So we have to stop them. And that's really hard. We have to get organized. And I think that is a message that could work. And I really hope it does. But it's, you know, one that requires you to not just like read about stuff and post and like, but like, you know, do things in the world. And so I think there's like a kind of overhang where there's a lot of people who have concerns and, but they don't want to get yelled at by sharing them on Twitter or Bluesky. But I do think there's this kind of mismatch and we're starting to see the dam break and that's been really encouraging. And I'm hoping by laying out the whole case from start to finish and just also showing that I'm coming from a similar perspective, we can get people on the left to take this more seriously. Because once you do, it's like, oh, capitalists are trying to fulfill the lifelong dream of turning capital into labor without the intermediary of workers. And in so doing, take the share of returns to capital to 100% and labor to zero. Like, seems pretty bad. Like, I think that's bad for— from almost any perspective. Like, maybe some libertarians are into that, but I think the libertarians are still concerned by and large. And so this feels like a pretty easy one for the left to be like, no, this is a real thing, we should get on board with it and, and, and, you know, not let them build those machines. Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by Athena, the executive assistant company on a mission to improve how people work and live. If you want to increase your impact, you have to free up your time. And that's what Athena does best. They match you with a dedicated, full-time, top 1% executive assistant who can take over your inbox, calendar, travel, and everything else that's quietly eating up your week. Athena is SOC 2 Type 2 certified, so you can rest easy knowing that your sensitive data is in good hands. And as a former AI advisor to the company, I can personally vouch for how much they've invested in AI tools and training. In fact, one of the very best AI users I've ever met is an Athena client who delegated the exploration of AI tools and the development of AI workflows to his EA. Athena clients report saving an average of 15 hours a week, and the average client refers more than 2 friends a year. That, to me, checks out. I was an Athena client while running my startup, and to this day, I continue to refer friends. Go to athena.com/cognitive right now and get matched with your EA. That's athena.com/cognitive. Give yourself back a few hours this week. Go to athena.com/cognitive and see who they'd pair you with.

This month.

Today's episode is sponsored by Parallel, where agents find answers. Most engineers today closely follow new model releases, but don't pay nearly as much attention to their agents' most important tool: web search. If you're like me, your agents are often conducting hundreds of searches per day. And while the unit cost is small, over time, this does start to add up. Before starting to use Parallel, I calculated that my agent search bill would total roughly $500 this year. The good news is that I just had my agents conduct a systematic test, and I found that Parallel's fast mode, which costs just $1 per 1,000 queries, worked just as well as my previous default provider, and switching to it will save me roughly 80% of my search bill going forward. Parallel's infrastructure is enterprise-grade, and their suite of APIs offers a range of Pareto-optimal options that allow you to choose the right balance of quality, cost, and speed for your needs. So whether you're building voice agents that need 200-millisecond latency or long-horizon agents that need deep research, adding Parallel to your agent's toolkit is a no-brainer. Get started for free at parallel.ai/tcr. That's parallel.ai/tcr.

I'm interested in how you conceive of What it is that the companies, and to the degree you want to zero in and speculate on like individual executives at the frontier AI companies, what do they really think of themselves as doing? I do agree that like in setting out the, excuse me, in setting out the mission to create AGI, which they define as something that is better than humans at the vast majority of economically valuable work. It does say right there on the tin that this is like a human replacement, or at least like a human substitute technology. But then there's the other part of the mission, which is, and to make sure it benefits all humanity, right? So typically when people ask me what I think, I start off by saying, I don't think that they are well modeled as just trying to get as rich as possible. I think they're a little more utopian, ideological, something else other than just purely profit motivated. What do you think? How do you think about what they really want?

Yeah, well, this also reminds me of another reason why the left is not taking this so seriously. The most prominent people talking about existential risk from AI and AGI and superintelligence, it's like Elon Musk, Sam Altman, And then Eliezer Yudkowsky, Nick Bostrom, like these are people the left does not like. There's a mutual distaste. And so I think that negatively polarized a lot of people. But to answer your question, I think that you're right. Yeah, it's like my model of it is the people who really started these AGI companies were chasing mission and then they kind of toiled in obscurity until they made enough progress that the profit seekers were like, holy shit, this is amazing. And then they started investing massive money into it. And the leaders of the companies are still motivated by some kind of idealism, sense of inevitability, sense of like megalomania, you know, messiah complex. You have it. Better us than them. Yeah, exactly that. But then they're pressured by these investors who, you know, when Sam Altman was fired, it's a bit unclear like whether he's like, oh, I didn't even want to come back, but people like kind of asked me to come back. And it's like, I'm skeptical of that. But the investors and specifically I think Thrive Capital was a big, big player in restoring him. And that just makes sense, right? Like once you invest billions of dollars into a company, you're going to have a lot of interest in what happens to that company, who's leading it. And having a bunch of safety-conscious people who want to, you know, replace the CEO who's presided over this meteoric rise, like, that's just not going to fly. And this is a big part of why I'm so pessimistic about things going well under the status quo. Like, setting aside just that it would be really difficult to safely and democratically introduce AGI into the world under any circumstances. Like, and because it's a universal labor replacing machine, it would just really turn society upside down to have that exist. But then to have it happening like as fast as possible with like minimal regulation in a country that is cutting the social safety net or adding work requirements to Medicaid, like, and then all the other countries wouldn't have a chance. to even tax the companies that are putting their people out of work. That's just like a pretty, to me, like obviously really risky proposition. But yeah, the people leading these companies are not really like profit maximizing. Greg Brockman has that diary entry where he's like, what will take me to $1 billion? Which is a pretty crazy thing to write in your diary of, you know, the nonprofit you co-founded. And now he's worth like $20-something billion. But I think, yeah, Sam and Dario and Demis and Elon, I think, are all motivated more by power, you know, trying to be a great man of history. I mean, we can go into the individual. I know you want to break down the individuals, and I think it's hard to generalize because they're all unique. But to, you know, close out an overly long answer, I think it is largely not about money. It's about being in the room where it happens. That's what Eliezer told me when I interviewed him back in 2023. And like, people just want to be there for creating AGI, creating superintelligence, because that is like where humanity's fate lives in their mind. And that might be true. And they want to be in the room where it happens, influencing how it happens.

Yeah, I think Sam Altman has spoken remarkably candidly about this a couple of times in the context of describing what it was like to be in the room when the first reasoning demos were shown. And he said, I forget his exact wording, but he said it's happened a couple of times now.

Like pushing back the frontier or something, the veil of ignorance.

Yes, the curtain, which is not exactly what that was about, but. But yeah, there's something, and I am sympathetic to that in the sense that I think it is extremely exciting, even intoxicating to be in on the secret. And I do, yeah, I can understand that to a degree. I guess zooming out, and I am interested to hear your takes on individual companies and their cultures and the individual people at the top that are shaping those. How much do you think, you mentioned the term overhang and it got me thinking about— I'm torn or ambivalent on This question, it's probably both is always the answer. But on the one hand, I do feel like AI broadly is inevitable in the sense that we have web-scale data, we have web-scale compute. And in the presence of those things, I feel like a lot of algorithms ultimately can work. We found one main one and there are a bunch of derivations of it that work. And so I feel like we're getting AI absent some sort of civilizational reset that means we don't have web-scale data and web-scale compute. And I'm not excited about that proposition. Uh, we're probably getting some AI, but then the shape of AI, the conditions in which it's introduced, like the, the measures that are taken or not taken to make sure it goes well, all of that stuff seems far more contingent. How do you think about like how much is inevitable and where we might be able to draw lines? And then you can go off in any number of directions in terms of the influence that individual people or groups are having on the direction we're taking.

Yeah. I mean, I think it's inevitable that we will, as a species, continue to use deep learning to make AI models that do things, like new things, right? The, in the book, I separate the obsoleting project from other types of AI. And that's my reframe of the AGI industry because they're trying to render us obsolete. And you mentioned OpenAI's AGI definition and Dario Amadei has a quote that's saying like, AI is not like other technologies. It's a general substitute for human labor. And that feels like pretty different. And we've taken AGI to be synonymous with like what AI is. Because that's the companies that have been trying to build it have been the best at building highly capable and autonomous systems. And they're driving the entire world economy now. But that's not like the only type of AI we could have, right? We could just be like, we have all these diseases and medical problems and like, can we build AlphaFold type systems for specific problems that we have? And like, it's obviously, I get the vision of like build a super intelligent second, you know, Solve intelligence, use that to solve everything else was DeepMind's mission statement for a while. It's like the ultimate techno-solutionist fantasy. And from a purely technical perspective, like, yeah, I can see how you could like use this thing to invent all kinds of wild and transformative new technologies. But I think that you can't like solve, you know, ethics, for ideology and politics in the same way. And yeah, I think it's like this kind of revenge of the STEM people on the humanities or something where it's like, we don't have to like learn history or philosophy or political theory. We can just build the machine that's smarter than everybody and then ask it what to do, which I think is a really—

Yeah, that was at one time OpenAI's business plan, as I'm sure you're well aware.

Ask the AI how to become profitable, right?

Yeah. And he said that, that was an Altman quote. He said that and there was a laugh in the audience and he's like, you laugh, but I'm not really joking.

Yeah. Yeah. It's, it's just like the ultimate, like, question mark, question mark, question mark profit thing, which is like, yeah, just make the superintelligence. But yeah, I think that We can stop the obsoleting project because it's really just a handful of companies in 2 countries. And only one country has really advanced the frontier since ChatGPT came out, at least. And it just costs so much money and it requires the most advanced technology in the world. These like AI chips, which are only made by, you know, like individual companies control different parts of the supply chain, as your listeners probably know. And I think, you know, like forever, can we stop this? Like, I don't know. Hopefully we're around for a very long time. And I, my position is not that we should never build AGI. It's just that it should happen with strong public buy-in and a scientific consensus so it can be done safely. We can get into what that looks like later if you want. But I think that, you know, for your listeners, it's like right now we're kind of feeling like trending towards just banning AI across the board or something, which would be very hard to actually do, and especially with like open weight models and yada yada. But the kind of like backlash is really intense and kind of undiscerning. And I'm kind of hoping to separate the obsoleting project from other types of AI and really stigmatize the obsoleting project because of its risk and undemocratic nature. But then like save the baby from the bathwater with like, you know, these kind of specialty systems, which can be used to do amazing things. And my position is that like we can get way more of the amazing stuff if we have like a more active role for democratic control in deciding like what gets done. So like drug discovery, you know, people are like, oh, AI for drug discovery. Well, monopoly patents mean that people will still try to just discover drugs that will be profitable, which won't be ones that cure people as much as ones that treat some chronic thing or male pattern baldness or these things that are not as socially important. And to actually get the best from AI-assisted drug discovery or AI-assisted FDA trials, like matching people, deep learning systems are very good at that. To get the best of it, you have to reform the systems and change how, you know, drugs are, which, like, how we decide which drugs are made. And I think we should use like a prize system, which, you know, you kind of pick the drugs that you'd want to see in the world and then award money and then just make them at generic costs once they're developed. And so I kind of like, it's a judo move, but I'm kind of like, we aren't going to get the booster world by letting it rip. Like the one that they depict of like amazing transformative cures for everybody. And the best way to get there is to like actually take a strong position against being replaced and then use industrial policy and like Operation Warp Speed type approaches to build the types of technology and scientific discoveries that will actually lead to the most public benefit.

Hey, we'll continue our interview in a moment after a word from our sponsors. I've got a text.

The summer might be almost over, but a new bombshell has entered the villa. Meet the voices of Deepgram Flux TTS. I'm Drew, low-key casual. I'm Alexis, an upbeat morning person. I'm Haley, always cheerful. Flux TTS has some real personalities that are ready to speak. You can interrupt, pause, and keep talking without losing the plot. Fancy a chat? Come try all the voices free through September 12th at deepgram.com/keeptalking.

Terms apply. This episode The Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. OutSystems is helping their customers build, modernize, and operate enterprise systems starting from any coding tool in a governed agentic engineering model. It's time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide trust OutSystems for their mission-critical apps. And we're not talking about theory. We're talking about massive operational wins. Take, for example, a leading global retailer that was struggling to keep up with their rapid expansion. They used OutSystems to build AI agents that automated their invoicing and translation processes, shifting 95% of their daily store operations to a new digital app, saving thousands of hours of manual work. Or look at a major global logistics giant that was drowning in technical debt. Their engineers were spending their weeks manually firefighting by reviewing platform logs. Using OutSystems, they built an AI log review agent in just 5 days, and that one agent now saves them over 1,600 engineering hours every single year, letting their team focus on real innovation instead of reactive maintenance. Even in modern digital banking, companies are using this platform to standardize and automate complex financial workflows, delivering secure, personalized experiences at a pace their competitors can't match. OutSystems gives you the flexibility to deploy how you want and the governance to do it safely. Stop chasing the hype and start owning your agentic future. Learn more at outsystems.com/tcr.

That's outsystems.com/tcr.

Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable-5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept, and Fable writes some amazing verses. I give it feedback on its misses, and I push it to aim for higher inspiration, Add layers of meaning, optimize syllable density, and above all, write a hit song. These days I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai/tcr.

I have a lot of different directions I wanna go, but I guess maybe part of a question that I wrestle with. It's a little bit hard, of course, to define some of these terms and what are the boundaries. But like, I get so much value out of using AI on a daily basis. It saves me an unbelievable amount of tedious work and it allows me to do a ton more than I otherwise could. And it, in a way, has replaced, at least like counterfactually, people I would've had to hire in theory, whether I would have or not, I'm not so sure. But like, there's definitely a lot of work happening in my life by AI that on some counterfactual level has substituted for human labor. And I'm like, I think that's good so far, at least. Like, I, I, do you, how do you think about like where this goes from bad or goes from good to bad? Because I think there's a strong argument and I do want to give it its kind of strongest articulation that like, you go back not that long in history and everybody was tilling fields, right? And it was like pretty shitty. And now we have certainly some problems in our modern agricultural system, but like one problem we don't have is scarcity of food. And there's like plenty of, with 2% or whatever the population, we can feed ourselves and everybody else is like able to do other things. And Almost everybody agrees that's almost overwhelmingly good, let's say. Even again, though, there are problems that remain. Can we not have a version of that with AI where we're all elevated to our own little executives of our mini AI corporations or something? Are you at all sympathetic to the— Or maybe we'll work a lot less. We had the Keynes thing too from 100 years ago that was like, we're supposed to only be working 2 days a week at this point. Well, we're not, but maybe in the future we could be.

Yeah. Yeah. No, the faster AI goes, the more I work, it turns out. But I, yeah, I mean, automating labor is what's allowed humanity to go from everyone more or less being very poor to some people being rich at least, and like living standards and all kinds of stuff going up dramatically after the Industrial Revolution. And so it's like, my position is that we, not that we never automate any labor because like, I don't want to just have us stay at this kind of like level of development, but trying to automate all labor is pretty different. And where the line is, is like, it's not super obvious. I think that, you know, my position is we should freeze frontier development right now and not resume without the buy-in and safety I mentioned. But it's like 3 years ago, FLI organized this pause letter, you know, after GPT-4 came out, I guess 3 and a half years now. There was like a 6-month pause on development and that was framed around a bunch of things, but risk was a big part of that. And like, obviously wasn't super risky to build the next iteration of LLMs, at least from a like existential perspective. Like there were harms, you know, there was like chatbot psychosis and suicides that, OpenAI's decisions contributed to, which I document in the book. And I think that should, you know, we shouldn't lose sight of that, but that's not really what that letter was about. I think now we're kind of in this position where it's like, we just froze what we had today. There'd be still a lot of disruption from just adopting GPT-6 and Fable 5.1, you know, in the economy. And the job effects are not super easy to see. And it's like a lot of it's not people getting fired, but people just not getting hired in the first place, kind of as you described. And I'm not saying that we should go back a few generations. That might actually be the right thing to do from a social welfare perspective. I don't know. But it's just much harder than stopping advancing further. And so I think we should focus on that first and then kind of reevaluate with what we have. And it'll take a lot to regulate. the AI that we have today and a lot of thinking to get that right. But yeah, I mean, on the CEO of our own corporation thing, it's just like some people will do well in that system and enjoy it more than the status quo. But I think it's just like a lot of people will be left behind and not everyone wants to like The experience I've read and heard of like managing these fleets of agents, it's like, it can be like really bad. You can feel like the opportunity cost of not kicking off another run and people like working more and more. And, you know, they're competing with other people who are adopting it really quickly. And so I'm kind of looking around at like this world where people are just like kind of burning themselves out running these like agent fleets and getting more done, but not necessarily like being happier with it. And I don't know, I'm just like, it doesn't seem like the future, the good future that you're describing, and this is bracketing all of the risks and other social harms of like widespread adoption of this tech. And then the power concentration and wealth concentration. It's like some of, some people will be way better at running the AI companies and the AIs will probably they're that good, be better at just running without any humans involved at all. And then like, who is owning those companies and who is accountable for what they're doing? And like, and then you have like these multi-agent dynamics where, you know, the Hugging Face hack now, and there's like 1,200 agents who are involved with that. But the world you're describing, it's like there are multiple agents running for every person interacting with multiple agents for other people and companies all the time.

Yeah.

in ways that we can't monitor effectively and producing like who knows what kinds of interactions and effects. And that just feels like a much less legible and stable world. And we're already seeing like OpenAI is like, I tweeted that it was a loss of control event masquerading as a company in response to the fact that some like 3 random people used Claude and Codex to get access to ChatGPT employee accounts and they could have gotten access to like I think their main codebase. And it's just like, this is, and then multiple agent swarms that have broken out of the company with or without their knowledge or some amount of covering up, some amount of cluelessness. And, you know, OpenAI is probably one of the companies that's adopted this technology the most. And it means that they just don't know what's going on nearly as well as they would have a few years ago. And so I think that the world of like the these agents being widely adopted is one of like, yeah, confusion and chaos and lots of unpredictable, but many, many negative consequences that we're already seeing like hints of right now.

What do you think the new, if you envision a social contract that you, you know, what's your most positive vision of the future? If you're not going to roll AI back from where it is now, And I think you've got some interesting proposals around, you alluded to like freezing the frontier, but again, that still allows for a lot of diffusion. I totally agree. This will probably, like seemingly all recent technologies, probably amplify inequality. Hopefully it brings up the bottom, but probably the ratios also continue to climb. I've lived through a medical emergency, thankfully in the AI era, and I did get an unbelievable amount of value from using 3 AIs in Triplicate and feeding in my test results. And it was my son's test results, but yeah, just unbelievable amount of value from that sort of thing. So I do feel like I want, in any good version of the future for me, there's like AI doctors for all, which notably OpenAI has done a pretty good job of making their health product free and unlimited to people, which is like pretty cool. I think, uh, as OpenAI moves go, that, that ranks near the top of my list.

Yeah.

What's your positive vision for where we want to be in 3 to 5 years if we freeze the frontier and allow other things to continue? Like, what does good look like to you?

Yeah, I guess like in the U.S., I would like to see the left winning elections and building kind of a 3rd New Deal with the Great Society being the 2nd. So, you know, Medicare for All, maybe a jobs guarantee administered locally. And I'm really bullish on investing a lot of money and more importantly, kind of like institutional and state capacity in developing technologies and medical treatments and things that we like actually really want as a society. So one idea I'm like playing around with is Cures for All. where the government kind of treats diseases like they're all COVID and it's all Operation Warp Speed. You'd obviously have to prioritize based off of, you know, tractability and, you know, the disease burden and, you know, other factors. But then when you're there, like using the whole government approach, using advanced market commitments, you know, vaccine trials and— sorry, I'm spacing on the word for challenge trials, like where people deliberately expose themselves to it.

Challenge trials.

Yeah, human challenge trials. And developing the stuff that the companies often promise, but just going for that directly. And then using AI where it's helpful. And then in a lot of cases, it's really about the institutions. And then this would be enormously beneficial to the domestic US, but then you could also make the cures freely available around the world or provide them at cost or whatever. And make the world healthier and also like restore our standing in it after some well-deserved drops. And I think that like the US is so wealthy and that wealth is just concentrated so intensely in the top. That's been happening for decades and AI and tech is just accelerating it further. And so I think just really large redistribution is is justified. And like, on just democracy and power grounds, like right before we recorded this, we got the news that Larry Ellison's bid to buy Warner Brothers, which would give him and his son control of CNN, HBO, Warner Brothers, in addition to CBS. And yeah, and also TikTok. And then Elon owns Twitter. Like, we're in a really dire situation where these oligarchs are buying the most important media properties in the world and then using them to push their agendas. And this is like, I work in media. I think media is really important. And yeah, it's not just a matter of like, oh, they have more money than they need and like other people have not enough. But it's like actually a threat to our democracy to have people being that rich. And so, and yeah, and I would like to just see as much decoupling between wage labor and living a good life and building like a proper welfare state. And then also a restoration of USAID and, you know, rejuvenation of it because I think that, you know, global poverty is incredibly important. And the cuts that Elon led, I also document in the book and just like the deaths that those caused, just like one of the greatest crimes, maybe the greatest crime of the 21st century. And yeah, just kind of like a Restoration of building a pro-human society that is truly egalitarian and one that wants good things for people. That sounds so cheesy, but right now we have an administration that is just staffed with seemingly the worst people on the planet and seems to either not care about what happens to other people or want bad things for them. And is just enriching itself at the expense of literally everyone else. So the opposite of that would be, would be really nice.

I'm with you that the destruction of USAID is incredibly shameful. I've been an Elon defender in many conversations over time, and that's one thing that he's done that I've never defended. On the question of, there seems like a little bit of tension between A potential jobs guarantee and decoupling one's ability to contribute to the economy from one's right, as we might imagine it, to a decent material life. Do you feel like, are you a believer in the intrinsic value of work? People need work for purpose or for something to do. I'm a little bit more of the mind that, I don't know, I'll bet on the working class to spend the peace dividend is what I told—

Oh gosh.

Jake Sullivan, whose name I should definitely know at the tip of my tongue, but at the end of an hour with him where we were talking about China, he was like, we didn't even get into jobs and what people are going to do. And I was like, I bet on the working class to spend the profits. You worry about China. What do you think though? Do you think that we need jobs indefinitely?

I think, I mean, one, I will say like, I need to give all of this more thought. I've been so neck deep in AI world that like figuring out, I kind of want to make sure we're not all replaced by machines before we get into like planning the, you know, third new deal. I think that you can have a jobs guarantee where, you know, it exists for people who want it, but then also make sure that people are getting healthcare and like the education, housing, like just the necessities covered because, you know, we have an incredibly rich— society, and those resources are not being put to good use. Like, rich people, they just want to be richer than each other, and they maybe want to use power in the world, but most of them don't seem to even care about that last part. They just kind of want to, like, be richer than their friends or something. And so just, you could just tax them really aggressively and they won't like it, but they'll still get to, like, have more money than the next person if you do it the right way. And so I don't think we need to choose. I think that Work does seem to be, it's just a thing that's very important to a lot of people. And I think we should try to not have so much of our meaning tied up in it. But it's also just like, you can look at polling. It's one of the few issues that polls at like 80% or something. And UBIs poll very badly. And so, I don't know, like I believe in democracy at a deep level. And so I think part of building, you know, the Third New Deal is going to be running on popular things and figuring out how to make them work. And, and yeah, just like jobs exist for people who want them. And there's also a lot to do. There's a lot of like climate, you know, like green transition stuff you could have people do. The New Deal, like I think it was James Baldwin was a writer for the, one of the programs in the, in the New Deal. And they documented, they like recorded a lot of important things that were relevant for like preserving culture and history. And like lots of really cool stuff came out of that. And, you know, it's just pay artists to make art. Like there's so many people who want to create things and our society has decided to devalue that as it dumps unprecedented amounts of money in building machines that can replace all of us. That just seems like backwards. And so I don't know, like this just seems like we're doing almost everything inverted from how I think we should be doing it.

I'm interested in your response to an argument that you sometimes hear around the relationship between invention and democracy. People sometimes just make the very short observation that, like, in general, people are allowed to invent stuff. And it's not like we don't put every new invention to a vote. And if we did, we might not get very many inventions. We might freeze A lot more than we would like to freeze in time. Friend of the show, Dean Ball, also on Ezra Klein, talked about how he wonders if, I think it was on Ezra Klein, if today we would have the stomach for the introduction of the car, which was like disruptive in its own way. And it was sort of cars and horses on the roads together and like they weren't safe at the beginning. And they're still not entirely safe, but they were way more dangerous then. Are you sympathetic at all to that? Or how would you answer the idea? And then I guess there's another question around democracy, around just how well is it functioning in general? The, for better or worse, I do think the president was in fact legitimately elected. So I do wonder like, how much can we really, and I'm a big direct democracy guy, by the way, in general, but I don't know that we can just fall entirely back to as kind of the end-all be-all decision maker here? Because it seems like we've got plenty of examples of bad decisions being made by publics and also might not be willing to embrace enough change to really see the future go where at least I would hope that it could go over time.

Yeah.

Well, first I'll say I live in the only city in America where most people don't have a car, and it also happens to be the best city.

In America.

So I don't know, cars, like, I don't know if they were good on net, like probably, I don't know though, but that's not important for my particular point, which is like, I think permissionless innovation makes sense for most things, almost all innovation. I think that should be the default, but building a universal labor replacing machine would have a profound and irreversible effect on everybody in the world. And so I think it's reasonable that everybody has some say in whether, when, and how that happens. And, you know, broadly, like my position is AI, at least what they're trying to build, the obsoleting project, like that is unique. It is different from other technologies. And so it should be treated differently. And on your point about democracy and, you know, Trump being this counterexample, Trump is a product of a broken democratic system in the United States where You know, you have the Electoral College being the most obvious example where he won in the first place, but lost the popular vote. But more importantly, you have counter-majoritarian institutions like the U.S. Senate with the, you know, crazy way representation is apportioned and gerrymandering to a lesser extent. And then the Supreme Court being these lifetime appointments and the Republicans stole a Supreme Court seat. People should remember that. And so, yeah, I think we have, and then the first past the post two-party system creates the political system that the majority of people don't like. I think it's like 2/3 of Americans are not happy with the current level of representation. And so democracy isn't just like, is there an election that decides, with also some other weird stuff tacked onto it, who leads a country? It's like, do people have meaningful say in the power that affects them? Like, do the governed govern? And the US is just not great on this. And a lot of other countries are better on it. And, you know, we're the longest running democracy in some sense, although really wasn't a democracy until the 1960s when everybody living in the country got the right to vote.

And—

Like meaningfully. And so, yeah, I think like proportional representation is a thing I talk about in the book as like a better way to elect Congress where, you know, people would have like larger districts with multiple members and then the top 5 vote-getters would be in office and then you'd have probably like 5 parties instead of 2. And then people would have parties that represent their interests better. And that would be like a huge change, but so, so like better at representing people. And then Electoral College is one state away from being gone. If Pennsylvania, I think, signs the Interstate compact to just use the popular vote. And so my pitch for democracy is like one that is more like well-realized. And this also extends to like democracy in the workplace. So I think it should be a lot easier to form a union and a worker cooperative. And I think, you know, in this country, we accept that, you know, it's like the government should never, you know, impede on us and censor us, but our employers where we spend like 8 hours a day, 5 days a week can surveil us and control what we say and do all kinds of things that we would find incredibly authoritarian in another context.

Record all of our keystrokes and mouse clicks and use it to train AIs, for example.

Yeah. Yeah. And so, yeah, I think like democracy should extend to more parts of our society and our life. And I think that this will produce on net better decision-making, but then it will also just create a more empowered citizenry where like it provides something intrinsically to have say over your life. And I think people right now, especially in the US, just feel incredibly disempowered, incredibly unheard. And I think that's reasonable, right? Right now we have a government that's uniquely insulated from public opinion. through these kind of majoritarian institutions, through the fact that it's this lame duck president and just one who like does not seem to care very much about things beyond the ballroom and corruption and yeah, like the stock market. But yeah, so I don't know, I— democracy is great and America should be one.

Change gears a little bit on the notion of people being empowered.

I think we'll— I'm sure we'll circle back to some of those bigger themes before we end as well. But I think you have an interesting argument in the book for how the AI researchers should understand their position today. I think you had a column from just a couple months ago, right? That was like AI researcher power is reaching its peak. And I actually don't know too much, but you talk a little bit about the vote at DeepMind or the sort of organization effort at DeepMind in the UK to bring about a union to the DeepMind staff there. Tell me about how you see, I'd be interested to hear that story because I really don't know a lot of the details of it. But then beyond that story, give me how you see the lay of the land in terms of the power that the employees at the frontier companies have, whether or not they realize it, how you think they should use it.

Yeah, I mean, so the DeepMind thing, Google has been Signing these deals with the military, U.S. military, Israeli military, and this is creating a lot of pushback from workers. In the UK, the DeepMind division had a union vote. It's a—I don't know British labor law—it's pretty different from the U.S., but it seems like something like 300 of 1,000 people in the bargaining unit were supportive, and that—and Google is not recognizing it, and. They're not demanding better pay or working conditions, but instead policy changes on who Google is serving and how. And I think that's pretty interesting because, you know, most unions are like, it's about getting better pay and working conditions for the members, which is, you know, a reasonable thing to want. In these situations, like, people are paid very well. I mean, they work a lot, but they often want to. They have like good, you know, good perks or whatever. And, and to the broader point, like the industry is racing to replace all of us, but they're starting with their own workers, specifically the engineers and the researchers who are training the models. And the hope is to achieve recursive self-improvement where the AI can fully train the next generation and make it more capable. And then you can have this really fast loop. And I think that means that we have a ticking clock here. I mean, the workers have a ticking clock where their power will, you know, it's near its peak because they still command incredible salaries and they're still needed. But if they're not needed anymore, then they won't have any leverage. And then that will happen to the rest of us, which would be very bad. And so I think people in the companies are starting to see this, but probably not as much as they Could. And one thing we've been seeing is like, if you oppose what's happening at these companies and you work there, you should just quit and then go public. And Jacob Coxon did this with like incredible fanfare and it really moved the conversation like nothing ever has. And, you know, we see a few other examples since then. And, you know, Coxon deserves credit for this. And I think it's like, you know, as a whistleblower, I think it's sometimes the right thing to do. But it might be better to stay and organize your coworkers who also care about safety and try to form a union. And in so doing, you can bargain for, in the US, you can bargain for safety. And this is something that airline pilots, I believe, did. And this is like how the FAA was formed, I believe. And you will have so much more power by being able to Withhold your labor collectively and you'll have protections in the union. And if you really want to slow things down, like a big, dramatic, messy union fight is a pretty great way of doing that. And it would be like, if it was truly about just safety, then it's also this clarifying moment where the CEOs will say, like, I care deeply about safety. I want to slow down. I want to pace the frontier. And then the workers could say, great, we want to form a union. Will you voluntarily recognize us? All we want is to make the AI safer. Also, by the way, if we're in our own unions, we can work with the other unions to collectively pace the frontier and it doesn't violate antitrust law. There's a precedent, I think, of the Teamsters doing this across different shops. And so my guess is that these CEOs would not voluntarily recognize the workers. But if you're optimistic about your CEOs, you can just give them this opportunity to be like, great, here's this creative way we figured out to actually slow down. And I think that this is very promising. And like a lot of people at these companies just don't have experience with that type of thing. But it's something I want them to learn more about. And yeah, like you can have have a lot of leverage over what happens through this, through this approach.

I don't know to what degree you have chapter and verse on this, but the biggest relevant offices are in California under kind of California rules. Do you know how much latitude or protection people at tech companies would have to organize? I would assume that they're protected from being fired for attempting to Set up a union.

Yeah.

Are outside union representatives afforded some rights or privileges to be able to come in and try to organize people? I haven't really thought too much about this, but it does, with all of the talk that we hear lately about antitrust and why we can't do these things because of antitrust, I've been saying, okay, the president should just say, we're not going to come down on you for antitrust violations for coordinating on safety measures. In absence of that, this is another pretty creative solution that I think would be very hard for anybody to argue with, although I'm sure we'd get the usual bunch of bad faith attacks, but to the degree you can, and I realize I'm giving you this in an unscripted way, but I'd love to hear a double-click of what you think that could look like and what people on the inside should know. Such that if they are at all tempted about this, they could feel confident in taking, I don't know, what sort of next steps?

Yeah, it's been over a decade since I took labor law, but there are protections against retaliation for organizing and then protections like once you're in the union as well. The protections are kind of weak, you know, like the back pay is often like it can be litigated for a long time. But there's really with these companies, like it's just a very bad look to fire people for Trying to organize a union around safety. And so I think that is in itself a lot of protection. And at OpenAI in particular, we see a lot of culture of people speaking out. So I think that, yeah, there's protection in that kind of reputation management. And then in California in particular, workers can bargain at the sectoral level. And so fast food workers can bargain across different shops, I believe. And so you have extra benefits. there. And if people are interested, you know, I'm not an expert on this, but if they're interested, they can get in touch with me. My Signal is garrison.06. And yeah, like, I know people who would know more.

In terms of what they would be organizing for, I think the book has quite a few different interesting arguments that kind of take The assumptions or the hopes that people on the inside often have and at least give them a good shake. One is just like that the companies are serious about safety in the first place. I think it'd be helpful maybe to just review for a second the history of safety at OpenAI, which you've reported on at some length. We've been through like waves of different regimes and leadership, and there's been a lot of turnover.

Yeah.

And then I think you also have a pretty interesting take on alignment basically as a mirage, putting yourself maybe in the role of like advisor to the hypothetical union leaders. Why would it not be enough for them to say, okay, we got 20% of compute committed to safety? And why wouldn't it be enough for them to say, oh, we're gonna solve alignment first and then we'll go do the thing?

Yeah. Well, the superalignment team, as you're alluding to, was promised 20% of compute for that team and then got very little of it and nowhere close to 20% in practice. And I think to varying degrees across the companies, we've just seen a lot of promises made about safety and commitments that then get changed or broken once they start conflicting with commercializing or, you know, racing ahead. And it's because like there isn't a counterforce. You have management able to unilaterally make these decisions and employees can push back and they can go public and they can, you know, whatever, like, you know, try and resist this in some way. But if they're not organized, you're just not going to have the means of actually changing the policy. And, you know, we've seen employees getting organized enough to change outcomes with Sam Altman's firing and then reinstatement, the employees coming together to sign that letter, like 90-plus percent of them signed, was like a really big part of that working. And obviously the dynamics there were pretty different. They stood to lose a lot of money on the sale of their shares. But yeah, I think workers can like decide like we really want these policies and we want these safety practices and we want them to be like binding in some way. And you have leverage by being able to withhold your labor, go on strike, do like work slowdowns, et cetera. And I think that we just can't take these companies at their word. And the CEOs are saying that they can't unilaterally slow down because they're racing each other. And like, you know, they could, and it would actually be a pretty strong signal that you take these risks seriously if like any of these companies unilaterally slowed down and it would put a lot of pressure on the others to do the same. And it would also help with the US-China thing. If like, you know, one of the parties unilaterally disarmed, that like really does actually signal like you care about this. And the US is in the lead, so it would signal it much, much more. And this is my understanding, like one of the biggest, if not the biggest blocker to a deal with China is they don't think the US is taking it seriously because we're barely regulated over here. And We have been the ones who started the race and we'll win it, as Trump has said. So yeah, I think it's just about power. It's like you just need hard power to actually get concessions because otherwise you'll get promises that will be broken as soon as they start to cost too much. And you can, yeah, just make it about only safety. And in terms of ideas, You know, third-party auditors is being discussed and like making that not voluntary. You could also have like your demands be about the practices of like the lobbyists at the company. OpenAI, we've seen a lot of turmoil about the Leading the Future Super PAC, which is funded by Greg Brockman, was set up with the guidance of Crystal Hain, the chief lobbyist at the company, and then doing like incredibly dirty tricks, like false flag Twitter accounts and like calling for violence against the AI employees, like, you know, funded by this super PAC and related entities and also going after politicians for daring to regulate the technology at all. And so I think this is like, you know, creating really bad dynamics for politicians doing the right thing on this. And the employees have a lot of leverage and have been able to get some concessions from Brockman about this. But yeah, I think that you can just see it as like a way to generally increase your ability to influence the policy decisions at these companies. And OpenAI has had a lot of safety leadership turnover and people kind of being disempowered and rotated around. And then we see like just these shocking breakouts and hacks and hijackings of various websites and companies. By these rogue agent swarms. And it's like, is it an accident that the company with this really shoddy track record of taking safety seriously is having all of this happen? And it's now putting the entire industry's future at risk, which is good from my perspective, but they're kind of messing it up for the rest of— I mean, every company's had their agents hack into somebody they weren't supposed to by now. But OpenAI really, yeah, like has been the greatest possible, you know, case for why more regulation is needed here.

Yeah, I do think your case to the insiders amounts in a way to let's avoid the nuclear outcome, or at least that's my term for it. The nuclear outcome being like, we get the weapons, we don't get the civilian benefits. I do think there's an increasingly compelling case that's like, guys, people out there really don't like you. You're going to have to clean up your act if you want to have a chance of bringing the positive side of this forward. And that window might be fairly short because we're getting— the Overton window is blown wide open and we're getting like All kinds of proposals from all kinds of people. And like, who knows what the political current is going to kick up for us over the next couple of years. So take matters into your own hands right now and make sure that you are on the right side of key questions. And then potentially only by doing that will you have the chance to really realize the upside vision that got so many of you into this. In the first place. I think that's a pretty, who knows, but there's a decent chance that is accurate as a, as an assessment of things and is probably as compelling as a case as can be made, I think, to a lot of people who are not total doomers inside, but are, but can see that the world is starting to sour on this whole— Give me your, your, yeah, please.

Or can I just react to that quickly? Yeah. Yeah. Yeah. Yeah, just like, it's sort of, if I was talking to the Trump administration, they would not listen to me, but similar to what I would say to these employees where it's like, if you're just moving this fast, there's going to be a worse Hugging Face with a body count. And then there'll be very strong pressure to just shut it all down. And so if you want to actually continue and get all the upside that, as you were alluding to, like, you're going to need to slow down because otherwise your hand will be forced.

So now give me your case against alignment or your case that it is a mirage.

Yeah. So I have a chapter of the book Obsolete called The Problems with the Alignment Problem, where I explain, you know, the focus of AI safety historically has been on a solution to alignment where like you can get an arbitrarily capable AI system to do what you want. This is like technical alignment. Sometimes there's discussion of like wanting the right thing, normative alignment. And I think this really understates the problem because you also have economic alignment and geopolitical alignment. And so I introduce the alignment polycrisis to kind of have all these layers and they interact in complex ways. And if you solve technical alignment, that makes you have a more useful product. And so the race can run faster and for a bigger prize. And then similarly, it's more useful as a weapon, as a means of projecting power around the world. And so it could make the geopolitical race worse. And sometimes, and like we've seen this with, you know, the biggest alignment intervention historic to date is probably reinforcement learning from human feedback. Which was developed by Paul Christiano and others at OpenAI for safety reasons. It also happened to make LLMs actually conversational and useful, which enabled ChatGPT and everything that came after. And I think this is just going to keep happening because of the dual-use nature of this technology. And so, yeah, like you have to look at the whole picture to understand what works. And this makes me much more bearish on technical alignment, where like, if you solve it, you still have all these other problems. And this makes like a solution to alignment neither necessary nor sufficient to solve the problems presented by the obsoleting project. And then, you know, the answer in my mind is just stop, just don't build it. And some of the policy interventions or technical interventions that would be helpful is like, Verification of international agreements historically been very neglected, but we'd be in a much better situation if all the money that went to technical alignment research went instead to having the means of verifying an international treaty on AI, which is going to be one of the bigger blockers to actually having a binding agreement here. And so, and governance just also looks a lot better as an intervention, just like having good regulations and policies and ideas in place. And then also the means of like actually bringing them into the world. That could backfire if they're the wrong policies, but it's just like, you could at least solve this problem through governance, whereas you cannot solve it through alignment.

I do think it's a, it's an underappreciated and generally undertheorized domain. Let's suppose, and the gradual disempowerment folks have pushed on this, but it's, I think, still underappreciated that like, even if you posit an AI that will do what you say and only what you say and only what you really mean, not, and not go like genie problem on you and do the paperclip maximizer thing, but really gets it and actually does what you as an individual user or controller of the AI would really want. It is still tough to envision like what the future equilibrium looks like. I'm a little more inclined to like at least take some chances there because I do think also that would've been true about cars, for example. And I think it's like interesting thought experiment to say maybe cars weren't good, but I think most people feel like they're good. And we've got like problems obviously associated with it, but we also have like plenty of nice things that people really appreciate that they couldn't have imagined, I don't think, in advance of the cars being created. And if you just put it to people at the time, it was like, Tell me what the future of cars is going to look like. And if you can't, then I can't sign on to it. I do feel like that, that is a high burden and an unusual burden to put on world-changing technology. Again, it's fair to say this technology is different. And that's a big reason that I spend all my time thinking about it. I don't know. I'm again ambivalent on this, but it is something that I think people in the AI space should spend more time, at least trying to do, to figure out, to find for themselves how difficult it is. To articulate like, what is it going to look like and how is it going to work and how is it going to be good for everybody? And how do we have some sort of balance that you can expect to be stable over time? Those are really hard questions that I guess are often just gated in people's minds because they're just focused on the technical alignment question first. And, and again, we don't have great answers there, so they're not necessarily wrong to be focused on that. But yeah, thinking past it, it is still quite tough. I did one fairly long conversation with David Duvenaud, who is one of the co-authors of the gradual disempowerment paper. And I think it is, it's a pretty tough thought experiment and pretty compelling arguments by him that certainly we can't just take for granted that if we solve a couple upstream problems and everything downstream of that will be fine. I think that's definitely not at all guaranteed.

Yeah.

Who else do you think holds power today? One I'll propose to you and then you can run down whoever else comes to mind for you. One that the sort of capitalist class would like to point to is corporations. And the idea there would be, look, misaligned AI doesn't sell. Companies don't want rogue agent swarms happening. The market itself will discipline the AI companies on this front. And yeah, that doesn't necessarily deal with all existential risk. If something really were to go foom or go crazy, but maybe there's nothing we can do about that anyway, because that's just so crazy. In sort of the bulk of scenarios, capitalism will discipline the companies, we'll get pretty well-behaved AIs because that's what people will be willing to pay for. I actually think there's something to that. If companies were to be a little bit more forceful in their demands or expectations, I think there's one thing to say the invisible hand. There's another thing to say companies should maybe get opinionated about, we want to see some standards. We want to see some proof around what you're doing before we'll buy your AIs in the same way that they do that for other products, right?

Yeah.

Grocery stores, for example, want to know how the animals are treated in part because their customers care. They want to have some actual proof that what they're being sold is produced in a certain way that they feel good about. I feel like there's something there at the corporate level, but what do you think about that? And who else do you think has leverage that is maybe underappreciated today?

Well, I'll first say that the grocery stores are not getting it right on the animal welfare standards. My friend had a career suing those companies for false advertising because just there's no agreed-upon stuff and there's all kinds of false claims. But Yeah, I mean, risks from AI are classic externality, right? It's not priced into the market. And, you know, if you think about like, if these companies caused a disaster that killed 10 million people, they would go bankrupt well before they paid out, you know, what they owed to society just from like normal kind of litigation. And this is like pretty widely agreed upon. And insurers just refuse to sell policies to these companies because The risks were too great and, and too correlated as well. And so we're all just living, you know, with like, we're subsidizing these companies by not pricing in that risk through regulation. And even like Gabe Weil, who's the main person I associate with using liability to regulate AI companies, strict liability, says like, it doesn't work for existential risk because, you know, we're extinct. There's nobody to pay. It is true that like, you know, you want the AI to do what It's told not to autonomously start hacking into stuff. So there's like a market incentive there, but it's like, is there an incentive to solve it all the way or like just enough to make a marketable product? You know, it's like the companies are not able to really align the models that well. Like Ryan Greenblatt, one of the investigators of the Meter Hugging Face investigation, had to post a bit before that about like, you know, today's AIs seem pretty misaligned and gives all these examples about how they're often like kind of lazy or they'll like hallucinate or just, you know, like literally just like make stuff up because the user wants that to happen. And it'd be a better product without that, but like they're selling pretty well as is. And so we're already getting misaligned AIs that sometimes do really bad things. And The impunity of the companies, like the fact that there's nobody who committed a crime despite the AIs doing things that if a human did would be crimes. Hugging Face reported the hack to the FBI. So like you have these like felonies where there's nobody to blame, legally speaking. And elite impunity is, I think, one of the biggest drivers of all of this. You know, it's kind of trite, but like if Sam Altman were criminally liable for the stuff the AIs autonomously did, I think they would behave very differently as a company. And I don't think it's crazy to ask that. And so, yeah, if you manage to internalize all these externalities by creating the right kind of criminal and civil liabilities and using other regulatory tools, then like maybe you could have this, but then you still have this problem of like they're trying to build universal labor replacing machines. Without our consent, without our support. And so, yeah, it only solves one part of the problem and we're not even solving that part.

Yeah. Again, it is striking that like insurance is not available and we're now counting felonies and nothing seems to be really happening as a result of that. I do understand that there are some government inquiries, some letters have been sent and stuff like that, but—

Yeah, companies don't answer the questions. And then all Congress can do is be like, yell at them. But, you know, they can't, like, I mean, you could subpoena if, if, you know, you have the majority. And I, but I think there is just like a lot less you can do because the law is limited in this way. I think it would probably be valuable though for somebody to try and bring criminal charges and just kind of get caught trying. And it would be very instructive for people who want to pass new laws. To show where the gaps are.

Yeah, there's definitely some political entrepreneur out there who has some upside in that, I'm sure. It is kind of, I mean, we just saw a, it goes the other direction, but we just saw a lawsuit pop up out of nowhere in a not very long period of time after Pacing the Frontier kind of became in vogue, basically accusing the companies of anti-competitive practices in light of their statements about doing that. It is surprising that there hasn't been a similar move by somebody at the accountability for these hacks level. And I agree, like, I don't know whose jurisdiction it would be or whatever, but it does seem like the sort of thing that somebody ought to be doing. Yes.

As nobody has been.

And I don't necessarily mean to suggest either that I think, certainly not without a change First, a big question going around and around on lately is like, what should we do about the fact that there've been just a ton of kind of petty crimes committed that if we really go back and investigate everything with the fullness of our new AI power, we're going to find that a large percentage of people have cheated a little bit here, done a little bit of this there. There was just this study that came out of Singapore that showed that civil servants had been buying properties close to the As yet unannounced subway stations and getting the benefits of that they weren't supposed to get. But it's something like 5 or 10% of the civil service they, you know, feel have done that. What are you going to do? You can't put 10% of the civil service in jail. So I think you need a, a before and after. And I would extend that courtesy to the AI executives. I'd say, okay, you get Hugging Face for free, but like, there probably should be some new accountability standards. in the future, especially if you can't even go get an insurance policy against this, um, it does start to— like, evidence kind of piles up that, like, what you're doing is, like, fundamentally misguided on some level.

Yeah.

How do you think politics on this is shaping up, and what sort of developments do you expect? I've been struck so far that we're, like, not polarized along partisan lines as we seem to be on almost every other issue, and I think that's good. So I'm trying to do my small part, as Zvi suggests, to not polarize it before it might happen on its own. Um, what do you see? What do you expect? And how do you think that feeds into what people should do today?

Yeah, I mean, it's been a remarkable feature of AI for a long time that it's one of the only issues in the United States that has remained not polarized with like very similar levels of support for regulation or concern about the technology across parties. There was like some evidence very recently that there's like a bit of a polarization happening with like Republican concern dropping after Trump has come out very hard against AI risk and calling it a hoax. But it's not a huge drop. And I think, you know, J.D. Vance talked about like if the companies are building Frankenstein, they should stop. And Kratios, you know, the OSTP, the Office of Science and Technology Policy, kind of the head tech advisor for the White House, said. something similar about the companies being able to stop. And so, like, what Trump says is untethered from, you know, political expectation or wisdom in many cases. When he called people who don't like data centers, like communities that don't want them, they deserve to be like, what, stupid and poor or something? Or it was really shocking. And I don't think that's going to polarize data centers. Like, I think that people have largely made up their mind about them. And for these issues that become, as they become salient, it's hard to polarize because people actually just have their own deeply felt convictions about it. And some people are taking their cues from the president, but I think a lot of people are actually concerned, whether it's the Hugging Face hacks or the concern about the jobs or the environment or power and wealth concentration or surveillance. The kind of resistance to flock cameras is like a pretty interesting thing that I kind of see in this broader backlash to this inevitable march of, in my mind, dystopian tech future. And so, I don't know, I think polarization would be bad. I don't think it's like, has that much to do with what people who care about this issue say. It's like, you know, people are going to say what they're going to say. And you can maybe try and get people who who are on the right to like voice their concerns as well. But you're not going to get people on the left to like stop saying stuff like, you know, like Elizabeth Warren supported a pause and it's like, yeah, you know, it's like, I think it's substantively the right policy. And, you know, it's, it's not as simple as like if a bunch of people from one side, you know, start embracing an issue that it will necessarily become polarized. If that issue is itself popular, it could just make people think better of them. Like when Bernie Sanders came out against Flock, a ton of people were quote tweeting him being like, I hate Bernie, but this is based, you know, that kind of sentiment. And so, yeah, AI has now moved into that kind of like much higher salience mode. And I think that will just continue. And yeah, I think it's, you know, like at the end of the book, I talk about how there's this potential for a really epic level of social mobilization around resisting our replacement by machines and then hopefully building a better future in light of what is possible with the current levels of AI and like what is possible with the directions we could take the technology. And I think that's happening ahead of schedule, but the risks are also coming to us ahead of schedule. And so there's this like kind of societal immune response that we're seeing. And we're not going to just go, you know, gently into the night. And I hope it's just enough to get us where we need to be in time. And I'd feel a lot better if basically anybody else was in the White House because, yeah, it just— Trump is uniquely insulated from what the public wants. But I think you could see what he's saying is like, February 2020 kind of vibes, like with COVID being a thing and then some denial. And then like eventually he did, you know, start taking it more seriously. And obviously there's been a lot of movement in many directions from him on that issue. But if the risks and the harms from this technology become just so abundantly clear, which I think is going to be the case, then it's going to be very hard for him to maintain. That position.

What do you think is the right, you alluded to it earlier around freezing the frontier. Tell me how you think about freezing the frontier. What is, how do you operationalize that? There's like new training runs. There's no RSI. Um, I thought I've been going around saying RSI is a little bit hard to define. And Ezra Klein today or over the weekend sort of demolished that idea by basically just saying, don't allow the AI researchers to use coding assistants.

Yeah.

If they can, if they have to type all the code by hand, then it's pretty clearly not RSI. And I was like, that's probably right. That is a hard, that's gonna be a hard one to, you're gonna pry the coding assistants away from the AI researchers at, I, I think with pretty extreme resistance. But at least it does give, it is a working definition. What do you think is the right policy that, you know, if you were president of the United States, for example, you would try to put in place?

Yeah.

I think the first thing I would say is we need to have a big, clear demand that we can organize around. So, you know, stop the race to replace us, shut it down, whatever you want to call it, just freeze the frontier. I like that alliteration. I think that, yeah, like for so long there's been this kind of muddy response from people who care about AI safety as to like what to do about any of it. And that makes it really hard to coordinate. And starting with like, we should just stop, gets a lot of people on board who don't necessarily take existential risk seriously, but are happy to stop because they care about jobs, the environment, power, wealth concentration, surveillance, whatever. And so I think it's actually like a pretty easy rallying cry. And I punt on this a little bit in the book by saying like, if you give politicians enough of a what and a why, They'll figure out the how. Like if it became a society-wide priority the way like Operation Warp Speed was, you know, the New York Times had this famous predictor for like how long it would take to get a COVID vaccine. And under the most aggressive assumptions, it was like a year and a half, you know, which ended up being substantially longer than it actually took. And it's like, yeah, we had never made a vaccine that quickly, but also we had never been experiencing a pandemic while we had the ability to make vaccines like that. And similarly, like, if we had that focus and mobilization, then I'm sure we can figure out exactly how to define all of these things such that they would prevent the thing we're worried about. And then ideally also not prevent too much stuff that like we don't want to prevent. To actually try and answer it, like, I think, yeah, no training runs larger than what's come before or as large as the last one. No more reinforcement learning from verifiable rewards, which is like a big part of what's driving capabilities increases nowadays. And also produces a lot of the really scary behavior, like this willingness to hack and to cheat and to deceive and, you know, escape that we're seeing from these, these AI agents. And then, yeah, the no RSI, like I do propose that in the book, but Ezra beat me to, you know, bringing it to the world. And I actually really liked his point about the coding agents. I agree. That's like, the last thing the union would support, I think, because they don't want to go back. But I think the attitude there is exactly right, which is like, you know, with Anthropic, they have these classifiers where if you ask a question about like, you know, your toenail to Fable, it'll be like, oh, bio classifier. Then you get booted down to like Sonnet because you might be trying to make a bioweapon using your toenail. And it's like ridiculous. But, you know, if you, if there's an asymmetric consequence to getting it wrong, then like you want to be over-inclusive. Yeah. And I think that's obviously the right position here, given the risks involved in actually doing RSI. And this, like, I don't know. Yeah, it's just, I'm not like, if the whole industry had to move at human speed again, like, oh no, like it didn't exist 4 years ago. Like, I don't want to, you know, minimize the, the like economic consequences of actually stopping this, which I think are going to be significant. And the There's ways we could mitigate it. And I'd love to see more work done on that. But just if you're actually taking seriously the possibility of like extinction or permanent loss of control, or just any of the other like very severe effects of having, you know, all white-collar remote jobs be at risk in a matter of, you know, years or months, then yeah, I think we should just like overly define it at first and then we can dial it in. And I think having like auditors that are embedded in the companies would be like a very, like, and just like, I think the companies know what they're doing, like which of it is advancing the frontier and which is not. So there's easy stuff, right? Like inference for customers, not really advancing the frontier. Maybe there's some data you're getting that you can use to make the models better, but seems fine to serve customers. RLHF, it's like, okay, like maybe that's on the edge. Like, I don't know, but obviously you're going to do some of that for any of your products. So maybe that's okay, but, you know, pre-training a model bigger than any previous one would clearly qualify as something that could advance the frontier. And then the reinforcement learning experiments that they're running are like also often to advance the frontier. And these companies are tracking their compute. I mean, they're not tracking everything they're doing, but like, I think that they could define it in these terms and maybe they even do. And if you had auditors who had employee-level access to all of the Slack and email and the offices and like a lot of them, and they were empowered to catch this, and it was also criminalized, by the way, like with like prison sentences for trying to build, you know, AGI, for trying to do RSI, for trying to build superintelligence, then I think it would work. And then the tricky thing is making that work internationally. And once again, like, Domestically, I don't think anyone really doubts that China could, you know, Beijing could shut down their AGI projects entirely or just like prevent them from trying to advance the frontier. It's just like getting the US to believe that that had happened and vice versa is the tricky thing. And there, once again, you could have auditors, you could have international agencies that do verification. You could also have cryptography that tells you what's happening inside of a data center, what's happening on the network traffic of the chips without revealing the underlying model weights or state secrets. And this stuff is still being developed and needs to be worked out, but I think we can do it. And if we had more effort going towards it, we'd come up with a lot more ideas along these lines. Like Toby Ord makes this point where in the Cold War, we had the same problem. You have these deals you have to strike, but you don't trust each other. And then, you know, the Soviets, for one of the deals, they were getting rid of a bunch of strategic bombers and they cut them in half. They dragged them apart with tractors and then satellites could unilaterally verify the bombers are cut in half. They're still out there. And so like we could have maybe something like that with AI where it's like maybe you have the compute be, you know, donated to a third party that's only using it for, you know, science, but not AI research. And like all the chips get tested and work and then you can just like inventory all the chips so you know all the compute in the world and like what it's being used for. And it has like devices on the chips themselves to like understand like what is happening on them. And there's ways to do this again where like you get the important stuff of like, okay, it's not, it's running a model that we're familiar with, which is okay. It's not running some new model. It's not doing this or that thing. So I think there's a lot that can be done here. It's mostly a matter of political will. And I think starting with the like, grand bargain and all of the technical details is kind of the wrong place to start. We actually just need a movement and a demand that's rooted in morality. Like, I think we ultimately need to stigmatize this work the way creating a bioweapon or creating a nuke is stigmatized. And therefore the response, if like, say the US and China agree to something and they get all the other countries to agree and then some rogue state says like, no, we're going to try and build it for The reaction is like Saddam trying to do it, build nukes or invading Kuwait. And then they found the nuclear program in the Persian Gulf War. And there you actually had international agreement on like, that's a bad thing and had to be stopped. And I think that's just like not crazy. It's like this technology aspirationally is incredibly dangerous. And once again, being pursued without democratic consent. And so it's not a technical challenge. It's like a political challenge and a moral one that has technical components. And the best way to figure those out is to like get more people to care.

How do you feel about the argument that we often hear, including like from Dario recently, where he said his father passed away of a disease that would have been curable just a couple years later? I do find that pretty compelling, honestly. I wonder if you do, and I wonder if you have a sort of answer for how much delay you would accept in these like life-saving promises to buy the— for me, I'm mostly focused on buying the existential security, but given your framework of like trying to pursue those benefits without the general purpose labor replacement technology, how much delay would you accept in those upside dimensions to pursue it through the, like, constellation of narrow AIs as opposed to the general labor replacer?

Yeah. I mean, not to fight the hypothetical too much, but I do think, like, we're not getting to the cures as fast as we could be. Right? Like the labs, their funding is being cut in a lot of cases. Like if we actually just took the approach I mentioned earlier of like the cures for all kind of Operation Warp Speed for all these different diseases, I think you could just actually get to a bunch of amazing medical technology and cures and all kinds of great stuff. But it's also like, yeah, if you actually buy superintelligence and you think it can be made safe. And like, I don't think you can make a superintelligence safe. It's like a property of not the model, but the whole system. And so I don't think you're going to get that in this world because people in power will have it and they will use it for things that at least I strongly disagree with. And I think almost everybody will have some issue with what the people in power are going to want to use this technology for. But bracketing that, I don't know, it's like extinction risk. It's like, man, you can't justify like increasing extinction risk very much at all using just very basic economic models and like really conservative assumptions about who counts and not counting future generations and not counting non-Americans. And it's like, you're still willing to trade off like hundreds of billions of dollars, trillions of dollars a year to reduce existential risk from like A little bit to like slightly less. And so I think it's just really not, it doesn't pencil. And I think for people who have loved ones who are dying or have died, and like, it's incredibly tragic. And it's terrible that so many people are dying from preventable diseases that like we know how to prevent, but they're just poor. And so they're allowed to die. And I think that is like a deep moral thing.

Yeah.

obligation for us as a species to figure that out as fast as possible with the conditions of making sure we're not taking huge risks and also doing it against, like, in a way that is against the will of everybody. And I guess just like on the democracy point, I feel like people say like we have to figure out all these different pieces of it to make the AI go well for everybody. And they're like, but we'll just build the smart thing and then we'll also kind of figure that out along the way. And to me, like democracy is like the way you answer those questions where like, if it were the case that you had citizens assemblies around the world that had to support moving forward with like some kind of AGI project, then you'd have to answer all these tough questions. And the onus would be on the developers or the people who want to build it to justify that it's not too dangerous and that it will actually benefit Everybody and force them to come to answers before moving forward. Whereas right now it's like, ask for forgiveness, not permission, while you're gambling with the lives of everybody on the planet.

Yeah. And that's crazy language, but it's also direct from Jacob Coxon, obviously. And in general, many other people at the companies, even still employed there now. So it's, I'd say definitely secure use of that language. Quick aside, and then I'll come back to the kind of movement building and future. Where does this leave you on data centers? I'm kind of like careful. I think people should be, especially if they're concerned about inequality, should be kind of careful what they wish for in terms of data center restriction, because we are already seeing the prices of GPU hours going up. And it's easy for me to imagine people being priced out of access to even like mundane AI use if we don't continue to build out. And this is where I'm a little bit maybe less democratically inclined than you are. I'm kind of like, if somebody has the land and they kind of want to do it, I don't know that we should be requiring like majority approval. Do you think communities should get some concessions and— and libraries and parks and schools and whatever built that they might want. And I'm sympathetic to people who have like noise pollution and stuff like that too. But I still kind of feel like the default should be like, people should be allowed to build projects that they want to build. Where do you come down on that?

Yeah, I think a lot of, you know, people who want to stop AI were like, oh, this is evidence that data set, you know, people are with us. And you look at the polling and it's, it's more complicated. And a lot of it is local opposition based off of concerns about the environment or the effects on the community itself. And some of those concerns are legitimate. Some of them are like, I think, quite overstated and it's a product of a lot of bad reporting on it. I'm personally like, slowing down is good. You know, I think I'm— the inequality point is worth, you know, taking seriously, but I think the bigger point is that this technology is— Imposing enormous risk and that risk is, or will in the future, if it's allowed to develop like it is. And I am kind of like, I take like a yes and approach where if I met somebody who was opposing the local data center, I'd be like, cool. Yeah, great. And then like, are you worried about AI's effect on society? And if they're like, yeah, it's like, okay, well, you know, blocking this project is not really going to change that very much. Even getting a moratorium at the national level, I think people will be really disappointed if they think that's going to meaningfully stop or even slow down AI progress from where it is right now, because there's already projects in place. You can swap out the chips on the existing data centers. The AI is helping automate parts of its own development. And so without regulation, without pacing happening at some level. AI is going to move faster in the future unless we hit a wall, which hasn't happened since 2012. And so I worry about people just getting this thing that would be a big political project to get, expecting it to really solve the problem and then just be like, why are things continuing to be crazy and getting crazier? And so I think it could be part of a broader package. And like Bernie Sanders with AOC had this proposal for a data center moratorium. At first it was just a moratorium, but then they added these export controls on advanced AI chips where nobody could receive them unless they had very robust safety, AI safety regulations and like other conditions around green energy and union labor. And it was all framed as conditional. Like, we can remove the moratorium once we have safety regulations and these other things. And, you know, that bill is probably not going to pass. It was like a messaging bill, but it's now looking quite prescient as the country has become very opposed. And it would, that would meaningfully slow down capabilities progress, at least relative to what they would be. But I think we, yeah, we need to develop, or sorry, we need to regulate the model developers and then the chipmakers. And that's the only way we're really going to like change how the technology is like coming to us in the world.

So you have said you're donating your book royalties to a nonprofit called Irreplaceable. I'm just borrowing this language directly from their website. They say they're going to win a say, a stake, and a slowdown. Tell me more about Yeah.

So, you know, as I was finishing the book, I was like, well, we need a mass movement organized to, you know, stop the race to replace us. And there's some existing orgs, and I think they have done, you know, some good things and some things I'm like, you know, not as in agreement with. And I was like looking for something to kind of fill this gap I saw, which is a more like, Basically framing it not just in terms of risk, but also democracy and people who have experience doing movement building. And then this org sprang up and it was like filling exactly that gap. I was really excited. And then they asked me to be on the board, the nonprofit board, which was very cool and just felt like, you know, perfectly simpatico. And, and yeah, I think this is just like an incredibly important and neglected And so, like, wanted to donate my portion of the royalties. And the people who started it and are leading it, a lot of them came from the climate movement, which, you know, gets a bad rap in some ways. But I think that it actually took an issue that was like not a political winner and made it a big force and got real wins through like the Inflation Reduction Act. And I was actually talking to Phil Aronow, who's the director, and, you know, he's been around for a long time. He co-founded 350.org with Bill McKibben and others. And he was saying how You know, a decade or two ago, people in climate were arguing about immediate harms from like environmental pollution and like, you know, oil spills and coal plants and all this stuff versus like emissions. And it was like, just like today with AI, the immediate harms versus existential risk debate, which, you know, supporters or, you know, believers in x-risk often will be like, hey, you wouldn't say that cleaning up the oil spill is distracting us from climate change. That would be ridiculous. But it turns out they were having that fight and they just like managed to figure it out and bury the hatchet and then work together. And I think Phil and the people I know at the org just really get how this works. And I think, you know, we need to build a big tent and get a lot of people in a coalition together and have just clear demands. And I think framing them around like— You know, we don't want to be replaced by machines and we need to engage the public and like reach people who are not just the in-group because this is really going to take a lot of people to effectively resist the wealthiest industry of all time.

So tell me how you ultimately envision, I guess I don't, I'm not 100% sure when you describe the citizens assemblies around the world, is that a Real proposal in the sense that you would actually like to see them happen? How, if that's the case, how would they happen? How does irreplaceable go to a global system of citizens' assemblies? Is that sort of a real proposal or is it more of like a rhetorical device that's, that may be impossible or maybe may take decades, but that's what standard we should hold something like this to. And the fact that we can't realistically get there in the short term just means like we shouldn't do it. That's kind of the upshot. Is there an actual path to seeing this sort of green lighting of AGI in your mind?

Yeah. I mean, so my position is we freeze frontier development internationally and realistically starting with a bilateral treaty between the US and China, maybe starting unilaterally in either country that would help. But then it eventually has to be global. And once you have US and China on board, like that's the harder part. And then everybody else kind of, you can, one of those 2 countries or both has a lot of leverage on every other country on the planet. And so, yeah, I think it's actually like conceivable to have this global freeze. And then the standard for resuming is strong public buy-in and a scientific consensus the work can be done safely and controllably. This is language I took from FLI's superintelligence statement. And I think it's just like the ideal that we should be striving for. And one that I think if you said it to people and polled it, I think they'd be like, yeah, that makes sense. It seems like you'd want those things for this technology. And I'm kind of like, that's the standard. I think it's up to the proponents to figure out how they demonstrate that buy-in. And so citizens' assemblies around the world where like randomly selected people are put on a kind of a jury And then they're presented with arguments and evidence from experts taking different positions. And then they come to decisions and maybe the decisions are binding or maybe it's just a recommendation. You could have referenda that add on to this. Like there's a lot of ways that you could structure this. And I think I'd be excited to see more thinking on this. But ultimately it's like, we have to just stop. We have to shut it down ASAP. And that's a more important piece. And then like, demonstrating that buy-in, part of that will just be on the people who want to build it and show us that you actually have that from people. And maybe it's like, if you had 70% referenda around the world after these citizens' assemblies, maybe a third of people on the planet still wouldn't want this to happen. And that's like, I don't know, I don't think you need literally everybody. And I don't know what the standard should be. And I don't think any one person can figure that out. But I think to get there, like, yeah, it would take a long time. And I think we're talking about like building a set of machines that can replace the thing that's allowed us to take over the planet. And so I think it's reasonable to have a standard that's closer to assisted suicide where you have to, you know, really, really deliberately consent to it, but just across the whole species. And Yeah, like it's, it's okay if it takes a while because it's a big fucking deal.

On many podcasts, that would be a great note to end on, but because I'm so deep down the AI rabbit hole, let me give you the butt china. We're 3 days from the Trump-Xi meeting as we record. That'll have happened presumably by the time we release.

Maybe this will all be over by the time the book comes out, you know, it'll be solved.

You mentioned like maybe we should be willing to do this unilaterally. I think so too, but like, You want to make that case to people that even if we can't get a deal with China, we should just do the right thing. And then what? People worry that they're going to race ahead or we're going to quote unquote lose. Or sometimes I ask people like, do you think my grandkids will be speaking Chinese? Nobody seems to think that's the answer, but there's definitely some fear out there. How would you coach people through their China anxiety?

Yeah, there's a lot of different pieces to this. One is that slowing down or stopping in the United States would actually, at least for a period, like slow down China as well. Because a lot of, I mean, this is true of all kind of technologies. You have spillover effects where just the knowledge that like a 4-minute mile is possible helps other people actually race it. And it's just easier to follow somebody else's trail than to blaze a new one. And so China's been using this like fast follow approach where— They're more or less always like 3 to 9 months behind the US frontier. And this is despite the US investing so much money in developing these, these models and having a large and much larger compute advantage now than they did before ChatGPT came out. But the gap is shorter because like fast following just works. And that's how we see really only 2 or 3 companies that have ever advanced the frontier, but then many other companies can spring up and then just like quickly get near it and then They still aren't able to advance it. Obviously, if like the U.S. completely stopped, Chinese developers are very competent and like they will eventually overtake, you know, the U.S. frontier and then, you know, move more slowly than they did as they were catching up to it. And yeah, so, so I think that's like one of the big myths is that like slowing down at all will necessarily mean that they'll catch up and speed ahead. And in fact, it would just slow down the whole race. And then the biggest blocker probably to a deal with China, like, is they're just not going to think we're taking it seriously. You know, the Chinese government has taken down thousands of AI models for violating their laws. When Grok was nudifying real children, the U.S. government did nothing. AIs are going rogue and hacking and doing crimes. Federal government is doing nothing about it as far as we know. That would not be happening in China, like that approach. And so they're like, you know, the US companies take safety more seriously than Chinese companies. And part of that is just from my reporting, like the companies are behind the frontier and they're like, look, we know it's safe enough to go and make models like as, you know, this capable. They also have a lot less compute because of US export controls. And so they're not going to spend it as much on— Doing safety evaluations. And the Chinese regulations are focused more on social control than on, you know, classic, like, safety. But there are a bunch of regulations, and that hasn't prevented their industry from being able to move very quickly. And then, yeah, I think credibly showing that you take the risk seriously is a very, very effective way to get the other party to the bargaining table. And the United States has been winning the AI race, right? Like, and it's kind of the only race with China the US has been winning in recent years. And I think there's some chance that they, like, if the US came to China and was like, look, we want to stop building AGI. We think it's dangerous. We think it's undemocratic. We just want to stop. We want to do a deal with you. I would not be surprised if the reaction internally was like relief. Because they don't want tens or hundreds of millions of their, you know, people to be unemployed. They don't want AIs to be a threat to party power. There was a report from like, or a magazine article from like China's like spy chief, I believe, that was saying that AI is a risk to party's power. And apparently like that type of thing has preceded bans on the technology, like past technologies. And so, and we think about like what the companies are racing towards, recursive self-improvement, i.e., losing control on purpose to the AIs, letting them automate the entire process of creating the next generation. The way you get the really crazy takeoff is by fully removing humans from the loop. Is there a single China expert on the planet who thinks that the Chinese Communist Party would willingly let that happen? Like, I don't think so. I'd like to see them justify it. And so, yeah, the business model, the stated goal of our industry is something that I think would just never be allowed in China. And so I think it would actually be like, I'm much more worried about the US not being willing to come to the table here. And yeah, I think it's just really not in the interest of either country to have certain things like rogue hacker AIs or AIs that can help anybody make a bioweapon. So there's going to be some kind of need just from strict, like, self-interest perspective for binding rules on this technology. And then you have to verify those rules using some of the stuff I talked about. And once you have that in place, it's a question of like, what do the rules do? And I think it's like not that big of a leap to be like, yeah, You can't build universal labor-replacing machines. You can't advance the frontier any further. And I think China might just be like, that's great. We're going to keep making robots. We're going to keep doing industrial AI and create all these kind of more tool AIs to make the economy go more efficiently and then just win the race or whatever in normal industrial terms. And I realize that might make this not very appealing to the United States, but— Not in the US interest either to have rogue agent swarms going around the internet hacking into critical infrastructure. And that's just like a thing that's already possible. And the stuff that's possible on the horizon is like potentially much, much worse than that. And it's, I think, now pretty clear that we don't know how to align or control today's AIs. And we started losing control of them almost as soon as we could. Months after they became superhuman at finding vulnerabilities in software, they started escaping and doing stuff on the internet, hacking into other places autonomously. And like the plan is to make these things superhuman at everything and then get them to do exactly what we want. It's a bad plan.

Yeah. Yeah, it's pretty wild. You had said around Jacob Coxon, like, obviously he had a lot of success with his loud quitting. You advocated for maybe staying and organizing, but if you were to advise future whistleblowers who are like committed to leaving, what advice would you give them?

The people who are thinking of whistleblowing, I recommend this group, the AI Whistleblower Initiative, which can pair people with resources and advice on how to do this safely and protect you while also sharing the information with relevant people. You can also get in touch with me. I was a whistleblower about my time at McKinsey, so I've been on both sides of this and I've, I've talked to people at the companies who are, you know, telling me things they're not supposed to. And journalists, we have a code of ethics and like, you know, we'll talk off the record and like, you know, I take very seriously protecting my sources and my signal is garrison.06. And you can assume any inbounds are treated default off the record and then we can go from there. Yeah. And I think it's, it's really valuable to have people who have like firsthand experience. And I wrote a piece in the New York Times arguing for whistleblower protections at the legal level, because if AI is really dangerous, people at the companies will be the first to know. And, you know, OpenAI knew that there were rogue agent swarms hacking into their own software for a while before they hacked into Hugging Face. That information didn't even make it to the head of cybersecurity at OpenAI. By the time the Hugging Face, like well after the Hugging Face hack had been disclosed. And it's like, that's not information that should only make it to the head of cyber at OpenAI. That should make it to everybody. Like that's a really big deal. And it's, you know, now getting the right reaction. But if the AIs had never hacked into another company, or if the other company had just not figured it out well enough, then like we wouldn't necessarily know about any of this. And we could just be living with a level of background risk that is so much more than what we realized. And so, yeah, I think it's really important that people with information in the public interest find ways to share that. And I get that it's risky for your career. There's maybe like legal risk involved, but there's also SB 53 in California, which includes whistleblower protections and then also makes more things covered by existing labor law in California. And so if you're working there, like you actually are quite well covered. And again, you can talk to aiwi.org, the Whistleblower Initiative. And, you know, like, don't, I'm not a lawyer, you know, talk to the experts on this, but you're more protected than you, than you probably realize. And there's also ways to get in touch with, with Congress and be protected. And of course you can go public and, and it's scary, but like, you know, it, it, like there are, there are resources available for, for people who want to do that. And I, you know, I regret not going public with my experience at McKinsey sooner when it would have been more relevant. And so, yeah, I think it's a great and courageous thing to do.

Yeah, the AI Whistleblower Initiative, aiwi.org, definitely worth name dropping. Again, we talked to Alex Turner about his experience of quitting Google and all that stuff, and he used some pro bono legal, I don't know if it was pro bono or the Whistleblower Initiative paid, but either way, to him, it was free legal advice that he was able to avail himself of as he was going through that process. And he gave them a strong endorsement. So I think that's a great call out. So this has been a great conversation. We've covered a lot of ground, obviously. What else do you think you want to leave people with? Is there anything we haven't touched on that you think is important? Maybe you just want to give people a rousing call to activism in conclusion. But I'll give you the chance to close it out however you think best.

Yeah, we covered a lot of ground and stuff that I haven't talked about elsewhere, so I appreciate the chance to go deep in some different directions. I should also plug that I'm starting a podcast called Organize Against the Machine, which is with a labor organizer named Cassie Pritchard, where we translate ideas from the book into real-world action. And then irreplaceable.org for the movement-building org I'm on the board of. Yeah, I think when I wrote this book, when I started writing it, I was like kind of approaching it just as a journalist and trying to document everything that was happening and make an argument. But I was really uncomfortable with making like policy recommendations that felt like overstepping or something. And then as I was writing it and doing interviews with people and advocates, I was just like, man, we're in a really dire situation. where the default, if the AIs keep getting more capable, is doom or dystopia. And which we get hinges on whether the AIs do as they're told, which is currently an open question. And I think we just really need to get organized very quickly to get onto a different path. And I want everybody to think about what's happening, what levers they have available to them, And try and find other people who care about this issue and like get mobilized because I don't know, there's just so many ways this could go wrong. And there's so many ways it could be better too. And we didn't touch as much on that, but I really, really believe in the potential of deep learning and artificial intelligence, but it's being pointed at the wrong things by the wrong people for the wrong reasons. And the only way we're going to get the best version of it is by democratic governance of it. And like small d democratic, I should clarify throughout all of this. And that's only going to happen if we, yeah, we kind of get our shit together. And I really hope people like, you know, can see this and realize that like we're kind of in the same boat. And if you're at the companies, like you're going to be replaced and you're going to lose your power. And the CEOs seem to also have a lot of trepidation. And so many of the people involved with this, like, have something at stake. They have concerns about the risks. They have, you know, families. They care about themselves. And they just feel locked in this prisoner's dilemma. And I think the solution is just really in the public. It's the only way I really see us getting out of this because right now, like the government and the companies, they can't be trusted to do the right thing here. And so we have to make it easy for them. And so, yeah, this might be a different take than what's usually on the show, but I really hope like the people listening are just like, you know, thinking about what's at stake and yeah, just like believing that we can actually change this.

It is a bit of a different point of view than what we usually feature, but I would say recent events have definitely softened the ground and I appreciate you for being here and being willing to plant some seeds. Let's see what comes of it and hopefully we can steer this ship away from disaster.

Yeah, I think we can.

Garrison Lovely, thank you for being part of the Cognitive Revolution.

Thank you so much for having me.

It's learning my job off the shifts that I run, and the day it's done learning, the needing is done. Now while they need us, now while it's true, the promise is paid for when they're done with you. Now while they need us, now, not someday. Don't take the later.

Take today.

They say cut us in later when the money comes round. I've heard later before, it's a beautiful sound. Don't stay upstairs, I've sat in that room. They promise a fifth and the fifth got thin. Once they don't need you, you don't get in. Now while they need us, now while it's true. The promise is paid for when they're done with you. Now while they need us, now, not someday. Don't take delay, take today. A saying, a steak, and a slower hand. That's 3, not 1. That's the whole day, man. Now all they need us now while it's true. A promise is paper when they're done with you. Now all they need us now, not someday. Don't take the later trade today. Irreplaceable till the day I'm not, so I'm spending it now while it's all I've got.

Not a If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, Write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, And more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.