The Case for Trump and the Future of AI – Part 1, with Samuel Hammond, Senior Economist, Foundation of American Innovation

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg.

Hello and welcome back to The Cognitive Revolution.

This weekend we're running two episodes which originally appeared on the Moment of Zen feed, focusing on the election, in which I attempt to give an honest hearing and questioning to two different AI-forward cases for Trump. I like this exercise because, in my honest opinion, this election is ultimately a referendum on Trump. My interlocutors are, in the first episode, Samuel Hammond, senior economist at the Foundation for American Innovation and And a thinker that I generally very much respect. We cross-posted his appearance on the Future of Life podcast last year, and I also really appreciated his 95 Theses on AI, most of which I agreed with. In the second episode, I speak with Joshua Steinman, Trump's National Security Council Senior Director for Cyber Policy and Deputy Assistant to the President from 2017 to 2021, the entire Trump term. Before launching into it, I'm going to briefly share where I've landed on Trump, as I expect a possible Trump presidency might relate to AI development. If you see AGI as a real possibility in the 2027 timeframe, it seems totally reasonable to consider the election's impact on AI as a major decision-making factor. Of course, I understand that people have other priorities, but this is not a politics channel, so I'm not going to offer my opinion on every issue, just AI and AI-adjacent issues. Interestingly, as you'll hear, I find that on a number of these AI adjacent issues, I agree with the Trump supporters that I talked to. To name a number of them, nuclear energy is good. We should build more nuclear plants. Population decline is a problem and does merit real concern. Freedom of speech is valuable and should be protected. On the current margin, we should have fewer rules and more right to build on one's own property. We should cultivate a culture of achievement. We should aim for an age of abundance and dismiss degrowth. And it makes sense to prioritize high-skilled immigration, at least to some degree. Finally, American companies like Amazon, Google, and Tesla should not be allowed to abuse their market power at the expense of consumers, but neither should they be subject to government harassment just because they're outcompeting many legacy businesses. Fortunately, it does seem that the Democratic establishment is coming around on at least a number of these. But regardless, there are 3 big reasons that I cannot ultimately support Trump despite these important points of agreement. Those are: 1, he is far too inclined to escalate tension with China, accelerate decoupling, and prioritize his own narrow domestic political interests over the national and global interest. 2, the lack of rigor in his thinking and discipline in his communications seems like a recipe for unnecessary risk-taking, in an increasingly volatile environment. And 3, I believe we are far better off approaching the future with a positive-sum and inclusive mindset, not just within the US, but globally, if we're to have a healthy conversation about a new social contract that befits the AI era. On the question of US-China relations, I think we have a general failure of leadership and vision on both sides unfolding slowly, but gathering more momentum all the time. People now see adversarial relations with China as a foregone conclusion. Joshua Steinman calls it the physics of the environment. To put it plainly, I do not accept this. Conflict with China would be a disaster, and an arms race would take us ever closer to that disaster. But I don't see this as an inevitability because I don't see China as a meaningful threat to America, Americans, or the American way of life. That's not to say that the Chinese government hasn't wronged us at times. Their cover-up of early COVID, whatever its origins, was definitely shameful. And obviously Chinese agencies and companies have stolen a lot of intellectual property from American companies. I don't think we should ignore that, and of course we should take steps to make ourselves less vulnerable, but I think we should stay level-headed about it. The possibility that our grandkids could be speaking Chinese one day seems far more remote to me than that AI destroys the world. I would say that the Biden administration has done okay on AI policy domestically. The 10 to the 26 threshold for reporting requirements has aged pretty well for a 2023 rule, and I do believe in some strategic industrial policy as well. Subsidizing the building of new chip fabs in the US so that we're not so easily disrupted by a Chinese attack on Taiwan seems, under the current circumstances, a prudent step for a great power to take. At the same time, the chip ban still feels wrong to me. And I have to admit that Kamala's rhetoric on China also depresses me. There's simply no way for China to understand recent U.S. statements and actions other than as an attempt to keep them down. And for me, this escalation seems premature at best. Given the shape of an exponential curve, we could have retained the option value of cutting them off from chip sales later, and still the bulk of the total hypothetical chip sales would have remained in the future. Relatedly, I've been really interested to see Miles Brundage, OpenAI's recently departed head of policy research, saying that we need to make a much more proactive effort to demonstrate that Western AI development is benign, which of course is much easier to do if it actually is benign and to some degree open or otherwise shared. If we must frame our relationship with China as a competition, I would love to see us race them on metrics like life expectancy improvements, the number of diseases we can eradicate, Or perhaps the tonnage that we can send to Mars. Of course, I do understand that it's a complicated situation, that naive solutions won't work, and that real solutions will be both nuanced and difficult to find. And I do intend to invite more China experts onto the show going forward in an effort to more substantively contribute to a positive vision for the future. For now, I wish that Kamala and Democratic leadership in general were both less hawkish and more visionary. But considering just how much trust has already broken down and how difficult it's going to be for the two countries to make credible commitments to one another, I'd rather have. a steady hand, who is more predictably going to follow a sane consensus path, and might actually make some arms control type deals, even if politically costly for them, to help us navigate a tricky period as well as we possibly can. Trump, I think it's well established, will do anything to avoid looking weak, has credibility with rival countries perhaps when it comes to threats, but not when it comes to positive follow-through. And for me, his withdrawal from the Iran nuclear deal, general taste for inflammatory rhetoric, and stated plans for blanket tariffs, which will hurt the American consumer in order to vaguely somehow stick it to China, all suggest to me that he is overwhelmingly likely to make things worse. Zooming out from US-China relations, while I agree with Sam Hammond when he says that the more fucked you think we are, the more willing you should be to roll the dice with Trump, I don't actually think that we are all that likely to be fucked. I've been a bit ambiguous about this over time. Often saying that my P doom is 5 to 95%. And I've meant that to reflect the fact that while nobody has convinced me that we don't need to worry about AI risk, neither has anyone convinced me that it's inevitable. After all, while we don't really know how they work and we see all sorts of surprising and sometimes alarming capabilities, today's best AIs do understand human ethics and values quite well and seem to be getting at least a bit more aligned with each passing generation. This may not continue, and we should absolutely be vigilant about it. But this is a much better position for 2024 than most AI safety people expected 5 or 10 years ago. Today, facing a decision like this referendum on Trump, I recall the words of a wise friend who told me that we should think less about what the probabilities currently are and more about what we can shift them to. Here, I have to say, with competent, stable leadership, I believe we can steer towards scenarios on the lower end of that range, where the nature of the risk is more intrinsic to the technology itself and less the result of letting domestic political incentives lead us toward imprudent escalations, AI arms races, or catastrophic impulsive decisions. I often think of the role that Kennedy played in the Cuban Missile Crisis. My understanding is that he overrode the recommendations of his military advisors to insist that the United States would not escalate to nuclear war first. That was heroic, but scenarios in which executive authority really matters can cut both ways. When I imagine Trump versus Kamala in moments of intense crisis where a single decision could alter the course of history, I have to say, I find it much more likely that Trump would impact things substantially for the worse than substantially for the better.

After all, we all saw how he handled COVID. To be clear, Kamala has not impressed me on the topic of AI either.

And in general, her track record suggests not the great foresight of excellent leadership so much as the tendency to follow local trends and incentives.

We absolutely could hope for better.

But still, if I have to choose a leader for a potentially highly volatile period of time, I'll take the stable, sane person who will listen to the expert consensus, even acknowledging that the experts could be wrong, rather than betting that Trump will somehow manage to override the experts in a positive way. You'll hear my conversation partners make the case, which I won't attempt to summarise here for fear of doing it too poorly, that Trump represents our best case to break out of a broken consensus and revitalise the American state for the AI era. In the end, I just don't see it. It sounds to me like chaos when we need capability. Finally, when it comes to the future of American society and the world at large, I think we have a never-before-seen opportunity to adopt a positive-sum mindset for Create a world of abundance, and ultimately update our social contract. I think OpenAI and other leading companies do have roughly the right vision when they talk about benefiting all humanity. And I think Sam Altman, for all the other criticisms that I've made of him, absolutely should be praised for his experiments in universal basic income. While neither candidate has shown this kind of vision, Kamala at least aims to speak and extend opportunity to all Americans. I thought her best moment of the recent debate was when she said that when Americans look at one another, quote, we see a friend. This is at least something of a foundation on which to start building a shared positive vision for the future. Trump, of course, is far more zero-sum in his thinking and negative in his outlook, and that does have real consequences. I grew up in Macomb County, Michigan, one of those bellwether counties that swung hard from Obama to Trump. And I also have family in Ohio. My beloved Mama and Papa belong to the same cohort as J.D. Vance's grandparents. They moved from rural Kentucky to southern Ohio for jobs, the whole thing. To be totally honest with you, one thing that I have seen for myself is that Trump has brought out the worst in a lot of people. Sure, J.D. Vance, Elon Musk, and others in Trump's orbit are no doubt more sophisticated thinkers about technology than Trump himself. But still, I cannot imagine this brand of cynical populist politics could possibly lead to a healthy national conversation about adapting to AI, as it, let's face it, is going to be disrupting a lot of people's jobs, let alone reimagining what it means to contribute as a citizen or to live a good life. It would be shameful if we ended up hoarding the benefits of AI or restricting access for non-Americans due to some sense of scarcity that isn't even justified by the fundamentals. But unfortunately, That's the direction that I would expect Trump to take us. Ultimately, the idea that Trump could be president as AGI is first developed strikes me as an imprudent move, to say the least, with far more and more likely downside than upside. By all means, listen to these conversations with an open mind and form your own judgment. But for my part, I can't support putting a loose cannon in power as we head into such a potentially pivotal period. And so I will be voting for Kamala, primarily as a rejection of Trump.

Hello.

There we are.

Yo.

There we are.

Hey guys.

So, so sorry about the slow start or the miscalculation on my end. Really stoked to have this conversation, guys. Thanks for joining.

We just had a chance to chat a little bit and get familiar. Actually, first time we've ever spoken, although we've exchanged DMs occasionally appreciating one another's various outputs. So it was good to have a little warmup.

You two thinkers are two of the thinkers I respect the most. You can't fit you two in a box. You both get into arguments, heated arguments, good-natured but deep arguments with people who are on your side of the— who are kind of in your peer group. So you don't fit neatly into any category. So I wanted to invite you both on. And Sam, let's start with the piece you wrote, but kind of the broader thinking that you've been undergoing as it relates to the case for Trump. That's been what we've been debating the last few episodes, and you brought the effective altruist perspective to it as well as your other perspectives. So let's start there.

Sure.

And for the record, I'm not an effective altruist. I just, I can put on that hat and try to think about what an EA would say. I am a kind of rationalist and one of the things rationalists are known for is trying to see through convention and social desirability bias. Is supporting Trump going to hurt you in your social circles? Well, then maybe you should have a degree of meta-rationality and try to interrogate that and overcome it potentially. At least it doesn't mean support Trump, but it means factor that into how you understand yourself. So from my lens, to take a more utilitarian look at how 2 candidates would play out, what matters most, at least in the long run, is innovation and innovative capacity, especially if you care about future people. Innovation compounds on itself, productivity improvements compound on themselves. So the question becomes, who is better for innovation, at least at first order? And in my piece, I break down some of the most obvious areas, drug and medical innovation, is a huge one. Kamala Harris, piggybacking on some Biden initiatives, wants to cap pharma prices based on a basket of international prices. This, I think, would be a huge mistake. I mean, there's been tons of research into how the US is a kind of provider of global public goods because the fact that we pay exorbitantly for all kinds of drugs enables big pharma companies and small pharma companies for that matter to actually do the R&D that goes into designing and manufacturing new drugs and then jumping through the FDA approval hoops. And so there's been estimates that even just the pharma caps that were put in under the Inflation Reduction Act have delayed or maybe killed up to 170 new drugs. Untold costs associated with that policy, even if they're relatively unseen. And I don't think it really stops there. I mean, later in the piece I get into AI, which is maybe where me and Nathan can get into it, but— I think the 2 administrations, a Trump or Harris administration, would take a very different tack on AI. And that might be the biggest issue of all, depending on how you see it playing out and where you think the technology is headed. Getting AI policy right could easily swamp the costs or benefits of every other domestic policy we think about.

Well, I agree with the last statement most, that the AI policy seems to matter most. I guess the way I think about Trump is As a total wildcard. And when I think, and I do agree, I think EAs, and do I call myself an EA is an interesting question. I'm definitely highly influenced by EA thought. I would say if I'm not an EA, it's probably because I'm not virtuous enough to count myself among them. I generally have a lot of respect for the EA community, and especially those that really hold themselves accountable to living up to EA values. And I don't always feel like I do that. So I sometimes want to say, like, I'm not worthy of the EA label. But I think the EAs are right to think long term, right to care about future people, right to not discount the future too much. Whether you should go all the way to zero discount is an interesting question. But then when I think about Trump in that way, I kind of can't even get down to the level of resolution on these super specific policies. I would agree that we want to get more drugs. I don't necessarily always like paying the higher prices for drugs, but I do buy the argument that there's something good about subsidising drug development for the rest of the world. And yet I'm like, that seems so small compared to the really big questions. I think the EAs are also right to worry a lot about the tail risks. And when I talk to EA thought leaders, I really don't hear anything along the lines of this policy, that policy, tax incentives, what's the sort of amortization going to be? Instead, what I hear is like, how do we make sure we don't have nuclear war? How to make sure we are as prepared as we can be if a really bad pandemic comes our way? and how we make sure we have the best possible thinkers in place to make whatever judgment calls might need to be made as AI really starts to potentially heat up, maybe as soon as within the next presidential term. So I guess when we get to— we can unpack all those things, but I think all of them are kind of tail risks. If you were to tell me that another pandemic would happen, but it would only be as bad as COVID, then I could sleep reasonably well at night. If you were to tell me we might have a small nuclear war, but it'll only be like one nuclear weapon, then again, I could be like, okay, well, the world can probably deal with that. The real risk is like the pandemic that is super lethal or the nuclear war that goes all out. And it's in those sort of scenarios where I'm just like, I have no idea what Trump would do. I don't trust that he is like a very stable genius at all.

Quite the opposite.

I would say he seems like a very unstable and not genius person. He has managed to, in his first term, he did seem to kind of get away with crazy in most cases. But I sort of look at that when I look at the better angels of our nature graph of fewer conflicts and lower casualties in war. And then I look at the spikes, I'm like, I really don't want to see that next spike. And Trump, to me, feels like the kind of guy who might tamp down the small spikes, but inadvertently create the conditions for a really big spike. And so I don't know even what I should expect from an AI policy from Trump. I don't think Harris is a genius on AI. Her famous clip of just 2 little letters is not super confidence inspiring. But Trump feels like he's totally all over the map. I have no idea what to expect from him one day to the next. And I I don't feel like we can really even talk about this policy or that policy or whatever, because he's just so erratic. And there is a lot of power vested in the presidency, right? I mean, at the end of the day, if he holds the office, people are supposed to listen to him. And that just seems like a very unstable situation to put ourselves in, given all the other uncertainties that we're likely to face. So that's the hump. I mean, I would be interested to know, what do you think Trump AI policy would be and how confident are you in that assessment? Because I don't have a lot of confidence in my ability to predict what Trump will do from one day to the next.

Yeah, I mean, before maybe getting into the concrete policy details of how different administrations could pan out, I think maybe we could zoom out to just the different governing philosophies and the kind of coalitions behind each party. First of all, from an EA perspective, especially the more doomer you are, it's probably because you think things just go bad by default. And how do we get off a default path without embracing some higher variance option? And so that's point number one. The high variance of a Trump administration actually should count in its favor because there could be very high highs and very low lows. And it definitely leads to some discomfort or some uncertainty, But if they were going to do something that's sort of out of left field, it's, I think, much more likely to come out of administration. Second, Matt Grossman and David Hopkins, these 2 political scientists, have an excellent book that I always recommend called Asymmetric Politics. And it talks about how to understand the Republican and Democratic Party from a sort of more analytical structure-based lens. And their argument is essentially Democrats should be thought of as a coalition of social groups. You have the teachers unions, the trial lawyers, the Joe Biden and the credit card companies. There's all these different stakeholders and they kind of have to balance each other's interests and often pursue policies that are very concrete and directed at a particular interest group. But ultimately it's sort of through the stakeholder negotiation path. Republican Party, in contrast, is much more of an ideological vehicle. It's based around the conservative movement. There's obviously different factions within the Republican Party, But because it's more ideologically driven, it can also do things if you are able to couch a policy within that vision and show why it's consistent that are in some ways more first best and in some ways even cut against their narrow interests. So the fact that they appointed justices that overturned Roe v. Wade has cost them in the election, but it was driven out of an ideological view of what the Supreme Court was for. The fact that Paul Ryan during the Tax Cuts and Jobs Act repealed the home mortgage— severely capped the home mortgage interest deduction, the SALT deductions, which are very popular credits, it instigated a war with the realtors associations. That was because he had this ideological view of the tax code should be simpler. It wasn't driven from an interest group, it was driven from this sort of supply-side ideology. Similarly, if you look at the personnel, and to a large extent personnel is policy, within a Republican administration, at least in the last Trump administration, they drew heavily from more libertarian circles. So we talked about the tail risk from pandemics. Operation Warp Speed occurred under the Trump administration, and I think it only could have occurred under a Trump administration in part because they had a couple of Chicago School economists that were at FDA and that were basically given the mandate to apply Chicago School theory The Chicago School of Economics has been thinking a lot about the hidden costs of FDA drug regulations, like the drugs that don't go to market. How do we streamline these things? And so they were already in the process of deregulating FDA for the explicit purpose of accelerating vaccine approval. And part of that was an architecture for public-private partnerships, which then was deployed with Operation Warp Speed. And that pulled forward the vaccine by 5 or 6 months and saved Up to 200,000 lives, probably has all kinds of second-order effects because now there's been a boom in activity around mRNA and those mRNA candidates for malaria, for cancer, for— there's like a pan-influenza vaccine that's in development. And so like on the surface, yes, now politics have polarized. And during the pandemic, Trump suddenly became, you know, went from being pro-vaccine to anti-vaccine. And you saw the exact opposite on the Democratic side where like Kamala Harris, before the first vaccines were approved, was saying, I don't know if I put that in my body. And then now they're out and then all of a sudden things repolarize. That doesn't mean I think in the event that we had another pandemic that they wouldn't also just pull out that same playbook and accelerate a vaccine because ultimately the way Trump appears on TV is much different from how he ends up governing. You probably make a similar point about World War III and nuclear risks and stuff like that. In interviews, Trump will say, it's the N-word, the other N-word, nuclear. And he talks about World War III. the chance of World War III being at its highest in our lifetimes. He said that AI is— he called it super duper intelligence— is alarming and scary. Meanwhile, you hear Biden saying that our biggest existential risk is climate change and Harris saying that the existential risk from AI is it denies you healthcare because healthcare is existential. And I think there's just a different level of attention to these potential black swans through the different lenses. And that's also partly downstream, again, of these different party structures where you have something that's more ideas-driven. And I think you can only really tackle AGI risks through a more ideological lens precisely because they're still theoretical, whereas something that comes through a stakeholder lens is going to be connecting AI risks and these other risks to particular constituencies.

Hey, We'll continue our interview in a moment after a word from our sponsors.

Yeah, I mean, I don't want to be the person who's going to defend the stakeholder-based approach to AI. It is pretty easy to imagine the scenario where lots of good things that I want to see come online are blocked by interest groups. And you can look at all the Cost disease segments of the society and imagine what AI could do for those segments. And then you could imagine how the people who might stand to lose out in relative income and status and whatever would want to block those things. I've been, as an aside, I've been pretty pleased with how the medical establishment has responded to AI so far. I would have expected a much harsher anti-AI stance from doctors in general, at this point than we've seen. There's been perhaps just because they're all so overworked and exhausted and just fed up with all the paperwork that they have to do, anything that could reduce the paperwork burden is maybe welcome. But one can still imagine that could change. And certainly I don't have a high level of confidence in what teachers unions are going to do when it comes to implementing AI in classrooms. And I definitely would like to see my kid have the benefit of the best AI that we can bring to bear on his education, or my only one is so far in school. But so, sure, I'm with you that could suck. At the same time, I also really don't like the sound of an ideological approach to AI. I feel like one of the biggest things I've learned, or I feel like I've learned in the last couple of years of obsessive study of it is that these things have to be understood on their own terms, that we can't really bring preloaded frameworks or we can't really reason by analogy all that much. We really have to get down to kind of ground truth on what is this thing, how does it work, ideally as much as possible mechanistically. That's why I'm so excited about the actual study of what is going on in these systems and why are they doing what they're doing and can we engineer ways to control them? If we're really going to get to any sort of sustainable safety, it's presumably going to be with engineering solutions. But in the meantime, I really don't like the idea that we would have a sort of ideological approach. And most of the people that I think are kind of the smartest or have the most insightful things to say about AI are, first of all, very quick to admit their high level of uncertainty on where we're going in the biggest picture. Even Anthropic has their sort of core views on AI safety blog post, which is kind of like their canonical statement on the matter. And even in that, they're just like, we really have no idea. We don't know how hard this problem is going to be. It could be relatively easy, it could be near impossible. We're going to just try to do our best and update this view as we go. I think that's basically the right view to take right now. It's also interesting to note that a lot of these people are like techno optimists and libertarians in every other way than AI, right? They're all kind of eager to see more drugs come out. I mean, I think among people that are AI safety hawks, you would find many FDA reformers, for example. And so I think these people who have broken their normal— and I count myself among them, I'm broadly a libertarian-minded person— but I think these people who have been willing to recognise that, like, yikes, this is not something that we have precedent for, it's not something we have good analogies for, it is something we need to understand on its own terms. Those are the people who's analysis I find most compelling. And the idea that we would have, like, a— I mean, what ideology are we even talking about in the Republican Party? It does seem like there are several, right? There is this sort of Chicago School market maximalist. There's also, like, a Christian thing. I don't know, there's, like, neo-reactionary elements. There is— I mean, I don't know. I'm not sure whose ideology it would be, but none of those 3 seem like they're giving us an actual grounded in the substance of technology understanding, which feels to me like critical to making good decisions as the situation evolves. So, yeah, I don't know which ideology are we talking about and why should we trust that?

Yeah, I mean, the first point of clarification is I don't mean ideology in a pejorative sense. I just mean driven by ideas. In this case, broadly the conservative movement, and it does contain many factions. Ideologically, it's like, let's follow the Constitution. Let's be concerned about surveillance and censorship. Let's preserve basic civil liberties. Let's try to simplify government and roll back programs that we think were part of a progressive social engineering project or whatever. And so there's no interest group behind that per se. Sometimes they will be tied to an interest group. But I guess a good way to illustrate this is during the first Insight Forum that the Senate held on AI, Republicans and Democrats both get to invite people. Republicans invited Elon Musk and Mark Zuckerberg. The Democrats invited Randi Weingarten, who's the head of the Federation of Teachers. And they also have had hearings with Emily Bender and And so there's other— the people who are informing the left on AI policy are also ideological in a certain sense, maybe a more pejorative sense in this case. I think to the National AI Advisory Committee, which advises the White House, the Democratic appointees on that include a woman named Janet Haven, who runs this program called the Digital Society Initiative that includes a land acknowledgement for their servers. They call it a digital land acknowledgement. I was at a conference with her where she said that AI is just Web3 all over again, and we really need to be concerned about the race inequity and discrimination implications of AI. So I think that's the lens, the more ideological lens, the prism that AI policy will be passed through in a Harris administration. You already kind of see that as well with some of the policies that the Biden administration has already adopted, right? Because Biden made Harris his AI czar. She was the representative at the AI summit in the UK. Part of what came out of that was this AI Bill of Rights. And then that was somewhat formalized with the White House executive order, which included a directive to OMB, the Office of Management and Budget, which is sort of the brains of the executive branch. And now OMB has this directive that says any application of AI in government has to pass through this minimum practice of standards, and they include external consultations, audit of risks, ongoing quality management. And this applied retroactively to all kinds of existing uses of AI. So there's worries now that the post office, because the US Post Office uses a simple AI tool for routing mail, that this has to then retroactively go through this process and verify that it's not going to be rights impacting. And so in one sense, that is a kind of safety. I do worry that it's a form of safety that comes with the cost of dramatically reducing US state capacity and ability to adapt to new technology. Because on the ideological side for the right, there's been a long project to try to deconstruct the administrative state and challenge administrative law. And part of that is trying to figure out how do we actually make government more efficient and do the things that it needs to do without having a Trump person, what to call them, agents of the deep state interfere. And so that kind of energy can actually be applied towards more rapid diffusion of AI within government to the extent that we could either augment existing civil servants or substitute civil servants with AI. And in many cases, A lot of what government does is sort of a kind of fleshy API passing PDFs back and forth and checkboxes. Now that's a longer-term project. I think one of the ways you have to motivate and how do you actually get that project done is by building some kind of counter-establishment. And this is where someone like J.D. Vance may come in. I think back to the first Trump term, I was sitting in a Starbucks and this was during the transition. And Trump, of course, didn't expect to win, so they barely had a transition team in place. And then the news came out that—

This is the kind of preparedness we need for AI.

Well, this time it's different. This time it's different. But news came down the wire that none other than Balaji Srinivasan was on the shortlist for FDA commissioner. And I remember at the time I tweeted a link to his startup exit talk, Silicon Valley's Ultimate Exit. And I said, this is a surprising pick. Watch this infamous talk, and then my phone rang, and it was Balaji, and he said, "Infamous? I think you mean famous." And I had never—I'd never spoken to Balaji before at that point. I don't know how he got my number, but I was intrigued and started talking to him about like you know, and his background—you know—he's now known as a Bitcoin maxi, but his background is computational genomics. He taught the first comp genomics course at Stanford and developed a DNA counseling company with his brother. But I talked to him and he was like, what I want to do is— right now FDA has a pre-market approval regime where new drugs have to go through clinical trials, phase 1, 2, 3, and then pass through and then finally enter the market. This is anathema for personalized medicine. And if you just extrapolate falling costs of genomics and the potential for AI to revolutionize medicine in this way, what we need is a post-market surveillance regime. And there's a way I'm going to do this and we're going to use the post-market adverse reaction system. So anytime there's a new drug on the market that has this one-in-a-million interaction, doctors are supposed to report this to the FDA. We'll use that and a variety of other big data techniques and sort of Bayesian machine learning to have like a real-time update on the efficacy of a drug and how it correlates with idiosyncratic factors including your genetic profile. And I don't know whether or not he would have been able to enact that. We'll never know. But the fact that he was even under consideration was only because Blake Masters and Peter Thiel were running the transition. Now, I use that as an example of the kind of thinking that we're going to need if we're going to actually adapt and co-evolve with AI on an institutional level. We need the kind of community notes version of everything, where we move from a system where you have a bunch of people, flesh-blooded people, with their own ideological proclivities doing policy, deleting tweets, whatever the case may be, and transition that to something that is actually scalable. And that won't come through a stakeholder process, I'm afraid to say. I think it's only going to come through something that is a little more ideologically driven, provided that faction has the resources to call upon. Because right now, the problem with the Republican Party more generally is this education polarization means that they lack the kind of technocratic elites that you need to actually run agencies. And but the tech world can potentially fill that void.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Yeah, I mean, boy, you could paint a picture that sounds pretty attractive, but I kind of still come back to that doesn't seem to bear any resemblance to the actual Republicans that we have in today's world, right? I mean, This is sort of a— I think people are really investing a lot of hope in this sort of bromance between Trump and Silicon Valley and thinking that we're going to get a very high competence government out of it. That notably didn't really happen the first time around. I mean, with maybe a couple of— and I'm not somebody who can't recognise that there were some good things to come out of the Trump administration. happy to give credit for Operation Warp Speed. We deemed it that bad, and it's certainly very good that it happened. I don't know what the counterfactual would have been if that was a Hillary project. I think she was also pretty competent and very likely could have figured out a way to do an Operation Warp Speed. But who knows? We'll never know. But I don't I guess, first of all, still on just ideology for a second. What is the opposite of ideology? One way to describe it would be like being responsive to new evidence and being willing to update your beliefs on new evidence. Where I know you're not saying the pejorative form of ideology too much.

Sure, in the pejorative sense.

But this is where the rubber hits the road, though, right? The Republicans that we have are not those people.

And I wouldn't say they're not Ben Sasse.

Yeah, I mean, you just look at Congress.

I guess I think there's a version of the Republican Party that doesn't currently dominate the actual Republican Party that I think we could have this discussion about. But this discussion feels like it is sort of forgetting the fact that we have, by all accounts, a valueless, sort of purely self-interested maniac at the top of the party and a Congress that is full of people who are some mix of religious zealots on the one hand and sort of craven opportunists on the other hand, with a small section that is still sort of the old guard that's kind of institutionalist or is interested in a future of state capacity or whatever. But the actual Republicans in power, I don't see any signs that they're interested in state capacity. I see, like, they want to drain the swamp, they want to burn it down. I have no idea how to anticipate what they'll do with respect to AI, but I can't reconcile the people that are actually there, that are actually casting the votes, that would actually have the sort of top-level control of the administrative apparatus of the country with the characterisation of Chicago School or Balaji. I mean, these are very different people. Balaji wants to burn it all down too. If given power, I think he would do a much better job than a lot of other people. But they're not like— this project seems like quite fanciful, honestly. I mean, it seems like it would be great, maybe, but I just don't think that the people, as Americans go to the polls in a couple of months, I don't think the vision that you're giving them right now is actually on the ballot. I think what is on the ballot in almost every congressional district is like the person that Trump endorsed, who was the one that was like the most religious right, the most focused on abortion. I mean, that's what wins Republican primaries. These people are— they don't care about AI. They don't care about anything, right? They're outright— and they are ideological in the pejorative sense. They are not ideas motivated. I think if you visited every member of Congress, I would be very confident that you would have much more ideas-driven conversations with the Democrats than you would with the Republicans. The Republicans that you're going to meet actually in Congress are like ideologues. They are like, most of them are. A few of them, I think, are a little bit more sophisticated, but most of them are really not. And that's not to say all the Democrats are great. And certainly there's this sort of excess of wokeness over the last few years is well documented. But I still think at the end of the day, I think the real EA case, the real EA outlook, I think, is anti-Trump. And I think the real case for that is just like, we want people who are going to think carefully. We don't want people who are motivated by scripture. And that's like a lot of what the Republicans are. And scripture is just not going to be a good guide to a lot of these modern challenges, AI or otherwise. So I wish we had— I feel this dissonance where I'm like, as I listen to you talk, I'm like, man, that sounds pretty good. We could get FDA reform, we can get all these things going on and Chicago School, this and that, and great. And you're speaking to the 10 years ago me who was like, man, this Roundy guy makes a lot of good points. Right. But now we're like, that dude has been totally— he's like the last guy. One of the last few standing that— and he has basically no power, right?

Yeah. So I think this is a bit of a caricature, maybe through the media lens, media filter, because I work with congressional offices on a daily basis, both sides of the aisle. And I think you'd be shocked by the amount of ideas-driven hard discussion that takes place in Republican offices and how disappointed you'd actually be on the other side of the aisle. I've been in reading groups with Republican members where we read Joan Robinson's heterodox work on monopsony power and then we like, well, I know someone like Marco Rubio. I worked with Rubio and his staff on this Small Business Administration reauthorization where they wanted to turn it into a development bank to do hard manufacturing projects and stuff like that. Whereas what the SBA typically does is give loans to gas stations and nail salons. So that's another example of what I mean by they have a vision in mind and want to reform the agency, any given agency. And certainly I would agree that on the surface level, Republicans seem much more chaotic. They seem much more religious. I would make the counterargument that Democrats are also in their own way deeply theologically motivated. Maybe it's more of a progressive version of theology, but it's also one that is much more institutionalized rather than I think actually with the Republican National Convention, Trump made a gesture of actually pushing the social cons out of the coalition to some extent. So I think really what we're voting for— I'm not voting, I'm Canadian.

You're telling me he's not still playing to his beautiful Christians? I don't know if I quite believe that.

Yeah, he said the Bible's his favorite book after The Art of the Deal. And he couldn't quote a verse. I think in some ways Trump may just be an apatheist. He's not really— and I think he's actually kind of weirded out by them, right? Because what Trump actually is is a kind of populist New York City Democrat who has sort of paleocon views on immigration and trade, who's somehow found himself leading the Republican coalition supported by his cult of personality. And in some ways that's better than the alternative. If not for that cult of personality, then you might actually have to pander even more to those various interest groups. But even on this point, there's good eschatological reasons for the evangelical movement to be actually terrified of AI. You've seen Steve Bannon has done episodes on what he's called the cyborg theocracy and the singularity. Tucker Carlson has actually called for Bombing the data centers. Strangling it in the crib. But more fundamentally, when it comes down to if we enter a crisis, and I do think even though this is not on the ballot, I fully agree, I think we're sort of talking at a more esoteric level of what's actually going on versus what is projected to the masses who may just be voting on inflation or kitchen tabletop issues. I think to this recent Dwarkesh podcast with Patrick McKenzie, Where Patrick was explaining, you know, he, that he kind of ran vaccine logistics out of a Discord server during the pandemic because there was no one that was in charge, right? There was no adult in the room to say, you're the person running logistics. And so it ended up falling to this sort of polymath. And, you know, in that conversation, he said, you know, what should have happened was like when the pandemic started going on, like they should have said, you know, found the smartest person and said, you are now a colonel of the US Army. You have all the authorities of a colonel. Go figure out vaccine logistics and don't worry about racial quotas. Don't worry about like all this other junk. Like, let's just get shots in arms and like, let's not throw them into the waste because that's, that's, that's super stupid. And, you know, if, if this really is— if, if there's a reason to believe that like the next major technological inflection point in human history could could plausibly happen within the next 5 years. That means the next president could be presiding over an intelligence explosion or an AI takeoff or some transformative AI, whatever you want to define it as. And that could be a moment of crisis where you need to call somebody the smartest person in the room and make them a colonel to do whatever. And the question is, is that going to be Elon Musk or is it going to be designed by committee? And particularly when you start talking about what jobs, at least in the near term, are most threatened by AI. You listed some of the cost of these sectors. I don't think it's going to be that public schools adopt AI into the classroom. I think it's much more likely that they drag their feet and AI tutors end run the system and then we end up having a totally new kind of education system where people can just insource in the same way that we saw during the pandemic. The shift to remote schooling led to this 2 million student increase in homeschooling. There's going to be huge resistance from what you could call the professional managerial class to any kind of AI diffusion. And that's going to translate into a kind of decelerationism targeting use rather than actual safety. Whereas if we actually are in a crisis, an emergency situation, I would much rather have actual experts in the room, including someone like Elon Musk, who I know is incredibly polarizing, partly because He's a little bit on the spectrum and says things that he probably shouldn't and is not politic in that sense, but I think has demonstrated an operational and executive ability to handle very complex engineering problems.

Yeah. I mean, I guess again, there's a hopefulness to the description here that I have a real hard time really signing on to. I would agree that I'm a big Elon Musk fan, not of everything he's ever done. Certainly it feels like you have to sort of almost ritualistically say he shouldn't spend so much time on Twitter. And I would definitely advise him to get offline a little bit more often if I was an advisor. But yeah, I mean, he's clearly able to make things happen in the world. Not entirely clear that's what we'll need in a moment of crisis, maybe, but I'm not sure it's like a technical solution that is ultimately what we will need. It very much depends on the crisis, right? And when I think back to the pandemic, I recall these news conferences where we're getting like extemporaneous, oh, maybe we could inject bleach and whatever. And it's like, okay, sure, that was a lowlight, but It is a reflection of the fact that the dude at the top was not disciplined, was not ideas driven, was not in command of the facts, and just kind of went around making a mess for his people to clean up and kind of further polarising the issue. And I don't think it's hard to imagine at all a different president doing a better job on the pandemic than Trump did, you would have to imagine, I would think, that it would have become less polarised. Some of the fault of that definitely resides with Democrats. I do think that we did not need a statement from Kamala Harris that she maybe didn't trust the rushed Trump vaccine. That wasn't healthy. Unfortunately, nothing is healthy around Trump, though. That seems to be the problem, right? It's like, I do think when somebody makes everybody around them worse, both the people on their team who they're constantly throwing off of what they're trying to do and creating problems for them to clean up and distracting them, and the opposition that they're just driving insane. That's not a good person, right? That's not the person that we want to be in charge.

Oh, I totally— I'm not trying to defend his character by any means. Again, going back to utilitarian analysis, Someone like Norman Borlaug who kicked off the Green Revolution with his research into wheat varieties, he plausibly saved hundreds of millions, maybe even a billion lives incrementally through that research. He could have spent the rest of his life being a raving antisemite and playing the knockout game and punching babies. A utilitarian should still be happy that we lived in a timeline where he existed.

And likewise, But Trump hasn't given us anything like that. We don't— I mean, where's my green revolution from Trump? Sure, I guess maybe. But like, you know, if he were all those things and he had done the green revolution, I still wouldn't want him to be president. And Trump doesn't have a green revolution to his credit. I just don't— this— it feels like we're really being asked here to imagine a scenario that we don't really have evidence for. that it didn't happen last time, but this time is different, right? I think you said that earlier.

This time is different.

This time we're going to get competent executives to run all these things. I just don't see that that's likely to happen at all. If I did believe that was going to happen, I would have a very different outlook on this.

I think the competence of the first administration is underrated as it is. It's just was a rocky transition because, again, they didn't expect to win and had to, especially in the White House, have to staff up with a bunch of people who had very conflicted views of what to do. And the leaks were constant. I think that's the one big thing that will change. It'll be much tighter ship.

But in all the people, I mean, I'm not sure who you are referring to when you refer to the competence of the first administration, but there's like a long list of—

Head of HHS or Trump's NSC. I thought There are people in very important decision-making roles who were very competent and partly were able to express that competence because they came from a more right-center orientation. And again, going back to one of the things I think we need to do is try to make our beliefs consistent with each other. This is a big topic all the time when we talk about AGI timelines. Are you long the market or short the market? joining a 9-year fellowship program if you think AGI is 3 years away? There's all kinds of ways where our beliefs and actions may be internally inconsistent. But if you do believe that there is a technological inflection point within potentially the next administration, then all these things become open questions. It's not just a matter of, oh, we could prospectively make the FDA more efficient. It's that, no, we're going to have a Google Gnome project, but for drug discovery that is going to take us from 50 new drugs a year to 500,000 new drugs a year. And we do not have the state capacity. And state capacity in this context does not mean civil service protections. It might even mean the opposite. It may mean actually firing 70% of the people so you can bring in the 20% more who are the hyperscalers. And I think of this in terms of history because if you look at other major technological transitions, they have had major second-order effects on our institutions. The last major one we went through was the Industrial Revolution. In some ways, we're still going through it, but the Industrial Revolution took us from the world of 1900 to 1950 and the level of governance is like night and day. How did that transition take place? What we had was first in the early progressive era, the rise of the first large corporations, railway networks, the Carnegies and Rockefellers, those types of industrialists. And they developed new science of management. This was the first time where you could have big multinational companies and you had to understand supervisors and direct reports and all these basic things that we take for granted today. And that learning was then brought into government directly and indirectly through osmosis into how to build an administrative state. And we're going to go through another transition because AI directly implicates the machinery of government. And what we need is, and I use this more of a metaphor, but how do we get the Patrick Collison Secretary of Commerce? How do we get the Palmer Luckey Secretary of Defense? And that is much harder to see coming through the Democratic side of the aisle, partly because their horizons of possibility are much more constrained because in a weird way they've become the Burkean party.

Yeah, I mean, a lot of that holds water, I think. But then look at what all of the previous cabinet members are currently, and even the vice president. The number of people that were in the White House that served at the most senior levels under Trump in the first administration are now saying in very clear terms, this dude should never be back in a position of power. Right. He did this, he did that, all these things.

Right.

I mean, There's a laundry list of cabinet members that basically, to a degree that I would say is unprecedented in American history. I don't know that there's ever been a president, even Nixon, I think, mostly had people that were loyal to him and still thought highly of him despite his mistakes or crimes, if you want to be more blunt about it. But not a lot of returning starters, right, from the first administration. Most of those people, have fallen out of favor. They mostly hit their limit of what they were willing to put up with from Trump and finally spoke out about it, and now they won't be back. And so now we're going to have this whole other regime, and we can hope, I mean, if he does get elected, I certainly hope he picks wisely, and I certainly hope we get executive competence that can create great stuff for us.

We will at least get much less internal conflict, right? Because one of the ways that that lack of transition manifested was they had to pull in a bunch of neoconservatives and old Bush operatives and people who naturally reviled Trump throughout and had to basically falsify their preferences until they could achieve what they wanted to achieve. And then you have Miles Taylor and—

But isn't that what J.D. Vance is doing now?

I mean, isn't—

No, I think he's really doing that. He's very sincere.

You think he likes Trump now? I don't think he—

I think he's genuinely radicalized a little bit, maybe somewhat on some dimensions.

But you think he thinks Trump is a good leader for the free world after he said very clearly that he was not to be trusted?

Well, I think that was a reasonable— I think you actually tell a story where were coming through the Trump administration and seeing how things played out, that there is a case for that. Trump is the only person that could do certain things, right? He is the only person that can go before German members of the European Parliament and tell them that they need to increase their defense budget or they're going to pull out of NATO, right? I don't think he was ever actually going to pull out of NATO, but I think you need to have that credible threat. And certainly Trump was uniquely capable of that, right? But in another administration, he's not going to have this. It's sort of like if Elizabeth Warren became president and had to hire a bunch of Obama Democrats because she didn't have a transition prepared. You'd have a lot of people exiting saying that she's going overboard on things. It won't be the same that time around. Go ahead.

One quick analogy might be Elon and Twitter. A lot of Elon's former execs aren't going to say great things about Elon. He kind of had to clean house. I know Trump had more selection over those executives than Elon did, but he was fundamentally misaligned with the party, right? With Tillerson, with Mattis, with McMaster, with Kelly. He was a new regime. And the fact that they're not a fan of him in some circles is actually a plus that Trump didn't sort of bow down to people who are fundamentally misaligned with him. And you see very serious people like Bill Barr who are supporting Trump, who are aligned with him.

So he is bringing a new regime.

into the party, and that regime wasn't stocked up in 2016, wasn't ready.

Michael Lind, in his book The New Class War, talks about this as the first Trump administration was a counterculture. To be a Trump supporter, you were a counterculture. And what you really need is a counter-establishment, because a counterculture can't actually affect change, whereas a counter-establishment can. And a counter-establishment needs to be reasonably institutionally complete to actually move the ship of state because it's a 2.2 million person organization. There are tons of different agencies, tons of moving parts. You need expertise in administrative law and all kinds of stuff. And so you can't just do that based on cult of personality alone. And we have the green shoots of a new counter-establishment forming. And if Trump wins again, they will be able to solidify their position. And it won't be sunshine and roses overnight. It's not like we're going to— you can't rebuild the FDA or whatever overnight. But it does establish a new hierarchy within the right. And I think that it would be incredibly salutary, in part because when you look at who are the biggest detractors from the first Trump administration, it is the Mike Pompeos and so forth that were like, I didn't get a long enough leash on my kill list and I wanted to invade Iran, but all we had to do was a silly bomb. And they're mad at Trump, I think, for some of the right reasons.

Right.

Especially when you start talking about avoiding World War III, who else could broker a diplomatic channel with Xi Jinping? Someone who is aggressively hawkish on China rhetorically, but at the same time has this more transactional stance and wants to strike a deal.

But where's the deal, right? I mean, this is where we keep coming back to these stories. I noticed in the piece that you wrote, there was a link to A 2016 article from The Federalist, February 2016, and the link text is, only Trump can fix America's broken immigration system. But in reality, he was president for 4 years. He didn't fix it. Then he was not president, and there was allegedly— I mean, I guess we can't know for sure that it was going to pass, but by all accounts that I understand, there was a bipartisan deal to make some meaningful reform just in the last however many months. And what did he do? Comes in from left field and tries to destroy the deal, ultimately successfully destroying the deal because he wants the issue to remain live for his own electoral grievance campaign. So maybe he's the only one that can fix—

But that article was by me.

So you would not say he scuttled the bipartisan immigration deal? I mean, that's seemingly been reported A lot of places.

So the argument is that if we want to liberalize immigration in the long run and get to a more sensible system, not one that is based on dozens of different visa categories that each have their own bespoke interest group and that are easily abused, even like H-1Bs, as much as you may favor high-skilled immigration, as a Canadian working in America, the visa system is incredibly exploitative and Cluegeocracy. How do you transcend that? The Trump people wanted to do a points-based immigration system, and there was a proposal from Tom Cotton called the RAISE Act. But in the first term, Paul Ryan and what's his name? I'm just having a brain fart. The Senate Majority Leader both met— yeah, McConnell. Mitch McConnell both met with Trump in the Oval Office and said, look, I know you want to do immigration. We promise we'll do that the next Congress. This time around, we're going to do a big tax cut. And he got rolled. He had the majorities to do immigration reform in his first term, but he got rolled and he's not going to get rolled again. I think moving to a points-based immigration system would be beneficial in the long run, even if it came at the cost of short-run reductions in total numbers. Simply because it would massively clean up the system as it currently exists and lead to some semblance of order and control.

And points-based, just to make sure I understand, this policy is like you get points for having advanced degrees and for just having various positive traits that we want.

Right. Like a more Canadian-based system. And again, as a Canadian, I kind of like that system. I think it actually allows you to have higher rates of immigration in the long run because it gives a sense of democratic ownership over the policy rather than something that's— you look at what drives xenophobia and anti-immigrant sentiment, it's not necessarily rates of immigration, it's whether those immigration rates come through democratic consent or are a product of chaos and lack of control. And going to the point about this border deal, you don't want to put too much stock into this. It wasn't going to solve the problem. The main thing it was going to do was increase the number of judges for these asylum courts so they could process more of these claims. And the Republican argument was that this is another kludge, another patch to the system that is going to only further delay the more structural reform that we need because it's going to sort of take a— it's a release valve on the pressure for deeper reform. And this is actually a recurring pattern in a lot of how Republican votes get interpreted by the mainstream media where they say, oh, you're for the child tax credit, then why did you vote against it? And this thing was like, actually, there's more of a game theory behind this. And I'm not just like wishcasting or projecting. This is what they will tell you. Like, we want to get more deep structural reform done, and that will only be possible if we have like the motivation and the will to do that. And if we do little patchworks here and there that sort of give the semblance of fixing the problem, And that kills the political capital to get deeper change. And time and time again, from my personal experience working on the Hill, Republicans are the ones proposing actual deep structural reforms to things. Even something like healthcare, where Obama did the biggest healthcare expansion you could do without upsetting anybody. How do we do this big policy without actually doing anything structural? And that again is part of the problem, is all these policies get filtered through existing coalitions of interest groups that want a particular outcome without rocking the boat. And we're going to need to rock the boat.

Yeah, I mean, I think that time is coming. I don't disagree with that. My second son is named Teddy, in part after Teddy Roosevelt. And I think a lot of this comes down to, for me, do we have the right person? And Teddy Roosevelt did some amazing things that people thought somebody from his class or his party could never do because he was willing to cross those lines or be a— I wouldn't say he was a traitor to his class, but whatever, to be dramatic about it, you could describe him that way. Certainly he was willing to speak truth to his own constituencies when needed. I just don't see Trump as that guy. And I don't see the Republican Congress as largely full of those guys. And that's not to deny that you're having some substantive discussions behind closed doors. But I mean, at some point, if they're so substantive, why are they keeping it such a secret? Is it because they— I just have a very hard time believing that there's all that many of these Republicans who project as abortion-focused religious zealots. that are in fact very thoughtful state capacity structural reformers. I don't doubt there's a few, but I don't really buy that there's all that many. And I just don't think this is not the group that can reform America. They're like, if they are given the levers of real power, as we've recently seen, they didn't do it. They did a big tax cut. They didn't do the immigration. You know, this— and give me some, give me some relief too. You know, I mean, if we are in this, it's just also cynical, right? I mean, I, the idea that, okay, we're going to hold our, you know, we're going to hold this issue hostage and we're going to keep the pain alive because that's the only way we're going to get, you know, a real win in the future. I don't know.

How about Incremental progress?

How about taking wins when they present themselves? How about relieving pressure from a system that, by the Republicans' own account, is about to blow? If we're being invaded, if we have— it's recently been reported, I don't know if this is really true, that Trump is somewhat confused about asylum, the immigration concept, and asylums, the place that holds crazy people. And he certainly has said many times that they're sending criminals and crazy people into the country. So who knows what he believes? It's all very hard to read. But to bring it back to the AI question, I don't think it's a very healthy outlook on AI to say, oh, well, yeah, we could get a win here or do something sensible here, but it's only on the margin. Therefore, hold off or kill that deal because we want to let this pressure build until we can really do the right thing. That just does not seem like a healthy approach. I mean, Trump was not in command of the exponential of COVID I have the receipts for that, right? I mean, he was tweeting in February of 2020 that, oh, this is how many people have had it and only this many people have died, and compare that to the flu and whatever. And he just clearly did not get the fundamental underlying nature of the process that he was going to be forced to deal with. And if he brings that same sort of outlook to AI and is kind of like, oh, this thing is not that big of a deal, or I can sort of kick the can down the road, or I can sort of use this as like a wedge issue in the next election, I just don't think these are the guys. And they almost all are guys too, right? I do think we need to— we would do very well to have some meaningful reform in this country. But are these the guys that are really going to deliver us that reform? I still am just like, I don't see it at all. And I would rather kick the can down the road one more administration, frankly. I don't expect that the Democrats are going to change the system, but Trump seems to be a burn it down guy. I don't see the build it up part. I kind of think of him almost as like a tornado where it's like this extremely destructive force that sits at the middle of 2 giant opposed masses and is kind of this vortex of energy. It's largely destructive, right? I mean, some towns that got wiped out by tornadoes may years later look back and say, well, yeah, actually we have a nicer Main Street now than we used to because We had to rebuild, but I don't want to go through that process on a national level. And that seems to be the base case. I'm not, I don't know, I'm not sold on the—

So creative destruction is the engine of prosperity in the private sector, right? I guess Blockbuster had an opportunity to buy Netflix, but they passed on, right? But ultimately Netflix was a new company. That displaced Blockbuster. Uber and Lyft were new companies that displaced taxi commissions. This is how creative destruction tends to work. And creative destruction can be very dramatic and it does have costs, right? Like Uber example, there were taxi drivers in Paris that were throwing rocks off bridges and people committing suicide in New York because their medallion became worthless and so on and so forth. So creative destruction is dramatic even in the private sector. The problem is that in the public sector, we don't have a system of profit and loss that allows, you know, the NIH to fail and a new, better public health science funder to rise in its place. There's no competitive process. And as a result, we get this sort of buildup of entropy over time where we're kind of living on the capital that was developed from the New Deal era and then the Great Society eras. And That long 20th century is rapidly coming to an end, and there's going to be some necessary destruction to even begin the rebuilding process. And part of that, it looks like massive deregulation. It looks like gutting existing agencies, reallocating resources you want to keep, and setting a new framework. And again, it's much easier to do that if you have some philosophical motivation or some vision that you're trying to enact that's sort of detached or sits above pecuniary interests. And I agree, Trump on the surface certainly doesn't seem like he's totally in charge of the facts on any given issue. On the other hand, he does have this remarkable ability to absorb things and often he'll echo what he's absorbed in a kind of mangled way, but you kind of get what he's getting at. And one of the things that's leaked thus far on their potential AI policy is this executive order that's been drafted by the American First Policy Institute that calls for Manhattan Projects on AI, that calls for broad-based deregulation, and situates this under a framework of we need to make America first in AI. And speaking as somebody who took part in one of the Project 2025 policy committees, totally voluntary role, sort of an outside participant offering, submitting some memos on what their AI policy should be. I was actually quite surprised, and I can't really reveal too much, but surprised by how many AI safety-pilled people there were on these committees. We had actual memos on timelines to AGI, what are the risks, how do we respond. There's been some worry that the Trump administration would repeal the Biden executive order and then therefore repeal the compute thresholds for monitoring large models. My sense is that they will retain those provisions and do more of a repeal and replace, a day one executive order, keeping some of the monitoring capacity but getting rid of all the things that are hindering adoption within government and doing these Manhattan projects. Certainly within the EA world, there's a lot of debate about whether this is a good thing or a bad thing because you don't want AI to become A pure, a true arms race. On the other hand, my intuition is that if you're going to take AGI seriously and respond in an appropriate way, you at least need to believe it's a thing that can happen and have some realistic expectations of the timelines. And I can assure you that the people within the Trump world and within the Heritage world and who would be on his NSC and so on down the line are situationally aware in the Ashenbrenner sense. They see what's coming. They take it very seriously. Again, partly this is because a lot of conservatives come from, if they're not libertarian, they at least were forced to sit through some libertarian seminars at some point in their career. And there's some adjacency between the libertarian world and the rationalist world and the EA world where they think about these I came up through all that stuff. I was at Mercatus. We all read Jurgen Miklowski and studied the Industrial Revolution and Tyler Cowen's Great Stagnation. And we're always trying to think, where's our flying car? And that to me seems at least prerequisite to getting the policy right because at least gives you the sense of gravity. Now to the question of whether it's appropriate that we accelerate a kind of militarized approach to AI. I don't think that's what this is. I think overwhelmingly the talent and capacity to even build a superintelligence, let's say, is still in the private sector, and that doesn't seem to be going anywhere. And certainly if the Department of Defense or Department of Energy had some secret project, we'd probably be able to see it from space just because of all the energy it would need. But what we do want to be wary of is that to the extent that some of these dynamics are already baked in, it's probably inevitable that the intelligence community and security apparatus is going to become situationally aware if they aren't already. And you'd much rather that be done out in the open with some explicit conscious intent, and you wouldn't want to base your decision on who to support on something that was probably inevitable anyway. So then the question becomes, who gets there first? And this is the Ashenbrenner thesis as well. It's like, to the extent this is really baked in and the slow takeoffs are already taking off, do we want China to get there first or do we want the US to get there first? And if getting there first means having a decisive military, technological, economic advantage in perpetuity, I think you'd much rather the US get there first. And Moreover, if you're worried about Skynet scenarios, you probably also want the people who are governing that policy to be militantly anti-censorship and surveillance, which just so happens to describe a lot of the people in the Trump world who are terrified of big tech censorship and so on and so forth. And I think this is, even before we get to superintelligence, going to be a really important issue just because of all the ambient information we're constantly shedding. And the ways in which even existing financial reporting systems and big data that could be applied— we could have LLMs and multimodal models that are spying on everybody right now. NSA could have a dragnet surveillance program that is filtering data using an AI to detect whether it would pass a FISA warrant or not. And I would much rather have people in there who could plausibly pardon Edward Snowden than people— People who don't think it's even an issue.

Let me take it from the top. I think the top is if we are racing China to some notion of strategic dominance indefinitely, I think we're all kind of screwed. We are still living under the shadow of the last Cold War and the arms race. And I think the US government still has like 5,000 nuclear weapons that are in some form of operational readiness. That, first of all, by the way, raises the cost of creative destruction in the government sector a lot higher than in the taxi sector. Not to diminish the personal tragedies involved there, But I don't really think we have the luxury of a very chaotic creative destruction of the US government without it being truly catastrophic, just in virtue of the sheer weapons cache that we have that the president is, according to policy, independently authorised to make a decision on. Right. Now, hopefully there would be somebody in the chain of command or something.

Reforming an agency is much different than state collapse. I'm not talking about state collapse. That would be catastrophic if suddenly the US government just had total system failure and the world was flooded in deepfakes and drone swarms and we didn't know who was handling the nukes. That would be really bad. I agree. But I'm talking about something a little more mild. Right. And to me, again, as a Canadian and just looking at other parliamentary systems, it's totally normal in a parliamentary system for a new government to form. They literally call it forming a new government. And then for ministries to be shut down, new ministries to be created. My political idol growing up was Jean Chrétien. In the late '90s, he fired a third of the federal civil service in his unilateral budget. And so the US is uniquely ossified in this respect, partly because our system of checks and balances makes it very hard to do more dramatic kinds of change. And my basic thesis is just that this change will be forced upon us.

Yeah. So, I mean, we could eliminate the filibuster and start to have— I think there's all kinds of incremental steps we could take to unjam, to get ourselves unstuck. But I guess going back to the top, if we're racing China for indefinite strategic military dominance, I don't think that's a race either of us can really win, even if we last a while. There's various estimates, of course, of what is the annual risk of catastrophic nuclear war in today's world. But I think most people would have a very hard time saying it's less than like a 10th of a percent. And we're not dealing with it. We're just sort of living with that. And maybe it's even more than that. Maybe it's a half of a percent. I mean, these things really start to compound over time. AI feels like even if we can reach some sort of quasi-stable equilibrium, it's very hard to imagine an actually safe outcome that we arrive at through a race to strategic dominance. So I sort of feel like the only way that we get to a good place is to challenge that frame. And that's again where I just don't have— I don't have a lot of confidence in Kamala Harris about that either, but I have basically negative confidence in Trump. I mean, it seems like he will play the it's us versus China card to the end of the road, and he's not thinking far enough ahead to It's not going to be his problem. He'll just be dead, right? Presumably, unless it happens really fast, he'll age out. And the rest of us will be here for however many decades, and our kids will be here, and we'll be living under, at best, a sort of mutually assured AI destruction regime. And I just really don't want us to create that. If there was anyone who would say anything contrary to this notion that we have to race China for AI dominance, they would be my candidate. But Trump just seems like he's absolutely going to ratchet that up and he's going to play to those fears. And he's like, what is a Manhattan Project? I mean, the history of the Manhattan Project is not exactly a glorious one, right? I mean, it's a scientific triumph in some sense. But notably, a lot of the people involved immediately regretted what they had done when they actually unleashed it on the world. And then they were like, shit, can we get a mulligan on this? And you can't. So I don't know what Manhattan Projects we really expect, but it's kind of chilling to me to think that that's the sort of historical precedent we want to look back to, to say, back when America was really great in unleashing mutual assured destruction paradigm on the world. And I don't know.

I mean, the counterfactuals are always—

It created a century of peace, right?

Again, I look at the spikes. I don't know that it created a century of peace. I think it pushed a lot of sort of mid-level conflict risk into the tail. And the tail is potentially what ultimately kills us. I think all war is bad.

But do you think the A-bomb was going to be built at some point?

I don't know. I think that's very hard to say.

I would say there certainly are plenty of people who are more expert in this than I am, but I think there's at least a credible case that if the US had not built the A-bomb, that maybe nobody would have. Germany wasn't close. Nobody else was close at the time. The Soviet Union stole it from— stole the original IP from the US. There was no real other, as far as I'm aware, there was no other credible project that was on the path to the A-bomb. So maybe somebody else would have started one, maybe they wouldn't, especially if we had sort of gotten to a general period of relative peace. Who knows? I don't know. I think we can at least hope for a scenario where we don't end up with mutual AI destruction. I don't think we should take it as a foregone conclusion that, well, we're going to have mutual AI destruction. So MADE, I'm coining a new term here. We had MAD, now we're going to have MADE.

I think there's all kinds of ways you can deploy even existing AI in an offensive way. Stuxnet was, I think, initially developed in the mid-2000s. So there's lots of ways you can cause damage with existing capabilities. The way I think AI potentially gives us supremacy is not necessarily through wielding a superintelligence singleton that imposes US hegemony on the world. It's more that if we are the first to begin having a productivity boom, None other, where we have major automation of R&D and factories and services, that this gives us a huge dividend in terms of our economic prowess and lets us also do some of the defensive work. And I think if there's any ethic among the AI people in the Trump world, it is a kind of defensive accelerationist point of view. We need to have iron domes everywhere type of thing. If you look at what China's already doing, the main bottleneck for who gets there first is going to be energy, at least in the medium term. China is creating new coal fire plants and has tons of battery and electric capacity. Trump seems to be, again, situationally aware enough that he thinks that we need to double our electric energy production by the end of the decade. I think they'll do anything. That's going to require embracing natural gas and deregulating nuclear even more than has already been pushed for. And again, I think that's, you know, energy abundance is much more likely under a Trump administration. And I don't think this— I don't see this in terms of mutually assured destruction, more as how do we, you know, US infrastructure is already deeply compromised, right? There's tons of stuff that's probably burrowed into our critical infrastructure that China could turn on at any point. You know, the US government itself is incredibly fragmented. It's not like we have like some WeChat, single platform that everyone uses. It's like different agencies have different cloud contracts and Microsoft is constantly having leaks and so on and so forth. So I think it actually is quite existential for the US's position in the world. And I don't think it necessarily has to be a race to oblivion. It's a race to getting to a point where we can credibly defend ourselves against oblivion and have a degree of deterrence both technologically and through our economic strength. And I look at the political actors in the world right now and there's only one group that are actually sort of talking about this in a way that seems sort of at least proportional to what's coming down the line. And again, I would beg to differ on your characterization of who's been more hawkish on China. In some ways, I think that Biden administration has been much more escalatory vis-à-vis China, whether from Nancy Pelosi visiting Taiwan to the chip export controls, which I support. But we're sort of at a nadir in terms of our diplomatic relations with China, and that actually took place under Biden, not Trump. Trump cared about tariffs. He wanted a fair trade deal. He didn't really care a whole lot about human rights, and he said he had a lot of respect for Xi. He brokered a peace with North Korea, right? Like, he's like best buddies with Kim Jong-un. And so there's lots of ways I think a Trump administration on foreign policy could actually surprise us on the upside while still maintaining, you know, an America First posture.

I'd be interested to hear more of your take on the chip ban. I've never been sold on that policy, and I would say I'm broadly a China dove and quite just under— I feel like the case has not been made really that this is something that we need to do. I would totally agree the chip ban is very escalatory, and I'm not sure— it doesn't feel like the case has been made that it's actually going to help us. It feels to me like we sort of said this is like the one bipartisan thing, right? This is the a sad reality, in my view, that the one thing that the two parties can agree on is that we need to stick it to China at every turn. And to what end, really? Right. It seems like there is some sort of assumption in there that we're going to achieve some sort of kind of weaponized dominance, because if it's just about a productivity boom, Then they can have some productivity boom too, right? Why do we need to cut them off of the productivity boom? Like when you think about that from China's perspective, to me it seems that they can only interpret this as these guys will do anything to stop our rise, and they're probably trying to create strategic military dominance that we can never come back from. But at worst, they're just trying to stop our mundane application of development and application of AI. And that's also like a shitty thing to do if it's all kind of benign. Is there a story about this that makes sense that isn't predicated on the Leopold sort of narrative that we're going to use our— and there's a lot of chapters in that story, right? And they're not supposed to be playing out over that long of a time. It's like, we're going to first use our little lead to solve alignment, and then once we have this, then we can finally talk to China and have a nice conversation with them. And I'm just kind of like, maybe we could start having a nice conversation now. Maybe we could start building some trust. This is another issue where it's like, do I want to play the card of, yeah, we could deescalate with China now, but really it'll be better if we keep the tensions high. And then later, when the pressure is at its highest, that's when we'll have this great moment of reconciliation with China. That doesn't sound that great. I'd much rather pick up some wins along the way. But what would be your sort of non-made case for the chip ban?

Well, I view this as just securing American standard setting capacity over AI, right? Like, the fact that we're 90-something, something like 90% of cloud infrastructure and NVIDIA is a US company, gives us enormous leverage over how AI governance gets actually implemented at the compute layer. And if not for short timelines to AGI, I would probably maybe be against the export controls because they have been so escalatory and also easily evaded. But to the extent that they could add a year or 18 months or 2 years to our lead, I think they become potentially decisive, especially when you think about inference compute. Maybe we build the AGI in a lab and they steal the IP, but if they don't have the scaled infrastructure to actually commercialize it and make it useful, that gives us a huge upper hand. And this comes back to a line of thinking I've written a lot about in the essay series called AI and Leviathan, where I kind of see a kind of knife-edge scenario where AI can either go in a very totalitarian, authoritarian, panopticon kind of direction or into something potentially better and something that tries to preserve our liberty. That gets much more difficult when it is possible in principle to have a Big Brother watching you. What does that look like in practice? Well, the country that comes closest to what I think we need to become is Estonia. Estonia Because they had a blank slate after the fall of Soviet Union and a bunch of young policy hacker type people in government adopted all these e-government reforms in the late '90s, were the first to have e-banking. This was all built on a system called the X-Road, which was like a precursor pre-Satoshi version of the blockchain. So the Estonian government is like on this big distributed data exchange layer that has like an open API. You can develop tools for the same ID card that you use to ride the bus and pay your taxes and vote and start a business in a way that has the economies of scale of a big scale-up system, but with the civil liberties and privacy protections built in. The whole thing's like PGP encrypted and so on. So it's no longer getting to Denmark, it's like getting to Estonia, right? And I don't think we're going to literally put the US government on a blockchain, but who's actually thinking in those terms on that level? And they tend to be on the more libertarian right. They tend to be like the Balaji type people. And that comes with a lot of baggage, right? Because Balaji might be great on the FDA, but he's totally zany ideas on the Federal Reserve. And you can say that about a lot of different actors on that side. And this is why it becomes a high variance play. And I'm not saying this is dispositive. I'm saying that it's The gamble that's on the table. In the kind of political system we have, we have the turd or the douchebag, right? And we need to assess expected values over those 2 options. And one has a much higher variance, but I think a low-variance incrementalist approach is the default.

Right.

The default path, if you look out at the stars according to Fermi's paradox, is not a viable path. We need something that sort of takes us off that default.

Yeah, I would love to see something. I mean, I think I'm still inclined to fold this hand on this particular gamble and wait for somebody that I think I can actually trust. On the chip end, for A second more. I don't know if I quite find the standard setting argument compelling, and I'll try a different argument out on you. I don't know if I necessarily believe this argument. I broadly am very uncertain about a lot of these things. I guess the highest level summary of my worldview here, and the reason that I don't want to put Trump in a position of power, is that I don't think we know what we need to do. I do think we need to be very keenly in tune with what is happening on the ground and very responsive to new information, very willing to update our worldviews. And I just don't trust him to do that. And even if he has good people around him, I don't trust him to listen to those people. So that's kind of my Trump view. But now going to the chip thing and maintaining that same level of humility on. I really don't know if what I'm about to say carries the day or not. I think there's an argument to be made that China is not going to be denied AI. They will make their own chips. They've seemingly made pretty good progress on that already. I don't think they're caught up, but I've seen them build a hospital in like 7 days. I wouldn't underestimate their state capacity to really mobilise and create this industry if they need to. They also have tons of great AI researchers. So my guess is they'll figure out a way to have enough chips. And if we say to them, hey, you can't use the same hardware that we use, and then the next thing might be the sort of classification of research, then, and even by all accounts, even the frontier labs are getting more and more secretive all the time. Even OpenAI internally is now said to be quite secretive, to the point where most of the people that work there don't have all that much more information than we do about what the status of GPT-5 is or what Q* is or Strawberry or what have you. The more secretive we become about all that sort of stuff, perhaps that pushes China toward a significantly different branch of the tech tree. And I actually quote you on this from your '95 thesis. I thought one of the most compelling was that RL-based threat models have been prematurely discounted. I'm not sure that was exactly the right quote, but that's kind of the worry that I have for if we put China in a position where they're like, okay, we can only understand the United States as an adversary. They're trying to keep us down. They're trying to gain decisive advantage over us. They're cutting us off from hardware. They're cutting us off from research. we've got to go our own way. Who's to say that they don't spin up enough compute to pursue some, like, RL-based killer bot scenario?

Reward is all you need.

And then, yeah, then we're back to screwed again, right? And did we really set standards in that case? I would say that on the contrary, we sort of balkanize right now. We're still in this, miraculously, We're still in a zone where I see US-China papers every week, and it's often Microsoft, but it can be other things too, where we have leading minds across the US-China divide working on AI technology together. And it seems like we're going to end that. This trajectory is like, we're going to cut them off. They're going to have to go their own way. They will go their own way. Who knows what comes out of that? But the more we sort of aren't talking to each other and we're racing and we don't even have a great sense of what one another's technology progress looks like, it just seems like it feeds that dynamic where how does one pull back from that? The only way I can see to pull back is to begin to ease up on them a bit and say, hey, why don't we share some benefits sooner rather than later? Let's maybe try to stay on the same tech tree. If we were really all inclined to take AI risks seriously, one shared global tech tree that we can all kind of be on and understand. And the fact that LLMs have a decent understanding of human values relative to your— I know you understand this from that one 9-word sentence. These The LLMs that we have, for all their flaws, I can have a very sophisticated conversation with Claude about any number of ethical quandaries, and I would say it's more ethical than most people that I know. And there's like a very different type of AI that they could create to try to take an asymmetric strategy on us. And I just think we run the risk of In our effort to standard set, it seems like we run the risk of pushing them to a totally dark portion of the map that could have some very unpleasant surprises for us. I'm again, not super confident in all that.

I like export controls partly because they're kind of robust to many different worlds. Right. And we don't know what the final architecture will be for AGI, but my expectation is that it will still benefit from accelerated computing and scale. And if you want to have many AGIs that are revolutionizing the economy, you need lots of inference compute on top of that. And the export controls that were introduced in 2022 and strengthened in 2023, it's a complex threshold, but roughly around H100 level. Nvidia produced on the order of half a million H100s last year. This year they're projected to produce 4 million. And then the new series of chips coming behind that. And so there's been The thresholds where they're set today are exerting a cost on China, but it's a relatively manageable cost. That cost will escalate over time as the delta between where the threshold is set and the volume of chips being produced above that threshold increases dramatically. And so they're ramping up their efforts around smuggling and, and so on and so forth. So, you know, a big part of state capacity here is like investing in the Bureau of Industry and Security, which enforces export controls. So they actually have the capacity to monitor supply chains and clamp down on smuggling and identify new Chinese entities and add them to the list and so on and so forth. But I do think that the exponentials are really binding and they're binding regardless of what— if you take a more RL approach or transformer-based approach or whatever the case might be, it's all going to need hardware. I think there's a longer-term risk that this forces them to maybe down the tech tree of doing some energy-based chip, like an extropic kind of thing that's a totally new paradigm that induces a platform shift. I think that's a ways away. The Semiconductor Manufacturing Industrial Corporation, SMIC, the China state chip national champion, I think broke ground in 2000. So they've been working on this for 24 years and not only have they not caught up, I think they're still at least a decade behind. That's my impression. They have made progress with the 7-nanometer chip with Huawei that was basically repurposing old semiconductor manufacturing equipment. And so they're really— that was impressive, but they were really squeezing, you know, juice out of like existing equipment. And they're now completely cut off from like newest stuff. And those controls could go even further because ASML is still servicing their equipment and that equipment breaks all the time, right? And so I think it's much harder to just like indigenize your own chip production especially at the frontier. It's probably one of the hardest things any country could ever do. They've been trying to do it with chips, they've been trying to do it with wide-body airplanes, and they've failed on both accounts. So I'm not so worried about inducing a search down some dark path part of the phase space, at least on that dimension, especially when we're talking about potential transitions in the next 5, 6, 7 years maybe. Anything that pushes that lead out, I think, redounds to to the US just to give us at least some breathing room, partly because our political process moves so much slower. So going back to the question about diplomacy and having olive branches and collaborations, I'm all for that, especially before if it was done in a way that had some reciprocity and didn't have this asymmetric cost of the fact that China has no qualms about taking our IP and using these joint partnerships to basically transfer US technology into China and we get very little in return other than citations and our H-index goes up, but—

What could matter more than that?

Right. But the question, who could actually broker that? I think what pisses off China more than a tariff war is being scolded on their human rights record and questioned about their internal politics. And while Trump is hawkish on China in one sense, he comes at it from a more purely economic standpoint. He was on the same bandwagon about Japan back in the '80s, right? And for that reason, I think they could— yeah, again, this is maybe wishcasting. I think there's more opportunity, more potential to have stronger diplomatic ties while still turning the screws on the economic front.

Yeah, something feels a little— I'm not one who goes down this path of whose values are going to prevail. This is like the new framing that I am extremely allergic to. Now we got Sam Altman on this vibe as of recently with, I think it was a Washington Post op-ed, stark departure from what he was saying even just one year ago. I went back and looked at just one year ago. He had an interview in which he said, so often in the West, we basically act like there's no hope of cooperating with China, and I don't think we should give up on that too soon. And now he's in the Washington Post saying there's no third way. It's our values or their values. First of all, I'm like, what are Chinese values? Can anyone who's saying this articulate Chinese values? What are they? Do we have a summary? It feels like we're basically just talking about Xi in many cases there.

I don't know what China—

sometimes I say, like, 5,000 years of civilization can't be all wrong. I think there are probably some Chinese values that we would also find to be valuable. But then there just seemed to be something weird, too, about this. And I feel like this is kind of the theme that I'm just not able to get over the hump on is that somehow it's this big contest of values between our values, their values. We're the good guys, they're the bad guys. We're for freedom, they're for not freedom. But then we've talked ourselves into this position where Trump, because he's not going to push them on values, can actually do a better job of somehow getting to something good. Where I'm like, I don't know, it all seems very bankshotty to me. If we actually believe that our values are better than their values, then why don't we just—

That's all we get, though.

Well, I don't know.

We could play it straight. If I was president, my proposal would be, let's de-escalate with China. Let's try to stay on a similar branch of the technology tree. Let's have a shared sense of what we're building and where we're trying to go. And then let's tell them to let the people out of the camps at the same time.

And if they're annoyed by that, fine.

But I would think they would be a lot less annoyed by some chiding at the UN or whatever than they are by being cut off from chips. I don't know if you mean to say that you think otherwise, but if I think anything about Chinese leadership, it's that they're like, Quite practical. I would think they would be able to handle some rhetorical abuse if we had an actual technology sharing and just vision for a positive future of shared prosperity. How we get to this point where we're like, Trump is going to be better for our values because he doesn't even mention them to the Chinese. And so that's going to somehow lead to better relations?

I don't know.

Well, I'm not saying that Trump would be better for the hegemonic imposition of our values on them. I think if you want that, then you should vote for Harris, right? I'm saying it'd be in terms of having some separation, some firewall where we're able to preserve our values and build an open society without being overrun by an AI takeoff occurring in a country with very different values, where even however hard it is to define, they have in law that no AI system may be developed that does not reflect socialist values. I think it's something that we should at least take very seriously. And when it comes to multilateralism, I think if you look at the history of the last century, most acts of multilateralism are really just wrappers on American hegemony. There's been very few binding, strong multilateral peace treaties, agreements, arms control agreements, so on and so forth that didn't ultimately depend on US buy-in. And we only bought into those things from a position of strength.

Well, I guess 2 reactions to that. One is we don't have that level of strength anymore.

We are not in a position to dictate global terms for all that much longer.

It seems to me like China is going to have their seat at the table, barring some dramatic AI development that changes everything. And I guess I also think on the point of the idea that a Chinese AI would run off and impose socialist values on us, we are the ones doing this. They are not leading. We are leading. It's our companies that are talking about AGI in the next couple of years. It's not their companies.

I think we just need to look at ourselves.

We're the ones, if anybody's going to have a runaway AI in the short term, overwhelming odds would be that it comes out of the United States, right? There's not. So I just don't like the idea that we're going to hold up this Chinese socialist AI bogeyman as the reason that we need to do all these things when it's almost for sure going to be us. If anybody destroys the world with AI, it will be us. And I would say that's easily an order of magnitude more likely that it would be us than that it would be them. So we just got to keep that in mind, right? I mean, we're doing all these things out of this sort of paranoia of maybe that's too pejorative, but this fear that China has a policy about socialist values must be encoded in the AI. Sure. But if there's actually an AI that comes out of anywhere that's powerful enough that if it were to have come out of China, that it would impose socialist values on America, then we're probably all screwed, right? I think the general AI safety view on this is like, it's not your values or my values, it's do we survive this thing at all? If it gets that powerful, do we survive it at all?

And I think there is just some—

you look back in deep history, I always think about the— this is a bit of a non sequitur, but I always think of the oxygenation of the universe, of the planet, not of the universe. The oxygenation of the planet was the first mass extinction event. And I just think we have no way of predicting just how weird an AI future might be. It's probably going to be way— if it gets that powerful that fast, that in the next 4-year presidential term, something on the scale of a socialist AI taking over America is at all in the realm of possibility, then my sense is that the Overton window is way more open than that, and that it's going to be way weirder than socialist values. And whether the bot can talk honestly about Tiananmen Square is not going to be important anymore. Instead, we need to just get prepared for super weird, super disruptive. And again, look at ourselves. We're the ones that are going to do it almost for sure.

Yeah, you're kind of making the case for me though, because things are going to get super weird and super disruptive. And it's the Red Queen dynamic from evolutionary biology, right? You have to run just to stay in place. You have to co-evolve with the technology if we're not going to be completely subsumed by it. And that gets back to the discussion around institutions. Going to the question of my picture of the world is not that some hegemonic Chinese superintelligence takes over the world. I think that's maybe a little more fanciful than what I have in mind. I think more in terms of just raw economic power. Using that power to then establish some kind of new world order that is consistent with federated views of how to run their society. That's more or less what the history of the 20th century was. And AI could be that fulcrum for that prowess. And I totally agree. If it is possible that we could train a world-destroying model on a single GPU, then We're damned anyways, either way. But if there's a possible world where we could have location verification or other kinds of kill switches on GPU hardware that we have a de facto monopoly over and cloud infrastructure that has to run these systems due to their scale that has our governance standards within them, then we could potentially even export that technology to the global south and provide services that respect liberties and freedoms in a way that by default we wouldn't do because most of their telecom infrastructure is Huawei. In my own piece on the Manhattan Project for AI, I proposed— well, less a proposal and more of what I think is going to be a kind of inevitability— is a kind of joint venture between the major labs and the US government. Because the US government is not going to do this on its own. But if there is a risk that superintelligence gets developed in the lab and is super, super powerful, super duper AI, then I think it's imperative that that be done within some kind of democratic structure. We don't want OpenAI to become a sovereign citizen and to be a total new source of sovereign power vis-à-vis the state. And you start to see this a little bit already. OpenAI has announced that they're working with Los Alamos of all places on red teaming their models. Anthropic already has deep relationships with the national security establishment. And I think this is actually good. I think partly it's good because it shows that democratic institutions are now getting involved. Partly it's good because it shows that there could be a roadmap towards some kind of joint venture where the companies work together on some of the most sensitive parts of the research and have the scaling capital from the government that only government has to sort of have a glimpse into the future. But it's also good because whether or not we want this to be militarized, for better or worse, the defense vertical within the US government is the one vertical that has lots of discretion and ability to pivot and work flexibly. And that's why the Manhattan Project happened at all, was because it was sort of run like a startup. And if I look around the world, who is our Leslie Groves today? It's not in the US government. It's leading one of our major tech companies.

So what do you think is actually going to happen with— I mean, this is a hard question, but partly when you go through that analysis, I'm like, well, maybe we should just bring those interest groups back to the table. If we're headed for this Seems like you take all the way up to intelligence explosion quite seriously as a live possibility. I think if I believed that, if I was confident that was going to happen soon, I would be moving more toward advocating for a pause. It seems crazy to me that we would push to that level of power given our current level of understanding. I've been very encouraged over the last 2 years by how Much progress has been made in mechanistic interpretability, and we haven't seen the next model yet. So part of what's weird is we have this sort of— some progress is happening in the open and some progress is happening in places where we can't see it. So both with respect to open source versus closed source models, and also just capability of systems versus ability to interpret and control them, we have these major step changes seemingly happening, or this certainly has been in the past, and I am guessing probably will be again in the not too distant future. There'll be like another model that takes a big step up and it'll be like, okay, whoa, everybody has to recalibrate. And then in the meantime, that'll sort of open up a gap where it's like, oh yeah, no, well, closed source is way ahead of open again. And then open will kind of chip away at that lead until maybe it gets close. Right now they're kind of very similar. Same thing with sort of interpreting what's going on. There's been a lot of progress there for sure. But you can't interpret something you don't have access to. So by almost definition, if there are AI systems under development that can reason at a human level, only a very precious few people have had any chance to study how that works under the hood. We're at this moment right now where it seems like the interpretability has kind of caught up, but another shoe may be about to drop. At some point, I kind of expect myself to move into advocating for let's stop pushing the frontiers of this capability. It would be a mistake to race China. It would be a mistake to push it any farther, not beyond the current point, but beyond some point until we really have a robust understanding of what we're dealing with. And that's again why I'm just so allergic to all these other things where it's like, oh, China this, or whoever that. We're the ones that are going to do it. We're going to make the decision to either push this thing to the point where it becomes genuinely dangerous to us, or we're going to be the ones to decide we need to understand what we're doing better before we continue. Is that your worldview, or how do you—

Yeah, I mean, in my Manhattan Project piece, I talk about it more in terms of how to scale interpretability research and alignment research. And defensive applications. I think there's a subset of research that you could describe as gain of function that you'd much rather that be done in some kind of joint public-private framework that included very secure facilities, more top secret protocols in place, more compartmentalization. Obviously another leg of the Ashenburner Thesis is that we need much better operational security at these labs. And we can talk about the fact that OpenAI started to compartmentalize more and so on and so forth, but I still imagine that they're doing these pitch meetings on the new training framework from a conference room where their security protocol is to close the blinds. And so there's still a lot more to be done there. And I think that to do that, you have to start imagining worlds where— and to coordinate a pause in the first place, where the labs come together, the big 3, and agree to work jointly and are given some safe harbor from antitrust considerations and so on and given access to the scaling capital to actually see into the future. And I really think this is where there's a Yudkowsky view that scale is all you need to the point where simply scaling up a model, the N+1 model could spontaneously learn to wiggle psionic bonds and escape from the computer somehow. And I don't buy into those sort of more hard takeoff scenarios, but I do think there are worlds where we could trip into some kind of self-play mechanism or some other super scalable approach that immediately reaps the benefits of the scale we already have and will be scary powerful. And again, I'd rather that if that is the warning shot that leads us to pause, a pause will be much easier to coordinate if companies are already working together and in some kind of formal structure and have some, some kind of blessing from the, from the Defense Department, right? You know, the US Defense Department spends $1 billion a year on directed energy weapons. And, you know, I don't think we've ever actually used them. You know, these are like big microwave guns. You know, their main output has been to like generate conspiracy theories about Havana Syndrome and so on. So we pour billions of dollars into boondoggles that go nowhere. We could surely spend equivalent sums of money on a major alignment and peering into the future effort to try to understand where this technology is going. Because if you can scale the $100 billion model you get some glimpse of what will be possible in a decade for much less. Right. And that is only something that nation states can do.

So the argument there is peering into the future. Yeah. I mean, I would feel a lot better about that plan with a genuine, very stable genius at the top of the chain of command. who does this work and how competent they are and what is influencing their decisions and whether they have the courage to call a halt or to blow the whistle. I think so much depends on that sort of thing. And I just can't say, as much as I think I could imagine somebody I could be inclined to gamble the future on and trust enough to do that. It's like, this is maybe the— would be like at the very bottom of my global power rankings of people that I would probably bet on Xi first, honestly. I don't see eye to eye with Xi about a lot of things, and I don't want to live in Xi's China. I've said that many times. But who do you think is better able to oversee a private-public partnership to peer into the future and to channel billions of dollars of resources? Like Xi or Trump? Honestly, it's a sad—

I mean, that's what Operation Warp Speed was. Now, advanced market commitments are all the All the craze, but that was deep. That was like a Milton Friedman idea. Yeah. So I mean, we'll see. Civil-military fusion is happening on both sides of the ocean, and that's somewhat priced in. The question is, are we going to do this in a way that is proactive or reactive? And my sense is that if you have Elon Musk and Vitalik Buterin and some of these people who Could plausibly have the ear of the National Security Council in a Trump administration, whereas a Harris administration, well, superficially much more competent, much more credentialed and expert-driven, will be turning to the ethicists and major hospital networks and other stakeholders in terms of how they think about the technology. And so there's no genius without a little bit of madness, right? And it's a package deal. And I say that as somebody who temperamentally, I love walls, like the Paris VP pick. If we were at the end of history and I didn't think there was going to be another major technological transition point in the near future, let's have competent managerial people to increase SNAP benefits a little bit. All for that, right? But that's not the world we're running into. And there's a segment of the Democratic Party that basically sees itself as kind of managing a relatively stable social technical equilibrium. And that's just not the world as it really is.

Yeah, Lord knows. I mean, they may have a real wake-up call in store for them if they are actually in power as AI becomes what it might very well become. So, yeah, I guess the choice that you're putting to me is, would I bet on the Democrats to update their worldview in response to new evidence and kind of get it together on a rolling basis? Or should I put it all on black with Trump and hope that the right people are going to whisper to him in the right way and he's going to make the right decisions? And I mean, there's no, we don't have any great answers. Maybe we could conclude by drafting somebody for president. Do you have any, if you were going to put another name on the ballot, who do you think would be a good leader that has the right mix of grounded temperament, responsiveness to evidence? Idea-driven but not ideological.

Yeah, I think we put too much stake into the person. And that's sort of been the bigger message of this is we're not just voting for individuals. And with that, we're not just voting for their epistemics. We're voting for blocks of power, establishments, and the political economy that comes with that. So I think it doesn't even matter if Harris reads Superintelligence or The Singularity Is Near and has a wake-up. If the political economy of the Democratic Party constrains her in what she's able to do. And so I don't know who I would draft for my presidential pick. We had our chance of Andrew Yang. He was kind of ahead of his time maybe, but it has to be someone relatively outsider-ish, the kind of Dominic Cummings profile, because you need to be an exogenous force. If you're working from within an existing political establishment, you're going to be tied and constrained in what you're able to do because those is sort of the classic innovator's dilemma, right? You're making incremental progress while the world changes around you.

What do you think happens if Trump wakes up and says he wants to nuke somebody? Do you think that he can do it or?

I don't think that's him. I think he's actually a bit of a pacifist. Like he's—

Well, okay.

I mean, we're projecting a lot of hopes and dreams onto Trump during this exercise. I hope that's right. I'm not so sure. I think when he bombed Soleimani on the runway, he had no idea where that was going to go. And they wanted to do a lot more than that.

Right. And you had Tucker Carlson on the phone with him pleading him not to invade Iran.

But, I mean, I don't know. We could even cut this off.

He wasn't going to in the first place. Right.

We could even cut this off the podcast. I don't I don't know to what degree I believe the individual matters or not. I think the line that the public is broadly given is that the commander in chief makes these decisions and that we're ready to respond and that we carry a nuclear football around. And it is supposed to be one person's decision. I guess you could say, well, that won't happen. But Trump's done a lot of crazy things. That's not really in dispute. He's said a lot of crazy things. If he were to say something like that, do you think that the chain of command acts on his instruction or refuses him in that moment?

I mean, this is a total hypothetical. I don't think this would happen in the first place, right? His number one priority at getting into office is to broker a peace with Ukraine and Russia. Right. And I think he's uniquely capable of doing something like that. He's terrified of World War III and nukes.

He's like, I mean, he was threatening nukes against North Korea. He's gone further than any other president has gone in invoking— he understands madman theory. He is a madman. He has gone further than— I think it's undisputable, right? Yes, he's definitely convinced me that he is a madman. He has gone further than any president in 60 years, at least in terms of threatening nuclear war. I would say that's clear.

He walked across the DMZ and now they're like bosom buddies. I think Trump has psyoped the world into thinking he's one thing when he's really not. And You look at the actions, again, like the NATO example, how are we going to actually get Europe to pay their fair share if not to make some kind of credible threat? And how do you motivate that credibility without having some kind of actual capacity to pull out, or even if it's feigned? Same with the Taiwan situation. We've benefited from a policy of ambiguity, whether it's its own independent country or not. But those are the kind of things, the sort of game theory that having someone like Trump, I think, is actually an asset. But again, I don't think this is where things are going to hinge. The president obviously has a lot of unilateral power. Individual decisions made by a president can reshape the world order. On AI policy in particular, I think it's going to come down to more how things are delegated. And if you had to appoint a colonel, in a moment of crisis, who would that be type of thing? And what sort of broad social base are you drawing on? Are you drawing from a bunch of ethicists and sociologists and union representatives or people who actually know the technology and know how to scale large organizations and large responses? Trump's CTO, Kratios, is now head of policy at Scale. He ran his OSTP. These are very, very smart, capable people who I think are below the fray in terms of what you see on CNN. You don't hear this level of granularity and how the Trump administration the last time around actually operated. And it was much more rational than sort of met the eye, and that was partly Trump's own fault that that didn't get communicated because he was again telling people to drink bleach and mired in lawsuits. But look at what he did in spite of that.

I do want to be mindful of time. This has been a great discussion, but maybe Nathan will give you the last word and we'll wrap on that.

Boy, how to land this plane after all this.

There are 2 pills before you.

I'm inclined to gamble at some point. I think that there is a strong case, and I've been one to make it in years past. I've only kind of stopped making it because of the current crop of leaders of the Republican Party. I buy a lot of the analysis that we're not going to handle a technology revolution by calling in union leaders from every different sector to ask what they think should be done about it. I definitely buy that. I think it's undeniable that we need a reform of many parts of government. It seems like you do need somebody who has some sort of independent power base to do that. It can't just be a total— or it seems unlikely that it would be a total creature of the machine that can affect those sorts of big changes. So I do buy a lot of the overall story that you're telling. And I think for me, it ultimately does come down to, do I want to bet on this now? We don't have full control of the timing. I've said to people in the past, I'm sorry that the timing of the AI revolution is in convenient for you. So I can't move that timeline. But, man, do I want to gamble on this group to do this? Do I want to put Trump in the ultimate say-so position, or would I rather keep with the incrementalists for another term, hope that they'll rise to the occasion if things really get crazy? and hope that we might get a better reformer at a future date. I can't get myself to want to bet on Trump.

Well, I'm Canadian, so I can't vote, but you're in Michigan where your vote actually matters. So think hard.

Awesome. Well, that's a good place to wrap. Sam, Nathan, this has been a fantastic conversation. Thank you as always, and until next time.

Thanks, guys.

Thank you.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.