AI Meets Geopolitics w/ Samo Burja and Nathan Labenz

Hey, Upstream listeners. Today we're releasing a conversation between Sam Oburia from Turpentine Show: Live Players and Nathan Labenz from Turpentine Show: The Cognitive Revolution. They discuss how AI will impact geopolitics, the effects AI will have on scientific progress, AI progress in different countries, and more. Please enjoy.

Sam O'Buria, founder of Bismarck Analysis. Welcome.

Great to be here.

So you are known for your strategic analysis and for your super crystal clear monologues. I'm an active listener to your show that you do with Erik, Live Players, and definitely recommend folks check that out for in-depth analysis of many of the nation-state level live players. Obviously, one of the hottest topics in the world today, and one that is gradually infusing all of geopolitics, is AI. That's a many-dimensional topic, but I wanted to start off before we go deeper into different live players and what their interests and positions and strategies are likely to be, just with a little bit of context on kind of your AI worldview. I find that many conversations fork early. On questions of just like, what do you actually expect to happen? Do you think that AI is going to get super powerful or, you know, not so much? And we'll leave any debate of that topic for another time, but I just wanted to at least give people a sense for kind of what your outlook is so that they can, you know, better contextualize the rest of your analysis.

I think that artificial intelligence is a fascinating test for people. It really, whenever you talk about artificial intelligence, you end up having to look at basically all of the world models that one has. You have to employ them, you have to reconsider them, you have to check which assumptions hold and which assumptions should be dropped. I think it's, uh, fascinating how much people's expectations of the future of AI reflect their general thinking on technology, or even their general thinking on what is a mind, and ultimately what human beings are and what is special about human beings. You might have people who greatly focus, say, on the lack of agency that the current AI models display, right? You might say that actually that rather than intelligence is the most valuable thing. And then others would brush that aside and say, you know, all you really need is a for loop on, you know, already there's generative intelligence. First off, that's shown to be not quite true, but it is a fundamental question of whether something much, a very different process would be needed to produce agentic AI, or whether it's just a small tweak that we're going to get through, through the normal application of generative AI. So first, to get it out of the way, I think there's been real scientific progress in the early 2000s on just the computer science of artificial intelligence. The term AI scientist did actually make sense then. It still sort of makes sense. I think since the last 4 or 5 years, there have been new breakthroughs as the larger and larger models have been trained. However, to a great extent, we're merely applying these theoretical insights from the early 2000s. That's why there is a lot of disputation as to who ultimately should properly be credited for the so-called transformer architecture that is now used to basically generate most of the text that's behind LLMs. LLM is in a way a much less descriptive marker, right? It's a marker for an application rather than the technology itself. It's like talking about a car when of course there could be an internal combustion engine inside the car or there could be an electric motor. I'm certain that now that there is demand for basically generation of text with comes with generation of knowledge, that there will be architectural revolutions that will optimize for it even further than what transformers have done. So I do expect some continued progress in sort of the science of artificial intelligence, that is the theoretical side, the architecture side, and the discovery of what do larger and larger and different kinds of datasets do. We are in a way discovering it. We don't have strong theory behind what's happening. That's an important thing that has to be stated. This is not similar to the development process of, say, supersonic flight, where let's say 80 to 90% of the science is just resolved. We have our models of physics. It's not pushing physics that much forward, except maybe some details of fluid dynamics, which are— chaotic. We're just doing engineering on the basis of a theory. Here, the engineering has outpaced theory long ago, and we're continually surprised by what are essentially not just business ventures. They're not just launching an app. A training run is actually an experiment. I think we should think of it as if we've built dozens of Large Hadron Colliders around the world. You know, the scientific experiment on the border between Switzerland and France at CERN, where they accelerate particles at great cost. Now we're kind of living in this world where, you know, $100 billion valuations have become boring for some of these new software companies. No one's that excited. They're interested when yet another AI scientist goes off to fund their own lab. But that seems like table stakes now. So these very, very large training runs are legitimate, in my opinion, scientific experiments. They're not just normal technological development. Now, why did I talk so much about sort of the epistemology of this? Well, I think we have to be extremely cautious when making predictions. I don't think this is actually like Moore's Law. At all. I think that we are continually surprised by what these systems do and do not do. There've been already so many deviations from the so-called scaling laws that I think we have to admit that that was perhaps not quite the correct hypothesis, right? To give an example, already we're not seeing huge improvements in output for improvements in compute. But we're suddenly finding ways to reduce the amount of compute to get the outputs that we already had. So in a way, AI is becoming more compute efficient. That's not what people were predicting 6 months ago. That's not what people were predicting— experts, by the way, or at least entrepreneurs, let's say, if we make a distinction between expert and entrepreneur— 18 months ago. So truly, I think we do not understand the fundamental science of any of this. And I've tried, you know, in my own thinking, my research, both for personal interest and in a professional sense, deeply interviewing the best AI scientists I could find. And they have no consensus. They have no consensus with each other. So I think the enthusiasm over scaling laws was trying to invoke something like, oh, we kind of understand the physics of semiconductors. So it's only a matter of Moore's Law is kind of like Intel's business plan, right? And it's an economic and social and technological law. But here we could encounter scientific barriers at any point. So, you know, there is no guarantee that simply throwing more compute will continue to work. We can expect more and more compute to be available. But perhaps as soon as there's enough data on what these training runs are doing, there will be some sort of scientific understanding of what this is. And I think that, you know, in computer science, there's always been this interesting question of how much useful computer science is there actually, and how much is it just engineering and just application of the computer science to new and new problems. Here we have an open wide field of computer science. So this leads me sort of to the second point. Why is it important to carefully think about the theoretical underpinning, not just the empirical outcome of AI? Well, this is my theory of progress. I would literally not even track who has the most compute. I think that's important to know. You should have a little chart, a little table with the best information you can find. You need to update it not quite quarterly, but you need to update it every 6 months because it changes pretty fundamentally. And people are going in depth trying to find all of the compute they can, all of the GPUs that they can. They're trying to strike deals, favoritism, all of this corporate politics stuff. Super interesting. Yet at the end of the day, I'm not going to bet On the company that has raised most money to have the most compute. I'm going to bet on the company that has the best AI scientists, right? So Demis Hassabis, for example, at the end of the day, when you read what he writes, when you look at what he says, he ultimately wants to answer questions about the nature of intelligence. Ilya Sutskever. He almost has this like religious air about him when he talks about the models, because to him, this is about the beauty of the universe. This is about the beauty of intelligence, right? This is about the discovery of it. And I think when you hear scientists talk this way about their craft, about their field, you realize that they have a very deep motivation. Now, this motivation might be assisted by a financial motive.

Mm-hmm.

A profit-making motive. But let's put it this way. If I see an AI company where everyone is just trying to make money, and you know, it is possible to make money with AI, it's harder than people might think, by the way. Most of the AI companies are clearly losing a lot of money very quickly. So I expect most of these to fold within the next few years with the giants remaining because the giants just have way more capital to throw around. Like whoever's working with Amazon. Cover's working with Microsoft and of course Google itself. They're not going to run out of money, right? They're basically like Saudi Arabia. Saudi Arabia can build a line in the desert, right? That giant skyscraper that just goes through the desert, this pharaonic gesture. Well, Google can build like pyramids of compute and it would still be among the most profitable companies in the world. Microsoft can build great cigarettes of compute And they could still be a very successful company. So that's important to keep in mind who has money to burn, but merely having money to burn does not produce the breakthroughs. The breakthroughs will be actually from who has the best AI scientists. And I do mean scientists. I do not mean software engineers. I don't mean data scientists. I mean people who you can figure out. Have both an intellectual excellence, an in-depth passion, and are not easily confused by marketing or social pressure. There has been in Western civilization a fundamental tension between science, engineering, and commerce. The commercial mindset always involves talking up the current business plan, talking up the current approach to things. Science involves undermining it, right? Science involves questioning the current approach. Science involves disproving the current approach. So you can see how in a corporate culture these two can be in tension. And most people, also very intelligent people, if they go to elite universities, by default they're trained in agreeability. The purpose of the educational system is to train you up in agreeability. We basically destroyed the Prussian science university It took a century for the destruction to happen. But when we made the Prussian research university model, when it was adapted, adopted in the United States and adapted to scale to most of the population, it obviously can't scale. So something that was designed to sort of protect your PhD autists ended up being a degree mill and ended up being a necessary career step, not just here, but back in Germany, And in the United Kingdom and in France, and actually now more and more so in China. If the talent distribution mechanism of your society is the same thing as the scientific mechanism of your society, that is also a contradiction. And that is a fundamental problem of Western civilization. We do not know what to replace universities with. So why am I talking about universities in the context of AI? Like I said, AI is a good test. You have to go deep. So many of these top AI scientists were poached from universities that had good computer science departments, but these weren't even first-tier universities. Can you name a brilliant AI scientist from Harvard? Has a background, did a PhD there or something? It's kind of interesting. You see that Canadian universities punch above. Their weight. MIT, well, that's pretty predictable. British universities, French universities, for some reason French mathematicians seem to have a knack for it. Anyone who was trained in the former Soviet Union, for some reason, you know, and by the way, that's not surprising. A lot of the mathematics underpinning modern models was actually developed in a compute-starved environment as academic hobbies. for mathematicians who are kind of trying to cope with the lack of good computers and trying to figure out new and more efficient ways to do things and working things out on paper rather than on huge training runs. Now, that tradition has developed immensely in the 1990s and the 2000s after the fall of the Soviet Union, but it's notable that a lot of these, like, academic lineages go back there. So, After the launch of ChatGPT, I think we have stripped mind academia for all the good AI scientists. I don't think you have any truly knowledgeable questioners of the nature of the universe of the Ilya Sutskever tier left. I think they've all been stripped mind. There was just too much money. There are mathematicians who for ideological reasons were not inherently interested in AI. But were scared of AI by basically Eliezer Yudkowsky and basically things around that. Arguably Terence Tao is in that category to some extent. And those mathematicians, I think, will make modest or decent progress because ultimately, if you have a holy terror of AI, which causes you to change your mathematics, what you apply your mathematical expertise to, I think it can make you a very good engineer, But it makes you like kind of a mid-scientist. I think the curiosity, not the terror, is the actual underlying deep motivation which causes these like very flawed human brains prone to various interesting cognitive flaws like wishful thinking and so on to actually end up locking onto truth. This means that if someone wanted to, they could compile a list of all the AI scientists that are likely to be paradigm breakers, you wouldn't get 100% of them. You could probably get 90% of them, and you can chart which company has the most. And at the end of the day, you can chart which company is likely to not be constrained by needing to develop a product. Let's remember that this breakthrough of application of of ChatGPT came out of a company that people used to make fun of for releasing white papers instead of products. Like, this is important to remember before it's memory holed that OpenAI, you know, it wasn't just a corporate drama that came out of structuring it as a nonprofit. They basically gave the best AI researchers they could find tenure. Like, they basically were just publishing white papers.

Yeah.

And they were working very hard and they were motivated by this vision that AI will transform the world. They were recruited by literally Sam Altman and literally Elon Musk working together. Like that's in the prehistory, including things like giving Tesla gifts to the early employees to encourage them, right, to go do that. And I think that cannot be recreated in the current environment. So it's sort of decaying, right? And if you look at the key people like Dario Maday and like Ilya Sutskever and, you know, maybe others in the future, this dream team has sort of split apart. Now, I think they have likely trained up significant human capital. So anyone who was working with them in that initial period before the entire planet decided OpenAI was a huge success. Those people, if they were AI scientists, not software engineers, I think they're underrated. And I think people should be trying to poach those because, you know, like I said, academia is strip-mined. Which places were still producing and training people in the scientific mindset rather than an entrepreneurial mindset, a hype mindset?

We'll continue our interview in a moment after a word from our sponsors. How deep do you go to seek out an answer to a question? Maybe you've spent hours clicking the source links on an obscure Wikipedia page, or maybe you're even the type of person who checked out the entire shelf on the topic at your library. If you're nodding along, then check out GiveWell, an organization that researches questions about global health and philanthropy, even if a satisfying answer might require years of reviewing studies, talking to experts, and over 300 footnotes. GiveWell has now spent over 17 years researching charitable organizations and only directs funding to a few of the highest impact opportunities they've found. Over 125,000 donors have used GiveWell to donate more than $2 billion. Rigorous evidence suggests that these donations will save over 200,000 lives and improve the lives of millions more. GiveWell wants as many donors as possible to make informed decisions about high-impact giving. You can find all of their research and recommendations on their site for free. You can make tax-deductible donations to their recommended funds or charities, and GiveWell doesn't take a cut. If you've never used GiveWell to donate, you can have your donation matched up to $100 before the end of the year, or as long as matching funds last. To claim your match, go to givewell.org and pick podcast and enter Econ 102 with Noah Smith and Erik Torenberg at checkout. Make sure they know that you heard about GiveWell from Econ 102 with Noah Smith and Erik Torenberg to get your donation matched. Again, that's givewell.org to donate or find out more. Hey everyone, Erik here. In this environment, founders need to become profitable faster and do more with smaller teams, especially when it comes to engineering. That's why Sean Lanahan started Squad, a specialized global talent firm for top engineers that will seamlessly integrate with your org. Squad offers rigorously vetted top 1% talent that will actually work hard for you every day. Their engineers work in your time zone, follow your processes, and use your tools. Squad has frontend engineers excelling in TypeScript, React, and Next.js, ready to onboard to your team today. For backend, Squad engineers are experts at Node.js, Python, Java, and a range of other languages and frameworks. While it may cost more than the freelancer on Upwork billing you for 40 hours but working only 2, Squad offers premium quality at a fraction of the typical cost, without the headache of assessing for skills and culture fit. Squad takes care of sourcing, legal compliance, and local HR for global talent, Increase your velocity without amping up burn. Visit chewsquad.com and mention Turpentine to skip the waitlist.

Let's be real, like, I feel like generative AI is now as much of a buzzword as Web3 ever was. It's Web4 basically, right? It's like what is supposed to save the balance sheets of all the investors and all the major corporations and actually BlackRock. And when you add it all together, really the US has no plan for solvency except for an AI revolution, right? Like that's the only way in which the Western world can achieve collectively 4 or 5% economic growth per year. Now, of course, if AI worked in this promised way, we wouldn't have 4 or 5% a year. We might have like 10% growth a year or 20% growth a year, truly singularitarian numbers. But, you know, if it works only the way it's working right now, I don't think there's any chance that this results in a 5 or 6% rate of GDP growth. There's no chance it automates labor fast enough to counteract the global labor crunch. And I don't think we are focusing enough on the science of artificial intelligence and asking the deepest questions. And investing in the most esoteric mathematics. One way to put it is that imagine a world where it is easy to raise money to build particle accelerators and employ experimental physicists, but it is difficult to employ and protect for longer periods of intellectual development theoretical physicists. That is a world where we would hit a wall in physics very quickly. And I think artificial intelligence, it's much closer to physics than it is to, I don't know, fintech or Moore's Law or something like this. So that's my sort of contrarian thesis. We've undervalued AI science. We've overvalued AI compute. We overvalue raising huge amounts of money to like quickly enter the AI race and undervalued longer-term plays to cultivate top AI talent that is intrinsically motivated by their curiosity to push forward this frontier of understanding. And I think it's just because a single AI product went viral. And here's the spiciest take I have, my spiciest take. Had OpenAI not gotten ChatGPT, had ChatGPT not been viral for whatever reason, Probably 2 or 3 UX decisions could have killed it, to be honest. It would've enthused experts and some top programmers, but the general public would ignore it. I think we'd actually be making more theoretical progress because the dream team would still be together and the intent would still be on the white papers. Ironically, you should interrogate these 5 or 6 heresies because I don't think anyone talks about it this way. I've been like surveying the entire discourse landscape and I see very few people Arguing these points, so I'm happy to be challenged on them.

Yeah, I think there's you've touched on like ten different things that could be full discussions unto themselves. Some of the ones that I think I agree with, assuming I'm interpreting you correctly. I always say the transformer is not the end of history. Yes, and it's important to remember that. You know, we've had for much of our lives the sort of end of history vibe, and obviously we're. now shaken out of that. I think the AI community, maybe not the top thinkers, not the top scientists that you're talking about, but the community at large has experienced a somewhat similar thing where there's been this sort of, well, we just scale up the transformer and we'll keep doing that forever, and that will deliver the sort of transformative outcome. And I personally don't rule that out, but I expect that there will be more theoretical and like, we'll probably never find out because I think we will see more theoretical advances that will mean we're never gonna run the trillion-dollar transformer experiment because there will just be other things that will be sort of better options.

It's good to remember that diffusion models are what's generating all the images. It's not transformers. So what is it if transformers are fully general intelligence in the bedrock of Why are they so much worse at generating images than diffusion? I think that we are seeing the Cambrian explosion of the different definitions of intelligence, right? The Cambrian explosion with this event on prehistoric Earth hundreds of millions of years ago, where suddenly there was a vast diversity of multicellular life that came to be. It's like visible in the fossil record. I think for most of our history, we have had only only human and animal intelligence, and we added a tiny bit of computer intelligence, which I think is different in kind than human and animal intelligence. And now we have added even more forms of computer intelligence, right? I think that already the Turing machine was already a form of artificial intelligence. I think we're eventually going to have a periodic table of elements of different things that can be called intelligence. That are fundamentally different processes. So I think we'll have qualitatively different processes that can produce outputs that we would call intelligence, but they will be fundamentally different processes with different properties, different strengths, different weaknesses. And I think that'll be surprising to us because we're used to thinking of intelligence as a single thing because the evidence is pretty strong in human beings. do psychometrics and cognitive science, that humans are just kind of doing one thing super special. But, you know, maybe the universe doesn't work that way, right? Maybe human intelligence was like the hydrogen atom, the first atom that comes into existence in the universe. But then very quickly, the stars start making helium and iron, and then the supernovas start making uranium and all these heavier elements that mere star fusion can't produce. So I really do think that we are going to be continuously surprised at the strength, unreasonable effectiveness of various forms of intelligence as we discover them, but also surprised and shocked by their weakness, right? Why do AI agents, when you put generative AI, when you put transformers on a loop, why do they melt down into incoherence and don't pursue goals and cannot actuate and cannot act on the world. Like, that's a surprising thing. I don't think people expected that. If you told someone 20 or 30 years, the kind of verbal behavior this intelligence is capable of, they would assume it's trivial to use the words to command a robot and have the robot do things, right? I think they would assume it's trivial, but it turns out it's not. Or they would assume it's trivial to have a representative. Wasn't that the thesis of like A Hunger at Startups? 5 minutes after ChatGPT came out, the personal assistant, and they're just not, you know, you can use them, but you can use them the way you can use a Calendly link. You can't use them the way you can work with an executive assistant, a human executive assistant.

I really agree with, and I think you're sharp to emphasize, I use the term Cambrian explosion sometimes as well. I also have started to use the term mixture of architectures era. to signal sort of the end of the transformers era and the beginning. And I think we're just very much still in the beginning. But yes, folks who listen to my feed know that I have an obsession with state space models. And the big reason for that is that I see it as one of the first new core mechanisms that performs basically on par with the attention mechanism, but notably, as you said, does have different strengths and weaknesses, which you sort of have to drill down to a very small level to really distinguish. It's quite interesting that at the high level, when they train like a state-space model versus a transformer model at scale, they look pretty similar. Like qualitatively, you can chat with them. They seem to have reasonable memory. When you drill down into the micro skills, you find that there are kind of two sides of a coin, sort of different strengths and weaknesses. And not surprisingly, based on that, some of the most exciting new things coming out these days are hybrid structures where you have an integration of these things. When I just think about, and I'm no expert in neuroscience, but when I just look at sort of a picture of a brain in an anatomy textbook, I sure think, you know, yes, there's like some parts of this that are kind of highly repeated. You've got your kind of cortical column that's like definitely a main piece that is repeated a lot and doing a lot of work. We also see just a lot of other parts. And so it does seem like we're headed for, I think a periodic table is a pretty good mental model for it. We'll have a lot of these different specialised, not even full architectures, but components that you can kind of snap together in all sorts of different ways. Going back to just the sort of how, let's say Erik always asks questions in this way, so I'll do the same. If you're advising a national government, President Harris comes to you and says, okay, A lot of hype out there. Obviously, though, a lot of undeniable progress too. How should I think about how extreme things could be? I hear you that the agents aren't quite ready and that GPT-4 is still kind of a slog to implement and probably is not going to be transformative. But there's a lot of buzz about a possible GPT-5 and the GPT-6 behind that. How would you kind of characterise You know, for somebody that doesn't have time to get into the science, like what the sort of range of plausible outcomes might be, or like what they need to be prepared for?

Well, I think the possibility of a compute race should be kept in mind, but a compute race is not a software company race. It is a hardware race. So if you want to preserve the US capacity to engage in a hardware race. I would advise the president, and especially, you know, in an ironic twist, maybe former President Trump has already received this advice because recently he went on air saying, oh, we're going to need to double the energy supply for compute. I'm like, wait, where did you get that talking point? I don't think he read about it. I don't think he asked ChatGPT. Someone who kind of knows what they're talking about must have spoken to him. So I would say to President Harris, if Kamala becomes President Harris, I would say that you will have to pursue an industrial policy on semiconductors and an industrial policy on energy. And the thing that my libertarian friends will most disagree with me, I'll just say it will require government action, not just the government getting out of the way, but the government buying compute. In America, even TSMC cannot run a chip fab in America and have it be as profitable. No matter what they do, the unit costs are 50% higher. Like Morris is on record for having said that. So if TSMC can't do that, how could Intel? And if we're doing free trade, even with our allies, this means the second largest chip fab. Might be in America, but not the largest. The free market will not provide. So I think the answer has to be something extremely politically difficult. I think we need to create a new Silicon Valley. Let's remember, Silicon Valley originally meant silicon, right? It meant hardware, right? When I saw, you know, there's a Wired article a few years ago, it was titled Shenzhen. is the Silicon Valley of hardware.

I was like, I felt my heart drop.

I was like, oh no, oh no, the Valley has no idea how much trouble they're in. And my friends at the time made fun of me because I was saying, no, no, you don't understand. If they dominate hardware so thoroughly, they will easily reverse engineer software too. And they're gonna have their own unicorns and they're gonna have their own software giants. Well, my friends don't find it so strange. My clients don't find it so strange. Either now that we are discussing whether to ban TikTok, right? Like obviously China can build software startups now. No one's wondering that. Obviously Baidu is honestly, it is about as good as Google actually. They're like keeping pace, not just with the stuff we did 3 or 4 years ago. They're keeping pace with things like self-driving cars and driving their own progress just as fast. So really the hardware thing is going to be difficult. It's going to be difficult because You won't be able to do it like NASA. The political economy of NASA is that NASA has to do something in all 50 states because otherwise it can't protect its budget. It has to give something to every single state or states will go against it. However, that's not a good way to build rockets, right? SpaceX doesn't have to have a facility in all 50 states, right? SpaceX's biggest client, of course, you know, I think it remains the government, though Starlink is genuinely going to be a very different model. But I think there has to be a move to do the politically difficult thing of designating a zone, probably a port city. And it doesn't have to be, by the way, the Bay Area. Maybe California is actually difficult to deal with and you want to do it in a different state, but it has to be a federal designation. Get every single regulation out of the way. You know, the Arizona plant and the delays there, you've heard of that?

I've heard a bit about it for sure. I know the—

Yeah.

Daycare requirements and so on and so forth.

Yeah. So much of it is stuff that has nothing to do with manufacturing the chips. It's literally like, how do we comply with the building codes? I'm like, well, what if we didn't worry about the building codes and just build the chip fab as soon as possible? Right? It would take that kind of mentality where you create an American special economic zone and you demand, you buy a lot of Compute there. Honestly, given the density of data centers in Virginia, heck, maybe you do it in Virginia. Maybe you go retro; you put it on the East Coast. I'm not sure if Virginia would play nice with that, but maybe it would. Maybe the federal government partners up with that one individual state, and then we build full stack, somewhat subsidized, somewhat expensive chip fab setup there, and then we have the capacity. To win a hardware race. Let's be real, we shouldn't be seeking a hardware race with China because we'll lose it. If it was actually an existential military exercise, China can destroy Taiwan. It doesn't need to take Taiwan. It can destroy Taiwan and can make it hell to get ships out of there with chips. And what are you going to do then? Are you going to invade the Chinese mainland? Like, What the hell? Like, are you going to nuke China? Well, if you do that, China's going to nuke you right back. The people who are calling, you know, shout out to Leopold, right? Like the people who are calling for this like massive buildup. I'm like, no, no, buddy. We have to be much slower with this. If America wants to win an industrial race, it just needs some industry first. And America's industry is like, it's pretty good. But the fact that, you know, let's put it this way. America is not the industrial peer of China. It is the industrial peer of Germany. So every single thing you can say about the German economy and how it sucks and how they've been deindustrializing, well, guess what? We can say that about American industry. It's just that America is not just a Germany. It also has like a giant software sector. It has finance, it has service sector, all of this other stuff that Germany has less of. So we sometimes forget that You know, when we look at these GDP numbers, it's not really American manufacturing driving those. And American manufacturing does not outperform Dutch or German or South Korean or Japanese manufacturing. That's why ASML is in the Netherlands. Okay. That's why Samsung is like a Korean company, not an American company. So that's what has to be dealt with. So because of that, I actually think it's a hopeful message geopolitically that this might be yet a scientific race. However, I'm sure that if the intelligence explosion theories are correct, that there's like a key event after which there's a beautiful feedback loop that then results in kind of a single singleton-style future where whoever starts this will just be so far ahead exponentially every few years, they'll be hopeless to catch up. We have to preserve industrial capacity for that era because at the end of the day, it's not like it's easy to coordinate allies. So even friendshoring does not fully work. And you know, I'm not sure which president has the capital, the political capital to push for a functional industrial policy rather than like a pretend industrial policy where we have to do something in all 50 states.

Yeah. I want to get into the chip ban in a second. And sure, I definitely—

By the way, the chip ban is of course a subsidy for American companies too. When I say subsidy, I don't just mean literally giving them a trillion dollars, though, hey, maybe we should try that, right? Do you remember that infrastructure bill? Can anyone point to a single bridge built from the trillion dollars of the infrastructure bill? A single bridge, right? We could have a trillion-dollar chip bill and maybe even some chips would get built. Who knows?

Yeah. I mean, so I'm very allergic to, I guess, the whole US-China conflict. I mean, we could do hours probably on this from kind of beginning to end. But in brief, my view on it—

I think we should embrace economic competition with China and stop emphasizing the geopolitical angle because that's really how China thinks of it. They think that they will win the geopolitics. Through winning the economics, like every single Chinese Party document that comes out, you know they talk a little bit about Belt and Road, but they sort of know it doesn't work. What they really want is to be the economic hegemon, and because they're orthodox Marxists, if you you know here's a funny one: they were such orthodox Marxists that they decided to own the means of production, and they thought that. Results in global dominance by literally owning the factories that build all the stuff. And they weren't averse to using free market capitalism to get to the point where China owns the means of production. So I think that's their theory of geopolitics. And they don't think about it as this like imperial way that we in the West think of it. The Chinese approach to dominance, it's more like, it's not that you go out to the periphery and you conquer all the periphery, which we could say is the Roman way. of building an empire. It's more the, you consolidate and unify China, and then you outshine and extract tribute from all of the periphery. That's the Chinese model of empire historically. And I think these different set of assumptions are there, right? Like for everything we can say about China is like, can we name a single country that China has invaded since their last war with Vietnam? Like that's been over half a century ago, right? Like it's been a long time. The last place they annexed, you could argue maybe it's Hong Kong, but really it was Tibet. And from their perspective, Tibet, Hong Kong, Taiwan is just unifying all of the Chinese lands. So I think the economic competition and having a smarter trade policy, I think that is the right approach to think about China. And like militarizing it or like having a rhetorical point where you emphasize the national security angle, I think that is likelier to cause them to also build up military capacity. Right. And I think the US in the 21st century should strap in for the long game of beating China economically over the next 50 years. And it will take 50 years. China will not implode in 5 years. It'll not implode in 15 years. It'll take 50 years to beat China economically and once more manufacture high technology here in the United States rather than in East Asia.

So before going a little bit deeper on China, I just wanted to zoom out for a second and ask if you see any other live players on the global stage that you think we should factor into This analysis. I look around at the US. I agree with your general sense that like most of the second tier of AI companies, even those that have raised in the low billions, their days seem to be numbered. You know, that, that used to be sort of a spicy take, to borrow your term, but increasingly as these sort of acquihire type deals are happening, it's actually now becoming, I think, sort of consensus. That leaves us with maybe like 5 companies at the top, depending on exactly how you'd want to count. I would certainly still put OpenAI, Anthropic, and Google in the top 3 just by the kind of sheer excellence of the science. Obviously they've scaled, but they seem to have a know-how that is kind of still a cut above. Meta has definitely crashed that party in a meaningful way, although I wouldn't put them quite at the top tier.

Yes.

And then I would never count out an Elon company, so I definitely would put an xAI Tesla, whatever, in that same category. Yeah, same. Let's call that the top 5. When I look around the rest of the world, then I sort of have China as like a bucket. And then maybe we could see a European champion. There's Mistral. Europe maybe wants a European champion, but obviously they're a bit of a harder environment to do business in, and it's unclear Exactly how that's going to play out. Maybe somebody from India, maybe somebody else I'm not thinking about. Any other, you know, before we zoom back in on China, which I think is inevitably where we're going, is there anybody else that you would say really has to be watched on the global game board?

I think not quite yet, but I think 10 to 15 years from now, not 10, maybe 5 years from now, 10 years from now, there will be a formidable Indian giant. I think that India is very likely to develop its own software ecosystem. And if they decide to eventually go the route of favoring Indian companies over Western companies, that's 1.5 billion users right there. That's large enough. That's the same sort of initial captive audience that Chinese companies enjoy and that actually even a Russian company Yandex enjoys, right? Yandex kind of is the Russian software conglomerate. They basically do almost everything that Western software companies do, including having a self-driving program of their own. They're not leaders in any sense, but they demonstrate that even if you have like a very small market of users, which honestly like 100 million users is really not that much, but if it's behind a digital wall or a regulatory wall, well, then you can do very similar things to Silicon Valley. Even if you have much less money than Silicon Valley companies, you can do for a few billion, what in Silicon Valley is done for tens of billions or hundreds of billions. And I think the Indian situation will be very similar. Something will happen there. I think for Europe, I'm more bearish on Europe After Macron stepped down, because I think Emmanuel Macron was personally convinced of the importance of artificial intelligence, had personal contacts with Silicon Valley, championed and pushed against German regulators, ironically using French pull and weight to have French diplomats argue against regulation. You know, think how dire it is if French diplomats are going around Europe Arguing against AI regulation, while the Belgians and the Germans are going wild and having fun restraining them. And it went as far as the French state was trying to do the best it could to recruit people into Paris. This was a smart strategy. With the exit of London from the European Union, Paris could have become the startup capital of Europe. Now, if a future French government. picks this up. And let me explain a little bit. France still has excellent mathematics education. They still have lots of field medalists and so on. You have the raw talent there, right? So they don't need to poach from MIT. They have their own mini MIT. So they have a university there. It's a big city, so you don't feel like you're in a backwater. And if the French government gets off your back, you're not even getting in trouble. Regulatory-wise, like in theory, you could actually do a startup if the French government got out of your way and gave you some tax breaks and all of that stuff, and also actively invested into seeding an ecosystem there. So I think it remains an option for the next 4 to 5 years for French government to continue this strategy. I'm just not sure if they will. Right.

Okay.

So then that brings us back to the US and China. I would love to get a, maybe the brief version of how you understand why the US and China are at such odds in the first place.

Mm-hmm.

I don't have a great account of it. You know, I start like you with a sort of naive observation that like we're on opposite sides of the globe. And it seems like, you know, neither one is a threat, as you said, like we're not going to invade the Chinese mainland and they're not going to invade North America. So we should seemingly be able to coexist, each comfortable in the fact that we've got these huge ocean buffers. And yet all I hear, I feel these days is out of like the DC establishment in general, China this, China that, China, China, China. And it's also now moving into the AI discourse in a way that I think is quite unhealthy, where it's like, well, if we don't race to develop the most powerful technology that we can as fast as we can, then we're just ceding the future to China and we're going to have to live under Chinese values. And I swear to God, people who say that, I always want to know, what is your definition of Chinese values that you are expecting us to be living under? Because I think that's just Often such a vapid comment. But anyway, why do we have this? And I guess the next question that you can maybe carry on into is, is there any way for us to avoid the AI arms race or AI Cold War with China? The chip ban, to put my cards on the table, seems like a step in the wrong direction in that it can only be interpreted from their side as at best an attempt to keep them down, and at worst, an attempt to create decisive strategic dominance over them. And obviously neither one sounds very good from their perspective. So yeah, why do we have this? Is there any way we can avoid an arms race? And what would you, you know, to maybe go back to your strategic advising frame, what would you advise the next president to do about the chip ban?

You know, I think the chip ban is too little, too late. I think that had it been something that was pursued in like 2010, China could have been taken off the track of developing cutting-edge chips. What the CHIPS Act has done now, it's not sanctioned China really. It's given a subsidy to Chinese chip manufacturers. So if you can't buy the best Nvidia chip, or if you can only buy a nerfed version of it, well, guess what? A Chinese chip designer or manufacturer only has to compete with the nerfed version, not your cutting-edge version. They just need to do the nerfed version better, slightly better, and a Chinese giant or company becomes viable because there is enough Chinese commercial demand for GPUs, right? They have a huge tech ecosystem, like dozens of huge companies, their equivalent of Google, their equivalent of Amazon, their equivalent of OpenAI, right? Their equivalent of Waymo, their equivalent of Meta. They have their own demand for this. And the government in China hasn't really even started buying GPUs at all. If they wanted to, they could make it a national priority. And we've basically announced, hey, it's a national priority for you to develop your own high-end chips because we're not going to sell them to you, not now and not in the future. And the Chinese government is rich enough to afford that. I think people really don't understand that America is not the only country in the world that can drop a trillion dollars on a serious problem anymore. China could drop a trillion dollars on this problem if they choose to, or $100 billion or $50 billion, right? In the US context, the US has not dropped a trillion dollars on this. It's only dropped $100 billion. We dropped $100 billion on it and we get an Arizona chip fab. Meanwhile, we drop a trillion dollars on building bridges and it's not clear we built any bridges. So I find that super funny. Anyway, the future here will rely on a general industrial revival in the US where I do think the energy angle is the right one. I think we should seek to radically increase energy production. electricity production in the US, because then it makes sense to run data centers in the US. And energy is one of the few things where labor costs are not a big cost in producing electricity, right? So I think that would be a good medium-term advantage. And I think that also the scientific question is an interesting one As well. I do think we have a scientific advantage over China, but not a technological advantage. So I think something very good can be done in terms of smarter scientific grants, even if the Chinese inevitably copy whatever the science discovers or produces a few years later. I think that one of the US advantages has been DARPA and having a good understanding of how DARPA works. Well, maybe you should have a DARPA focused exclusively on artificial intelligence, and you have to staff it with ideologically diverse people. So don't just hire all the EAs. Even if Jason Matheny's listening to this podcast, you know, the current CEO of RAND and Holden Karnofsky and all of the EA royalty, they can certainly have a few program directors at the AI DARPA. They just should not be allowed to appoint all of them. So that's my other advice to the president would be, you know, Madam President or Mr. President, whatever it might be, you know, we have to have ideological diversity because this is no longer just a technology. People are pursuing their end of the world cosmologies here. It's like, you know, sure, the Manhattan Project can be 50% Marxist. But maybe keep it just 50% Marxist. So similarly, the AI DARPA, maybe it can be 50% effective altruist, but keep it at 50%. Okay. Not to pick on EA, but they're like a big— you also wouldn't want it to be more than 50% EAC, whatever EAC even is ideologically, right? It's just the case that a lot of people will pay lip service to the United States, but ultimately, if you look at their ideology, look, if have a utilitarian ideology, you are not always a friend of the US. You are mostly a friend of the US, but you're supposed to be thinking globally and the future light cone and all of that stuff. You are incidentally not fundamentally patriotic. And I don't want to overburden the patriotism because patriots can be dumb. Patriots often do very noble and excellent things, but it's very easy to weaponize patriots against free speech. Like that's happened many times in the history of many countries where countries embark on a stupid course of action because proposing the smart one sounds like a little bit less nationalist than the alternative. Hey, that's why the Empire of Japan failed. It was too unpatriotic to suggest that maybe Japan cannot beat the United States and maybe Japan should not bomb or destroy US ships, right? That was just unpatriotic and The outcome was not one that I think Chinese nationalists appreciated when the full course of history ran. So let's put it this way. I think currently the United States is— it has to play a very careful game because it is not the world's industrial giant as it was. So the game has to be scientific, low energy costs, very smart government spending. We'll have to have much smarter government spending on strategically relevant science, not more of it, smarter. Because the way we're currently spending scientific money is that we're literally giving the Wuhan Institute of Virology money to produce new plagues for the world. That's kind of insane, right? That was partially US-funded. That, mind you, is now well documented. It's long ago left the Alex Jones or even Joe Rogan world. It's now just congressional testimony world.

So a lot of threads there that I want to pull on.

Again, I invite pushback. I put provocative things there because we need to have real and honest conversations.

So I just, so I'm broadly very much with you when it comes to, okay, let's get our own house in order. Let's like throw up some nuclear plants, get the energy, you know, flowing more freely. All that stuff sounds great when it comes to the I do understand the chip ban in a similar way that like we're inviting them to develop a strong industry and I don't doubt their ability to do it. It may take some time. I don't know how long it's going to take, but, you know, I always say I watched them throw up a hospital in a handful of days, you know, at one point. So I don't want to be caught underestimating their ability to create a chip industry. I also think there's just an interesting question of sort of, do we want to be developing this technology in tandem, or do we want to be running on sort of parallel tracks and having like secrets from each other and all of the things that that implies? I don't necessarily know that you meant to imply this with DARPA, but when I think DARPA, I sort of think like secret technology development. Some of that stuff obviously finds its way into the broader world, but some of it is secret.

I think the US government kind of sucks at developing secret technology. I think the US has great strength in its commercial sector, which is by its nature public. Even if you have trade secrets, right? You know, Nvidia has trade secrets, TSMC has trade secrets. Secrecy in that sense makes sense. Secrecy in a top classified, you know, top secret classified something. Well, let's say, I think the World War II era US government could do that well. We are not dealing with a World War II era government. If you declare something classified and top secret, the result will not be a technological breakthrough. It'll just be waste shielded from oversight. So I think that there can be clandestine projects. It's sort of like, well, how exactly we apply artificial intelligence to the NSA's work. Okay, that can be secret, but commercialization, and by the way, the intelligence world has done this through Institutions like In-Q-Tel, right? Any technology you can declassify and hand over to Silicon Valley or America's commercial sector, you should. You should just do as much of that as possible, even if it means that China catches up to Silicon Valley 3 or 4 years from now because of what you declassified to Silicon Valley, because the true advantage of the United States is its vast wealth. And the speed of its scientific progress. And when you make something secret, you stall scientific progress. You don't accelerate it.

I also worry that this combination of trying to deny chip access, which is not going to work entirely, but, you know, may make for at least some asymmetry. You know, China might be looking at a situation where they're saying, geez, we can only get to 20% the compute capacity that the United States can get to. And also they've got some secret military science programmes that we don't have much— of course, they might hack into those perfectly capably, who knows? But let's say they are also genuinely worried that there are secrets that they are not able to learn. It worries me that might send them down a totally different branch of the technology tree.

Mm-hmm.

And when I think about how do we get into an arms race that would be a really bad situation versus how do we avoid it? One key factor there seems to be, are we kind of on the same general technology path such that there is kind of a shared understanding of what everybody's working with versus to what degree do we look at the other side as kind of the great unknown that we can project our fears onto and then motivate ourselves to take higher risks, be more militaristic in our technology development?

Historically, that's always been the case, right? Even when the US and the USSR signed a treaty banning biological weapons development in the late Cold War, both sides' scientists were of course told that now you should get to work because of course the other side is ignoring this treaty and we don't know what they're doing, so you best develop the worst plagues you can think of. And even during World War II, the fact that Los Alamos scientists greatly overestimated the state of German nuclear research is what allowed the project to be completed before the end of the war and be used in Japan. I think the weaponization of AI is like troubling, and I think part of it is that, you know, scientists are like everyone else on some level. They think weapons are cool. And they want to develop cool weapons. So if they find a way to make physics a weapon, people get very excited. And I think this is one of the problems where we crave peace as a species, but we also crave violence. And I think a lot of the time we should actually be very mindful of our motivations for developing breakthroughs. I think that it is best if artificial intelligence is a peaceful technology for 2 reasons. First, we do invite existential risk, like, and human extinction into the equation if we're trying to develop literally weaponized or malevolent AI. Dude, Skynet is not just a movie, it's a Pentagon pitch deck at this point, right? Like, if you could deliver half of what Terminator films depict, if you could deliver half of that, the Pentagon would just give you a trillion dollars like right away, right? Skynet but a pitch deck would be, you know, that'll probably work. But that's like kind of, we're developing the worst technology first when we think of it in a weapons context. We will not be prioritizing safety and we will be teaching an artificial intelligence to win at zero sum And adversarial games and to win at logistical and military games. And I think it's completely predictable that if you could have an agentic artificial intelligence that decides humanity is its enemy, like that's literally the plot of Terminator and all of the AI safety people made fun of it for decades. But if you just look at how the military world is perceiving AI, you realize that that's like actually extremely bad. And if you make it a geopolitical conflict, the generals will ask you to just build Skynet, please, right? They will just ask you to do that. Don't, don't build Skynet, bro. Just don't. I know you think it's cool. I know it feels really good, but you're gonna regret it, just as Oppenheimer regretted. Now, after he got the high of the detonation, then he regretted it. And that's the hypocrisy of human nature. But You know, hopefully we can grow from that. So let's not make it cool that way. Let's not fetishize the Manhattan Project. Let's not do all of that. And instead, let us be curious explorers of the universe and figure out what is the nature of these new forms of intelligence we're discovering. And let's apply them to our economy to compete with the Chinese economy in who can get richer faster.

Yeah, I love that vision. Just to put one little bit, we're gonna have a full episode on this coming soon, but a little bit of meat on the bone on— because I think some people might be thinking, okay, sure, but what does that actually mean to have weaponised AI versus non-weaponised AI? One proposal for AI safety that I just have been reading through is to train AIs so that in general you have these loss functions, right, in the training process where there's some objective that you're being optimised you, the AI, are being optimised toward. And a huge trend in general in the literature is to have a supplemental element to that loss function that in addition to, you know, so you have the one component of the loss function that's, can you do the task? And then there's some other thing that's like, but we also want to emphasise this. And that could be like sparsity for efficiency or for interpretability or what have you. So this really interesting one that I just came across was about minimising self-other distinction and trying to create an AI that had the same internal state regardless of whether it was referring to itself or referring to someone else. And this is like very early proposal, but they've come up with all these different scenarios in which they think there's really no need for the AI to have a different representation of these concepts. Just because it's referring to itself versus someone else. So let's try to minimize those differences. And they've been able to show in simu— very simple reinforcement learning scenarios that applying this technique can be the difference between agents emerging that deceive other agents or that don't deceive other agents. And I think this is like One of the best ideas that I've heard in a while for how to create AIs that are sort of the like kumbaya, you know, all is one, we're all sharing prosperity sort of mindset that we might hope to have. But obviously that's going to be like a real problem if the goal of the AI is to like distinguish between, you know, friend and foe on the battlefield. So, or at least it seems like those things might be quite incompatible. So I do think that there is like real We're so early in it. We've kind of just got these things to work, right? We've just ended the transformer for everything era, and we're just barely getting into the lots of different components to make tons of different architectures era. And we don't know that much about this stuff yet, but it sure seems like we could optimize for telling friend from foe and killing foes, or we could— take some more time and be more creative, like minimizing self-other distinction. And that just seems like such a much better idea to me that I would really love to see us go down that path. So the Leopold Plan, I want to kind of give you a chance to maybe articulate the Samo plan. The Leopold Plan, in brief, as I understand it, is we should do everything we can to keep China down for the time being. So if we have a couple years lead, we should harden our defences. We should get the national security establishment and the AI labs into some sort of public-private fusion partnership. We'll keep all the secrets from China that way. They'll be denied the chips. Then, because we are the good guys, of course, we will use this lead that we have to solve alignment. Obviously, details will be filled in later. I honestly don't really like it. And then once we've done that, once we can then approach China from a position of decisive strategic advantage, then because again, we are the good guys, we can approach China with the offer of benefit sharing. And then we can sort of maintain American hegemony and American, you know, Western civilizational values will prevail. And China will kind of recognize that better to take the deal at that point. Than to, you know, what alternative will they have? We'll have superintelligence. How would you, what's your alternative vision for how leadership can conceive of a plan for the, you know, next however many years in AI that would contrast against that?

Let me start at the end of the plan and work backwards. The U.S. cannot credibly offer benefit sharing of this kind. The U.S. allowed Germany and Japan to rebuild their economies after World War II because it was interested in the containment of the Soviet Union, and the Soviet Union was a threat. Had that not been the case, perhaps Germany would've been deindustrialized in 1946 rather than 2024, which, mind you, is partially the US is doing through the destruction of Nord Stream. I'm just happy to say it's overwhelmingly clear the US did that. The second step. So the offer cannot be credibly made, therefore the Chinese will not accept it. A step before then, we solve alignment. Because who is we? Who is we? What set of institutional or political forces in the US government would result in an accurate scientific understanding of alignment, assuming that is a coherent problem and make rational bureaucratic committee-driven decisions in line with that. I mean, sure, if we appoint a benevolent dictator of AI, which of course, if we take the premise, is a benevolent dictator of the United States and of the planet. So, you know, so much for American freedom-loving values. Perhaps that is the true convergence all along. And then we go a step earlier. Where it's sort of, you know, this kind of hardening. The only way you can truly make the American technology stack secure against China is to make it secure against US citizens. So the idea of an open digital society, a creative digital society, I know open source, you know, people talk about open sourcing the weights, but I feel the vision of the social revolution that was supposed to happen because of the digital revolution is so lost at this point. I think we all want to have a closed internet. We want to have a monitored internet. We want to have a censored internet. Like, that's what it would require. Do you scrub AI leaks? Leaks would happen. There would be an Edward Snowden of AI. Would you scrub that from the Western internet, perhaps using AI to hunt down all instances and mentions of it? Like, our problem is without an open source, open access citizen participation vision of our internet, if as soon as we abandon that, we actually go on the treadmill first towards being like the EU and then even worse being like China. I've written article on this. Unfortunately, this article has aged well. It is called The Centralized Internet Is Inevitable. And for example, the only break in the escalating trend towards social media censorship that is suggested— and by the way, this is documented by things like WikiLeaks and so on— there are various parts of the US government just suggest, oh man, sure is a nice software monopoly you got here. It would be a shame if someone antitrust it. Can you remove the social media post or can you unbank this person, this Alex Jones person? He's a problem. Do you remember that happened in one day? All the major social media networks decided to deplatform him. I don't like Alex Jones, but that should have been a red flag for the state of internet freedom. And as soon as we abandon sort of the open source vision, I'm open to us Developing a new vision of what the internet could be. But let's not pretend that we are not slow walking into digital totalitarianism of our own kind. When I make a China comparison, it is a warning. It is not praise, right? So I'm not praising the US when I say that the US internet every year gets closer and closer to the Chinese internet. Or to put it differently, if one were to speak their mind politically, On socially taboo topics in the US today, I think your business gets in about as much trouble as it does in China. And perhaps the only difference is you personally are not arrested, you are just made unhireable, and maybe you're put on an intelligence agency watch list. And this happened to many concrete documented cases of US citizens in a way that's unconstitutional, in a way that should be corrected. So I'm just going to talk a little bit about that. So the hardening Can't happen if the civilian infrastructure is built to be open. So I'm open to hardening the infrastructure. We should not be enabling, at least for the military side of things, to be accessed. However, the commercial side kind of has to be open or it becomes structurally tyrannical. So we have to figure out how to thread that needle. What kind of security will we have on the base layer of our economy? Economy, society, politics, and communication. And like, how can we do that? And I know many people have many disagreements, but I'm talking here broadly more about the institutional incentives of the companies that provide us our very tools of thought. At this point, we are cognitively outsourcing as individuals. We're cognitively outsourcing a lot to corporations, and it is extremely risky to push a Closer corporate-government partnership than already exists, because then who shall restrain it in a realpolitik way? Who will restrain these organizations and the government from being tyrannical in a way that is illegal? You can't just say, oh, that's illegal, you can't do that. Law has to be backed by power, and usually it's backed by a balance of different powerful actors where law is sort of the operating protocol between them. Right? So rule of law could be destroyed through significant centralization of the digital space. In fact, we saw this in Canada. In Canada, if you gave $50 to the truckers revolt, you were unbanked. Your transaction was tracked and your bank account was canceled. Okay? Like, that's a level of tyranny I've not actually heard of China do. I've heard China locking up hundreds of thousands of people, but I've not heard them tracing, like, individual bank transactions to find every single small-time donor who gave money to Falun Gong or something like that. I've not heard of China doing that. I've heard of Canada doing that. So I think we give a pass to Western totalitarianism too often. Still, Chinese totalitarianism is worse, but that's not what we should be competing on. So that was a bit of a tangent, but let's go to the earlier stages of the plan. I don't think we can prevent the rise of China, and I think I probably just disagree on AI timelines. If what Leopold was saying on the earlier steps that we should try to slow down China, if you think superintelligence in this AGI sense is like 5 years away, then okay, maybe that makes sense. But if you think it's 20 years away or 200 years away, then it makes no sense. And I don't think it's 5 years away. I'm happy to bet with anyone that it's not 5 years away. So. Because of that, I think we should not be escalating geopolitical competition with China, and we should instead be pursuing a mercantilist trade policy. We should be thinking of it in terms of developing the US sector and US economy, and we should justify it in that language. And it should be a set of give-and-take negotiations with China over splitting the global economic pie. It should not be a set of give-and-take negotiations over splitting the light cone. It shouldn't. That invites the wrong kind of logic and invites the wrong kind of bureaucratic incentives. Like, it's almost like these people never seen how bureaucracies work, either the large corporate environment, let alone a government environment. You have to be extremely careful. The first misaligned artificial intelligence was arguably bureaucracy. Let's prompt engineer it really, really carefully.

It's poetic. I like it. What do you think is the— I have maybe 3, 3 and a half more questions if you have that much time. One is going to be, how do you understand the role of open source in this situation? That is at least at the moment, you know, a sort of irrevocable benefit sharing proposal among other things.

Yep.

Then though, I also wonder, let's say we do, you know, continue to see more and more powerful AI and we start to get to a point where it's like, actually, this is starting to look dangerous. Is there any sort of precedent that we should look back to or any framework that you would look to for how do we set up global coordination and containment of the technology development? Another one was going to be on sovereign AI. This one, probably save for the future. But if you're like any of the other 100+ countries in the world beyond the 2 that we've spent most of our time talking about, there's this notion that we should have our own models, or we should take a base model maybe from Meta and train it on our bureaucratic information or whatever. I wonder if you think there's any value in that. And the final one, personal angle, is to what degree do you use generative AI in your personal work today and what sort of value, if any, are you getting from it? So you can tack the answer on if you have it, pick one, or you just come back next time.

I'll answer. I'll answer. I check in every now and then and use generative AI to see where it is. It's not really a big part of my workflow because it usually provides me the consensus or the general thinking in a field, and almost all of my work is trying to find spots where the economic or political consensus or industry consensus is flawed or overlooking things. So in a way, high schooler interns, not yet useful. Maybe I'll check in when it can give me a Harvard intern who honestly, between you and me, are not that useful either. You kind of have to be even pickier than that if you're doing research. So I don't think it is useful for novel social science research yet. And because of that, I don't use it. And stylistically, sometimes it writes, puts things into words about as well as I do. The problem is it does it in a recognizable way. So I can't even use it in my personal writing because it can't really do my voice. And, you know, I can always tell since one of my part-time One of the hats I wear, I occasionally am an editor for other people's writing, right? For Palladium Magazine, I sit on the editorial board. I can always tell when they use AI and almost always it makes the writing worse because just as people have become sensitized to so-called AI slop in a visual sense, people have become sensitized to reading stuff that sounds like AI. I've actually had to change my own style. To be a little bit different so that people aren't confused about how this text is generated. So yeah, I think it's not even useful for all white-collar work. Not at all. I'm sure it can write probably a good rescheduling email. I think that's about the level where if Google makes a feature where they like tweak or improve the UI with, you know, my inbox. Like get the suggested text a little bit better, maybe I'll integrate in my workflow there. But yeah, not yet making use of the available intelligence. It seems too low grade.

Well, we will continue to watch that closely and I'll share a couple ideas with you perhaps offline. I'm working right now on fine-tuning some of the latest models to write as me. And I haven't completed this project yet, so I don't know how well it's gonna work. In all previous generations that I've tried to fine-tune, it did not work. But there have been some notable improvements in the capability, actually, most importantly, probably the context window, because I found in the fine-tuning it could pick up my style okay, but it couldn't learn the facts and it, it still did not do—

It couldn't learn unique ideas or theories and so on. And perhaps with the larger context window, it'll be capable of it.

Yeah.

Yeah.

The hope is I can stuff all my stuff in there and it might not have to hallucinate anymore. For starters, I think I will still have to give it like the nut, the kernel of the original idea that I have in the moment. I don't think it's gonna, at the current generation, substitute for me there, although we'll find out. I'll definitely let you know if it does. But my hope is I can give it the kernel and then I can give it like a lot of background and then train it on my style and get kind of interesting outputs. But I'll let you know if that starts to materialize. That's my current AI obsession. I know you gotta go. This has been fantastic. A lot of really interesting things to think about, and I will look forward to maybe getting into sovereign AI, and I'm sure we'll have plenty more developments on the US-China— I don't want to say conflict because I want to frame it as— let's frame it as friendly rivalry.

Hopefully—

Competition.

Constructive. Yeah. But yes, this will be certainly a very active area for continued strategic analysis. For now, Sam Albarea, founder of Bismarck Analysis. Thank you for being part of the Cognitive Revolution.

Till next time.

Thank you. Great job, as always.

Upstream with Erik Torenberg is a show from Turpentine, the podcast network behind Moment of Zen and Cognitive Revolution.

If you like the episode, please leave a review in the Apple Store.

Hey everyone, Erik here. At Turpentine, we're building the first media outlet for tech people by tech people. We're the network behind the show you're listening to right now. We have a slate of hit shows across a range of topics and industries, from our AI and investing cluster of podcasts to shows that drive the conversation in tech with the most interesting thinkers, founders, investors, and influencers like Econ 102 with Noah Smith. We're launching new shows every week, and we're looking for industry-leading sponsors. If you think that might be you and your company, email me at erik@turpentine.co. That's erik@turpentine.co, and let's partner together.