Balaji Srinivasan and Nathan Labenz on the Future of AI, AI Gods, and AI Control

This week's Upstream is a feed drop from one of Turpentine's biggest shows, The Cognitive Revolution, hosted by Nathan Labenz. Balaji and Nathan discuss the evolution, challenges, and role of AI in different realms like politics, environmentalism, medicine, and more. There's a lot of new content here about where Balaji thinks AI is going, and interesting parallels with religion. Nathan brings up some AI safety concerns in the course of the discussion. If you like what you hear, check out our other Upstream episodes with Balaji, and check out The Cognitive Revolution. Please enjoy.

Balaji Srinivasan, welcome to The Cognitive Revolution.

All right, I feel welcome.

Well, we've got a ton to talk about. You know, obviously you bring a lot of different perspectives to everything that you think about and work on. And today I want to just try to muster all those different perspectives onto this, what I see as really the defining question of our time, which is like, what's up with AI and how's it going to turn out? I thought maybe for starters, I would love to just get your baseline kind of table setting on how much more AI progress do you expect us to see over the next few years? Like, how powerful are AI systems going to become in, again, kind of a relatively short timeline? And then maybe if you want to take a bigger stab at it, you could answer that same question for a longer timeline, like the rest of our lives or whatever.

Sure. Let me give an abstract answer, then let me give a technical answer. If you look at evolution, we've seen something as complex as flight evolve independently in birds, bats, and bees, and even intelligence We've seen fairly high intelligence in dolphins, in whales, in octopuses. Uh, you know, octopus in particular can do like tool manipulation. They've got things that are a lot like hands, you know, with tentacles. And so that indicates that it is plausible that you could have multiple pathways to intelligence, uh, whether, you know, we have carbon-based intelligence. So we could have silicon-based intelligence that just has a totally different form where the fundamental thing is an electromagnetic wave. and data storage as opposed to, you know, DNA and so on. Right. So that's like a plausibility argument in terms of evolution as being so resourceful that it's invented really complicated things in different ways. Okay. Then in terms of the technical point, I think as of like right now, I should probably date it as like December 11th, 2023, because this field moves so fast. Right. My view is, and maybe you'll have a different view, is that the breakthroughs that are really needed for something that's like true artificial intelligence that is, um, Is human independent, right? Maybe the next step after the Turing test. I've got an article that, you know, we're writing called The Turing Thresholds, which tries to generalize the Turing test to like the Kardashev scale. You know, have you got energy thresholds? Like, what are useful scales beyond that? And right now, I think that what we call AI is absolutely amazing for environments that are not time varying or rule varying. And what I mean by that is, uh, so you kind of have, let's say, 2 large schools of AI, and obviously there's overlap in terms of the personnel and so on, but there's like the DeepMind school, which has gotten less press recently, but got more press, you know, a few years ago. And that is, uh, game playing, right? It is, uh, you know, superhuman playing of Go with AlphaGo. It is, um, you know, all the video game stuff they've done where they learn at the pixel level and they don't, they just teach the very basic rules and it figures it out from there. And it's also, you know, the protein folding stuff and what have you, right? But in general, I think they're known for reinforcement learning and those kinds of approaches. I mean, they're good at a lot of things, but that's what I think DeepMind is known for. Of course, they put out this new model recently, the Gemini model. So I'm not saying that they're not good at everything, but that's just kind of what they're maybe most known for. And then you have the OpenAI ChatGPT school of generative AI, and it'd include Stable Diffusion and, um, just as a pioneer, even if, you know, they're not I don't know how much they're used right now, but basically, you have the diffusion models for images and you have large language models, and now you have the multimodals that integrate them. And so the difference, I think, with these is the reinforcement learning approaches are based on an assumption of static rules, like the rules of chess, the rules of Go, the rules of a video game are not changing with time. They are discoverable. They're like the laws of physics. And similarly, like the body of language where you're learning it, English is not rapidly time-varying. Uh, that is to say, the rules of grammar that are implicit aren't changing, the meanings of words aren't changing very rapidly. You can argue they're changing over the span of decades or centuries, but not extremely rapidly, right? So therefore, when you generate a new result, training data from 5 years ago for English is actually still fairly valuable, and the same input roughly gives the same output. Now, of course, there are facts that change with time, like who is the ruler of England, right? The Queen of England has passed away. Now it's the King of England, right? It's just facts that change with time. But I think more fundamentally is when there's rules that change with time. You have, for example, changes in law in countries, right? But most interestingly, perhaps changes in markets because the same input does not give the same output in a market. If you try that, then what'll happen is there's adversarial behavior on the other side. And once people see it enough times, they'll see your strategy and they're going to trade against you on that. And I can get into other technical examples on that, but I think, and probably people in the space are aware of this, but I think that is the true frontier, is dealing with time-varying, rule-varying systems, as opposed to systems where the implicit rules are static. Let me pause there.

Yeah, I think that makes sense. I think in the very practical, just trying to get, as Zvi calls it, mundane utility from AI, That is often kind of cashed out to AI is good at tasks, but it's not good at whole jobs. It can handle these kind of small things where you can define what good looks like and tell it exactly what to do. But in the sort of broader context of handling things that come up as they come up, it's definitely not there yet. And I agree that there's likely to be some synthesis, which is a Kind of the subject of all the Q* rumours recently, I would say, is kind of the prospect that there could be already within the labs a beginning of a synthesis between the— I kind of think of it as harder-edged reinforcement learning systems that are small, efficient, and deadly versus the language model systems that are kind of slow and soft but have a sense of our values, which is really a remarkable accomplishment that they're able to have even an approximation of our values that seems reasonably good. So, yeah, I think I agree with that framing, but I guess I would still wonder, how far do you think this goes in the near term? Because I have a lot of uncertainty about that, and I think the field has a lot of uncertainty. You'll hear people say, Well, it's never going to get smarter than its training data. It'll kind of level out where humans are. But we certainly don't see that in the reinforcement learning side, right? It usually don't take too long at human level of these games, and then it blows past human level. Interestingly, you do still see some adversarial vulnerability. Like, there's a great paper from the team at FAR AI, and I'm planning to have Adam Gleave, the head of that organisation, on soon to talk about that and other things. where they found basically a hack where a really simple but unexpected attack on the superhuman Go player can defeat it. So you do have these very interesting vulnerabilities or kind of lack of adversarial robustness. Still kind of wondering, where do you think that leaves us in, say, a 3 to 5 years' time?

Obviously, huge uncertainty on that. It's really hard to predict something like this. Just to your point, Generative AI is generic AI, right? It's like generically smart, but it doesn't have specific intelligence or creativity or facts. And as you're saying, just like we have, you know, adversarial images that can fool programs that are trained on a certain set of data and they just give some weird, you know, pattern that looks like a giraffe, but the algorithm thinks it's a dog. Uh, you can do the same thing for game playing and you can have out of sample input that can beat These very sophisticated reinforcement learners. And an interesting question is whether that is a fundamental thing or whether it is a workarroundable thing. And you'd think it was workarroundable because there's probably some robustification, because these pictures look like giraffes, and yet they're being recognized as dogs. So you would think that the right proximity metric would group it with giraffes, but maybe there's some result there. My intuition would be we can probably robustify these systems so that they are less vulnerable to adversarial input. But if we can't, then that leads us in a totally different direction where these systems are fragile in a fundamental way. So that's one big branch point is how fragile these systems are, because if they're fragile in a certain way, then it's almost like you can always kill them, which is kind of good, right? In a sense that there's that, you know, almost like the, you know, the 50 IQ, 100 IQ, 150 IQ thing.

Like the meme?

Yeah, the meme, right? So the 50 IQ guy's like, these machines will never be as creative as humans or whatever. 100 IQ is look at all the things they can do. The 150 IQ is like, well, there's some equivalent result that's some impossibility proof that shows that the dimensional space of a giraffe is too high and we can't actually learn what a true giraffe— I don't think that's true, but maybe it's true from the perspective of how these learners are working. Because my understanding is people have been trying, and I mean, I'm not on the cutting edge of this, so maybe someone will But my understanding is we haven't yet been able to robustify these models against adversarial input. Am I wrong about that?

Yeah, that's definitely right.

We'll continue our interview in a moment after a word from our sponsor. Real quick, what's the easiest choice you can make? Taking the window instead of the middle seat? Outsourcing business tasks that you absolutely hate? What about selling with Shopify? Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of all e-commerce in the US, and Shopify is the global force behind Allbirds, Rothy's, and Brooklinen, and millions of other entrepreneurs of every size across 175 countries. Whether you're selling security systems or marketing memory modules, Shopify helps you sell everywhere, from their all-in-one e-commerce platform to their in-person POS system. Wherever and whatever you're selling, Shopify's got you covered. I've used it in the past at the companies I've founded, And when we launch merch here at Turpentine, Shopify will be our go-to. Shopify helps turn browsers into buyers with the internet's best-converting checkout, up to 36% better compared to other leading commerce platforms. And Shopify helps you sell more with less effort thanks to Shopify Magic, your AI-powered all-star. With Shopify Magic, whip up captivating content that converts, from blog posts to product descriptions. Generate instant FAQ answers. Pick the perfect Perfect email send time. Plus, Shopify Magic is free for every Shopify seller. Businesses that grow grow with Shopify. Sign up for a $1 per month trial period at shopify.com/torenberg. Go to shopify.com/torenberg now to grow your business, no matter what stage you're in. That's shopify.com/torenberg.

There's no single architecture, as far as I know, that is demonstrably robust. And on the contrary, even with language models, we did a whole episode on the universal jailbreak, where especially if you have access to the weights, not to change the weights, but just to kind of probe around in the weights, then you have a really hard time guaranteeing any sort of robustness.

The conjecture is, see, for humans, you can't mirror their brain and analyse it. Okay. But we have enough humans that we've got things like optical illusions, stuff like that, that works on enough humans. Um, and our brains aren't changing enough, right? A conjecture is if you had, as you said, open weights, open weights mean safety, because if you have open weights, you can always reverse engineer adversarial input and then you can always break the system. Conjecture.

Yeah, there's, I, that's again with Adam from, from AI. I'm really interested to get into that because they are starting to study, as I understand it, kind of proto scaling laws for adversarial robustness. And I think a huge question there is, what are the kind of frontiers of possibility there? How do the orders of magnitude work? Do you need another 10x as much adversarial training to half the rate of your adversarial failures? And if so, can we generate that many? It may always sort of be fleeting.

So FarAI, and they're working on cutting edge of adversarial input.

Yeah, they're the group that did the attack on the AlphaGo model and found that— and what was really interesting about that, I mean, multiple things, right? First, that they could beat a superhuman Go player at all, but second, that the technique that they used would not work at all if playing a quality human. Or it's a strategy that is trivial to beat if you're a quality human Go player, but AlphaGo is just totally blind to it.

That's why I say the conjecture is, if you have the model, then you can generate the adversarial input. And then, so if that is true, and that itself is an important conjecture about AI safety, right? Because if open weights are inherently something where you can generate adversarial input from that and break or crash or defeat The AI, then that AI is not omnipotent, right? You have some power words you can speak to it, almost like magical words that'll just make it power down, so to speak, right? It's like those movies where the monsters can't see you if you stand really still, or if you you don't make a noise, or something like that, right? They're very powerful on dimension X, but they're very weak on dimension Y. A kind of an obvious point, but. I'm not sure how important it's going to be in the future. Your next question was on humanoid robots and so on. Before we get to that, maybe obviously, but all of these models are trained on things that we can easily record, which are sights and sounds, right? But touch and taste and smell, we don't have amazing datasets on those. Well, I mean, there's some haptic stuff, Right. Uh, there's, there's probably some, you know, some work on taste and smell and so on, but those— there's 5 senses, right? I wonder if there's something like that where, uh, you might be like, okay, how are you going to outsmell, uh, you know, a robot or something like that? Well, dogs actually have a very powerful sense of smell. That's been very important for them, you know. And it may turn out that there's— maybe it's just that we just haven't collected the data and it could become a much better smeller or whatever, or, you know, taster than anything else. I wouldn't be surprised. It could be a much better wine taster. because you can do molecular diagnostics. But it's just kind of, I just use that as an analogy to say there's areas of the human experience that we haven't yet quantified. And maybe it's just the operative term is yet. Okay. But there's areas of the human experience we haven't yet quantified, which are also an area that AIs at least are not yet capable in.

Yeah, I guess maybe my expectation boils down to, I think the really powerful systems are probably likely to mix architectures in some sort of ensemble. When you think about just the structure of the brain, there certainly are aspects of it that are repeated, right? You look at the frontal cortex and it's like there is kind of this unit that gets repeated over and over again. In a sense, that's kind of analogous to, say, the transformer block that just gets stacked layer on layer. But it is striking in a transformer that it's basically the same exact mechanism at every layer that's doing kind of all the different kinds of processing. And so whatever weaknesses that structure has, and with the transformer and the attention mechanism, there's some pretty profound ones, like finite context window, you kind of need, I would think, a different sort of architecture with a little bit of a different strength and weakness profile to complement that in such a way that kind of more similar to a biological system where you kind of have this dynamic feedback where we have obviously thinking fast and slow and all sorts of different modules in the brain, and they kind of cross-regulate each other and don't let any one system go totally down the wrong path on its own, right, without something kind of coming back and trying to override that. It seems to me like that's a big part of what is missing from the current crop of AIs in terms of their robustness. And I don't know how long that takes to show up, but we are starting to see some possible— I think people are maybe thinking about this a little bit the wrong way. Just in the last couple of weeks, there's been a number of papers that are really looking at the state space model kind of alternative. It's being framed as an alternative to the transformer. But when I see that, I'm much more like, it's probably a complement to the transformer, or these 2 things probably get integrated in some form, because to the degree that they do have very different strengths and weaknesses, ultimately you're going to want the best of both in a robust system. Certainly if you're trying to make an agent, certainly if you're trying to make a humanoid robot that can go around your house and do useful work, but also be robust enough that it doesn't get tricked into attacking your kid or your dog or whatever, you're going to want to have more checks and balances than just kind of a single stack of the same block over and over again?

I know Boston Dynamics, with their legged robots, is all control theory and it's not classical ML. It's really interesting to see how they've accomplished it. They do have essentially a state-space model where they have a big position vector that's got all the coordinates of all the joints and then a bunch of matrix algebra to figure out how this thing is moving and all the feedback control and so on there. And it's more complicated than that, but that's, I think, the V1 of it. Sorry, I wasn't following this, though. Are you saying that there's papers that are integrating that with the kind of generative AI transformer model? What's a good citation for me to look at?

Yeah, starting to. We did an episode, for example, with one of the technology leads at Skydio, the US's champion drone maker. And They have kind of a similar thing where they have built over a decade, right, a fully explicit, spanning multiple orders of magnitude control stack. And now over the top of that, they're starting to layer this kind of— it's not exactly generative AI in their case because they're not generating content, but it's kind of the high level, can I give the thing verbal instructions? Have it go out and kind of understand, okay, this is a bridge. I'm supposed to kind of survey the bridge and translate those high-level instructions to a plan and then use the lower-level explicit code that is fully deterministic and runs on control theory and all that kind of stuff to actually execute the plan at a low level, but also at times surface errors up to the top and say, like, hey, we've got a problem, whatever. I'm not able to do it. Can you now, at the higher level, the semantic layer, adjust the plan? That stuff is starting to happen in multiple domains, I would say.

Yeah. And so I think that makes sense. Basically, it's like generative AI is almost the front end, and then you have almost like an assembly. Like, you give instructions to Figma, and the objects there are their shapes and their images. And so it's not text. You give instructions to a drone, and the Objects are like GPS coordinates and paths and so on. And so you are generating structures that are in a different domain, or it's like in VR, you're generating 3D structures again, as opposed to text. And then that compute engine takes those 3D structures and does something with them in a much more rules-based way. So you have like a statistical user-friendly front end with a generative AI, and then you have a more deterministic. or usually totally deterministic, almost like assembly language backend that actually takes that and does something. That's what you're saying, right?

Yeah, pretty much. And I would say there's another analogy to just, again, our biological experience where it's like I'm sort of in a semi-conscious level, right? I kind of think about what I want to do, but the low-level movements of the hand are both not conscious. And also, if I do encounter some pain or hit some hot item or whatever, there's a quick reaction that's sort of mediated by a lower level control system, and then that fires back up to the brain and is like, hey, we need a new plan here. So that is only starting to come into focus, I think, because obviously it's amazing. As you said, it's all moving so fast. What is always striking to me, and I kind of recite timelines to myself almost as like a mantra, right? The first instruction-following AI that hit the public was just January 2022. That was OpenAI's Text-Davinci 002 was the first one where you could say, like, do X and it would do X, as opposed to having an elaborate prompt engineering type of setup. GPT-4 just a little over a year ago finished training, not even a year that it's been in the public. And it has been amazing to see how quickly this kind of technology is being integrated into those systems, but it's definitely still very much a work in progress.

Yeah. I mean, the tricky part is, um, like the training data and so on with it, like a large existing scaled company, like a Figma or DJI that has millions or billions of user sessions will have a much easier time training. And, and they have a unique dataset and then everybody else will not be able to do that. So there is actually almost like I mean, a return on scale where the massive dataset, if you've got a massive clean dataset in a unique domain that lots of people are using, then you can crush it. And if you don't, I suppose, I mean, there's lots of people who work on zero-shot stuff and so on and so forth, but it still strikes me that it'll probably be an advantage to see those sessions. I find it hard to believe that you could generate a really good drone command language without lots of drone flight paths. But we could see.

And where it doesn't exist, people are obviously need deep pockets for this, but the likes of Google are starting to just grind out the generation of that, right? They've got their kind of test kitchen, which is a literal physical kitchen at Google where the robots go around and do tasks. And when they get stuck, my understanding of their kind of critical path, as I understand they understand it, is robot's going to get stuck. We'll have a human operator remotely operate the robot to show what to do, and then that data becomes the bridge from what the robot can't do to what it's supposed to learn to do next time. And they're going to need a lot of that for sure. But they increasingly have— I don't know exactly how many robots they have now, but last I talked to someone there, it was like into the dozens, and presumably they're continuing to scale that. I think they just view that they can probably brute force it to the point where it's good enough to put out into the world. And then very much like a Waymo or a Cruise or whatever, they probably still have kind of remote operators even when the robot is in your home. When it encounters something that it doesn't know what to do about, raise that alarm, get the human supervision to help it over the hump, and then Obviously, that's where you really get the scale that you're talking about. This raises a couple of questions I wanted to ask that are conceptual. So obviously there's huge questions around, again, highest level, how is all this going to play out? One big debate is to what degree does AI favor the incumbents? To what degree does it enable startups? Obviously it's both, but I'd be interested in your perspective on that. Also really interested in your perspective on offense versus defense. That's something that a lot of people now and in the future, that seems like it probably really matters a lot, whether it's a more offense-enabling or defense-enabling technology. So love your take on those 2 dimensions.

Hey, we'll continue our interview in a moment after a word from our sponsor. Have you ever wondered where your donation could have the most impact? In 2007, a group of donors had that exact question, but when they sought out information from charities to help them answer this question, they instead received cute pictures or unhelpful stories. Their experience led them to create GiveWell, an organization providing rigorous, transparent research about the best giving opportunities they've found. GiveWell has now spent over 15 years researching charitable organizations and only directs funding to a few of the highest impact opportunities they've found in global health and poverty alleviation. Over 100,000 donors have used GiveWell to donate more than $1 Rigorous evidence suggests that these donations will save over 150,000 lives and improve the lives of millions more. GiveWell wants as many donors as possible to make informed decisions about high-impact giving. You can find all of their research and recommendations on their site for free. You can make tax-deductible donations to their recommended funds or charities, and GiveWell doesn't take a cut. If you've never donated through GiveWell before, you can have your donation matched So like offense or defense in the sense of does it enable disruptors or incumbents?

Both in business and in potentially outright conflict. I'd be interested to hear your analysis on both.

All right, a lot of views on this. So obviously, if you've got a competent existing tech CEO who's still in their prime, like Amjad of Replit or Dylan Field of Figma, those are 2 who I've thought of who are very good and are will be on top of it. Amjo is very early on integrating AI into Replit, and it's basically built that into an AI-first company, which is really impressive. Those are folks who cleanly made a pivot. It's as big or bigger than, comparable to, I would say, the pivot from, uh, desktop to mobile that broke a bunch of companies in the late 2000s and early 2010s. Like Facebook in 2012 had no mobile revenue, roughly, at the time of their IPO, and then they had to like redo the whole thing. And it's hard to turn a company 90 degrees when something new like that hits, you know, those that are run by kind of tech CEOs in their prime, uh, will, will adapt and will AI-ify their existing services. And the question is, obviously there's new things that are coming out like Pika and Character.ai. There's some like really good stuff that's, that's out there. The question is, uh, you know, will the disruption be allowed to happen in the US regulatory environment? And so my view is actually that, you know, so this is from like the Network State book, right? I talk about, you know, people talk about a multipolar world or unipolar world. The political axis is actually really important in my view for thinking about whether AI will be allowed to disrupt. Okay. Because we'll get to this probably later, but the 640K of compute It's enough for everyone executive order, you know, 640K of memory, the apocryphal, he didn't, Bill Gates didn't actually say it, but that, that quote kind of gives a certain mindset about computing that should be enough for everybody. So the 10 to the 26 of compute should be enough for everyone bill. Um, I actually think it's very bad and I think it's just the beginning of their attempts to build like a software FDA. Okay. To decelerate, control, regulate, red tape the entire space. Just like how the threat of nuclear terrorism got turned into the TSA, the threat of Terminators and AGI gets turned into a million rules on whether you can set up servers. And this last free sector of the economy is strangled, or at least controlled within the territory controlled by Washington, DC. Now, why does this relate to the political? Well, obviously, this You know, you can just spend your entire life just tracking AI papers, and that's moving like at the speed of light like this, right? What's also happening, as you can kind of see in your peripheral vision, is there's political developments that are happening at the speed of light, much faster than have happened in our lifespans. Like there's more, you just notice more wars, more serious online conflicts. Like, you know, there's a sovereign debt crisis. All of those things I can show graph after graph of things looking like their own types of singularities, you know, like military debts are way up, you know, the long piece that Steven Pinker showed, it's looking like a U that's suddenly way up after Ukraine and some of these other wars are happening, unfortunately, right? Interest payments, whoosh, way up to the side. What's my point? Point is, I think that the world is going to become from the Pax Americana world of just like basically one superpower, hyperpower that we grew up in from '91 to 2021, roughly. that we're going to get a specifically tripolar world, not unipolar, not bipolar, not multipolar, but tripolar. And those 3 poles I kind of think of as NYT, CCP, BTC, or you could think of them as— and those are just certain labels that are associated with them, but they're roughly US tech in the US environment, China tech in the China environment, and global tech in the global environment. And why do I identify BTC and crypto and so on with global tech? Because that's the tech that decentralized out of the US. And right now people think of crypto as finance, but it's also financiers. Okay. And in this next run-up, it is, I think, quite likely about, depending on how you count, between a third to a half of the world's billionaires will be crypto. Okay. Around, you know, I calculated this a while back, around Bitcoin at a few hundred thousand, around a third to a half of the world's billionaires are crypto. That's the unlocked pool of capital, and those are the people who do not bow to DC or Beijing. And they might, by the way, be Indians or Israelis or every other demographic in the world, or they could be American libertarians, or they could be Chinese liberals like Jack Ma who are pushed out of Beijing's sphere, okay? Or the next Jack Ma, you know, Jack Ma himself may, may not be able to do too much, okay? That group of people who are, let's say, the dissident technologists who are not gonna just kneel to anything that comes out of Washington, D.C. or Beijing. That is the— that's decentralized AI. That's crypto. That's decentralized social media. So you can think of it as, you know, what we talked about on the recent Pirate Wires podcast: freedom to speak with decentralized censorship-resistant social media, freedom to transact with cryptocurrency, freedom to compute with open-source AI and no compute limits. Okay, that's a freedom movement, and that's like the same spirit as the Pirate Bay, the same spirit as BitTorrent, the same spirit as Bitcoin, the same spirit as peer-to-peer and end-to-end encryption. That's a very different spirit than having Kamala Harris regulate a superintelligence or signing it over to Xi Jinping Thought. And the reason I say this is I think that that group of people Of which I think Indians and Israelis will be a very prominent, maybe a plurality, right? Just because the sheer quantity of Indians are like the third sort of big group that's kind of coming up and they're relatively underpriced. You know, China is, I don't want to say it's priced to perfection, but it's something that people, when I say priced, I mean, people were dismissive of China even up until 2019. And then it was after 2020, if you look, that people started to take China seriously. And what I mean by that is the West Coast tech people knew that China actually had A+ tech companies and was a very strong competitor, but the East Coast still thought of them as a third world country until after COVID, when now, you know, the East Coast was sort of threatened by them politically. And it wasn't just blue collars, but blue America that was threatened by China. And so that's why the reaction to China went from, oh, who cares, it's just taking some manufacturing jobs, to This is an empire that can contend with us for control of the world. That's why the hostility is ramped up, in my view. There's a lot of other dimensions to it, but that's a big part of it. So India is also kind of there, but it's like the third, and India is not going to play for number one or number two. But India and Israel, if you look at like tech founders, depending on how you count, especially if you include the diasporas, it's on the order of 30 to 50% of tech founders, right? And it's obviously some, you know, very good tech CEOs, and, you know, Satya and Sundar and investors and whatnot. Those are folks— Indians do not want to bow to DC or to Beijing. Neither do Israelis, for all kinds of reasons. Even if Israel has to, you know, take some direction from the US now, they're bristling at it, right? And, and then a bunch of other countries don't. So the question is, who breaks away? And, and now we get to your point on— the reason I had to say that is that, that's preface the political environment, this tripolar thing of US tech, And U.S. regulated, Chinese tech and China regulated, and global tech that's free. Okay, of course there's even though I identify those three poles, there's of course boundary regions. EAC is actually on the boundary of of U.S. tech and decentralized tech, you know, and I'm sure there'll be some Chinese thing that comes out that is also on the boundary there. For example, Binance is on the boundary of Chinese tech and global and decentralized tech, if that makes any sense, right? There's probably others. Apple is actually on the boundary of U.S. tech and Chinese tech because they make all of their stuff in China, right? So these are not. Totally disjoint groups, but there's boundary areas, but you can think of them. Why is this third group so important? In my view, both the Chinese group and the decentralized group will be very strong competition for the American group for totally different reasons. China has things like WeChat, these super apps. I mean, obviously not like— but like, super— WeChat is a super app, but they also have, for example, their digital yuan. Right. They have the largest, cleanest datasets in the world that are constantly updated in real time that they can mandate their entire population opt into. And most of the Chinese language speaking people are under their ambit, right? So that doesn't include Taiwan, doesn't include Singapore, doesn't include, um, you know, some of the Chinese diaspora, but basically anything that's happening in Chinese for 99% of it, 95%, whatever the ratio is. They can see it and they can coerce it and they can control it. So they can tell all of their people, okay, here's $5 in digital yuan, do this microtask. Okay. All of these digital blue-collar jobs, both China and India, I think can do quite a lot with that. And I'll come back to that. So they can make their people do immense amounts of training data, clean up lots of datasets. Once it's clear that you have to build this into this, they can just kind of execute on that. And they can also deploy. I mean, in many ways, the US is still very strong in digital technology, but in the physical world, it's terrible because of all the regulations, because of all the NIMBYism and so on. It's not like that in China. So anything which kind of works in the US at a physical level, like the Boston Dynamics stuff, they're already cloning it in China and they can scale it out in the physical world. You already have drones, little sidewalk drone things. that come to your hotel room and drop things off. That's already very common in China. In many ways, it's already ahead if you go to the Chinese cities. So the Chinese version of AI is ultra-centralized, more centralized, more monitoring, less privacy, and so on than the American version. And therefore, they will have potentially better datasets, at least for the Chinese population. And so WeChat AI, I don't even know what it's going to be, but it'll be probably really good. It'll also be really dangerous in other ways. Then the decentralized sphere has power for a different reason, because the decentralized sphere can train on full Hollywood movies. It can train on all books, all MP3s, and just say, screw all this copyright stuff, right? Like what Sci-Hub and, you know, LibGen are doing. Because all the copyright, first of all, it's not— it's like Disney lobbying politicians to put like another 60 or 70 or 90. I don't even know what it is, some crazy amount on copyright so you can keep milking this stuff and it doesn't go into public domain, number one. And second, you know how Hollywood was built in the first place? It was all patent, copyright, and IP violation. Essentially, Edison had all the patents. He's in New Jersey-ish, okay, that East Coast area. And, um, Neil Gabler has this great book called An Empire of Their Own where he talks about how, uh, immigrant populations, you know, the Jewish community in particular, also others, went to Southern California in part so they could just make movies without Edison coming and suing them for all the patents and so on and so forth. And they made enough money that they could fight those battles in court. And that's how they built Hollywood. Okay. So, you know, one of my big theses is history is running in reverse and I can get to why, but it's like 1950s, a mirror moment. And you go more decentralized backwards and forwards in time is like these, you have these huge centralized states like the US and USSR and, and China, you know, all these things exist and then their fist relaxes as you go forwards and backwards in time. For example, backwards in time, the Western frontier closed, uh, and forwards in time, the internet frontier opens. Backwards in time, you have the robber barons. Forwards in time, you have the tech billionaires. Backwards in time, you have Spanish flu. Forwards in time, you have COVID-19. And I've got dozens of examples of this in the book. The point is that if you go backwards in time, the ability to enforce patents and copyrights and so on starts dropping off. Right? You have much more of a Grand Theft Auto environment and you go forwards in time and that's happening again. So India in particular, for many years, basically just didn't obey Western patent protections and all these stupid rules. Basically, you know, it's a combination of artificial scarcity on the patent side and artificial regulation on the FDI side. That's a big part of what jacks up drug costs where these things cost, you know, only cents to manufacture and they sell them for so much money. Um, all the delays, of course, that are imposed on the process, the only way they can pay for it, the manufacturers, is take it out of your hide. What India did is they just said, we're not going to obey any of that stuff. So they have a whole massive generic drugs and biotech industry that arose because they built all the skills for that. That's why they could do their own vaccine during COVID And they're one of the biggest biotech industries in the world because they said, screw Western restrictive IPs and other stuff. Right. So I was actually talking with the, uh, the founder of Flipkart. That's India's largest exit. And we were talking about this a few months ago. And what we want is for India and other countries like it to do something similar, not just generic drugs, but generic AI, meaning just let people train on Hollywood movies. Let them train on full songs. Let them train on every book. Let them train on anything. And you know what? Sue, sue them in India. Right, and have the servers in India and let people also train models in India because that's something that can build up a domestic industry with skills that the rest of the world, you know, people will want the model output. They'll want to use the software service there, and they'll be fighting in court on the back end. This is similar to how all of the record companies fought Napster and Kazaa and so on, but they couldn't take down Spotify. Do you know that story? Do you remember that? Basically because Spotify was legitimately a European company and that a combination of execution and negotiation, they couldn't take them down. They did take down Napster, they took down LimeWire, they took down GrooveShark, and Kazaa had Estonians. I don't know exactly how it was incorporated, but it was probably too US-proximal, and that's why they were able to get them. But Spotify was far enough away that they couldn't just sue them, and they actually genuinely had European traction. That's why the RIAA had to negotiate. So being far away from San Francisco may also be at an advantage in AI because it means you're far away from the bluest city in the bluest state in the union. This relates to another really important point. When you actually think about deploying AI, there's those jobs it can disrupt that are not regulated jobs. Like, you know, obviously programmers are not— thank God you don't need a license to be a programmer, but programmers adopt this kind of stuff naturally, So GitHub Copilot, Replit, we just boom, use it. And now it's amplified intelligence. Okay. But a lot of other jobs, there's some that are unionized and then some that are licensed, right? So Hollywood screenwriters are complaining, right? Journalists are complaining. Artists are complaining. This is a good chunk of blue America. If you add in licensed jobs like lawyers and doctors and bureaucrats, right? You know, especially lawyers and doctors, very politically powerful MDs and JDs. They have Strong lobbying organizations, AMA and, you know, ABA and so on. Basically, AI is part of the economic apocalypse for blue America. Okay, it just attacks these overpriced jobs. And I say overpriced relative to what an Indian could do with an Android phone, what a South American could do with an Android phone, what someone in the Middle East or the Midwest could do with an Android phone. Now those folks have, uh, you know, been armed with generative AI. They can do way more. They're ready to work. They're ready to work for much less money. And they're a massive threat to blue America. Blue America is now feeling like the blue collars of 10 or 20 years ago, where, um, the blue collars had their jobs, you know, going to China and other places, right? And they were mad about that. Factories got shut down and so on. That's about to happen to blue America. Already happening. Okay. And so that's going to mean a political backlash by blue America of protectionism. Again, already happening. And the AI safety stuff, that's a whole separate thing, but it's going to be used. I'm going to use a phrase, and I hope you won't be offended by this. Have you heard the phrase useful idiots, like by Lenin or whatever? Okay. It basically means like, okay, those guys, you know, they're useful idiots for communism. And so So there's, let me put it like naive people who think that the US government is interested in AI safety are trying to give a lot of power to the US government. And the reason is they haven't actually thought through from first principles, what is the most powerful actor in the world? I come back to that. They're trying to give power to the US government to regulate AI safety. But the government doesn't care about safety of anything. They literally funded the COVID virus in Wuhan, credibly alleged, right? There's, at least it is a reasonable hypothesis based on a lot of the data Matt, Matt Ridley wrote a whole book on this. There's a lot of data that indicates, a lot of scientists believe it. I'm, I'm actually like a bioinformatics genomics guy. If you look at the sequences, there is a gap and a jump where it looks like this thing could have been engineered or partially engineered or evolved. There's the Peter, you know, Peter Daszak, there's Zhengli Shi. There's actually a lot of evidence here. So the US government and the Chinese government are responsible for an existential risk. You know, by studying it, they created it. Okay. They're responsible for risking nuclear war with Russia over this, you know, piece of land in eastern Ukraine, which, you know, probably is going to get wound down. Okay, so they don't care about your safety at all. They're not like— these are immediate things where we can show, and there's nobody who's punished for this. Nobody is fired for this. You know, literally rolling the dice on millions, hundreds of millions of people's lives has not been punished. In fact, It's like, it's not even talked about. We're past the pandemic and, you know, this, these institutions can't be punished. So they don't care about AI safety. What they care about is AI control. And so the people in tech who are like, well, the government will guarantee AI safety, that's actually what we're going to actually get is something on the current path, like what happened with nuclear technology, where you got nuclear weapons, but not nuclear power. Or at least not to the scale that we could have had it, right? We could have had much cheaper energy for everything. Instead, we got the militarization and the regulation and the deceleration. Worst of all worlds, where you can blow people up, but you can't build nuclear power plants. And like even getting into nuclear technology, forget about just nuclear power plants. We don't have nuclear submarines. We don't have nuclear planes, all that kind of stuff. I don't know if nuclear planes are possible, but I do know nuclear submarines are possible. You can do a lot more Cruise ships, a lot more stuff like that. You could probably have nuclear trains. You have to look at exactly how big those are. I don't know exactly how big those engines are and what the supplies are, but I wouldn't be surprised if you could. We don't have that. Why don't we have that? Because we had the wrong fear-driven regulation in the early '70s. Putting it all together, I think that the current AI safety stuff is similar to nuclear safety stuff, that the US government has a terrible track record on safety in general, It doesn't care about it. It funded the COVID virus, credibly alleged. It definitely risked nuclear war with Russia recently. Hot war with Russia was the red line we were not supposed to cross, and we're now like way into that. So it doesn't care about AI safety, doesn't care about your safety, and it's also not even good at regulating. And so what it cares about is control. And we are going to have potentially a bad outcome where Silicon Valley in San Francisco is the Xerox PARC of AI. Maybe that's too strong. Okay. But basically it develops it and there's a lot of things it can't do because it lobbied for this regulation that is going to come back and choke it. And then the other 2 spheres will push ahead because it's not about the technology. It's also about the political layer. You know, the Steve Jobs thing, actually Alan Kay by way of Steve Jobs. If you're really serious about software, you need your own hardware.

Right?

So if you're really serious about technology, you need your own sovereignty. Because what the AI people haven't thought about is there's a platform beneath you, which is not just compute, it is regulate. It's the law. Okay? And if the law doesn't allow you to compute, so much for all of your stuff above that. And I know you're saying, oh, it's only a 10 to the 26 compute band and so on and so forth. Have you seen the first IRS tax form? It's always, always super simple. It's only the super, super, super rich who's who are going to get in first. Doesn't matter to you. So that's called boiling the frog slowly. There's a million you know slippery slope. Slippery slope isn't a fallacy. It's literally how things work, right? You know Apple one of the reasons they you know they talk about not setting a precedent. Zuck starts a he has a very hard line on setting precedents because he understands the long term. equivalent of setting a precedent, right? The precedent setting is that they're setting up a softer FDA, and they're gonna— and, and DC is so energized on this because they know how much social media disrupted them. That's why they're on the attack on crypto and AI. That's why they're on the attack on self-driving cars. They want to freeze the current social order in amber domestically and globally. So they think they can sanction China and stop it from developing chips. They think they can impose regulations on the US And stop it from developing AI, but they can't. And also, by the way, they're they're totally schizophrenic on this. Where when they're talking about China, they're like we're gonna stop their chips to make sure America is a global leader. This is Gina Raimondo who's saying this. And then domestically, they're like we're gonna regulate you so you stop accelerating AI. We're not about AI acceleration. EAC is weird or whatever. Okay. So think about how schizophrenic that is. Okay. You're gonna be far ahead of China. We're also going to make sure to control the US. So they want to try and slow— what they actually want is to freeze the current system in amber, try to go back to pre-2007 before all these tech guys disrupted everything. But that's not what's going to happen. But they're going to try to do it. And so everybody who's still loyal to the DC sphere, which includes an enormous chunk of AI people, because a lot of them are in San Francisco, right? The political chaos of the last few years was not sufficient for them to relocate yet. Not all of them. I mean, Elon is in Texas, and that it may turn out that Grok, for example, and what they're doing there, because he's a very legit. I mean, you know, he's Elon, so he's capable of doing a lot. He was very early on OpenAI. He understood. He understands. You know, the right. It may turn out that Grok becomes Red AI, or. the community around that. And OpenAI and DeepMind are still blue AI, and we have Chinese AI, and we're going to have decentralized AI. Okay, let me pause there. I know there's a big download.

Well, for starters, I would say broadly, I have a pretty similar intellectual tendency as you. I would broadly describe myself as a techno-optimist libertarian on just about every issue. And I think your analysis of the dynamics is super interesting, and I think a lot of it sounds pretty plausible, although I'll kind of float a couple of things that I think may be bucking the trend. But I think it's maybe useful to kind of try to separate this into scenarios, because all the analysis that you're describing here, if I understand it correctly, it seems to have the implicit assumption that the AI itself is not going to get super powerful or hard to control. It's like, if we assume that it's kind of a normal technology, then you're off to the races on this analysis, and then we can get into the fine points. But I do want to take at least one moment and say, how confident are you on that? Because if it's a totally different kind of technology from other technologies that we've seen, if it's more you raised the gain of function research example. If it's that sort of technology that has these sort of nonlocal possible impacts or self-reinforcing kind of dynamics, which need not be like an Eliezer-style snap of the fingers foom, but even over, say, a decade, let's imagine that over the next 10 years that AI is kind of multiple architectures develop and they sort of get integrated and we have something that kind of looks like robust silicon-based intelligence, maybe not totally robust, but as robust or more robust than us and running faster and the kind of thing that can do lots of full jobs or maybe even be tech CEOs, then it kind of feels like a lot of this analysis probably doesn't hold, right? Because we're just in a totally different regime that is just extremely hard to predict. And I guess I wonder, first of all, do you agree with that? There seems to be a big fork in the road there that's like, just how fast do these AIs become super powerful or do they not? And if they don't, then yeah, I think we're much more into realpolitik type of analysis. But I'm not at all confident in that. To me, it feels like there's a very real chance that AI of 10 years from now is— and by the way, this is like what the leaders are saying, right? I mean, OpenAI is saying this, Anthropic is saying this, Demis and Shane Legg are certainly saying things like this. It seems like they expect that we will have AIs that are more powerful than any individual human, and that becomes like the bigger question than anything else. So do you agree with that kind of division of scenarios, first of all? And then maybe you could kind of say how likely you think each one is. And obviously that one where it takes off is super hard to analyse. And I also definitely think it is worth analysing the scenario where it doesn't take off. But I just wanted to flag that it seems like there's a big— if you talk to the AI safety people, any world in which it's like, we're suing Indian AI firms in Indian court over IP is like a normal world in their mind, right? And that's not the kind of world that they're most worried about.

I think that there have been some plausible-sounding things that have been said, but I want to just kind of talk about a few technical counterarguments, mathematical or physical, that constrain what is possible. Okay. And actually, Martin Casado and Vijay and I are Working on a long thing on this where, you know, Vijay did Folding@Home. He's a physicist. Martin sold Nasira for, you know, a billion dollars and knows a lot about how a Stuxnet-like thing could work at the systems level. And I've thought about it from other angles and, you know, um, and some of the math stuff, which I'll get to. So for example, one thing, and I'm going to give a bunch of different technical arguments and then let's kind of combine them. Okay. One thing that's been talked about is if you have a superintelligence, it can deliberate for a million years and then it can make one move and it's going to outthink you all the time and so on and so forth. Okay. Well, if you're familiar with the math of chaos or the math of turbulence, there are limits to even very simple systems that you can set up where they can become very unpredictable quite quickly. Okay. So you can, if you want to, engineer a system where you have very rapid divergence of predictability so that it's like the heat death of the universe before you can predict out n timestamps. Do you understand what I'm saying?

This is sort of akin to a Wolfram, even simple rules can generate patterns such that you can't know them without literally computing them.

Yeah, exactly. At least right now, with chaos and turbulence, you can get things that are extremely provably difficult to forecast without actually doing it. Okay. You know, I can make that argument quantitative, but that's just something to look at, right? It's almost like a delta epsilon challenge from calculus. Like, okay, how hard do you want me to make this to predict? Okay, I can set up a problem that is like that, right? It's basically extreme sensitivity to initial conditions lead to extreme divergence in outcomes. So you could design systems to be chaotic that might be AI immune because they can't be forecasted that well. You have to kind of react to them in real time. The ultimate version of this is not even a chaotic system, it's a cryptographic system where— I've got a whole slide deck on this, how AI makes everything fake, easy to fake. Crypto makes it hard to fake again, right? Because crypto, in the broader sense of cryptography, but also in the narrower sense I think crypto is to cryptography as the internet is to computer science. It's like the primary place where all this stuff is applied, but obviously it's not the equivalent. An AI can fake an image, but it can't fake a digital signature unless it can break certain math and solve a factorization problem or something like that. So cryptography is another mathematical thing that constrains AI, similar to chaos and turbulence. Constrains how much an AI can infer things. You can't statistically infer it. Okay. You need to actually have the private key to solve that equation. So that is another math. So I'm going to rules of math, right? Math is very powerful because you can make proofs that will work no matter what devices we come up with. Okay. You start to put an AI in a cage. It can't predict beyond a certain amount because of chaos and turbulence math.

It cannot.

uh, solve certain equations unless it has a private key is because of what we know about cryptography math. Okay. Again, if somebody proves P equals NP, some of this stuff breaks down, but this is within the bounds of our mathematical knowledge right now. Physics-wise, physical friction exists. A lot of physical friction exists. And, uh, a huge amount of the writing on AI assumes by guys like Eliezer, who I like, I don't, I don't dislike it, you know, But it is extremely, uh, it's— there's 2 things that really stick out to me about it. First is extremely theoretical and not empirical, and second, extremely Abrahamic rather than Dharmic or Sinic. Okay, why theoretical and not empirical? It's not trivial to turn something from the computer into a real-world thing. Okay, one of the biggest gaps in all of this thinking Is what are the sensors and actuators? Okay. Because like, if you actually build, you know, I've, I've built industrial robot systems that, you know, 10 years ago, I, you know, a genome sequencing lab with robots. That's hard. That's physical friction. Okay. And a lot of the AI scenarios seem to basically say, oh, it's going to be a self-programming Stuxnet that's going to escape and live off the land. And hypnotize people into doing things. Okay. Now, each of those is actually really, really difficult steps. First is self-programming Stuxnet. Like, this would have to be a computer virus that can live on any device, despite the fact that Apple or Google can push a software update to a billion devices, right? A few executives coordinating almost certainly can. I mean, the off switch exists. Right? Like, this is actually like the core thing. Lots of AI safety guys get themselves into the mind state that the off switch doesn't exist. But guess what? There's almost nothing living that we haven't been able to kill. Right? Like, can we kill it? This thing exists. And this is getting back to living off the land. Even if you had like something that could solve some of the other technical problems that I'll get to, it exists as an electromagnetic wave. kind of thing on a certain, you know, on chips and so on and so forth. Taking it out in the environment is like putting a really smart human into outer space, right? Your body just explodes and you die. It doesn't matter how smart you are. That's strength on this axis, but you're weak on this axis. And, you know, so strength on the X-axis is not strength on the Y or the Z-axis. An AI outside, you know, pour water on it. This is why I mean the 50 IQ, 150 IQ thing. You know, the 150 IQ way of saying it is it's strong on this X and weak on this X. And the 50 IQ way is pour water on it, disconnect it, you know, turn the power off. Okay, right? Like, it'll, it'll be very difficult to build a system where you literally cannot turn it off. The closest thing we have to that is actually not Stuxnet. It's Bitcoin. And Bitcoin only exists because millions of humans keep it going. So you, you need— so that gets the second point, living off the land. For an AI to live off the land, meaning without human cooperation, okay, that's the next Turing threshold. An AI to live without human cooperation, it would need to be able to control robots sufficient to dig ore out of the ground, set up data centers and generators and connect them, and defend that against human attack. Literally a Terminator scenario. Okay. That's a big leap in terms of, I mean, is it completely impossible? I can't say it's completely impossible, but it's not happening tomorrow. No matter what your AI timelines are, you would need to have like a billion or hundreds of millions of internet-connected autonomous robots that this Stuxnet AI could hijack that were sufficient to carve ore out of the earth and, you know, set up data centers and make the AI duplicate. We're not there. That's a huge amount of physical friction. That's AI operating without a human to make itself propagate, right? A human doesn't need the cooperation of a lizard to self-replicate. For an AI to replicate right now, it would need the cooperation of a human in some sense, because otherwise those humans can kill it because there's not that many different pieces of operating systems around the world. I'm just talking about the practical constraints of our current world, right? Actually existing reality, not AI safety guys' reality where all these things don't exist. There's just a few operating systems, just a few countries. If everybody's going with torches and searchlights through the internet, it's very hard for a virus to continue.

Okay.

So A, on the practical difficulties, there's the technical stuff with chaos and turbulence and with cryptography itself where AI can't predict. And it can't solve certain equations. B, on the physical difficulties, it probably— I mean, like, to be a Stuxnet, Microsoft and Google and so on could kill it. The off switch exists. Can it live off the land? No, it cannot, because it doesn't have, you know, drones to mine ore and stuff out of the ground. And, um, can it, like, exist without humans? Can it be this hypnotizing thing? Okay. So the hypnotizing thing, by the way, this is one of the things that's the most hilarious self-fulfilling prophecy in my view. Okay. And no offense to anybody listening to this podcast, but I think the absolutely dumbest kind of tweet that I've seen on AI is, I typed this in and oh my God, it told me this. Like, I asked it how to make sarin gas and it told me X or whatever, right? That's just a search engine. Okay. What, what basically a lot of these people are doing is they're saying, what if there were people out there that were so impressionable That they would type things into an AI and, and follow it as if they were hearing voices. And that's actually not the, the, the model or whatever that's doing it. That's like this AI cult that has evolved around the world, like, uh, Aum Shinrikyo, you know, that, that hears voices and does like the sarin gas. The point is an AI can't just like hypnotize people. Those people have to like participate in it. They're typing things into the machine or whatever. Okay. Now you might say, all right, let's project out a few years. In a few years, what you have is you have an AI that is not just text, but it appears as Jesus. What would, what would AI Jesus do? What would AI Lee Kuan Yew do? What would AI George Washington do? So it appears as 3D. Okay. So it's generating that. It speaks in your language and in a voice. It knows the history of your whole culture. Okay. That would be very convincing. Absolutely be very convincing. But it still can't exist without human programmers who are like the priests tending this AI god, whether it's AI Jesus or AI Lee Kuan Yew or something like that. The thing about the hypnotization thing that I really want to poke on that, are you familiar with the concept of the principal-agent problem? Basically, in every, every time you've got like a CEO and a, and a, uh, a worker, or you have an LP and a VC, or Or you have an employer and a contractor. Every edge there, there are 4 possibilities in a 2x2 matrix: win-win, win-lose, lose-win, lose-lose. Okay? And so for example, win-win is when somebody joins a tech startup, the CEO makes a lot of money and so does a worker. Okay? That's win-win. Lose-lose is they both lose money. Win-lose is the CEO makes money and the employee doesn't. Lose-win is the company fails, but the employee got paid a very high salary. So what equity does is it aligns people. That's where the concept of alignment comes from. It aligns people to the upper left corner of win-win. That's when you have one CEO and one employee. When you have one CEO and 2 employees, you don't have 2 squared outcomes. You have 2 cubed outcomes because you have win-win-win, win-win-lose, win-lose-lose, et cetera, right? Because All 3 people can be win or lose. Okay. CEO can be win or lose. Employee can be win or lose. Employee number 2 can be win or lose. If you have N people rather than 3 people, you have 2 to the N possible outcomes and you have essentially a 2 by 2 by 2 by 2 by 2 by N hypercube of possibilities. Okay. It's all literally just 2 dimensions on each axis. There's tons of possible defecting kinds of things that happen there. So that's why in a large company, there's lose-win coalitions that happen where M people gang up on the other K people and they win while the other people lose. That's how politics happens. When you've got a startup that's driven by equity and the biggest payoff, people don't have to try to think, okay, well, I make more money by politics. They'll make more money by the win-win-win-win-win column because the exit makes everybody make the most money. That's actually how the OpenAI people were able to coordinate around, we want an $80 billion company. The economics helped find the sell that was actually the most beneficial to all of them, helped them coordinate. Okay. So you search that hypercube.

Okay.

That's a point of equity as lining. Still, Despite all of this, that's one of our best mechanisms for coordinating large numbers of people in the principal-agent problem. Despite all this, the possibility exists for any of these people to win while the others lose. Right with me so far? And I'll explain why this is important. What that means is those 1,000 employees of the CEO are their own agents with their own payoff functions that are not perfectly aligned with the CEO's payoff function. As such, there are scenarios under which they will defect and do other things. Okay. The only way they become like actual limbs, see, my, my hand does not, is not an agent of its own. It lives or dies with me. Therefore, it does exactly what I'm saying at this time. I tell it to go up, it goes up. Tell it to go down, it goes down. Sideways, sideways, right? An employee is not like that. They will do this and this and sideways, sideways. Up to a certain point, and if you if you have them do something that's extremely against their interests, they will not do your action. Do you understand my point? Okay, that is the difference between an AI hypnotizing humans versus an AI controlling drones. An AI controlling drones is like your hands; they're actually pieces of your body. There's no defecting. There's no loose wind. They have no mind of their own. They're literally taking instructions. Okay, they have no payoff function. They will kill themselves for the horde. Right. An AI hypnotizing humans has 1,000 principal agent problems for every 1,000 humans, and it has to incentivize them to continue and has to generate huge payoffs. It's like an AI CEO. That's really hard to do. Right. The history of evolution shows us how hard it is to coordinate multicellular organisms. You have to make them all live or die as one. Then you get something along these lines. Like, an ant colony can coordinate like that because if the queen doesn't reproduce. All the ants, it doesn't matter what their haplogroup or genetic material. Okay. We are not currently set up for those humans to not be able to reproduce unless the AI reproduces. Do I think we eventually get to a configuration like that? Maybe. Where you have an AI brain is at the center of civilization and it's coordinating all the people around it. And every civilization that makes it is capable of crowdfunding and operating its own AI. That gets me to my other critique of the AI safety guys. I mentioned that the first critique is very theoretical rather than empirical. And the second critique is they're Abrahamic rather than Dharmic or Cynic. Okay. And, you know, our background culture influences things in ways we don't even think about. So much of the paperclip thinking is like a vengeful God will turn you into pillars of salt, except it's a vengeful, you know, AI God will turn you into paperclips.

Okay.

The polytheistic model of many gods as opposed to one God is we're all going to have our own AI gods and there'll be war of the gods, like Zeus and Hera and so on. That's the closest Western version, you know, the paganism that predated, you know, Abrahamic religions. But that's still there in India. That's still how Indians think. That's why India is sort of— people have gotten so woke that they don't even make large-scale cultural generalizations anymore. But it's true that India is just culturally more amenable to decentralization, to multiple gods rather than one god and one state. And then the Chinese model is yet the opposite. Of course, they have their tech entrepreneurs and so on, but if India is more decentralized, China is more centralized. They have one government and one leader for the entire civilization. And the biggest thing that China has done over the last 20 or 30 years is they've taken various US things, they've made sure that they have their own Chinese version where they have root. So they take US social media and they made sure they had root over Sina Weibo. They make sure they have their own Chinese version of electric cars, the most Chinese version. So the private keys, in a sense, are with Xi. So that means that they also, at a minimum, you combine these 2 things, you're at a minimum going to get polytheistic AI of the US and Chinese varieties. And then you add the Indian version on it, and you're going to get quite a few of these different AIs around there. And then you have War of the Gods, where maybe they are good at coordinating the humans who, who, uh, you know, take instructions from them, but they can't live without the humans, and the humans are giving input to them. That's a series of things. I could probably make that clearer if I just laid it out in bullets in an essay, but just to recap it, A, technical reasons like chaos, turbulence, um, Cryptography, why AI is limited in its ability to predict timeframes and to solve equations. B, practical limits. An AI cannot easily be a Stuxnet because Microsoft and Google and Apple can install software on a billion devices and just kill it, right? Like basically guys with torches come. All right. It can't easily live off the land without humans because it would need hundreds of millions of autonomous robots out there to control, to mine the ore and and set up the data centers. It can't just hypnotize humans like it can control drones because the principal agent problem and the degree of human defection. To make those humans do that, you'd have to have such massive alignment between the AI and humans that the humans all know they'll die if the AI dies and vice versa. We're not there. Maybe we'll be there in like, I don't know, n number of years, but not for a while. That's a total change in like how states are organized. Okay. Finally, let me just talk about the physics a little bit more. There's a lot of stuff which is talked about at a very sci-fi book level of it'll just invent nanomedicine and nanotech and kill us all and so on and so forth. Now look, I like Robert Freitas. Obviously Richard Feynman's a genius and so on and so forth, but nanotech somehow hasn't been invented yet. Okay, meaning that, you know, there's a lot of chemists that have worked in this area. Okay, and a lot of, quote, nanotech is like rebranded chemistry because those are the molecular machines You know, for example, DNA polymerase or ribosome, those are molecular machines that we can get to work at that scale, the evolved ones. To my knowledge, and I may be wrong about this, I haven't looked at it very, very recently, we haven't actually been able to make artificial, you know, replicators of the stuff that they're talking about, which means it's possible that there's some practical difficulty that intervened between Feynman and Friedis and so on's calculations, right? Right? Just the sheer fact that those books have came out decades ago and no progress has been made indicates that maybe there's a roadblock that wasn't contemplated, right? So you can't just click your fingers and say, boom, nanomycin. It's sort of like clicking your fingers and saying, boom, time travel, right? Nanomycin exists. That was, that was a good poke that I had a while ago in a conversation like this where the AI guy, AI safety guy on the other side was like, well, time travel, that's too implausible. I'm like, yeah, but you're waiting on, on the nanotech thing. You're thinking is like, here, and you're making so many assumptions there that I want to actually see some more work there. I want to actually see that nanotech is actually more possible than you think it is. Uh, as for, oh, we just need to mix things in a beaker and make a, you know, virus and so on and so forth— you know what is really, really good at defending against novel viruses? Like, the human immune system. That's something that's within envelope, right? Like, you have evolved to not die and to fight off viruses. Is it possible that maybe you could make some supervirus? I mean, maybe, but again, like, humans are really good and the immune system is really good at that kind of thing. That is what we're set up to do, right? To adapt to that. Billions of years of evolution have been set up for that. Physical constraints are not really contemplated when people talk about these super powerful AIs. Mathematical constraints, practical constraints are not contemplated. And I could give more, but I think that was a lot right there. Let me pause right there.

Yeah, let me try to steelman a few things. And then I do think before too long, I want to kind of get back to the somewhat less radically transformative scenarios and ask a few follow-up questions on that too. But I think for starters, I would say the sort of Eliezer, he's updated his thinking over time as well, and I would say probably doesn't get quite enough credit for it because he's definitely on record repeatedly saying, yeah, I was kind of expecting more something from the DeepMind school to pop out and be wildly overpowered very quickly. And on the contrary, it seems like we're in more of a slow takeoff type of scenario where we've got these, again, super high surface area, kind of suck up all the knowledge, gradually get better at everything. Some surprises in there, certainly some emergent properties, if you will accept that term. Surprises to the developers, if nothing else, right, that are Definitely things we don't fully understand, but it does seem to be a more gradual turning up of capability versus some super sudden surprise. But okay, so then what is the alternative? I'm going to try to kind of give you what I think of as the most consensus, strongest scenario where humans lose track of the future. or lose control of the future, maybe starting by kind of losing track of the present and then having that kind of give way to losing control of the future. And I think within that, by the way, I'm not really one who cares that much about whether AIs say something offensive today. I'm not easily offended and whatever.

That's not world-ending. I understand your point. That's not like, who cares, whatever. That's within scope. That's within envelope.

Within this bigger kind of what is the real most likely path to AI disaster as understood, I think, by the smartest people today. I think that is still a useful leading indicator because it's like, okay, the developers, whether you agree with their politics, whether you think their commercial reasons are their sincere reasons or not, they have made it a goal to get the AI to not say certain things, right? They don't want it to be offensive. The most naive kind of down the fairway interpretation of that is like, hey, they want to sell it to corporate customers. They know that their corporate customers don't want to have their AI saying offensive things, so they don't want to say offensive things, and yet they can't really control it. It's still pretty easy to break. So I view that as just kind of a leading indicator of, okay, we've seen GPT-2, 3, and 4 over the last 4 years, and that's a big delta in capability. How much control have we seen developed in that time, and does it seem to be keeping pace? And my answer would be, on the face of it, it seems like the answer is no. We don't have the ability to really dial in the behavior such that we can say, okay, you can expect, you can trust that these AIs will not do A, B, and C. On the contrary, it's like, if you're a little clever, you can get them to do it.

You can break out of the sandbox on it.

Yeah. And it's not even like, I mean, we've talked about things where you have access to the weights and you're doing counter optimizations, but you don't even need that. The kind of stuff I do in my red teaming in public is literally just like, feed the AI a couple of words, put a couple of words in its mouth, and it will kind of carry on from there. So with that in mind, it's just a leading indicator. I don't know how powerful the most powerful AI systems get over the next few years, but it seems very plausible to me that it might be as powerful as an Elon Musk type figure, somebody who's really good at thinking from first principles, really smart, really dynamic across a wide range of different contexts. And he's not powerful enough to, in and of himself, take over the world, but he is kind of becoming transformative. Now imagine that you have that kind of system and it's trivial to replicate it. So if you have one Elon Musk, all of a sudden you can have arbitrary, functionally arbitrary numbers of Elon Musk power things that are clones of each other.

Maybe I can pause you there. So that's my polytheistic AI scenario. But here's the thing, that is, this is background, but I want to push it to foreground. You still have a human typing in things into that thing. The human is doing the jailbreak, right? What we're talking about is Not artificial intelligence in the sense of something separate from a human, but amplified intelligence. Amplified intelligence, I very much believe in. The reason is amplified intelligence. So here's something that people may not know about humans. There's this great book, Cooking Made Us Human. Okay. Tool use has shifted your biology in the following way. For example, and I'll map it to the present day. There's a book by Richard Wrangham, Cooking Made Us Human, where the fact that we started cooking and using fire meant that we could do metabolism outside the body, which meant it freed up energy for more brain development.

Okay.

Similarly, developing clothes meant that we didn't have to evolve as much fur. Again, more energy for brain development. Evolving tools meant we didn't have as much fangs and claws and muscles, Again, more energy for brain development, right? So encephalization quotient rose as tool use meant that we didn't have to do as much natively and we could push more to the machines. In a very real sense, we have been a man-machine symbiosis since the invention of fire and the stone axe and clothes, right? You do not exist as a human being on your own. Like the entire Ted Kaczynski concept of like living in nature by itself. Humans are social organisms that are adapted to working with other humans and using tools. And you have for, and we have been for millennia. Okay. This goes back not just human history, but like hundreds of thousands of years before hunter gatherers are using tools. Okay. So what that means is man-machine symbiosis is not some new thing. It's actually the old thing that broke us away from other primate lineages that weren't using tools. Okay. This is the fundamental difference between what I call Uncle Ted and Uncle Fred. Uncle Ted is Ted Kaczynski. It's a Unabomber. It's a doomer. It's a decelerator, the degrowther who thinks we need to go back to Gaia and Eden and become monkeys and live in the jungle like, like, you know, Ted Kaczynski, right? The, the, uh, the Unabomber style. Uncle Fred is Friedrich Nietzsche, right? Nietzschean. We must get to the stars and become Ubermen and so on and so forth. This, I think, is going to become, and I actually tweeted about this years ago before the current AI debates, the, you know, between anarcho-primitivism, degrowth, deceleration, okay, on the one hand, and transhumanism and acceleration and Human 2.0 and human self-improvement and make it to the stars on the other hand. This is the future political axis, the current one. And roughly speaking, you can cut, it's not really left and right. Because you'll have both left statists and right conservatives go over here. Left statists will say it's against the state, and the right conservatives will say it's against God. And you'll have left libertarians and right libertarians over here, where left libertarians say it's my body, and the right libertarians say it's my money. And so that is a re-architecting of the political axis where Uncle Ted and Uncle Fred, which is a kind of clever way of putting it, Okay. And the problem with the Uncle Ted guys, in my view, is, as I said, yeah, if they go and want to live in the, you know, the woods, fine, go get them. But once you start having even like a thousand— forget a thousand, 100 people doing that, your, your trees will very quickly get exfoliated. You know, the leaves are going to get all picked off of them. Humans are not set up to just literally live in the jungle right now. You've had hundreds of thousands of years of evolution that have driven you in the direction of tool use, social organisms, farming, et cetera, et cetera. The man-machine symbiosis is not today. It's yesterday and the day before and 10,000 years ago and 100,000 years ago. And how do we know we've got man-machine symbiosis? Can you live without, uh, even if you're not living, even if you're not using the stove, somebody's using a stove to make you food, right? Can you live without the tractors that are digging up the grains? Can you live without Um, indoor heating? Can you live without your clothes? Frankly, can you do your work without your phone, without your computer? No, you can't. You are already a man-machine symbiosis. Once we accept that, then the question is, what's the next step? And right now we're in the middle of that next step, which is AI is amplified intelligence. So what you're talking about is not that the AI is Elon Musk. It is that the AI-human fusion Means there's another 20 Elon Musks or whatever the number is. Okay. And that's good. That's fine. That's within envelope. That's just a bunch of smarter humans on the planet. That is amplified intelligence. That is more like, uh, you know, I mentioned the tool thing. Okay. The other analogy would be like a dog. You know, a dog is man's best friend, right? So that AI does not live without you. Humans can turn it off. They have to power it. They have to give it sustenance, right? Eventually, that might become like a ceremonial thing. Like, this is our god that we pray to, right? Because it's wiser and smarter than us, and it appears in an image. But the priests maintain it. You know, just like, you know, you go to a Hindu temple or something like that, and the priests will pour out the ghee, you know, for the fires and so on and so forth. And then everybody comes in and prays. Okay. The priests believe in the whole thing, but they also maintain the back of the house. do the system administration for the temple. Same, you know, in a Christian church, right? The, the, the, you know, like, it's not like it appears out of nowhere. Somebody, you know, uh, went and, and assembled this cathedral, right? They saw the back of the house, the fact that it was just woods and rocks and so on that came together. But then when people come there, it feels like a spiritual experience. You see what I'm saying? Okay. So the equivalent of that, the priests, uh, or the, uh, the, you know, the people maintaining temples, cathedrals, mosques, whatever, is, um, engineers who are maintaining these future AIs, which appear to you as Jesus. They appear to you maybe even a hologram. Okay, you come there, you ask it for guidance as an oracle. You've also got the personal version on your phone. You ask it for guidance, but guess what? Uh, you're still a human-AI symbiosis until And unless that AI actually has the Terminator scenario where it's got lots of robots and it can live on its own, I'm not saying that's physically impossible. I did give some constraints on it earlier, but for a while, we're not going to be there. So that alone means it's not Foom because we don't have lots of drones running around. The AI has to be with the human. It's a human-AI symbiosis. It's not AI Elon Musk. It is human-AI fusion that becomes Elon Musk. And frankly, that's not that different from what Elon Musk himself is. Elon Musk would not be Elon Musk without the internet. Without the internet, he can't tweet and reach 150 million people. The internet itself made Elon what he is, right? And, uh, and so this is like the next version of that. Maybe there's now 30 Elons because the AI makes the next 30 Elons.

Yeah, I mean, again, I think I'm largely with you with just this one very important nagging worry that's like, what if this time is different? Because what if these systems are getting so powerful so quickly that we don't really have time for that techno-human fusion to really work out. And I'll just give you kind of a couple of data points on that. You said it's still somebody putting something into the AI. Well, sort of, right? I mean, already we have these proto agents, and the super simple scaffolding of an agent is just run it in a loop, give it a goal, and have it kind of pursue some plan, act, get feedback, and loop type of structure, right? It doesn't seem to take a lot. Now, they're not smart enough yet to accomplish big things in the world, but it seems like the language model to agent switch is less one right now that is gated by the structure or the architecture and more one that's just gated by the fact that the language models, when framed as agents, just aren't that successful at doing practical things and getting over humps. So they tend to get stuck. But it doesn't seem that hard to imagine that if you had something that is sort of that next level, that you put it into a loop, you say, okay, you're Elon Musk LLM, and your job is to make us, whatever us exactly is, a multiplanetary species. And then you just kind of keep updating your status, keep updating your plans, keep trying stuff, keep getting feedback. Like, what really limits that?

There may be like a really good program, but the whole AI kills everyone thing is so— it's like, where's the actuator? Okay, I hit enter. What kills me, right? Is it a hypnotized human who's been hypnotized by an AI that he's typed into and he's radicalized himself by typing into a computer? Okay, that's not that different from a lot of other things that have happened in the past. Right. So who is actually striking me? Right. Who's striking the human? It's another human with an ax that he's been radicalized by an AI. Okay. He's not actually, that's not even the right term. We're giving agency to the AI when it's not really an agent. It is a human who is self-radicalized by typing into a computer screen and has hit another human. That's one scenario. The other scenario is it's literally a Skynet drone that's hitting you. Those are the only two. How else is it going to be physical? Right. How does AI— the actuation step is a part that is skipped over, and it's a nontrivial step.

Well, I think it could be lots of things, right?

I mean, if it's not one of those two, if it's not another human or a drone hitting you, what is it?

Just habitat degradation, right? I mean, how do we kill most of the other species that we drive to extinction? We don't go out and hunt them down with axes one by one. We just change the environment more broadly to the point where it's not suitable for them anymore and they don't have enough space and they kind of die out, right? So we did hunt down some of the megafauna, literally one by one with spears and stuff. But most of the recent loss of species is just like, we're out there just extracting resources for our own purposes. And in the course of doing that, whatever bird or whatever thing just kind of loses its place and then it's no more.

And I don't think that's totally implausible Wait, so that is though, I think, within normal world, right? What does that mean? That means that some people, some amplified intelligence, and maybe we might call it HAI, okay, human plus AI combination, right? Some HAIs outcompete others economically and they lose their jobs. Is that what you're talking about?

I think also the humans potentially become unnecessary in a lot of the configurations. Just a recent paper from DeepMind.

Zero marginal product workers or negative.

Yeah. DeepMind has been on— Google DeepMind has been on a tear of increasingly impressive medical AIs. Their most recent one takes a bunch of difficult case studies from the literature. I mean, case studies, this is like rare diseases, hard to diagnose stuff. And asks an AI to do the differential diagnosis, compares that to human, and compares it to human plus AI. And they phrase their results in a very understated way, but the headline is the AI blows away the human plus AI. The human makes the AI worse.

So here's the thing. I'll say something provocative, maybe. Okay. Like I have in a way. Fine. I do think that the ABCs of economic apocalypse for blue America are AI, Bitcoin, and China. Where AI takes away their, a lot of the revenue streams, the licensures that have made, uh, medical and legal costs and other things so high. Bitcoin takes away their power over money and China takes away their military power. So I, I foresee total meltdown for blue America, um, in the years and, you know, maybe decade to come. Already kind of happening, but that's different than being at the end of the world.

Right.

Like blue America had a really great time for a long time. And they've got these licensure locks, but because of that, they've hyperinflated the cost of medicine. It's like, how much, how, so what you're talking about is, wow, we have infinite free medicine, man. Doctor billing events are going to get a hit. That's the point.

Yeah. And to be clear, I'm really with you on that too. Like I want to see one of the things when people say like, what is good about AI? You know, why should we, why should we pursue this? My standard answer is high-quality medical advice for everyone at pennies per visit, right? It is orders of magnitude cheaper. We're already starting to see that in some ways it's better. People prefer it, that AI is more patient, it has better bedside manner. I wouldn't say if I was giving my own family advice today, I would say use both a human doctor and an AI, but definitely use the AI as part of your mix.

Absolutely. That's right. That's right. But you're prompting it still, right? The smarter you are, the smarter the AI is. You notice this immediately with your vocabulary, right? The more sophisticated your vocabulary, the finer the distinctions you can have, the better your own ability to spot errors. You can generate a basic program with it, right? But really amplified intelligence is, I think, a much better way of thinking about it because whatever your IQ is, it surges it upward by a factor of 3 or whatever the number And maybe the amplifier increases with your intelligence, but that internal intelligence difference still exists. It's just like what a computer is. A computer is an amplifier for intelligence. If you're smart, you can hit enter and programs can go to, like, think about the Minecraft guy, right? Or Satoshi. One person built a billion or in Satoshi's case, trillion dollar thing, you know, obviously other people continued Bitcoin and so on and so forth, right? So what I feel though is this is what I mean by going from nuclear terrorism to the TSA. Okay, we went from AI will kill everyone, and I'm like, what's the actuator? To, okay, it'll gradually degrade our environment. What does that mean? Okay, some people will lose their jobs, but then we're back in normal world.

Well, hold on, let me paint a little bit more complete picture because I don't think we're quite there yet. So I think the differential diagnosis recent paper, that's just a data point where it's kind of like chess. This came long before, right? There was a period where Humans are the best chess players. Then there was a period where the best were the hybrid human AI systems. And now, as far as I understand it, we're in a regime where the human can't really help the AI anymore. And so the best chess players are just pure AIs. We're not there in medicine, but we're starting to see examples where, hey, in a predefined study differential diagnosis, the AI is beating, not just beating the humans, but also beating the AI-human hybrid or the human with access to AI. Okay, that's not it, right? There's a paper recently called Eureka out of Nvidia. This is Jim Fan's lab where they use GPT-4 to write the reward functions to train a robot. So you want to train a robot to twirl a pencil in fingers. Hard for me to do. Robots definitely can't do it. How do you train that? Well, you need a reward function. The reward function basically, while you're in the early process of learning and failing all the time, the reward function gives you encouragement when you're on the right track, right? So there are people who have developed this skill, and you might do something like, well, if the pencil has angular momentum, then that seems like you're on maybe sort of the right track. So give that a reward, even though at the beginning you're just failing all the time. Turns out GPT-4 is way better than humans at this, right? So it's better at training robots.

So all of that is awesome and it's great. And, but here is, here's the thing is there's a huge difference between AI is going to kill everybody and turn everybody into paperclips. Okay. Versus some humans with some AI are going to make a lot more money and some people are going to lose their jobs.

Yeah, I'm not scared of that. I'm not scared of that scenario. I mean, it could be disruptive. It could be disruptive, but it's not existential unto itself.

Big no. Okay, so that's why I went right. There's a— the, the beta to me, it comes if I, if I ask just one question is what is the actuator, right? You know, sensors and actuators, right? What is the thing that's actually going to plunge a knife or a bullet into you and kill you? It is either a human who has hypnotized themselves by typing into a computer, like basically an AI terrorist, you know, which is Kind of where some of the EAs are going, in my view. Or it is like an autonomous drone that is controlled in a StarCraft or Terminator-like way. We are not there yet in terms of having enough humanoid or autonomous drones that are internet-connected and programmable. That won't be there for some time. Okay. So that alone means fast takeoff is— and what I think by the time we get there, you will have— Uh, cryptographic control over them. That's a crucial thing. Cryptography fragments the whole space in a very fundamental way. If you don't have the private keys, you do not have control over— so long as that piece of hardware, the cryptographic controller, you've nailed the equations on that. And frankly, you can use AI to attack that as well to make sure the code is perfect, right? Crypto— remember you talked about attack and defense? AI is attack, crypto is defense. Right, because, um, one of the things that crypto has done— do you know what the PKI problem is? Public key infrastructure?

I'll say no on behalf of the audience.

This is good. We should do more of these, actually. I feel it's a good, you know, fusion of things or whatever, right? But the public key infrastructure problem— the, the public key infrastructure problem is something that was sort of lots of cryptography papers and computer science papers in the '90s and 2000s assumed that this could exist. And essentially meant if you could assume that everybody on the internet had a, uh, public key that was public and a private key that was kept both secure and available at all times, then there's like all kinds of amazing things you can do with privacy-preserving messaging and authentication and so on. The problem is that for many years what, what cryptographers try to do is they try to nag people into keeping their private keys secure and available. And the issue is, it's trivial to keep it secure and unavailable, where you write it down, you put into a lockbox, and you lose the lockbox. It's trivial to keep it available and not secure, okay, where you, uh, you put it on your public website and it's available all the time, you never lose it, but it's, it's not secure because anybody can see it. When you actually ask, what does it mean to keep something secure and available, That's actually a very high cost. It's precious space because it's basically your wallet, right? Your wallet is on your person at all times, so it's available, but it's not available to everybody else, so it's secure. So you actually have to like touch it constantly. Yes. Right. So it turns out that the crypto wallet, by adding a literal incentive to keep your private keys secure and available, Because if they're not available, you've lost your money. If they're not secure, you've lost your money. Okay, to have both of them— that was what solved the PKI problem. Now we have hundreds of millions of people with public-private key pairs where the private keys are secure and available. That means all kinds of cryptographic schemes, zero-knowledge stuff— there's this amazing universe of things that is happening now. Zero-knowledge in particular has made cryptography much more programmable. There's a whole topic Which is, uh, if you want something that's kind of, you know, like AI was creeping for a while and people, specialists were paying attention to it and then it just burst out on the scene. Zero knowledge is kind of like that for cryptography. Thanks to the, you know, you've probably heard of zero knowledge before.

Yeah, we did one episode with, um, Daniel Kang on the use of zero knowledge proofs to basically guarantee, to prove without revealing like the weights. that you actually ran the model you said you were going to run and things like that, I think are super interesting.

Exactly right. So what kinds of stuff, why is that useful in the AI space? Well, first is you can use it, for example, for training on medical records while keeping them both private, but also, uh, getting the data you want out of it. For example, let's say you've got a collection of, um, genomes, okay? And you want to ask Okay, how many Gs were in this dataset? How many Cs? How many As? How many Ts? Okay, like you just say, like, that's a very simple analysis. What's the ACGT content of this, you know, the sequence dataset? You could get those numbers. You could prove they were correct without giving any information about the individual sequences, right? Or more specifically, you do it at one locus and you say, how many Gs and how many Cs are at this particular locus? And you get the SNP distribution. Okay. So. So it's useful for what you just said, which is like showing that you ran a particular model without giving anything else away. It's useful for certain kinds of data analysis. There's a lot of overhead on compute on this right now, so it's not something that you do trivially, but it'll probably come down with time. But what is perhaps most interestingly useful for is, in the context of AI, is coming up with things an AI can't fake. So what we talked about earlier, right? Like, an AI can come up with all kinds of plausible-sounding images, but if it wasn't cryptographically signed by the sender, then, you know, it should be signed by the sender and put on-chain. And then at least you know that this person or this entity with this private key asserted that this object, existed at this time in a way that'd be extremely expensive to falsify because it's either on the Bitcoin blockchain or another blockchain. It's very expensive to rewind. Okay, this starts to be a bunch of facts that an AI can't fake.

So going back to the kind of big picture loss of control story, I was just kind of trying to build up a few of these data points that, like, hey, look at this differential diagnosis. We already see humans are not really adding value to AIs anymore. That's kind of striking. And similarly with training robot hands, GPT-4 is outperforming human experts. And by the way, all of the sort of latent spaces are totally bridgeable, right? I mean, one of the most striking observations of the last couple of years of study is that AIs can talk to each other in high-dimensional space, which we don't really have a way of understanding natively, right? It takes a lot of work for us to decode.

This is like the language thing.

We're starting to see AIs kind of develop, not obviously totally on their own as of now, but there is becoming an increasingly reliable go-to set of techniques if you want to bridge different modalities with a pretty small parameter adapter.

That's interesting. What's a good paper on that? I actually hadn't seen that.

The BLIP family of models out of Salesforce Research is really interesting, and I've used that in production at Salesforce.

Really?

Yeah, Salesforce Research. They have a crack team that has open sourced a ton of stuff in the language model computer vision joint space. And you see this all over the place now. But basically what they did in a paper called BLIP-2, and they've had like 5 of these with a bunch of different techniques, but in BLIP-2, they took a pretrained language model, and then a pretrained computer vision model. And they were able to train just a very small model that kind of connects the two. So you could take an image, put it into the image space, then have their little bridge bridge that over to language space. And everything else, the 2 big models are frozen. So they were able to do this on just like a couple of days worth of GPU time, which I do think goes to show how it is going to be very difficult to contain proliferation.

Which is good. In my view, that's really good.

As long as it doesn't get out of control. I am broadly with you on that too. But by bridging this vision space into the language space, then the language model would be able to converse with you about the image, even though the language model was never trained on images. But you just had this connector that kind of bridges those modalities.

It's like another layer of the network that just bridges 2 networks almost.

Yeah, it bridges the spaces. It bridges the conceptual spaces between something that has only understood images and something that has only understood language. But now you can kind of bring those together.

As I think about it, it's not that surprising because that's what, for example, text-to-image models are basically that. They're bridging 2 spaces in a sense. Right. But I'll check this paper out. So on the one hand, it's not that surprising. On the other hand, I should see how they implemented it or whatever. So Blip-2.

Okay.

Yeah, I think the most striking thing about that is just how small it is. Like, you took these 2 off-the-shelf models that were trained independently for other purposes, and you're able to bridge them with a relatively small connector. And that seems to be kind of happening all over the place. I would also look at the Flamingo architecture, which is like a year and a half ago now out of DeepMind. That was one for me where I was like, oh my God. And it's also a language to vision where they keep the language model frozen. And then they kind of, in my mind, it's like I can see the person in their garage tinkering with their soldering iron because it's just like, wow, you took this whole language thing that was frozen and you kind of injected some vision stuff here and you added a couple of layers and you kind of Frankensteined it and it worked. And it's like, wow, that's not really— it wasn't like super principled. It was just kind of hack a few things together and try training it. And I don't want to diminish what they did because I'm sure there were more insights to it than that. But it seems like we are kind of seeing a reliable pattern of the key point here being model-to-model communication through high-dimensional space, which is not mediated by human language is, I think, one of the reasons that I would expect. And by the way, there's lots of papers too on language models are human level or even superhuman prompt engineers. Their self-prompting techniques are getting pretty good. So if I'm imagining the big picture of— and we can get back to, okay, well, how do we use any techniques, crypto or otherwise, to keep this under control? And I would say this is kind of the newer school of the big picture AI safety worry, obviously there's a lot of flavours, but if you were to go look at Ajeya Katra, for example, I think is a really good writer on this. Her worldview is less that we're going to have this foom and more that over a period of time, and it may not be a long period of time, maybe it's like a generation, maybe it's 10 years, maybe it's 100 years, but obviously those are all small in the sort of grand scheme of the future. We have, in all likelihood, the development of AI-centric schemes of production where you've got kind of your high-level executive function is like your language model. You've got all these lower-level models. They're all bridgeable. All the spaces are bridgeable in high-dimensional form where they're not really mediated by language unless we enforce that. I mean, we could say it must always be mediated by language so we can read the logs. But there's a tax to that, right? Because going through language is highly compressed compared to the high-dimensional space to space.

All right, so let me see if I can steelman or articulate your case. You're saying AIs are going to get good enough, they're going to be able to communicate with each other good enough, and they'll be able to do enough tasks that more and more humans will be rendered economically marginal and unnecessary.

I'm not saying I think that will happen. I'm just saying I think there's a good enough chance that that will happen that it's worth taken really seriously.

I actually think that will happen, something along those lines, uh, or in the sense of at least massive economic disruption. Definitely. Okay, but I'll give an answer to that which is both, you know, maybe fun and not fun. Have you seen the, um, you've seen the graph of the percentage of America that was involved in farming?

Yeah, I tweeted a version of that once.

Oh, you did? Okay, great. Good. So you're familiar with this and you're familiar with what I mean by the implication of it. Where basically Americans used to identify themselves as farmers, right? And manufacturing rose as agriculture collapsed, right? And here is the graph on that. But from like 40% in the year 1900 to like a total collapse of agriculture, and then also more recently a collapse of manufacturing into bureaucracy, paperwork, legal work. What has happened to the right since then is, uh, you know, the, um, the, the lawyers. What is up and to the right? What is replacing that? Starting in around the 1970s, um, we used to be adding energy production, and energy production flatlined once people got angry about nuclear power. So this is a future that could have been. We could be on Mars by now, but we got flatlined, right? What did go up and to the right? So construction costs— this is the bad scenario where The miracle energy got destroyed because regulations, the cost was flat and then went vertical when regulations were imposed. All the progress was stopped by decels and degrowthers. And then Alara was implemented, which said nuclear energy has to be as low risk as, as reasonably necessary, as reasonably achievable. And that meant that you just keep adding, quote, safety to it until it's the same cost as everything else, which means you destroy the value of it. Right. Um, but you know what was up into the right? What replaced those agriculture and manufacturing jobs? Look at this. You see this graph?

For the audio only, we will put this on YouTube. So if you want to see the graph, do the YouTube version of this. For the audio only group, it's an exponential curve in the number of lawyers in the United States from, looks like maybe 2/3 of a million to 13 million over the last 140 years.

Yeah. And in 1880, it was like, like sub 100,000 or something like that. Right. And then it's just like, especially that 1970 point, that's when it went totally vertical. Okay. And it's probably even more since. So, you know, if you add paperwork jobs, bureaucratic jobs, you know, every lawyer is like, you know, sorry lawyers, but you're basically negative value add, right? Because it should— the fact that you have a lawyer means that you couldn't just self-serve a form, right? Basically government is platforms where you can just self-serve and you fill it out. And you don't have to have somebody like code something for you custom. You know, lawyers that's doing custom code is because the legal code is so complicated. So, you know, the whole Shakespeare thing, like first thing we do, let's, you know, kill all the lawyers. First thing we do, let's automate all the lawyers, right? Only something that's the hammer blow of AI can break the backbone, and it will. That's the— it's going to break the backbone of blue America, right? It's going to cause That's why the political layer and the sovereignty layer is not what AI people think about, but it's crucial for thinking about AI. Because what tribes does AI benefit? And again, we got away from— why does AI kill everybody? Well, it's going to need actuators. Who's going to stab you? Who's going to shoot you? It's got to be a human hypnotized by AI or a drone that AI controls. A human hypnotized by AI is actually a conventional threat. It looks like a terrorist cell. We know how to deal with that. Right? It's just like radicalized humans that worship some AI that stab you. It's like the pause AI people are one step, I think, away from that. All right, but that's just like Aum Shinrikyo. That's like Al Qaeda. That's like basically terrorists who think that the AI is telling them what to do. Fine. If it's not a human that's stabbing you, it is a drone, and that's like a very different future where, like, five or ten or fifteen years out, maybe we have enough internet-connected drones out there, but even then they'll have private keys. So there's going to be fragmentation of address space. Not all drones will be controllable by everybody, in my view. Okay, that's what AI safety is. AI safety is, can you turn it off? Can you kill it? Can you stop it from controlling drones? That's what AI safety is. Can you also open the model weights so you can generate adversarial inputs? Can you open the model weights and proliferate it? You're saying, oh, proliferation is bad. I'm saying proliferation is good because if everybody has one, then nobody has an advantage on it, right? Not relatively speaking. Okay.

I have very few super confident positions, so I wouldn't necessarily say I think that proliferation is bad. I'd say so far it's good. And even most of the AI safety people, I would say, if I could speak on the behalf of the AI safety consensus, I would say most People would say even that the Llama 2 release has proven good for AI safety for the reasons that you're saying.

But they opposed it.

Well, some did and some didn't. I would say the main posture that I see AI safety people taking is that we're getting really close to, or we might be getting really close. Certainly if we just kind of naively extrapolate out recent progress, it would seem that we're getting really close. to systems that are sufficiently powerful that it's very hard to predict what happens if they proliferate. Llama 2, not there. And so, yes, it has enabled a lot of interpretability work. It has enabled things like representation engineering, which there is a lot of good stuff that has come from it.

The big thing that I want to kind of establish is you agree with me on the actuation point or not? Like, the, the thing is, this thing, like, oh, Llama 2 proliferates, and so businesses are disrupted, and people, you know, maybe, maybe they, they paid a lot of money for their MD degree and they can't make as much money. That's within the realm of what I call conventional warfare, you know what I mean? That's like we're still in normal world as we were talking about, okay? Unconventional warfare is, uh, you know, Skynet arises and kills everybody, okay? And that is what is being sold over here. And when you think about the actuators, we don't have the drones out there. We don't have the humanoid robots they can control. And hypnotized humans are a very tiny subset of humans, probably. And even if they aren't, that just looks like a religion or a cult or a terrorist cell, and we know how to deal with that as well. The super intelligent AI with lots of robots that can control in a StarCraft form, I would agree, is something that humans haven't faced yet. But By the time we get that many robots out there, you, you won't be able to control all of them at once because of the private keys things I mentioned. So that's why I'm like, okay, everything else we're talking about is a normal world. That's— that is the single biggest thing that I wanted to get, like economic disruption, people losing jobs, proliferation so that the balance of power is redistributed. All that's fine. The other— the reason I say this is people keep trying to link AI to existential risk. A great example is one of the things you actually had in here. This is similar to the AI policy institute thing. It's a totally reasonable question, but then I'm going to, in my view, deconstruct the question. What would you think about putting a limit on the right to compute? Are there capabilities an AI system might demonstrate that would make you think open access is no longer wise? Most common near-term answer here seems to be related to risk of pandemic via novel pathogen engineering. So guess what? You know who the novel pathogen engineers are? The US and Chinese governments, right? They did it, or probably did it, credibly did it. Credibly been accused of doing it. They haven't been punished for COVID-19. In fact, they covered up their culpability and pointed everywhere other than themselves. They used it to gain more power in both the US and China, with both lockdown in China and in the US, and all kinds of COVID-era trillions of dollars was printed and spent and so on and so forth. They did everything other than actually solve the problem that was actually getting, you know, the vaccines in the private sector. And they studied the existential risk only to generate it. And they're even paid to generate pandemic prevention and failed. So this would be the ultimate fox guarding the henhouse. Okay, the only reason the 2 organizations responsible for killing millions of people with novel pathogen are going to, uh, prevent people from doing this by restricting compute— no, it— you know what it is actually? What's happening here is one of the concepts I have in the network state is this idea of God, state, and network. Okay, meaning what do you think is the most powerful force in the world? Is it Almighty God? Is it the U.S. government? Or is it encryption? Right, or eventually maybe an AGI. All right, if what was happening here is a lot of people are implicitly, without realizing it, even if they are secular atheists, they're treating GOV as God. Okay, they treat the U.S. government as God, as the final mover.

No, I appreciate your little— I take inspiration from you, actually, in terms of trying to come up with these little quips that are memorable. So I was just smiling at that because I think you do a great job of that. And I try to incorporate— I have less success coining terms than you have, but certainly try to follow your example on that front.

It's like a helpful mnemonic. If you can compress it down, it's like more memorable. So that's what I try to do, right? Right. So exactly. A lot of these people who are secular, think of themselves as atheists, have just replaced God with GOV. They worship the US government as God. And there's 2 versions of this. You know how like God has both the male and female version, right? The female version is the Democrat God within the USA that has infinite money and can take care of everybody and care for everybody. And the Republican God is the US military that can blow up anybody. And it's the biggest and strongest and most powerful America. F yeah.

Okay.

And everybody who thinks of the US government as being able to stop something is praying to a dead god. Okay. When you say this, you actually get an interesting reaction from AI safety people where you've actually hit their true solar plexus. All right. The true solar plexus is not that they believe in AI. It's that they believe in the US government. That's a true solar plexus because they are appealing to— they're praying to this dead god that can't even clean the poop off the streets in San Francisco, right? That is losing wars or fighting them to stalemates, has lost all these wars around the world, that spent trillions of dollars, has been through financial crisis, coronavirus, Iraq War, you know, total meltdown politically, okay? That has interest— that now has interest payments more than the Defense budget that is you know that spent $100 billion on the California train without laying a single track. It's like that you know that Morgan Freeman thing, you know the clip from Batman where he's like, so this man is a billionaire blah blah blah this and that, and your plan is to threaten him, right? And so you're going to create this super intelligence and have Kamala Harris regulate it. Come on, man! So to speak, right? Like, these people are praying to a blind, deaf, and dumb God that was powerful in 1945, right? That's why, by the way, all the popular movies— what are they? It's Barbie, it's Oppenheimer, right? It's, uh, it's Top Gun. They're all throwbacks to the '80s or the '50s when the USA was really big and strong. And the future is a black mirror.

Yeah, I think that's tragic. One of the projects that I do like, and you might appreciate this, I don't know if you've seen it, is from the Future of Life Institute, a project called Imagine a World, I think is the name of it. And they basically challenged their audience and the public to come up with positive visions of a future where technology changes a lot, and obviously AI pretty central to a lot of those stories. And what are the challenges that people go through and how do we get there and whatever? But a purposeful effort to imagine positive futures, super underprovided. And I really liked the investment that they made in that.

One of the things I've gotten in the Never Exceed It book is there's certain megatrends that are happening, right? And megatrends, I mean, it's possible for like one miraculous human maybe to reverse them. Okay. Um, 'cause I think both the impersonal forces of history theory and the great man theory of history have some truth to them. But the megatrends are the decline of Washington, D.C., the rise of the internet, the rise of India, the rise of China. That is like my worldview. And I can give 1,000 graphs and charts and so on for that, but that's basically the last 30 years. And maybe the next X, right? I'm not saying there can't be trend reversal. Of course, it can be trend reversal. As I just mentioned, some hammer blow could hit it, but that's what's happening. And so because of that, the people who are optimistic about the future are aligned with either the internet, India, or China. And the people who are not optimistic about the future are blue Americans or left-out red Americans, okay? Or Westerners in general who are not tech tech people, okay? If they're not tech people. They're not up into the right, basically. Because the internet's— if you— I mean, one of the things is we have a misnomer, as I was saying earlier, of calling it the United States, because it's the disunited states. It's like talking about America is like talking about Korea. There's North Korea and South Korea, and they're totally different populations. And communism and capitalism are totally different systems. And the thing that is good for one is bad for another and vice versa. And so like America doesn't exist. There's only, just like there's no Korea, there's only North Korea and South Korea. There's no America. There's blue America and red America and also gray America, tech America. And blue America is harmed, or they think they're harmed, or they've gotten themselves into a spot where they're harmed by every technological development, which is why they hate it so much, right? AI versus journalist jobs, crypto takes away banking jobs, you know, everything, you know, self-driving cars, they just take away regulator control, right? Anything that reduces their power. They hate, and they're just trying to freeze in amber with regulations. Red America got crushed a long time ago by offshoring to China and so on. They're they're making you know inroads to ally with Tech America or Gray America. Tech America is like the one piece of America that's actually still functional and globally competitive, and people always do this fallacy of aggregation where they talk about the USA, and it's really this component that's up and to the right, and the others that are down and to the right, or at best. Flat, like red, but they're like down, right? Like red is like, okay, functional. Blue is down. Point is, tech America, I think we're going to find, is not even truly— or how American is tech America? Because it's like 50% immigrants, right? And like a lot of children immigrants, and most of their customers are overseas, and their users are overseas, and their vantage point is global. Right? And they're basically not. I know we're in this ultra-nationalist kick right now, and I know that there's going to be there's a degree of a fork here where you fork technology into Silicon Valley and the internet. Okay, where Silicon Valley is American, and they'll be making like American military equipment and so on and so forth, and they're signaling USA, which is fine. Okay, and then the internet is international. Global capitalism. And the difference is Silicon Valley, or let's say US tech— let me be less, you know— US tech says ban TikTok, build military equipment, etc. It's really identifying itself as American, and it's thinking of being anti-China. Okay, but there's— US and China are only 20% of the world. 80% of the world is neither American nor Chinese. So the internet is for everybody else who wants actual global rule of law. Right? When, as the US decays as a rules-based order and people don't want to be under China, people want to be under something like blockchains where you've got like property rights, contract law across borders that are enforced by an impartial authority. Okay. That's also the kind of laws that can bind AIs, like AIs across borders. If you want to make sure they're going to do something, cryptography can bind an AI in such a way that it can't fake it. It can't, an AI can't mint more Bitcoin. You know?

My, here's my last question for you. AI discourse right now does seem to be polarising into camps. Obviously, a big way that you think about the world is by trying to figure out what are the different camps, how do they relate to each other, so on and so forth. I have the view that AI is so weird and so unlike other things that we've encountered in the past, including just like unlike humans, right? I always say AI, alien intelligence. that I feel like it's really important to borrow a phrase from Paul Graham, keep our identities small and try to have a scout mindset to really just take things on their own terms, right? And not necessarily put them through a prism of whose team am I on, or does this benefit my team or hurt the other team or whatever, but just try to be as kind of directly engaged with the things themselves as we can without mediating it through all these lenses. I think about, you mentioned the gain of function, right? And I don't know for sure what happened, but it certainly does seem like there's a very significant chance that it was a lab leak. Certainly there's a long history of lab leaks, but it would seem to me a failure to say, okay, well, What's the opposite of just having a couple of government labs? Everybody gets their own gain-of-function lab, right? If we could, and this is kind of what we're doing with AI, we're like, let's compress this power down to as small as we can. Let's make a kit that can run in everybody's home. Would we want to send out these gain-of-function wet lab research kits to every home in the world and be like, hope you find something interesting. Let us know if you find any new pathogens, or, hey, maybe you'll find lifesaving drugs, like, whatever. We'll see what you find, all 8 billion of you. That to me seems like it would be definitely a big misstep. And that's the kind of thing that I see coming out of ideologically motivated reasoning or tribal reasoning. And so I guess I wonder how you think about the role that tribalism and ideology is playing and should or shouldn't play as we try to understand AI?

First is, you're absolutely right that just because A is bad does not mean that B is good. A could be a bad option, B could be a bad option, C could be a bad option. You may have to go down to option G before you find a good option, or there might be 3 good options and 7 bad options, for example. To map that here, In my view, an extremely bad option is to ask the US and Chinese governments to do something. Anything the US government does at the federal level, at the state level in blue states, at the city level has been a failure. And the way— here's a meta way of thinking about it. You invest in companies, right? So as an investor, here's a really important thing. You might have 10 people who come to you with the same words in their pitch. They're They're all, for example, building social networks, but one of them is Facebook and the others are Friendster and whatever. Okay. And no offense to Friendster, you know, those guys were like, you know, pioneers in their own way, but they just got outmatched by Facebook. So the point is that the words were the same on each of these packages, but the execution was completely different. So could I imagine a highly competent government that could execute and that actually did, you know, uh, like, you know, make the right balance of things and so on. I can't say it's impossible, but I can say that it wouldn't be this government. Okay, and so you are talking about the words and I'm talking about the substance. The words are, we will protect you from AI, right? In my view, the substance is they aren't protecting you from anything, right? You're basically giving money and power to a completely incompetent and in fact malicious organization which is Washington, D.C., which is the U.S. government, that has basically over the last 30 years gone from a hyperpower that wins everywhere without fighting to a declining power that fights everywhere without winning. Okay? Like just literally burned trillions of dollars doing this. Take maybe the greatest decline in fortunes in 30 years in maybe human history. Not even the Roman Empire went down this fast on this many power dimensions this quickly, right? So giving that guy, Let's trust him. That's just people running an old script in their heads that they inherited. They are not thinking about it from first principles that this state is a failure. Okay. And like how much of a failure it is, you have to look at the sovereign debt crisis. You look, have to look at graphs that other people aren't looking at, but like, you know, the, the, the domain of what blue America can regulate is already collapsing because it can't regulate Russia anymore. It can't regulate China anymore. It's less able to regulate India. It's less able even to regulate Florida and Texas. States are breaking away from it domestically. So this gets to your other point. Why is the tribal lens not something that we can put in the back? We have to put in the absolute front because the world is retribalizing. Like, basically your tribe determines what law you're bound by. If you think you can pass some policy that binds the whole world, Well, there have to be guys with guns who enforce that policy. And if I have guys with guns on the other side that say we're not enforcing that policy, then you have no policy. You've only bound your own people. Does that make sense? Right. And so blue America will probably succeed in choking the life out of AI within blue America, but blue America controls less and less of the world. So they'll have more power over fewer people. I can go into why this is, but essentially, You know, a financial Berlin Wall is arising. There's a lot of taxation and regulation and effectively financial repression, de facto confiscation, that will have to happen for the level of debt service that the US has been taking on. Okay, just— there's, there's one graph just to make the point, and if you want to dig into this, you can. All right, but the reason this impacts things is when you're talking about AI safety, you're talking about AI regulation. you're talking about the US government, right? And you have to ask, what does that actually mean? And it's like, in my view, it's like asking the Soviet Union in 1989 to regulate the internet, right? That's going to outlive, you know, the country. US interest payment on federal debt versus defense spending. The white line is defense spending. Look at the red line. That's just gone absolutely vertical. That's interest. And it's going to go more vertical next year because all of this debt is getting refinanced. And much higher interest rates. This is why— look at this. You want, you want AI timelines, right? The question for me is DC's timeline. What is DC's time left to live? Okay, this is the kind of thing that kills empires. And, and you either have this just go to the absolute moon, or they cut rates and they print a lot. And either way, you know, the, the fundamental assumption underpinning all the AI safety, all the AI regulation work is that they have a functional Golem in Washington, D.C., where if they convince it to do something, it has enough power to control enough of the world. When that assumption is broken, then a lot of assumptions are broken, right? And so in my view, you have to— you must think about a polytheistic AI world because other tribes are already into this. They're already funding their own. Right. The proliferation is already happening and they're not going to bow to blue tribe. So I, that's why I think the tribal lens is not secondary. It's not some, you know, totally separate thing. It is an absolutely primary way in which to look at this. And in a sense, it's almost like a, you know, in a well-done movie, all the plot lines come together at the end.

Okay.

And all the disruptions that are happening The China disruption, the rise of India, the rise of the internet, uh, the rise of crypto, the rise of AI, and the decline of DC and the internal political conflict. And, you know, various other theaters like what's happening in Europe and, you know, and, and Middle East, all of those come together into a crescendo of, ah, there's a lot of those graphs are all happening at the same time. And it, it's not something you can analyze by just, I think, looking at one of these curves on its own.

I think that's a great note to wrap on. I am always lamenting the fact that so many people are thinking about this AI moment in just fundamentally too small of terms, but I don't think you're one that will easily be accused of that. So with an invitation to come back and continue in the not too distant future, for now I will say, Balaji Srinivasan, thank you for being part Thank you, Nathan.

Good to be here.

Upstream with Erik Torenberg is a show from Turpentine, the podcast network behind Moment of Zen and Cognitive Revolution. If you like the episode, please leave a review in the Apple Store. Turpentine is a network of podcasts, newsletters, and more covering tech, business, and culture. From the perspective of industry insiders and experts. We're the network behind the show you're listening to right now. At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution to Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors, like Moment of Zen and my show Upstream. We're looking for industry-leading hosts and shows along with sponsors. If you think that might be you or your company, email me at erik@turpentine.co. That's erik@turpentine.co.