AGI and Institutional Disruption: Sam Hammond in Conversation with FLI’s Gus Docker

Hello and welcome back to The Cognitive Revolution. As we reach the end of 2023, we're still just 11 months into making The Cognitive Revolution, but I wanted to take a moment to say a big thank you to everyone who has made this show possible, starting with our sponsors. From our first presenting sponsor, Omniki, who agreed to support the show before hearing even a single episode, to the household names you've heard from throughout the year, including Oracle, MasterClass, Shopify, and NetSuite. We appreciate your support. I also want to say thank you to the entire Turpentine team, and in particular, our producers Natalie Torrin and Vivian Mang, our editor Graham Bessalou, and of course, our super connector and the mastermind behind the idea for a network of expert-hosted podcasts, Eric Thornberg. We've made nearly 90 episodes of this show, and I always emphasize that AI content ages poorly. With that in mind, we've turned many episodes around in just a day or two. Now, this is not the norm in the podcasting business, and it does create a lot of odd hours for the team, but they've been here for it every step of the way, and I have been extremely privileged to be able to focus 100% of my time and energy on the content itself, knowing that they have my back on all the logistics. Of course, I also want to thank our many incredible guests. They are an amazingly diverse group hailing from at least a dozen different countries around the world with all sorts of different academic and professional backgrounds, but all united by an appreciation for the transformative, dare I say revolutionary potential of AI. Collectively, they have helped me develop a far deeper understanding of AI technology than I had coming into the year, and I hope the same is true for you. Finally, I want to thank all of you, the listeners, for making this project not just worthwhile, but a success well beyond my expectations. Coming into the year, I had never created original content before, and I had no platform or audience anywhere. And to be honest, I wasn't really sure if anyone would want to listen to the deeply in-the-weeds, hopefully anti-polarizing content that I wanted to create. On that front, I'm glad to say that I have been consistently pleasantly surprised. Pleasantly surprised. Thousands of you have become regular listeners, and some of our most challenging episodes, including my recent 2.5-hour monologue about state-space models, have been among our very best received. We end the year as the 20th-ranked technology podcast according to Spotify, 31st according to Apple, and I take that as a sign that something is definitely working. Nevertheless, when a friend recently asked what metrics we use to track performance, I responded that for me, the best indicator is the number and the quality of people who reach out to share feedback with me directly. So with that in mind, if you have any topic or guest suggestions, or just general thoughts about how we can make the show more valuable, we would love to hear them. We have a ton planned for The Cognitive Revolution in 2024, including episodes about the rise of AI-automated scientific exploration, the intersection of AI and biology, the energy demands of AI, the future of robotics, and much, much more. So while we're working on all that, as a bit of bonus content, we're going to repost a few of my favourite AI Scouting episodes from other shows, episodes that I was jealous of when I first heard them. Starting today with an episode from the Future of Life Institute podcast, in which host Gus Stocker interviews economist Samuel Hammond. This conversation is super wide-ranging. covering the most likely default path to AGI, the economic and institutional transformations that AI will beget, the critical distinction between thinking about AI in isolation versus as part of a dynamic system, the proposal for a Manhattan or Apollo-like megaproject for AI safety, and lots more along the way. I hope you find this and the next couple of episodes to be helpful complements to my own AI scouting work, and I encourage you to subscribe to the Future of Life Institute podcast feed where you can hear lots more about AI and other issues of global importance. With that, happy new year from the Cognitive Revolution, and here's Gus Stocker and Samuel Hammond.

Welcome to the Future of Life Institute podcast. My name is Gus Stocker, and I'm here with Samuel Hammond, who is a senior economist at the Foundation for American Innovation. Samuel, welcome to the podcast.

Hi, Gus. Thanks for having me.

Fantastic. All right. I have so much I want to talk to you about, but I think a natural place to start here would be with your timelines to AGI. Why is it that you expect AGI to get here before most people?

Well, I don't really know what most people think. I think the world divides into people who are paying attention and people who are basically normies. And in my day job, I work on Capitol Hill and In Washington, DC, talking to folks about AI. And if you think about what people's implicit timelines are, you can read out people's implicit timelines by their behavior. I know Paul Christiano has short timelines because he's doubled up into the stock market. He's sort of practicing what he preaches. But then when you have Sam Altman testifying to Congress, I like to say people are taking him Seriously, but not literally, right? He's saying we're going to develop something like AGI potentially this decade and superintelligence thereafter. And then you have folks like Senator Marsha Blackburn being like, what would this mean for music royalties? And when the focus of policymakers is things like music royalties or the impact on copyright, it's not that those are invalid issues, it's that they belie relatively longer timelines. And then we also have this definitional confusion where folks like Yann LeCun would say AGI is probably decades away because he's using AGI to mean something that learns like a human learns in the sense that it's born as a relative blank slate and sort of can acquire language with very few examples. So people have these moving goalposts of what they mean. For me, I think we can sort of avoid Those definitional conflicts, if we just talk about human-level intelligence, and humans are quite general, we're generally intelligent. That's what separates us from animals in a lot of respects. And when you look at how machine learning models are being trained today, like large language models and now multimodal models, they're being trained on human data. And they're being trained to reproduce the kinds of behaviors and tasks and outputs that humans output. And so they're kind of like an indirect way of emulating human intelligence. And so if you benchmark AI progress to that, then you can sort of put information-theoretic bounds on what's the likely timeline to basically an ideal human emulator, something that can extract the sort of base representations, the internal representations of our brain. Through the indirect path of the data that our brain generates.

You have an interesting sentence where you write that AI can advance by emulating the generator of human-generated data, which is simply the brain. Do you think this paradigm holds all the way to AGI?

I think it holds this decade to systems that in principle can in context learn anything humans do. Again, this is a semantic question. Do you want to call that AGI or not? I think there are still outstanding issues around the limits of autoregressive models for autonomy and the question of real-time learning. The way we train these models, we sort of are freezing a crystal in place and humans are continuously learning. So there still are genuine potential architectural gaps, but from the practical point of view, from the economic point of view, We don't need to debate whether something is conscious or whether something learns strictly the way humans learn if it demonstrably can do the things humans do. And that goes to the original insight of the Turing test. It's sometimes presented as a thought experiment, but what Alan Turing was getting at was if you can't distinguish between the human and a computer, in some ways indistinguishability implies competence. And we can broaden that from just language because arguably we've surpassed the Turing test, at least a weaker version of it, to human performance on tasks in general.

If we have a system that can output a scientific manuscript that experts in the field can't distinguish from a human, then debating whether this is real AGI or not is, I feel, It is surprising in a sense that when you interact with GPT-4, for example, and it can do all kinds of amazing things and organize information, present information to you, but then it can't, or at least at some point it couldn't answer questions about the world after September 2021 or a date like that. That would be surprising if you presented that fact to an AI scientist 20 years ago. For how long do you think we'll remain in this paradigm of training a foundational model and then deploying that model?

I mean, it's worse than that. I think it would surprise people 5 years ago. Progress is sort of moving along 2 tracks. There's the industry track and the pure research academic track, and they're obviously having feedback with one another. The pure industry track is just looking to create tools that have practical value and can improve products and so forth. Meta has their own GPU cluster and they're training models so they can have fun chatbots in their Messenger. And so those kinds of things are going to progress, I think, well within the current paradigm because we know the paradigm works, basically deep learning and transformers. And there's lots of marginalia on the side, but that basic framework seems to be quite effective. And just scaling that up because we haven't sort of hit the range of irreducible loss and what transformers can do. Meanwhile, there's also this parallel pure research track where people on seemingly a weekly basis are finding better ways of specifying the loss function, ways of improving upon power law scaling and all these different— sometimes they're new architectures, but often they're just bags of tricks. And those bags of tricks then To the extent that they comport with the paradigm industry is running with, they can be reintegrated and end up accelerating progress in industry as well.

So do you think scale of compute is the main barrier to getting to human-level AI?

Yes. I mean, it's not all we need, but it's the main unlock.

To what extent can more compute be used to trade off for lower quality data or for lower quality algorithms? Can you just throw more compute and solve the other factors in that equation?

It depends on the thing you're trying to solve for. In principle, if we're talking about mapping inputs to outputs, then transformers are known to be universal function approximators. And so the answer is yes. That doesn't mean that they're necessarily efficient at approximating everything we want them to approximate. And sometimes universal function approximation theorems can be kind of trivial because they'll be like, okay, if your neural network has infinite width, then yes, we can approximate everything. The key fact is both that they're universal approximators and also that they're relatively sample efficient. At least relative to things we've found in the past. And so that to me suggests that yes, they can compensate for things that they're bad at. On the other hand, the way research is trending is towards these mixed models, ensembles of different kinds of architectures, things like the recent Q-transformer nets from Google DeepMind, which just sort of uses a combination of transformers and Q-learning to have sort of the associational memory and And sample efficiency of transformers with the ability to assign policies to do tasks that you get from reinforcement learning. So I imagine that there's going to be all kinds of mixing and matching. The key point is that in that space of architectures, it's a relatively finite search space. And as an economist, economists believe that supply is long-run elastic. And so there's this famous bet from between Paul Ehrlich and Julian Simon vis-à-vis the population bomb and whether population growth would lead to a sort of a Malthusian purge. And Julian Simon, being the economist, recognized that if prices rise for these core commodities, then that will spur research and development into extracting new resources. So he didn't have to know that fracking would be a technology. He understood that If oil prices went too high, people would find new oil reserves. And I think there's a— I have an analogous instinct when it comes to progress in deep learning, meaning you can become too anchored to sort of the current state of the literature, but over a 10-year horizon, you can say, well, there's a huge search on, a huge gold rush to find the right way of blending these architectures. And I don't need to know in advance which is the right way to do that to have high confidence that someone will find it.

Yeah, we can sometimes, if we're too deep in the literature, we might lose the focus on the forest for the trees in a sense. And if we zoom out, we can just see that there's more investment, there's more talent pouring into AI. And so we can predict that something is going to come out of that. You have lots of interesting insights about information theory and how this can help us predict What's the most important lessons from information theory?

The reason I start there is because within the conceptual realm, it's sort of like the most general thing that bounds everything else. And when you look back at the record of, say, Ray Kurzweil, I first read The Age of Spiritual Machines when I was a kid, and in there he makes a prediction that we'll have AIs that pass the Turing test by 2029 or so.

And when was this book written?

1999.

Yeah, that's pretty good.

Right. And people will complain that he got things wrong because he said we'll all be wearing AR glasses by 2019, when in fact Google Glass came out in 2013 and now we have Meta Glasses 5 years later. So he was wrong on the exact timing, but sort of right where the technology was, wrong where the Minimal viable product was.

Hey, we'll continue our interview in a moment after a word from our sponsors.

But nonetheless, like, if you look at his track record, it's quite good for a methodology as relatively stupid as just staring at Moore's Law and extrapolating it out. And I think that reveals the power of these information theoretic methodologies to forecasting because they set bounds on what will be possible. The team at Epoch AI have a forecast called the Direct Approach where it's sort of, you can think of it sort of like a way of putting bounds on when we'll have AIs that can emulate human performance through an information theoretic lens where they're looking at sort of how much entropy does the brain sort of process and how much compute will we have over time and what's implied by AI scaling laws. And you sort of Put those 3 things together and you can sort of set bounds on when we'll basically be able to brute force human-level intelligence. And of course, that's an upper bound because we're going to do better than brute force. We're going to also have insights from cognitive science and neuroscience and also ways of distilling neural networks and so forth and better ways of curating data. So their modal estimate for human-level AI is 2029. And their median is like 2036. And I've talked to the authors and they lean towards the 2029, 2030 for their own personal forecasts. And so going back to, is this an outlier? Am I out on a limb here? I think among our circles, probably not, but among Congress and among the broader public, I think people are seeing sort of, they think everything's an asymptote, right? They're imagining, okay, we have these chatbots and this They're not seeing the next step. I see a very smooth path from here to systems that can basically in-context learn any arbitrary human task. And so what does that look like? It looks like systems that can basically sit over your shoulder or can monitor your desktop, your operating system as you work and watch you for an hour or 2 and then take over.

And that'll be key to overcoming lack of training data or why is it important that they can learn in context?

Well, in-context learning is sort of the secret source of the power of transformer models. They learn these inductive biases and induction heads and so forth that let them few-shot learn different tasks. So GPT-4 is very good at zero-shot learning on a variety of different things, but it's incredibly good at few-shot learning. If you give it a few examples, it can kind of pick up where you left off. When I think about myself, when I want to learn a new recipe, I can go read a recipe book, but often what I prefer to do is to go on YouTube and watch someone make that recipe. And just by watching that person put together the stir fry, I have enough of a world model and enough knowledge of how to cook in general that I can sort of in-context learn how to pick up from there and do that recipe myself. LLMs do that already. Multimodal models are increasingly doing that. Some of the recent progress in robotics, Like I mentioned, the Q-transformer paper, it shows that you can basically build robots with a basic world model and then have it learn new tasks with fewer than 100 examples of a human demonstration. So the human demonstrates the task and the robot can pick it up and take it from there. And why that's important is both for understanding the trajectory of AI, but also its economic implementation. Because we're sort of used to automation being this thing where you get a contract from IBM and you spend many millions of dollars with consultants and they build you some bespoke thing that doesn't really work very well and requires lots of maintenance. And so people have this sort of prior that AI, even if it's near, will be rate limited by the real world because of all the complexity of implementation. But the point is, if you have things that can in-context learn, and perform sort of as humans perform, then you don't need to change the process. You can take human-designed processes and have the AI just fill in for the human. And so it leads to this paradox where we're probably going to have AGI before we get rid of the last fax machine.

Yeah. When we think of, say, old IT systems in large institutions, we might think of Moving from analog storage of information to the cloud, that's still going on in some institutions. That transformation has taken over a decade now. And so what exactly is it that makes AI different here? It is that AI plugs in directly where the human worker would be?

Yeah, precisely. You don't need to redesign an existing process to sort of plug into the automation. And that applies both for the structure of tasks. So much of mechanical automation takes something like the artisanal work of a shoemaker and has to translate it into something repetitive that a machine or an automatic seamstress can do over and over and over again. Our older school kind of automation requires collapsing a task into a lower dimension so that so that simple automations can handle it. But when you have AGI, the whole point is generality. It's a flexible intelligence that can map to existing kinds of processes. So that's sort of why I think this will catch people by surprise because it's not just that AGI could be this decade, but that when it arrives and sort of crosses some thresholds of reliability, the implementation frictions could be very low.

And do you expect— would AI have to get all the way there in order to substitute for a human worker? I mean, I would expect it to be a bit more gradual than that, taking over, say, 20% of tasks before 40% of tasks, 60% of tasks, and so on. But here we're imagining that the AI kind of plugs in for the human worker for all tasks, or what do you have in mind?

These things are— yeah, you're right, much more continuous. It's not an on or off switch. In part because the requisite threshold of reliability varies by the type of task. Arguably, self-driving cars like Waymo or Tesla have matched human performance, but regulators want them to be 100x better than human before they're let loose on the road because of safety. Codex and coding models are arguably still much worse today than elite programmers, but everyone is using them because even if it generates 50% of your code and you have to go back in and debug, it's still a huge productivity boost. So I think it'll vary by occupation, by sort of task category, sort of modulo the risks and stakes involved in those tasks.

Yeah, I guess then the question is how many of our jobs fall into the— is more like self-driving cars and how many of our jobs is more like programming?

Right. I mean, I've been in a manager position before and I've had research assistants and interns and I know that they're a very lossy compression of the thing I want to do. And so they require oversight and sort of copiloting. We're sort of in that stage now with AIs in a variety of different kinds of tasks. I recently read a paper evaluating the use of GPT-4 for peer review in science. And it found that GPT-4 would write reviews of work that bore some striking correlations with the points raised by human reviewers, but also left some things out. And so it concluded by saying GPT-4 could be an invaluable tool for scientific review, but it's not about to replace people. And that's just a case of like, Okay, give it 5 years.

Yeah, this is, this is a phenomenon you often see with some AI models out there and it has some capabilities but lacks other capabilities. And then people might kind of over-anchor on the present capabilities and not foresee the way the development is going. I think people are continually surprised at the advancement of AI.

Yeah, absolutely. Ramez Naam, the sci-fi author and futurist and energy investor, he gives this talk on solar energy and other renewables. And he has this famous graph where he shows the International Energy Agency, the IEA. Every year they put out this projection of solar buildout. And every year it's like a flat line, but it's like a flat line on an exponential, like the real curve is like going vertical. And every year their projection is that it's just going to plateau.

Yeah.

And I feel like people make that same mistake. And it sort of has this sort of ironic lesson, to the extent that we're drawing sort of parallels with the way our brain works and the way these models work. It seems like humans have a very strong autoregressive bias.

So what's going on there? Is it an institutional problem or is it a psychological problem? Why is it that we can't project correctly in many cases?

Well, to what I just said, I think it's probably both, but largely psychological. Our brains are evolved for hunter-gatherer societies that didn't really change over millennia. And even the last 40, 50 years have been a period of relative stagnation where we have a lot of pseudo-innovation. And so I think people are just a bit disabused.

Okay. You have some super interesting points about the Comparing how the human brain works to how neural networks learn. What is universality in the context of brain learning and neural network learning?

Universality is a term of art. It refers to the fact that different neural networks independently trained, even on different data, will often converge on very similar representations in their embedding space. Of that data. And you can extend that to striking parallels or isomorphisms between the representations that artificial neural networks learn and that our brain appears to learn. Probably the area of the brain that's been studied the most is the visual cortex. And it seems to me as a layperson that the broad consensus in neuroscience is that the visual cortex is very similar to a deep convolutional neural network. Is basically isomorphic to our artificial deep convolutional neural networks. And you train a CCN on image data and our brain is trained on our sensory data, and it turns out they end up learning strikingly similar representations. And there are a few reasons for that. One is hierarchies of abstraction. It makes sense that early layers in a neural network will learn things like edges and simple shapes, and only later, only deeper in the network will you learn more subtle features. So there's that sort of sequencing part of it. And then there's also just the energy constraint. Gradient descent isn't costless. It requires energy. It requires a lot of energy. These data centers suck up a lot of energy. The same is true of our brain. Our brain consumes a lot of energy, like 25% of our calories. And especially when we're young, there's a very strong metabolic cost associated with neuroplasticity. Our brain being something shaped by evolution was obviously very energy conscious. And so those energy constraints greatly shrink the landscape of possible representations from sort of this infinite landscape of all the ways you could represent certain data to a much more manageable— set of representations. And that doesn't guarantee that we'll converge on the same representations. It's at least suggestive of a weak universality where even when we don't have the exact same representations, they're often a coordinate transformation away from each other.

It's actually a bit surprising. As you mentioned, when we train neural networks, we don't have the same energy constraints as the brain had during our evolution. And I would expect again from evolution, the human brain to have many more inbuilt biases and heuristics. But if we then compare the representations in a neural network to those in a human brain, we found that they are quite similar. Isn't that the whole point of universality? So does the neural network have the same heuristics and biases that we have, or what's going on here?

Well, one of the primary biases in stochastic gradient descent is Sometimes called a simplicity preference, basically an inductive bias for more parsimonious representations. Parsimonious in the sense of Occam's razor. And that's a byproduct of this information theoretic concept of Kolmogorov complexity, where Kolmogorov complexity means, is measured by, is there a short program that can reproduce this longer sequence? And if you can find a short program that's sort of a more compact or more compressed way of representing it. And when you're under energy constraints, you're looking for those more compressed representations. And so that simplicity bias seems to be also the origin of generalization, of our ability to go beyond merely memorizing data, overfitting our parameters, to finding a simpler way of representing those parameters. Where we go from fitting a bunch of data points to recognizing, oh, these data points are being generated by a sine function, so I can replace all these data points by a simple circuit for that sine function or something like that.

What can we learn about AI progress when we consider the hard steps that humans and our ancestors have gone through in evolution?

It's beyond evolution. This often comes up in the discussion of the Fermi paradox. Life on Earth to exist at all, let alone intelligent life, had to pass through many hard steps, right? We had to have a planet in a habitable zone. We had to have the right mix of organic chemicals in the Earth's crust and so forth. We had to have the conditions for abiogenesis, the emergence of the very earliest sort of non-living replicators.

Right.

Probably some kind of polymer type of crystal structure. Then we had to have the transition from single-celled to multicellular organisms, the transition through the Cambrian explosion. Every one of these steps you could think of as a very unlikely, improbable thing, all the way up to the development of warm-blooded mammals and sort of social animals that were heavily selected for brain size to— To then the sociocultural hard steps of moving from small group primates to settled technological cultures, then technological hard steps like the discovery of the printing press or the discovery of the transistor. You put those all together and life seems just incredibly unlikely. And this often goes to the point of view that creationists or intelligent designers would to put forward. But then you zoom out and then you recognize, oh wait, there are trillions of galaxies, each with hundreds of billions of stars and hundreds of billions or trillions of planets. There's an awful lot of potential variation out there. And then meanwhile, every one of these hard steps seems characterized by a search problem that is very hard. But then once you find the correct thing, like the earliest self-replicator, things kind of take off, right?

Hey, we'll continue our interview in a moment after a word from our sponsors.

So you imagine that before the earliest self-replicator, there were millions or billions of attempts to self-replicate like that that didn't succeed?

Yeah, it's just a huge search problem, right? And, you know, maybe there are more gradual intermediate stages where Everything in biology ends up looking way more gradual the more you learn about it. But there are these phase transitions where you tip over and you get the Cambrian explosion or you get the printing press and the printing revolution. And so those hard steps end up looking relatively— they look more easy in retrospect because even though the search was hard, once you've tripped over the correct solution, there's sort of an autocatalytic self-reinforcing loop. that pulls you into a new regime. And indeed, when you look at the emergence of life on Earth relative to the age of the universe, and Avi Loeb with some co-authors have done this, life on Earth is incredibly early. The universe is 13.7 billion years old, but life couldn't emerge really much sooner. The reason being, The universe started out as hot and dense, it had to cool down, stars had to form, those stars had to supernovae so they could produce the heavy elements that are essential to life. And then those solar systems had to then take shape and then had to further cool so the solar system wasn't being irradiated constantly. And when you put all those factors together, human life emerged basically as soon as it was possible for life to emerge anywhere. And so this is one way to answer the Fermi paradox is that we're just in the first cohort. But it also should give you strong priors that passing through those hard steps isn't as hard as it looks.

And what's the lesson for AI here?

Developing AGI is sort of a hard step. We're doing this kind of gradient search for the right algorithms, for the right what have you. And We seem to be now in a slow takeoff where we've figured out the core ingredients and there's now an autocatalytic process that's pulling us into a new phase.

And what do you mean by autocatalytic?

Self-reinforcing. Once it gets started, it sort of pulls itself. It sort of has an as-if teleology. You see this in nature, but you also see this in capitalism.

And you would expect us to get to advanced AI basically as soon as it's computationally possible.

It basically seemed that way, right? There was a kind of tacit collusion between Google and other players in the space to— they had transformer models since 2017, but really there's some of the precursors to transformers go back to the early '90s. But once you have this sort of profit opportunity that's in the background, it's hard in a competitive environment to stop an OpenAI from being like, oh, let's chase those profits. And then once that ball gets rolling, it's basically impossible to stop. That's just why, whatever the merits of the pause letter, it's virtually impossible to really have a pause in AI development because everything is structured by these game theoretic incentives to just keep going faster. Once you've stumbled on the gold reserve, it's hard to keep the Samuel, is the US government prepared for advanced AI? No. No. I mean, where do I start? I mean, the US government's in— if you think of it from a firmware level, many countries have national IDs. The US doesn't have a national ID. We have Social Security numbers. They're these 9-digit numbers that date back to 1935. We have the core administrative laws date back to the early '40s. Much of our sort of technical infrastructure, like the system the IRS runs on, date back to the Kennedy administration and are written in assembly code. There's also been this general decline in what you could call state capacity, sort of the ability for the US government to— execute on things. And you hear about this all the time. You hear about how the Golden Gate Bridge was built in 4 years or something like that, and now it takes 10 years to build an access road. One of the reasons for that goes to what the legal scholar Nicholas Bagley has called the procedural fetish. Really, since the '70s, the machinery of the US government has shifted towards a reliance on explicit process. And proceduralism has pluses and minuses. If you have a clear process, government can kind of run on autopilot to an extent, but it also means you limit the room for discretion and you limit the flexibility of government to move quickly. And moreover, in our adversarial legal system, you also open up avenues for sort of continuous judicial review and legal challenge. Where famously New York has taken over 3 years to approve congestion pricing on one of their bridges because that has to undergo environmental review and people who don't want to pay the congestion price keep suing.

Do you think having more procedures would make it easier for AI to interface with government?

I would say having fewer procedures would make it easier for government to adapt.

My assumption would be that having something written down, having a procedure for something would make it easier to plug AI into that procedure. If it's less opaque and more kind of almost like an algorithm step by step.

Yes. But the analogy I would give is to the Manhattan Project. The original Manhattan Project was run like a startup. You had Oppenheimer and General Leslie Groves sort of being the technical founder and the Type A, get things done founder. And they broke all the rules. They pushed as hard as they could. They were managing at its peak 100,000 people in secret and they built the nuclear bomb in 3 years. And so the way we would do that today under procedural fetish framework would be to put out a bunch of requests for proposals and have some kind of competitive bid. And then we'd probably get the lowest cost bid. And it would be like Lockheed Martin and they would build half an atom bomb and it would take 20 years and 5 times the budget. And so that's sort of what I'm getting at. It's not about process versus discretion per se. It's about the way process hobbles and straitjackets our ability to adapt and sort of represents a kind of sclerosis, a kind of sort of crystallized intelligence. We lay down the things that worked in the past as process and sort of freeze those processes in place, ossifying a particular modality. And when the mode of production shifts and you need to completely tear up that process root and branch, it's very difficult because often there's no process for changing the process.

Yeah. I wonder if there are lessons for how government will respond to AI in thinking about how governments responded to say historical technical innovations of a similar magnitude, like the Industrial Revolution or the printing press or maybe the internet computer, do you think we can draw general lessons or is it so specific that we can't really extract information about the future from them?

I think they're very powerful general lessons. I think one of the first general lessons is that every major technological transformation in human history has preceded an institutional transformation, whether it's the shift from nomadic to settled city-states with the Agricultural Revolution or the rise of modern nation-states or the end of feudalism with the printing press to, in the New Deal era, the sort of transition with industrialization from the kind of laissez-faire classical liberal phase of 18th century America to an America with the— a robust welfare state and administrative bureaucracies and really an all-new constitutional order. And so there's sort of better and worse ways for this transition to happen. There's sort of the internal regime change model, and you can think of Abraham Lincoln or FDR as inaugurating a new republic, a new American republic. Or there's a scenario where we don't change because we're too crystallized and sort of like an innovator's dilemma, get displaced by some new upstart. And different countries have different abilities and different sort of capacities for that internal adaptation. As a Canadian, I'm a big fan of Westminster-style parliamentary systems. And one of the reasons is because it's very easy for parliamentary systems to shut down ministries, open up new ministries, to reorganize the civil service because it's sort of vertically integrated. under the Prime Minister's office or what have you. In the US, it's much worse because given the separation of powers, Congress and the executive are often not working well together, just as an understatement. But then moreover, the different federal agencies have sort of a life of their own. Often they're self-funded and all these other things that make it very difficult to reform.

Do you think Canada responded better to the rise of the internet than the US, for example? Isn't there something wrong with the story because the US kind of birthed the internet and Canada adopted the internet from the US?

Let's compare, first of all, the impact of the internet on weaker states because Canada and the US are similar or sort of in one quadrant. They have differences, but the differences are small compared to other countries. If you think about internet safety discussions that would've been taking place in the early 2000s, people would've been talking about identity theft, credit card theft, child exploitation, these kind of direct first-order potential harms from the internet. They didn't foresee that concurrent with the rise of mobile and social media, that the internet would enable tools for mass mobilization simultaneous with a kind of legitimacy crisis where the sort of new transparency and information access that the internet provided eroded trust in government and trust in other institutions. So you have these 2 forces interacting, the internet exposing government and exposing corruption and leading to a decline in trust while also creating a platform for people to rise up and mobilize against that corruption. And it's something that kind of rhymes with the printing press and the printing revolution. Where you had these sort of dormant, suppressed minority groups like the Puritans or the Presbyterians, the nonconformists. And with the collapse of the censorship printing licensing regime, they actually had a licensing regime in the UK Parliament back circa 1630. That licensing regime collapsed, I think 1634 or something around there. And that was like 5 years before the English Civil War. And you see something like this in the Arab Spring, where the internet quite directly led to mass mobilization in Cairo and Tunisia and elsewhere, and led to actual regime change, in some cases sort of temporary state collapse. And that's because those were weaker states that hadn't democratized, that hadn't sort of had their own information revolution. earlier in their history the way we did. In some ways, the American Republic is sort of a founder country built on the backbone of the printing revolution. So we were a little bit more robust to that because it's sort of part of our ethos to have this open, disagreeable society. But clearly, the internet has also affected the legitimacy of Western democracies. I think it's Clearly one of the major inputs in rising populism, the mass mobilizations that we see, whether in the US context, the 2020 racial awakening or the January 6th peasant rebellion. These sort of look like the kind of color revolutions that we see abroad. And some people want to ascribe conspiracy theories to that. I think there's a simpler explanation, which is— which is that people will self-organize with the right tools. Our state hasn't collapsed yet, but there's clearly a lot of cracks in the foundation, if you will.

Would it be fair to say that the main lesson for you from history is that technological change brings institutional change?

Yeah. Not necessarily one for one. I'm not kind of a vulgar Marxist on this, but yes. And the reason for that is because institutions themselves exist due to a certain cost structure. And if you have general purpose technologies that dramatically change the nature of that cost structure, then institutional change will follow.

Yeah, and I think we want to get to that. But before we do, I think we should discuss AI's impact on the broader economy. So not just the government, but the economy in general. Economists have this fallacy they point out often, the lump of labor fallacy. Maybe you could explain that.

The lump of labor fallacy is essentially the idea that there's a fixed amount of work to be done. If you were thinking about the Industrial Revolution and what would happen to the 50% of people who are in agriculture, you couldn't imagine the new jobs that would be created, but new jobs were created. And the reason is because human wants are infinite. And so demand will always fill supply. The second reason is because there's a kind of circular flow in the economy where One person's cost is another person's income. Society would collapse if we had true technological unemployment because there'd be things being produced but no one to pay for them. And so that ends up kind of bootstrapping new industries and new sources of production. There's still this open question, is this time different?

Yeah, that's exactly what I want to know because, I mean, for me, it's— or in retrospect, let's say, it's easy to see how workers could move from From fields to factories into offices. But if we have truly general AI, it's difficult for me to see where workers would move, especially if we have also functional robots and perhaps AIs that are better at taking care of people than other people are. I'm not asking you to predict specific jobs, but I'm asking you whether you think This historical trend will hold with the advent of advanced AI.

The first thing to say is when Keynes wrote Economic Possibilities for Our Grandchildren, a famous text where he predicted that technological progress would lead to the growth of a leisure society.

And this was in the 1930s. Yeah.

People have dismissed him as being wrong. But actually, you look at time use data and employment data and people are working less. It didn't match the optimism of his projection because it turns out maybe if we fixed living standards at what he expected, people want more and people will work more for more. But overall, people are working less. People do have more leisure. We've sort of moved to a de facto 4-day work week. So there is one world where rapid technological progress continues that trend and we all work less. It's sort of a technological unemployment that's spread across people and is enabled in part because in a world of AGI, maybe you only have to work a few hours a day to make $100,000 a year. There's another possibility, which is that, well, AGI could in principle be a perfect emulation of humans on specific tasks, it can't emulate the historical formation of that person. So what I mean by that is if you had a perfect atom-by-atom replication of the Mona Lisa, it wouldn't sell at auction because people aren't just buying the physical substrate, they're also buying the kind of—

The story.

worldline of that thing. And that's clearly the case in humans as well. There are certain talking heads that I go and enjoy, not because they are the smartest or what have you, because I'm interested in what that person thinks on this because they have a particular personality, a particular worldline. And then the third factor is sort of artificial scarcity. Right. And so even in a world with abundance and supply in services and goods, there are still things that will be intrinsically scarce. Real estate being probably the canonical thing, but also energy and commodities and so forth. And the reason real estate is intrinsically scarce is because people want to live near other people and people want to live in particular areas of a city. They want to live in the posh part of town. And those are positional goods. We can't all live in the trendy loft. So that builds in a kind of artificial scarcity. And so people will still be competing over those things. This is sort of related to artificial scarcity, but there's also sort of break it out into a 4th possibility, which are sort of tournaments and things that are structured as tournaments. Having chess bots that are strictly better than humans at chess hasn't killed— people playing chess. If anything, more people play chess today than have had in human history.

Yeah, it's more popular than ever.

Yeah. And the reason is because people like to watch other humans playing. And also they're structured as sort of zero-sum tournaments where there can only be the best human. You look at other things that have been created just in the last 15, 20 years, like the X Games, right? I think people will still want to watch other people do the Olympics or do motocross and all these other things. And so maybe more of our life shifts into, you know, both maybe greater leisure on the one hand, more competition over positional goods and more, more production that is structured as a tournament.

Yeah. Yeah, I can see many of those points. I'm just thinking again, with, with fully general AI, you would be able to generate a much more interesting person playing chess, or at least a simulation of a very charismatic and interesting human chess player. Why wouldn't people watch that chess player as opposed to the most— to the best human?

Maybe they will. It's hard to know. The question is who's producing that video stream? Because you still need the human behind it that's— that had the idea. Right.

And you could imagine people being dishonest about the history of this chess player. That this simulated chess player could be fully digital, fully fictional, so to speak, and just pretending to be human.

Right. So it could fool people. That's the case too. No, I can't rule that out. But I would just say that however that person is monetizing their deepfake chess player, they're making money, which they're then spending back into the economy. And so they'll produce jobs somewhere.

Do you think more people will move into say, people-focused industries like nursing and teaching? Is that a possible way for us to maintain jobs?

Maybe nursing, at least in the short run. I'm not very long on education being labor-intensive for much longer.

But you don't think education is, at least say, grade school education, is that really about teaching people or Conveying knowledge, or to what extent is it about conveying knowledge and to what extent is it about the social interaction and specializing your teaching to the individual student?

Well, AIs are very good at customization and sort of mastery tutoring. Education is a bundle of things. And for younger ages, it's also daycare. It is socialization, like you said. At the very least, it suggests a reorganization of the division of labor because the types of teachers that you would select and hire for may differ if the education component of that bundle is being done by AI. Maybe you select for people who maybe don't have any subject matter expertise but are just highly conscientious and good around kids. Or maybe you unbundle from public education altogether and it rebundles around You know, a jiu-jitsu school or a chess academy because, you know, you'll have the AI tutor that will teach you math, but you'll still want to grapple with a human.

Yeah. Yeah. What about industries with occupational licensing like law or medicine? Will they be able to keep up their quite high wages in the face of AI being able to, you know, be a pretty good doctor and a pretty good lawyer?

It's easy to solve for the long-term equilibrium. With the rise of the internet, you can do a comparison of the wage distribution for lawyers pre- and post-internet. And circa the early '90s, lawyer incomes were normally distributed around $60,000 a year. After, in the 2000s, they become bimodal. And so you have one mode that's still around that $60,000 range. Those are the family lawyers, And then you have this other mode that's into the 6 figures, and those are like big law, right? It's the emergence of these law firms where you have a few partners on top and maybe hundreds of associates who are doing kind of grunt work using Westlaw and LexisNexis and these other legal search engines to accelerate drafting and legal analysis. So if that pattern repeats, I could imagine these various high-skill knowledge sectors to also become bimodal, where in the short run AI serves as a copilot, sort of like Westlaw or LexisNexis was for legal research, and enables the kind of 100x lawyer. And so there's a kind of averages over dynamic. Longer run, you start to see the possibility of doing an end run around existing accreditation and licensing monopolies where obviously the American Medical Association and medical boards will be highly resistant to an AI doctor. I tend to think that they'll probably end up self-cannibalizing because the value prop is so great even for doctors to do simple things like automate insurance paperwork and stuff like that. But to the extent that there is a resistance, to the extent that in 10 years there's still a requirement that You must have the doctor prescribe the treatment or refer you to a specialist, even though the AI is doing all the work and they're just sort of like the elevator person that's actually just pushing the button for you. It'll be very easy to end run that because AI is both transforming the task itself, but also transforming the means of distribution. And if you can go to GPT-4 and put in your blood work and get a diagnosis, That no regulator is going to stop that. And so GPT-4 becomes sort of the ultimate doctor without borders.

You write a lot about transaction costs and how changes in transaction costs change institutional structures. First of all, what are transaction costs and how do you think they'll be affected by AI?

Transaction cost is sort of an umbrella term for different kinds of costs associated with market exchange. And this goes back to Ronald Coase's famous paper on the theory of the firm where he asks the question, why do we have corporations in the first place? If free markets are so great, why don't we just go and spot contract for everything? And the answer is, well, market exchange itself has a cost. There's the cost of monitoring. If you hire a contractor, you don't know exactly what they're doing. There's the cost of bargaining. Having to haggle with a taxi cab driver is is a friction. And there's the cost associated with searching information. So taking those 3 things together, they're not all that companies do, but they structure the boundary of the corporation. They explain why some things are done in-house and some things are done through contracts. If there's high monitoring costs, you want to pull that part of the production into the company so that you can monitor and manage the people doing the production.

And some of the same effects go for the existence of governments, right?

Yes, because governments, with a certain gestalt, governments and corporations aren't that different. They're kinds of institutional structures that pull certain things in-house and certain things are left for contracting or outsourced. And you even see sort of different kinds of governments having different parallels with different kinds of corporate governance. Relatively egalitarian democratic societies like Denmark are kind of like mutual Insurers, whereas more hierarchical authoritarian countries are more like, you know, like Singapore, say, is more of a joint stock corporation. And indeed, you know, Singapore was founded as an entrepôt for the East India Company. So there are very deep parallels. And it's also essential— transaction costs are an essential lens to understand why governments do certain things and not other things. All Western developed governments guarantee some amount of basic healthcare, right? But most, outside of say the National Health Service in the UK, most of these countries guarantee the insurance. They don't necessarily nationalize the actual providers, right? And the reason goes to transaction costs and sort of an analysis of the market failure in insurance. Likewise with roads. It's possible to build roads through purely private means and indeed countries like Sweden, a lot of the roads are run by private associations. But if you have lots of different boundaries, different microjurisdictions and so forth, there can be huge transaction costs to negotiating up to an interstate highway system. And those transaction costs then necessitate public infrastructure projects.

So the transaction cost in this case would be being a private road provider. You'd have to go negotiate with 500 different landowners about building a highway, whereas the government can do some expropriation and simply build the road much, much faster or with less transaction costs, at least.

Yeah, precisely. And we're seeing this, this dynamic in the US with, you know, permitting for, for grid infrastructure and transmission. You know, we were We're building all this, all the solar and renewable energy, but to build the actual transmission infrastructure to get the electrons from where it's sunny to where, where it's cold requires building, you know, high voltage lines across state lines, across different grid regions. And there are all kinds of NIMBYs and negotiation costs involved, holdouts and so forth. And so that the more those kind of costs exist, the more it militates towards a kind of Larger-scale intervention that federalizes that process.

Yeah. The big question then is how will AI change these transaction costs? What will the effects be here?

It's easy to say that they will be affected and obviously the internet affected them to an extent. And we were talking about the ease of mobilizing protest movements or the sunlight that was put on government corruption. Those are Those are reflecting declines in the cost associated with information and coordination. I think AI takes us to another level, and I think it's important to think through, in part because right now the AI safety debate, at least in the United States, is very polarized between people who are like, everything's going to be great, and people who are like, this is like a Terminator scenario or an AI kill us all existential risk. Even if we accept the existential risk framing, there's still going to be many intermediate stages of AI before we flip on the superintelligence. And those intermediate stages have enormous implications for the structure of the very institutions that we'll need to respond to superintelligence or what have you. The ways we can see this is because all these information and and monitoring and bargaining costs are directly implicated by commoditized intelligence. Start with the principal-agent problem. There is no principal-agent problem if your agent does exactly as you ask and works 24/7, doesn't steal from the till. And so AI agents dramatically collapse agency cost. Monitoring. Now that we have multimodal models, in principle, we could have cameras in every house, that are just being prompted to say, is someone committing a crime right now? Whether we wanted to go that direction or not, it gives you a sense of how the costs of monitoring have basically plummeted over the last 2 years and are going to go way lower. And so you're starting to see this rolled out in the private sector with Activision has announced that they're going to be using language models for moderating voice chat in Call of Duty. And this is a more robust form of monitoring because in the past you would have to ban certain words, certain swear words or things associated with sex or violence, but then people could always get around those by using euphemisms. Like on YouTube, the algorithm will ding you if you talk about coronavirus or if you talk about murder or suicide, these things that throw off red flags. So what people have taken to doing is saying they were unalived rather than murdered. And that doesn't fool a language model. If you ask a language model, if you prompt it in a way to look for sort of broad semantic categories, not just a narrow word, it's much more robust. And so what that means, you already start to see it, like I said, with Activision and the use of LLMs in content moderation. You're going to see it in the use of Multimodal models for productivity management and tracking. Microsoft is unveiling their 365 Copilot where you're going to have GPT-4 in Word and Excel and Teams and Outlook. But at the same time, you're also going to have a manager who's going to be able to say, to prompt the model, tell me who's the most productive this week. Right. Something as vague as that. And so you see this diffusion in the private sector. The question is, does it diffuse in the public sector? There's obvious ways that it would be a huge boon. Inspector General GPT could tell you exactly how the civil service is working, whether there's corruption, whether there's a deep state conspiracy or something like that. And at first blush, a lot of what government does is kind of a fleshy API. Bureaucracies are nodes that apply a degree of context between printing out a PDF and scanning it back into the computer. It varies. There's degrees of human judgment that are required. But on first order, government bureaucracies seem incredibly exposed to this technology and in a way that could diffuse really rapidly because going back to Microsoft 365 Copilot, Microsoft is the biggest IT vendor in the US government.

Right.

And so you can imagine Once everyone has this pre-installed on their computer, that the person at the Bureau of Labor Statistics who's in charge of doing the monthly employment situation report, the jobs report, at some point he's going to be walking into work and hitting a button, asking Excel to find the 5 most interesting trends and generate charts, and the report is done. And in the private sector, that person would be reallocated and maybe doing things that the computer's not good at yet. But these positions are much stickier in government. To the extent that diffusion is inhibited on the public sector side, I worry about the kind of disruption and displacement of government services by a private sector that's adopting the technology really fast.

This is something we'll talk about in a moment. Before that, I just want to get to your complaints about isolated thinking about AI. You've sketched out some complaint about People thinking about AI only applying to one domain and then not really seeing the bigger picture. What are some examples here? Why do you worry about isolated thinking?

A few dimensions to this. One is what I've called the horseless carriage fallacy, the kind of view that what automobiles were was just a carriage without the horse. And so that anchors you to the older paradigm and it's like you're changing one thing and everything else stays the same. And you neglect all the second-order ways that the development of the automobile enabled the build-out of highway systems, the total reconfiguration of sort of the economic geography. And then implications for institutions like the state where once you have road networks or telegraph networks or any of these kind of networks, it suddenly becomes easier to monitor agents of the state in other parts of the country. And so you can build out more of a federal bureaucracy. And so all these things were second order and were kind of neglected if you just were too focused on the first order effects of displacing the horses.

And in a sense, the second order effects turned out to be much more consequential in the end.

Yes, they seem to always be. And likewise with the internet and sort of the— I think this comes up a lot in how to think about AI use and misuse. There's lots of valid discussions there, but they're always very first order. And when you think about the way the internet has disrupted legacy institutions, yes, there's disinformation, but often the thing that's disrupting is not fake news, it's real news that's being repeated with misleading frequency. That's throwing off our availability heuristic. Or it's valid things that— valid complaints, whether Whether the protests in Iran, right? The protests in Iran have this striking parallel to the protests following the George Floyd protests and protesting in other countries where they even have a 3-word chant, right? Or the case of the Arab Spring in Tunisia that started with a person self-immolating, right? There's sort of the structure that repeats where you have a martyr or some shocking event. And because of the way social media is organized, it synchronizes people around the event in a way that's kind of stochastic. It's like lightning striking. You don't know what event it's going to strike on, but once we're synchronized, then we start moving back and forth in a way that causes the bridge to buckle. Nothing about that is a misuse. Those are all valid uses, but they're uses under— collective action. It starts solving not just for the partial equilibrium, but the general equilibrium when everyone is doing this. And I think the person who wrote the best on this conceptually was Thomas Schelling. And one of his little books, Micromotives, Macrobehavior, had a big influence on me as a kid where he talks about all these toy models where you're at a hockey game or a basketball game and something is happening, something exciting is happening in the arena. And so people, the people in front of you stand up to get a better view, and then you have to stand up to get a better view of them over them, and so on. And so it cascades, and suddenly everyone went from sitting to everyone went to standing, and no one's view has improved, right? And so these sort of, uh, general equilibria where you sort of solve for everyone's, uh, micro incentives and, and the kind of new Nash equilibrium that emerges, that ends up being the thing that drives the kind of multi— multiple equilibrium shift from one regime to another. And throughout, there may be no actual examples of misuse involved. It may just be people following their individual incentives.

I think it's worth stressing this point you make about the effects of earlier AI systems on our institutions. They might have effects that deteriorate our institutions such that we can't handle later and more advanced AI. And ignoring this would be an example of isolated thinking and ignoring the second-order effects. Yeah.

And it also changes the agenda. The AI safety agenda shouldn't just be about the first-order things around alignment. Very important. But it's led to a discussion of, do we need a new federal agency? And if so, what kind of agency? Whereas it may be more appropriate to think, Not what new agency do we need, but how do all the agencies change? And how do we sort of brace for impact and enable a degree of co-evolution rather than displacement?

I don't know whether the question of how to get our institutions to respond appropriately is more difficult or less difficult than the problem of aligning AI, but it certainly seems very difficult to me. Are we making it harder on ourselves if we focus on the second-order effects on institutions?

I mean, it's unavoidable. We can't pick and choose what kind of problems, but the alignment problem, the hard version, is yet to be solved. But we have many examples of governments building state capacity and shifting from very clientelistic, sticky, corrupt governments to sort of modernized governments where state capacity is built and then that government can sort of break out of the middle-income trap and become rich.

You mentioned Estonia as an example of a country that's pretty advanced on the IT front, on the technology side. Maybe you could talk a bit about Estonia.

Yeah, I would just say in general, it's hard for any organization to reform itself from within when there is path dependency, but I would say at least we have examples of it being done. Where we don't have examples of alignment being solved yet. When it comes to Estonia, Estonia is an interesting case. It's sort of an exceptional case because after the fall of the Soviet Union and the breakup of the peripheral former Soviet states, they kind of had a blank slate. They also had a very young population and people who had a kind of hacker ethic within their civil service. And so with that blank slate and with that hacker ethic, they were very early to adopt and to foresee the way the internet was going to shape government through a variety of e-government reforms. So early in the late '90s and into the 2000s, they were some of the earliest to digitize their banking system, like e-banking, to build this system called X-Road, which is kind of like a cryptographically secured data exchange layer. It resembles a blockchain, but it was It was about a decade before blockchain was invented.

For exchanging information between different government entities. Your medical information could be uploaded to this system and then be available to all systems that have the right to see that information.

Exactly, in a way that's cryptographically secured and distributed. So if a missile hit the Department of Education, you don't lose your education records because it's distributed. And that also enabled an enormous amount of automation where, for instance, this is my understanding, a child born in Estonia, once you file that birth record, it more or less initiates a clock in the system that will then enroll your child in school when they turn 4 or 5 automatically because it knows that your child has aged, unless it had a death record to cancel that out. But it also means you can do taxes and transfers much simpler. You get your benefit within a week. You can integrate across different parts of public infrastructure, like use the same card to ride the bus as you do to launch a new business. And it also serves as a kind of platform for the private sector to sort of do government by API, right? To build new services on top of government as a platform and integrate with government databases.

Yeah. And so the point here for us is that institutional reform is possible. Modernizing government is possible, at least under certain circumstances. We have proofs of concepts of this happening.

The hard thing is the path dependency. There's always a strong instinct to want to start from scratch, and it's normally not advisable because it's too hard. And so this is why it's hard in the US. This is why you have African countries that leapfrog us in payment systems and so forth. The challenge of this decade or century is how do we solve that path dependency problem and how do we get to Estonia? It used to be get to Denmark. Now it's get to Estonia and find that pathway up Mountain Probable.

Great. Let's get to your wonderful series of blog posts on AI and Leviathan. In this context, What do we mean by Leviathan?

Well, this all interrelates. So Leviathan was the book Hobbes, Thomas Hobbes wrote at the start of the interregnum after the English Civil War. And it was his, basically his early political science, early defense of absolutist monarchy as a way to restore peace and order after a decade of infighting. And Hobbes kind of hit on some basic sort of structural game theoretic properties of why we have governments at all. He talked about life being nasty, brutish, and short in the state of nature, war of all against all. And peace is only restored when people who don't trust each other offload enforcement and policing responsibilities to a higher power. That can then restore a degree of peace and order. AI and Leviathan is talking about how does AI change the story? Does it reinforce the Leviathan? Does it lead to a digital police state à la China, or is it something that we impose on ourselves? And we talked about how multimodal models could in principle be used to put a camera in everyone's house and have it just continuously monitoring for people doing any kind of crime. That's something that North Korea might do. In the US context, it's something that we're very liable to just voluntarily do to ourselves because we want to have Ring cameras and Alexa assistants and so forth. And so that leads to a kind of bottom-up Leviathan that is potentially no less oppressive and maybe even more oppressive because there's no one that we can appeal to to change the rules.

Yeah, so Leviathan is one way to respond to technological change, but you mentioned 2 other ways we could alternatively respond.

Right. So basically anytime a technology greatly empowers the individual, it creates a potential negative externality. Hobbes called these our natural liberties. In the state of nature, I have a natural liberty to kill you or to strong-arm you. And governments exist to revoke those natural liberties, but for a higher form of freedom. And so anytime a technology greatly increases human capabilities vis-à-vis other humans, the 3 canonical ways we can adjust are ceding more authority to that higher power, the Leviathan option. And then the other 2 options are adaptation and mitigation and normative evolution. So the example I give is If suddenly we all had X-ray glasses and could see through walls and see through clothing, one option would be we have a draconian totalitarian crackdown that tries to seize all those X-ray glasses. Another option is we adjust normatively, culturally, that our privacy norms wither away and we stop caring about nudity. And then the other option is adaptation mitigation where we We put in mesh into our walls and wear leaded shirts and pants.

Yeah, I guess continuing that analogy a bit between the smart glasses and AI, you have this amazing write-up of ways in which AI can increase the informational resolution of the universe. So you give some examples that are— I'm thinking specifically of AI identifying people by gait, for example.

Right. So gait recognition is nothing new. China has had advanced forms of gait recognition for a while now. So even if you cover your face, it turns out we're constantly throwing off sort of ambient information about ourselves, about everything. And the way you walk, the particular gait that you have, is a unique identifier. Another example is galaxy surveys. We've had from Hubble Telescope to now the JWST tons of astronomical surveys of distant galaxies and so forth. And all of a sudden, all that old data, it's like that same dataset is now more useful because applying more modern deep learning techniques, we can extract entropy that was in that dataset, but we didn't have the tools to extract yet. And discover that There are new galaxies or other phenomena that we missed.

Another example you give is listening for keystrokes on a keyboard and extracting information about a password being typed in, for example, which is something that of course humans can't do, but we can do with AI models.

Yeah, so that was a paper showing that you can reconstruct keystrokes from an audio recording, including a Zoom conversation. So I hope you haven't typed in your password because people the future. And so this goes to the fact that it's sort of retroactive, that even if the technology wasn't diffused yet, any Zoom conversation, any recording where someone typed their password in the future will be like those galaxy surveys where someone will go backwards in time and turn up the information resolution of that data.

Yeah, this is pure speculation, but I wonder if— I mean, imagine anonymized people in interviews, say, 10 years ago, whether they will be able to stay anonymous or whether AI will be able to extract the data about their face or their voice that wasn't technically possible when the interview aired.

Yeah, exactly. There are already systems for depixelating. You probably do something similar for the voice modulation. And then also sort of, again, going back to this ambient information we're always shedding, identifiers in the way we write, where we place a comma, The kinds of adverbs we like to use and so forth. People just dramatically underrate how much information we're shedding, in part because we're blind to it.

Some people who are taking great efforts to stay anonymous online, people in the cryptography space, for example, will put their writings through Google Translate to French and then back to English to erase subtle clues to how they— that could identify them personally. Why is AI so much better at tasks like the ones we just mentioned compared to humans?

Well, it goes back to what we were talking about with putting information theoretic bounds on AGI. When you minimize the loss function in a machine learning model, you're trying to minimize the cross-entropy loss. Cross-entropy is how many bits does it take to distinguish between 2 data streams? And if it takes a lot of bits to distinguish between the 2, that means they're relatively indistinguishable. So this is going again to the Turing test. If we have a Turing test where I can tell right away that the AI is different from the human, that suggests a high cross-entropy. But if I have to talk to it for days and do all kinds of adversarial questioning, I might still be able to, in the end, tell the difference between the two, but we've minimized that cross-entropy loss. And so when you have any arbitrary data distribution that you're trying to predict, Whether it's trying to predict galaxies in astronomical data or passwords from fingerprint data on a phone screen, all these things embed a kind of physical memory of the thing in question and can often be reconstructed through this kind of loss minimization where you have a system that asymptotically extracts the entropy that was latent. And this can be done in a way that is often quite striking where we can, with Stable Diffusion, make fairly accurate predictions of what people are imagining in their mind using fMRI data. And fMRI data is blood flow data in the brain. It's a very lossy representation of whatever's happening in the brain. But there's still enough latent entropy in there that we can reverse engineer or Decompress it into a fuller picture.

And this could turn into a form of lie detection.

Yeah, I think it already basically has. If you have fMRI data or EEGs or other kinds of direct brain data, it's probably a lot easier, but we already have systems that are over 95% accurate at detecting deception. From just visual video recordings.

We can see how all of this information that we are continually shedding gives rise to the possibility of a leviathan, either of the private or of the government's kind. I wonder what role do you see open sourcing AI models playing here? What are the tradeoffs and risks in open sourcing AI?

Among the people who are most bullish to open source, there's often a kind of libertarian ethic undergirding it.

Right.

Regardless of whether that's a good idea or not, one of the things I'm trying to communicate to that group is to say that be careful what you wish for because of these kind of paradoxical Hobbesian dynamics. The fact that in America, you never know if someone has a gun or not. On the one hand, the Second Amendment enhances our freedom. On another hand, You don't get the sort of like everyone's door is unlocked and people are— the police in England don't even have guns. There's a certain freedom that derives from us not all being heavily armed. And likewise, with open sourcing powerful AI capabilities, it empowers you as an individual, But in general equilibrium, once we all have those capabilities, the world could look much more oppressive either because we're all spying on each other all the time and we can all see through each other's walls or because there's a backlash and the introduction of Leviathan-type solutions to restrict our ability to spy on each other all the time. And my general sense is that we can only delay, we can't really prevent things from being open source over the long run because there's a trickle-down.

Yeah.

of compute requirements. But in the interim, there are definitely things that are valuable to open source. Having 70 billion parameter language models is not a threat. In fact, I think it's probably useful for alignment research for something like that to be open source. But if you are a researcher and you've developed an emotional recognition model that can tell with 99% accuracy whether someone is lying or not lying and whether your girlfriend loves you or not. These things, or the ability to see through walls using— I talk about the use of Wi-Fi displacement. There are people who have built pose recognition models using the displacement of the electromagnetic frequency of your Wi-Fi and they can see— and it's wall penetrating so that you can see through walls. What's the rush to put that on Hugging Face and to make it as democratized as quickly as possible? I would say that if we value the adaptation and mitigation pathway as opposed to the Leviathan pathway, then there's a value in slow rolling some of these things.

How do you think government power will be, or relative government power will be affected by So you write somewhere in this long series of blog posts that AI will cause a net weakening of governments relative to the private sector. Why is that?

Yeah, specifically Western liberal governments under constitutional constraints. So if you imagine society being on this kind of knife edge, I talk about this in the context of Daron Acemoglu's book, The Narrow Corridor, where he describes liberal democracy as sort of being in this corridor between despotism on the one hand and anarchy on the other. And we sort of have to stay in this saddle path where society and the state are kept in balance. If you veer off that path, you can, on the one hand, the state could become all-powerful, and that's the sort of China model, more authoritarian digital surveillance state. And indeed, China built up their digital surveillance state and their internet firewalls and so forth after watching the Arab Spring and seeing how the internet was destabilizing to weaker governments. And so I fully expect that AI will be very empowering and self-reinforcing of the power of the Chinese government. And indeed, their draft regulations for large language models stipulate that you can't use the model to undermine national unity or challenge the government. And so they're baking that in. In liberal democracies, we think of ourselves as open societies. And the issue is that we're only open at the meta level. There's a public sphere, there's freedom of information laws, we have freedom of speech. I don't have freedom of speech if I walk into a Walmart.

Right.

The Walmart is private property. In open societies, it's not that we don't have social credit scores and forms of thicker forms of social regulation. It's just that we offload those functions onto competing private actors, whether it's a church that has very strict doctrines to be a member or other kinds of social clubs. The fact that these days, if you want to go to a comedy club, they'll often confiscate your phone at the door because they don't want you recording the comedian's set and putting it online. My anticipation is that because of those constitutional constraints that that limit the ability of liberal democracies to go the China route because of our civil laws or bills of rights and so forth, and also because of a lot of procedural constraints, this will naturally shift into the private sector. And we see that already with the use of AI for monitoring and employment, for policing speech in ways that would be illegal if done by the state, but are fine if done by By Facebook. To the extent that the AI continues to increase these kind of negative externalities and therefore puts more value on having a sort of vertically integrated experience, a walled garden that can strip out the negative forms of AI and reinstate a degree of harmony between people, that more and more of our social life will be mediated through these sort of private organizations rather than through a kind of open public sphere.

Oh, you're imagining that government services will be gradually replaced by private services that are, that are better able to respond. Want governments fight to uphold individual rights? In Walmart or on Facebook, you are regulated in ways that the government couldn't regulate you, but you still have the choice to go to Target instead of Walmart or to go to X instead of Facebook. Isn't that the fundamental thing? So the fundamental thing is the choice between services and won't governments uphold citizens' rights to make those kinds of choices?

Yeah, no, I agree. And so this would be the defense of the liberal model is that we allow thicker forms of social regulation because it's moderated by choice and competition. And the issue with, with, you know, Chinese Confucian integralism isn't the, uh, the fact that it's, uh, super, uh, oppressive. It's the fact that you only have one choice and you don't have voice or exit. So, uh, yeah, the— but it's, it's, it's obviously a matter of degree, right? Um, when, you know, ride-hailing versus first arose, I remember back in 2013, 2014, it wasn't that long ago. I think Uber was founded in 2009, but it really only started taking off in the early 2010s. People thought it was crazy to ride a car with a stranger. And then within 5 years, it was the dominant mode of ride-hailing. And in that 5-year period, essentially we saw a kind of regime change in micro. Where taxis went from being something that was regulated by the state through these commissions that were granted legal monopolies and used licensing and exams and other sort of brute force ways of ensuring quality to competing private platforms where you have Lyft or Uber to choose from. And they replaced the explicit governance of legal mandates with The competing governance of reputation mechanisms, of dispute resolution systems, of structured marketplaces that collapse the bargaining frictions. You never have to haggle with an Uber driver, you just sort of get in. And that was obviously a much better way of doing ride-hailing. So even though there was sort of a violent resistance early on, literally in France they were throwing rocks off of bridges and— cab drivers in New York were killing themselves. So for the people affected, it was a very dramatic sort of regime change, but for everyone else, it was a huge positive improvement. And yet it's only made possible because Uber has a social credit score. If your Uber rating goes too low, you'll get kicked off the platform. And so we're fine with social credit scores. It's when you only have one and don't have an option, and it can follow you across all these different—

Do you imagine that because of rising danger in the world, you talk about the externalities from the widespread implementation of AI all across society, because of those dangers, those externalities, you know, you will either use Uber or whatever service or you kind of can't participate in society. Do you imagine increased pressure in that?

It does seem to be a longer-term trend. I don't know if AI will accelerate it. I have another series of essays that I call Separation Anxiety, and it's a reference to the fact that in insurance markets, there's kind of 2 equilibria. There's the pooling equilibria where we're pooled together into one risk pool, and then there's the separating equilibria where the insurance pool unravels and we break up into the gray power insurance for senior citizens who'd never had an accident and stuff like that. And it turns out that insurance markets are competitively unstable, that without government regulation or social insurance, that insurance markets will naturally tend to unravel because of adverse selection into the high-risk people being in one pool and the low-risk people being in another pool. And it turns out you can sort of use that as a mental model to look at other kinds of implicit pooling equilibria. So within company wage distributions, often there is 20% of the workers who are doing 80% of the work, but they're pooled together under one wage structure. And that was sort of the dominant structure of the period of wage compression in the United States in the '50s and '60s. And once we had better monitoring technologies and were able to tell who were the 20% that were doing 80% of the work, It suddenly became possible to differentiate pay structure. And a lot of the rise in inequality in the United States is actually between firms. So what happens is Ezra Klein is the most productive whiz kid at the Washington Post and he realizes, why don't I just go start my own website? And so that dynamic sort of played out across a variety of domains leads to a world that, to the extent that these features are correlated, that does separate, where you have the 1-star Uber riders driving the 1-star Uber drivers, the drivers driving the riders, and people who have the 5-star Uber ratings and the perfect credit scores self-sort into communities with other people with perfect driving records and perfect credit scores. And we see that to an extent already with the enclaves of rich zip codes with private schools and everyone is Sort of self-selected. AI could— it seems to me that AI would exacerbate that. I mean, at first blush, just because going back to the point about signal extraction, it can find all these different ways. You're a high-risk type and I'm a low-risk type and so forth that are probably latent in all kinds of data that we don't even need to give permission to the insurance company. They'll just— the same way that they use smoking or going to a gym as a proxy, there's all kinds of proxies they could use. And likewise for employers and how they pay people.

Society kind of runs on us not being entirely open and entirely honest all the time. Otherwise, you wouldn't be able to have kind of smooth social interactions and so on. Won't these norms be inherited by the way we use AI?

Yeah, I think this is a really big issue. I'm a big fan of Robin Hanson and a lot of his writing on social status and signaling is It's sort of presenting humans as basically hypocrites. We're constantly deceiving other people and we often deceive ourselves, so it's better to deceive others, as the evolutionary biologist Robert Trivers once pointed out. So all the kinds of polite lies that we tell are, I think, Critical lubricants to social interaction. And actually, it's good that there's a gap between our stated and revealed preference. I think a world where we all lived our stated preference could be hellish because we don't actually mean it. And AI has a direct implication on that because if I can have a pair of AR glasses on that will tell me if you're interested, if you're bored, if we're on a date and are you really Attracted to me. All that sort of polite veneer, that social veil, could be lifted in a way that we'll probably want to coordinate to not do. But again, it's this Nash equilibrium where it's in my interest to know whether you're interested or bored. And so I'll want to have the glasses on. And my ideal world is where only I have the glasses and you don't. And the other way that our hypocrisy is being exposed and challenged is the need to explicate the utility function that we want these models to work under. We need to formalize human values if we want to align these models. And so then we have to be honest and open about the fact that our stated preferences probably aren't our true preferences. And that's a very challenging thing because it cuts right to the nature of the human condition. and involves topics that are intrinsically things that we lie to ourselves about.

You have what you call a timeline of a techno-feudalist future, which I found quite interesting. Yeah, it's great writing and it's very detailed. We don't have to go through it in all of its detail, but maybe you could, you could tell the story of what happens in what you call the default scenario. This is the scenario in which Western liberal democracies are too slow to adapt to AI. And so we get something like a replacement of government services with more private services? What happens in the techno-feudalist future?

Right. And this sort of piggybacks on everything we've just been discussing, right? And I don't want techno-feudalist to carry too much of a pejorative. I'm sort of using it descriptively. And certainly some people would prefer this world. So the example of Uber and Lyft displacing taxicabs is sort of a version of this in micro. Where we go from this regulated taxi commission to competing private platforms that use various forms of artificial intelligence and information technology to replace the thing that was being done by explicit regulation. And as AI progresses and both creates a variety of new negative externalities, whether it's suicide drones or— Or the ability to spy on each other, there's going to be a demand for new forms of security and also kinds of opt-in jurisdictions that tie our hands in the same way that we give up our phone before we go into the comedy club. And so I think this leads to a kind of development of clubs, the kind of club structure maybe at the city level. As the vertically integrated walled garden that will police and build defensive technologies around the misuse of AI. And at the same time, provide a variety of new AI-native public goods that are only possible once AI unlocks them. And it's easy to see how this could very quickly displace and eat away at formal government services. Both because we saw it already with Uber, but also if you map that model to other areas of regulatory life, does it make sense to have a USDA farm inspector, a human person has to go to a commercial farm and maybe only goes to that farm once every few years because there's so many farms and only so many people and does a little checklist and says, oh, you're not abusing the animals and you do all the processes? in place and you get the USDA stamp of approval? Or does it make more sense to have multimodal cameras on in the farm 24/7 that are continuously generating reports that throw up a red flag anytime someone sneezes on the conveyor belt? And to the extent that government is going to be slow at adopting that, will there be a push for the kind of Uber model of governance as a platform where you have the kind of AI underwriter, the Consumer Reports that sells these firms the camera technology and the monitoring technology and builds their own set of compliance standards. And then you want to go to those firms or what have you that have the stamp of approval of that underwriter because it's much higher trust. It's sort of like the end of asymmetric information. And you can map that from food safety to product safety to OSHA and workplace safety. There's other parts of government that may be just rendered completely obsolete. Once we have self-driving cars that are 1,000x more safe than humans, do we need a National Highway Traffic Safety Administration? Once we have sensors that are privately owned everywhere and And can model weather patterns better than the National Oceanic Administration? Do we need a National Weather Service or could we bootstrap that ourselves? And then once we have AI-accelerated drug discovery, do we want to rely on the FDA to be a kind of choke point to do these sort of frequentist clinical trials that are inherently slow and don't capture the kind of idiosyncrasies? idiosyncrasies and heterogeneity that could be unlocked by personalized medicine? Or do we move to an alternative drug approval process that is maybe non-governmental, but much more rapid and much more personalized?

So that's the overall picture. I'll just run through the timeline here, picking up on some of your comments that I thought were especially interesting. You write, and this is in 2024 to 2027, you write that The internet will become, uh, Balkanized and, uh, you will have— it will become more secure and more private in a sense. Why, why does that happen?

We're already starting to see this a little bit, right? Um, once people realize that the data that's being generated on Stack Overflow or Reddit or whatever, uh, is valuable for training these models, suddenly everyone's closing their API, you know, and consequently Google searching Google Search and the Google index have sort of started to degrade already. So I think that will continue for the kind of privatization of data reasons. Then when you also think about how websites are going to handle the growth of bots and catfish attacks and cyberattacks and so forth, it makes sense that we're going to move from a sort of open everything goes kind of Twitter-esque platform to things that are much more closed because they require human verification and identity verification to sort of build the trust that you're talking to other people and not deepfakes. And then medium term, again, over this sort of 2024 to 2027 horizon, you could also start to see the emergence of intelligent malware, sort of modern AI-native cyberattacks that could be devastating to legacy cybersecurity infrastructure in a way that I talk about could hearken back to the famous Morris worm that in the late '80s basically shut down the early internet. They literally had to partition the internet and turn it off so they could rid the network of the worm. So for all those reasons, I think you start to see the internet balkanize. And then particularly at the international level, we're already starting to see Sort of the semiconductor supply chain become a critical part of national security, the growth of the Chinese firewall, the European Union is going to have to have their own quasi-firewall and they kind of already do with GDPR and the EU AI Act. And so the kind of nationalization of compute and telecommunications infrastructure that will take off once people understand both the security risks and the value prop of owning that infrastructure For the AI revolution.

Yeah. In 2028 to 2031, you write about alignment turning out to be easier than we thought with the increasing scale of the model. That was somewhat surprising to me. Why does alignment turn out to be easier?

And part of this is imagining a scenario where alignment is easy. So we can talk about what happens if alignment is easy. But I think there are reasons to think that the classic alignment problem will be easier than people think. I think that some of the early intuitions about the hardness of the alignment problem were rooted in a view of maybe AI turns out to be a very simple algorithm rather than a deep neural network that achieves its generality because of its depth. Clearly the kind of value I forget what Eliezer Yudkowsky called it, but there's a value alignment problem where how do we teach the model our values? That part of the alignment problem seems trivial now because our large language models aren't autistic savants. They're actually incredibly sensitive to soft human concepts of value and context. They're not going to have a— The paperclip maximizer sort of monkey paw kind of threat models don't really make sense in that world.

But there's a difference between the output of the model and the weights or what the model has learned. And so just because a model can say the right words that we want it to say, but what has it actually learned? We are not entirely sure. And so it has learned to satisfy human values to some extent, but has it learned to has learned to want to comply with human value out of distribution in other domains. And in a deep sense, I'm not sure about that.

No, I agree. So I'm sort of just laying some of my groundwork to explain my priors on this. I agree, reinforcement learning from human feedback is not alignment in the same way that you could argue that The coevolution of cats and dogs with humans led to a kind of reinforcement learning from human feedback in their short-run evolution that made them appear as if they experience guilt and shame and these human emotions when in fact they're just sort of a simulacra of those emotions because it means that we'll give them a treat.

But I've done plenty of episodes on deception in these models and so on. We don't have to go through that, but I just wanted to point out that, yeah, Maybe there's some complexities there.

So my first prior is that these models aren't autistic savants the way they might have been. The second is going back to universality. While it is true that it's possible through reinforcement learning from human feedback, for example, that you're not selecting for honesty, you're selecting for a deepfake of honesty, But in the bigger picture, the intuition that these models are converging or convergent with human representations should give you some confidence that they're not going to be as alien as we think they will be. It's also useful input for thinking about interpretability. There's some recent work discussing representation interpretability where instead of trying to interpret individual neurons, you interpret collections of neurons and circuitry through sort of human interpretable representations. And one of the lessons of universality is that some of these high-level human concepts like happiness or anxiety, these seem like vague psychological abstractions that there's no way they can correspond to the micro foundations of the way our brain works, but in fact, they may actually be very efficient, low-dimensional ways of talking about what's happening in our brain. And then the third thing is I think that I just have seen, my sense is that the work on interpretability is actually making some good progress. Whether it can scale is another question, but I think we'll get there. In my timeline, I talk about AGI-level models within the human emulator plus domain. I do later on talk about superintelligence emerging maybe in the 2040s, And that's another story. And so I think some of this stuff maybe goes out the window if we have models that are bigger than all the brains combined and have strong situational awareness. But I don't think that happens this decade. Certainly not with the current way we're building these models. The way we're currently building these models, I think, comes much closer to a simulacra of the human brain.

Got it. In 2036 to 2039, you talk about robotics being solved to the same extent, or maybe even in the same way as we are now solving language. I found that super interesting. Explain to me why would robotics suddenly, or quite relatively suddenly, become much easier? Roboticists have been fighting for decades to get these models to walk relatively unencumbered, and it's been an uphill battle. Yeah, why can we solve robotics in the 2030s?

This may end up happening sooner than I project, but if you look at LLMs, one of the stylized trends with large language models is that natural language processing went from being this study of how to make machines understand language, went from being a dozen different subdisciplines. You had people working on parsing, people working on syntax, people working on semantics, people working on Summarization and classification. And these are all different directions, research directions. And then along comes transformer models and it just supplants everything and LLMs can do it all. And I think robotics is sort of still in that ancient regime where a lot of what Boston Dynamics does is ad hoc control models, analytically solvable differential equations. Different kinds of object recognition modules and control action loops and so forth. And so it's still in that early NLP phase where they have 12 different subdisciplines and they're sort of mashing them together. And of course you get something that's not very robust. I think we're already starting to see that paradigm shift to end-to-end neural network trained models like Tesla, for instance, I think one of the reasons why Tesla cars had a sort of temporary decline in performance was because they were undergoing the transition from these ad hoc lane detectors and stop sign detectors and stuff like that to a fully end-to-end neural network transformer-based model. And that turned out to be a much more robust way to train the model because stop signs look different in different countries and maybe stop sign isn't the thing you care about really. So on and so forth. And so I think the transformer sort of scale deep learning revolution is only now coming to robotics. And people in that field are a little bit cynical because they're used to relatively small RL models, thinking that the fit with actuators and some of the hardware is a really challenging problem, and also believing that we don't have the datasets for it. But then you look at this recent RoboDog that you may have seen on Twitter, fully open-source robot model for a Boston Dynamics-style dog. It was trained on H100's 10,000 human years of training and simulation, and then some fine-tuning on real-world data. And they have a very robust robot control model that you could plug into all kinds of different form factors and have something that can hop gaps and climb stairs and do all the things that Boston Dynamics robots don't do very well outside of their distribution.

Do you think we'll have a general-purpose algorithm that we can plug into basically arbitrarily shaped robots that can then navigate our apartments or our construction sites or maybe our highways? That's an interesting vision, I think. Why is it that we achieve this level of generality?

If you look at humans, humans are very good at, if we've suffered an amputation or you have to go through physical therapy and it's not easy necessarily, but humans are able to adapt to different kinds of physical layouts of our body. I think there will be a trend towards unified robotic control models that aren't super tailored to 2 legs and 2 arms and so on and so forth. Once you've installed it through a little bit of in-context learning or fine-tuning or reinforcement learning, adapt to that particular form factor. And this will parallel the kind of pre-trained foundation model paradigm that is currently taking place in LLMs where you have the really big foundation model that can sort of do everything reasonably well and then you can fine-tune it beyond that.

If we get to the 2040s in your timeline, you talk about massive amounts of compute being available. You talk about post-scarcity in everything except for land and capital. And then you also talk about the development potentially of superintelligence at that point. What happens there? Who is in control of the superintelligence, if anyone?

Yeah, this is sort of where I start to get a little bit tongue-in-cheek, but first of all, I talk about how I tend to think that once we have exascale computing, and I think DOE just built their first exascale computer, maybe it was a private company, but we have one exascale computer in the world. By the 2040s, they'll be commonplace. And if we're worried about controlling the supply of GPUs, I don't know exactly how much compute will be in our smartphones, but it will definitely be possible to train a GPT-5 model from your home computer. And so any kind of AI safety regime that we build today that doesn't take into account that falling cost of compute, it would probably break down. And therefore, amid this broader sort of fragmentation of the machinery of government, the state, I expect more and more government functions to be offloaded into basically private cities, HOAs, gated communities. And likewise with the internet, I expect more and more of our sort of permissioning regime for new AI models and deployment to shift to the infrastructure layer where telecommunication providers will be monitoring network traffic for unvetted AI models and so forth. And we'll have Chinese-style firewalls, but that are specific to a particular local area network. And at that point, the world looks— the United States where this takes place looks more like an archipelago of microjurisdictions of— I tend to think that a post-scarcity political economy looks a lot like the Gulf states, Gulf state monarchies, right? Because Gulf state monarchies are basically living post-scarcity, right? They have a spigot of oil they can turn on and then they can go build mega projects in the desert and they have infinite labor because they can just import guest workers. And so you end up with this But if we can't have a Gulf state monarchy in the United States, instead we have a bunch of micro monarchies dotting the country. So I sort of jokingly say, who's going to stop the free city of California that's home to all the trillionaire ML engineers and tech founders from the decade prior from plugging in their humanity-sized supercomputer into a fusion reactor and turning it on?

Yeah. And this is really your— the kind of endpoint of the discussion or your main point of institutions being eroded and then afterwards being unable to respond to strong AI.

Yeah. And leading up to this, it sounds like a scary dystopian type of thing. It doesn't have to be, right? Uber is not dystopian. Airbnb is not dystopian. private airports in other countries are way better than the public airports in the United States. So privatization and the sort of techno-feudalist paradigm doesn't have to be bad, but what it is is more adversarial. And people have sometimes speculated, would we have the— did the crumbling of the Roman Empire, was that kind of prerequisite to a Renaissance?

Right.

Because it allowed for these principalities to sort of compete and to get the Florentine creativity and so forth. I think the next couple decades could similarly be a renaissance for science and technology and for understanding the world, but it's partly a renaissance because we'll be moving into a much more competitive adversarial world where these city-states and so forth will be hard to coordinate. And so to the extent that there are still these meta risks where we would value some large-scale intra- and international coordination, like peace treaties and so forth, the disintegration of the United States where this revolution is occurring would be bad for that.

You talk about or you hint at an alternative path. What we've been talking about, your timeline here, is the default path. You hint at a path where we have something you call constrained Leviathan. What is constrained Leviathan?

It's limited government, right? So this is Jaron Acemoglu's word for it from The Narrow Corridor. And if you trace the rise of what we associate with liberal democracy, it is part of a particular technological equilibrium. In particular, an equilibrium that favored centralized governments with impersonal rule of law and impersonal tax administration and so on and so forth. So we associate today libertarians with being anti-government, but the basic idea of liberalism is actually associated with strong government, a strong impersonal government that can impose the rule of law. And so if we want to maintain that kind of equilibrium, in a world where AI is diffusing on the society level faster than it is on the state and elite level, then we want to accelerate the diffusion of AI within government. And there's obviously lots of low-hanging fruit. We talked about how bureaucracies are basically fleshy APIs. Even today, I have a friend at the FTC, the Federal Trade Commission. They have a 30-person team that is part of the healthcare division and they're in charge of policing the entire pharmaceutical industry in the United States for competition. His day job right now looks like manually reading through 40,000 emails that they subpoenaed from a pharma CEO. And today you could take those emails and put them into Claude 2 or something like it with a big context window and ask, find me the 5 most egregious examples of of misconduct. And it would do that. It might not be perfect, but it's a hell of a lot more efficient than reading through them manually. And obviously big law is going to be doing that and the pharma CEO and his personal attorneys will be doing that conversely. To maintain our state capacity in the face of AI is to run in this arms race. And you can kind of liken it to what in evolutionary biology they call the Red Queen dynamic. Which comes from Alice in Wonderland where the Red Queen tells Alice that sometimes you need to run just to stay in place. And so I think our government needs to be adopting this technology as rapidly as possible so that they can basically tread water. And that means both diffusing it in existing institutions, but also being open to radical reconfigurations of the machinery of government and addressing some of those firmware-level constraints that we talked about, whether it's the lack of a national identification system or the outdated information technology infrastructure or the accumulation of old procedural kinds of methods of governance.

A focused way of doing this is what you've called for in a Politico article, which is a Manhattan Project for AI safety. First question here, would it be better to call it an Apollo Project as opposed to a Manhattan Project? The Manhattan Project created some pretty dangerous weapons, whereas the Apollo Project might have been more benign.

I mean, what the Apollo Project and the Manhattan Project have in common is that they came from an era of US government where we still built things, where we still had competent state capacity, where we still had a lot of in-house expertise and we weren't saddled with all these constraints. So today we couldn't go to the moon in 10 years. NASA couldn't. SpaceX can. And so our modern Apollo projects are being done by the private sector through competitive contracts. And so one of the messages of my piece on the Manhattan Project is to say, the reason I make this analogy is not just because AI is an Oppenheimer-like technology, but also because responding to it will require a throwback to those kind of institutional forms where we gave the people at the top a lot of discretion And sort of gave them an outcome and let them solve for that outcome without having a bunch of prescriptive rules about how to solve for that outcome. And then the second reason to make the analogy is OpenAI and Anthropic, they both have contingency plans for developing AGI and having runaway market power. In the case of OpenAI, it's their their nonprofit structure. In the case of Anthropic, it's their public benefit trust where they both are envisioning a world where they could potentially be the first to build AGI and become basically trillionaires. And so at that point, they need to become basically governed by a nonprofit board. At that point, and that's not where progress ends, obviously, there's going to be continued research. It would make sense for the US government to step in and say, let's do this as a joint venture. We're no longer competing. In fact, the basic structures of capitalism and market competition are starting to break down. Let's just pull this together into a joint venture, study the things that require huge amounts of capital that the private sector doesn't have, but the government can. The US government spent $26 billion on the Manhattan Project in today's dollars. When you think about the financial resources of nation-state actors to put behind scaling, It's nothing like what Microsoft or Google have. When's our first $200 billion training run, right? What kind of things can come out of that? I think that's something that you want to do with the Defense Department's involvement and working with these companies in a joint way through secure data centers and doing gain-of-function style research that really is dangerous. And more like, you know, more Manhattan Project than Apollo Project.

What would be the advantages here? We would be able to spend more to slow down capabilities research and spend more of our— of the resources on, say, mechanistic interpretability or evaluations or alignment in general, because now the top AI corporations have kind of combined their efforts under one government roof.

Yeah. And in my vision, they were still allowed to pursue their commercial verticals. And I have an extended version of the proposal where I talk about needing biosafety-style categories for high-risk, medium-risk, and low-risk styles of AI that closely parallels what Anthropic recently put out with their recommendations for a BSL categorization of AI research. So I'm really talking about that BSL-4 lab and beyond style stuff. And some of that stuff, some of it will be to accelerate alignment and interpretability research to sort of do versions of the OpenAI Superalignment project where they're dedicating 20% of their compute to study alignment. Another part of it will be to forestall competitive race to the bottom dynamics. So so that they can coordinate and not violate antitrust laws. And then the third thing is sort of the gain-of-function stuff that we really only want to be doing with very, very strict oversight, compartmentalization, kind of pooling of talent and resources so we can share knowledge on alignment and safety. But then also because government has this huge spending power relative to the private sector. Anytime you build a supercomputer, you're basically borrowing from the future. You're trying to see what the smartphones 20 years from now will be capable of. And so if we want to sort of get ahead of the curve and see where scaling is leading, then I think governments are really the only actor that can waste a bunch of money basically scaling up a system and seeing what comes out of it.

Yeah.

When we talk about gain-of-function research in AI, it's an analogy to the gain-of-function research that's done on viruses in biolabs, but done for AI models. And this could be experimenting with creating more agent-like models or inducing deception in a model and planting it in a simulated environment, seeing what it does, or enticing it to acquire more resources. But again, perhaps in a Safely, if this is even possible, in a safely constrained simulated environment. And this is the type of research that we could do in this Manhattan Project, this government lab, because we would have excellent cybersecurity and secure data centers and the combined efforts of the most capable people in AI research.

If you've watched Oppenheimer, the movie, A lot of that revolved around suspicions of communist spies and so on. And we really don't have great insight into the operational security of the major AGI labs. And that's something that bringing it in-house of the Defense Department, they would necessarily have to disclose everything they're doing, but also hopefully beef up their operational security.

Yeah, they're kind of stuck with a startup mindset, but they're not developing a startup They're developing something that, in my opinion, could be more dangerous than the average startup.

Yeah. And Dario Amodei has said as much, that we should just assume that there are Chinese spies at all the major AI companies and at Microsoft and Google.

When we think about gain-of-function research in AI, how do you think about the value of gaining information about what the models can do and what the models can't do versus the risk we're running? tragic and ironic death for humanity if we experimented with dangerous AI models to see whether they would destroy us, and then we hadn't constrained them properly and they actually destroyed us. So how do you, how do you think of that trade-off between gaining information and avoiding lab leaks?

Yeah, hopefully lab leaks are less likely than in the biology context where Getting a little bit of blood or urine on your shoes as you walk out the door. It's a difficult thing to talk about in part because we just went through a pandemic that very probably was caused by a BSL-4 lab leak. And so one saving grace is that AI models don't get caught in your respiratory system. And so hopefully there's forms of compartmentalization that are much more robust. than in the biology context. And to the extent that this research is going to be done anyway, it'd be much better to move it offsite and hopefully in facilities that are air-gapped and so forth rather than what Microsoft is doing right now. Microsoft just recently announced their AutoGen AI, which are sort of agent-based models, very similar to AutoGPT, but that work. And they're doing this through a Creative Commons, totally open source framework. All these capabilities work as gain-of-function research. Where we draw the line between doing things that are intentionally dangerous or doing things that are dangerous, but we're kind of pretending that they're not, is hard. I do think there's— and Paul Christiano has also agreed with this— sort of threat models that would be valuable to be running in virtual machines to see if the AI develops awareness, situational awareness, and tries to escape, but it escapes into a simulated world that we built for it.

Okay, let's end by talking about a recent critique of expecting AGI to arrive in a short time. This revolves around interest rates. The basic argument is, or the basic question is, if AGI is imminent, why are real interest rates low? I can explain it, but you're the economist, so maybe you can explain the reasoning here.

So it's really a question of how efficient are markets and how much foresight do markets have. You know, we're coming out of a world of very low interest rates, of ultra-low interest rates, near-zero interest rates. And one way to think about that is there's a surplus of savings relative to investment. And so one of the reasons interest rates have been in secular decline is because populations are aging. And so old people have a huge amount of savings built up. And meanwhile, we're going through this sort of technological stagnation. So the amount of savings relative to the amount of profitable investments was out of whack. And so that pushes interest rates down. In a world where AI takes off, it's a world where we have enormous investment opportunities, where we'll be building data centers left and right and not can't do it fast enough, where there's new products, new commercial opportunities left and right. And so you would expect in that world where the singularity is near, so to speak, to be one where the markets begin forecasting rapidly rising interest rates because the savings-to-investment balance is starting to shift. And in addition, there's a long-run stylized fact that interest rates, real interest rates, track growth rates. And so if GDP growth GDP growth takes off, you'd also expect at least nominal rates to also take off. And so some have argued that looking at current interest rate data, like the 5-year, 10-year, 30-year Treasury bonds, that the markets are not predicting AGI. The 2 responses to that are, one, first of all, interest rates are up quite a bit. Nothing's monocausal. There's lots of confounding factors. Is this to some extent the market's anticipating an investment boom? Maybe they're not anticipating full AGI, but they're seeing the way LLMs are going to impact enterprise and sort of baking some of that in. And then the second piece would be, okay, to the extent that they're not pricing in AGI, how much foresight do markets have anyway?

Before we discuss market efficiency, I just want to Just give a couple of intuitions here. If AGI was imminent and it was unaligned, say, and it would destroy the world in 5 years, well, then it doesn't make a lot of sense to save money. Similarly, if AGI is about to explode growth rates, well, then a lot of money will be available in the future. You're about to become very rich, so it doesn't make sense to save a lot now. And the pool of available savings determine what's available for lending, which determines interest rates. But let's discuss whether markets then are efficient on this issue or to what extent they're efficient.

Right. So this is the efficient market hypothesis, which comes in strong and weak forms. So the strong form of the efficient market hypothesis would say that markets aggregate all of the available information and are our best sort of point estimate of anything we care about. The weaker form, which I think is more defensible, is that markets can be wrong, but they can be wrong longer than you can be solvent. And so you can try to short a company that— like Herbalife, famously there was a big short position on that because Herbalife sort of looks like it's a multi-level marketing Ponzi scheme, but yet the hedge fund that did that lost several billions of dollars before they ended their position because the markets stayed irrational longer than they could stay solvent. The second factor is the weaker versions of the efficient market hypothesis are sort of based on a no arbitrage condition. They say markets are efficient only insofar as you can arbitrage an inefficiency. And so you look at some prediction markets, for example, they predict it. They'll often have very clear inconsistencies across markets that look like they're irrational, but then you realize, oh, I can only make like $7,000 total on the website and there are transaction fees and I have to— there's work involved. And so if the market isn't very deep or liquid, there may be inefficiencies that exist not because the market's inefficient, but it's as efficient as it can be under the circumstances. And when it comes to AI, how do you arbitrage? I've been thinking for a while now that Shutterstock, their market cap should be collapsing because we have image generation that is proliferating. And yes, people will make the argument though, Shutterstock has all this image data, they could build a better image model. Maybe. It seems like it's cannibalizing their business. It's sort of turning a moat into a commodity. And yet Shutterstock's market cap has basically held constant throughout this recent rebirth of image— birth of image generation models.

What if you borrow a lot of money cheaply and then put it into an index of semiconductor stocks or just IT companies in general, even just the general S&P 500, say? Would that be a way of arbitraging this AGI forecast?

Yeah, I would say if you have short timelines, you should be putting a lot of money into equities.

This is not financial advice, I should say. Right.

And I mentioned earlier that Paul Christiano has said in interviews that he's twice levered into the stock market. He basically owns a bunch of AI-exposed companies and he's borrowed enough money to double his investments. So that's putting your money where your mouth is. When you look at market behavior over the long stretch of time, markets didn't anticipate the internet very well. There was a short-run bubble that led to a boom and bust of dot-com stocks, but in terms of the real economy, the internet just kept chugging along. It kept being built out, and eventually a lot of those investments ended up paying off, even if you rode through the bubble. Markets are made up of people. Some of the biggest capital holders in the markets are institutional investors, pension funds, life insurance companies, governments like Saudi Arabia or the Norwegian Pension Fund. And often these are making safe bets. They're not taking very heterodox views on markets. And so as a result, markets can be a little bit autoregressive. They're a little bit biased to the past. Past is prologue and prone to kind of multiple equilibria where there's 2 prices that Shutterstock can be. Shutterstock could be a $50 stock or it could be a $0 stock. And at some point the market will update and will wander, go through the great repricing and all these asset prices will flip. In relatively short order. The efficient market hypothesis has to be false or else we wouldn't have Silicon Valley. We wouldn't have founders that— we wouldn't have Elon Musk. So I would just say the markets are wrong. And partly they're wrong because to be right would require having a bunch of relatively bespoke and kind of esoteric priors about the direction of the technology.

Yeah.

That are only now just sort of percolating into the mainstream.

Yeah. And that the big kind of capital allocators can't really respond to because they're risk-averse.

Exactly. Now, that doesn't mean Renaissance Technologies won't respond to it, but they're not going to move the market.

Samuel, thanks for this conversation, and I've learned a lot.

Thank you.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co Or you can DM me on the social media platform of your choice.