Best of: AGI | Dario Amodei / Balaji Srinivasan / Marc Andreessen / Samo Burja / Michael Levin

Hey, Upstream listeners. Today on Upstream, we're releasing a compilation of some of the best discussions we've had across the Turpentine network on AGI, scaling laws, AI safety, and the forces at play in AI development. Up first is Dario Amodei, followed by Balaji Srinivasan, Marc Andreessen, Michael Levin, and Samo Burja. Please enjoy the episode.

Maybe let's start there, Dario, because you have an interesting intellectual evolution. Why don't you briefly describe your intellectual evolution and why you're on an econ podcast and why it's fitting?

Yes, let's see. So I don't know if I go all the way back, like, you know, I think early before I got involved with AI, I was just trying to figure out what would be most interesting and impactful to work on. So I did my undergrad in physics. I believe Noah's the same, actually. And then considered a bunch of things. I actually considered going to grad school for econ, decided not to do that. Went to grad school for computational neuroscience and biophysics. I was very interested. I considered AI as well, but at that time, I just didn't believe that the AI of that day was all that interesting or had made all that much progress on important problems in intelligence. And so I figured I'd— Study that, you know, the one object in the universe we know of that is at least sometimes intelligent, the human brain, you know. And so worked on that, worked on basically networks of biological neurons and trying to understand what was going on in them, which turned out to be a very difficult task because, you know, just the biological access and the physical measurement, let alone even getting to the analysis, just turns out to be a very, very difficult task. And Spent most of my time on that and very little of it making tiny bits of progress on trying to understand, you know, the actual algorithmics of what is going on inside the human brain. And then around, you know, I did a postdoc at Stanford, worked on proteomics, and then around 2014 or so, you know, the deep learning revolution was starting. I saw the work on AlexNet and QuocNet, and so decided to get involved in that field. I worked with Andrew Ng at Baidu for a year, and then I was at Google for a year. Then I joined OpenAI very shortly after it started, was there for about, was there for about 5 years, developed a lot of the ideas around scaling laws, helped build some of the early GPTs, as well as inventing the RL from human feedback method along with others that, you know, would eventually be used to produce some of these, these commercial chatbots. And then left around the end of 2020 to, to found Anthropic and, you know, have been running Anthropic ever since.

Here's a question in general, you know, this is an econ podcast. We like to talk about a little more of the econ side than the, than the pure tech side. But yeah, so is Google the Bell Labs of the artificial intelligence age? Because they, you know, they produced the research that led to modern deep learning and transformers, et cetera. And they didn't really manage to commercialize it very well, kind of like Bell Labs, and they funded it from their, you know, IT monopoly, kind of like Bell Labs. And then, you know, a bunch of interesting people, like, you know, worked there and then sort of went off and started companies, you know, like the eventually the Fairchild people did from Bell Labs. Do you think this is an apt analogy?

Yeah, I mean, while nothing is ever a perfect analogy, I certainly think there's something to it, right? When, when, when, when I was at Google, I mean, you know, for, for many people there, They thought of it as a kind of continuation of their academic careers, which was you know the same the same as the same as you know the same as Bell Labs—an industrial you know an industrial environment that that has many things in common with an academic environment except that it has kind of more more resources to make things happen. And so you know people working on a lot of things, right? The transformer, which is kind of one of the key inventions that you know that drove the field. You know that was one of maybe a hundred things that that were being. that were being worked on. And if you went high enough up, if you went high enough up in the organization, you know, there was no reasonable ability to distinguish between that and the 99 other things that were being invented. It was like, it was let a thousand flowers bloom. And, you know, it was, I think during that time that I first started having ideas around things like the scaling hypothesis that, hey, we needed to take some of these innovations, we needed to really scale them up and we needed to put them together. And in theory, Google was the best place to do that. They have like the biggest clusters in the world. They have, you know, a lot of kind of talented engineering staff. You know, they kind of had all the ingredients, but, but the Google machine was organized in a certain way. It was organized to do search. I don't think it was necessarily organized to kind of put all these pieces together and scale up something radically different from, from what, from what had been done before, from what the business was about.

Right, like Bell Labs wasn't set up to, you know, invent computers and give everyone a computer. It was set up to like wire everyone up.

Yeah, exactly. It was a telephone company. And so, you know, I mean, you know, I can't speak to Google, obviously. Obviously now they've, you know, in addition to inventing all of this, these, these, you know, these, you know, in addition to inventing all these amazing things, you know, they have, they're one of the 4 companies that has, you know, that has the frontier models. They're, you know, both a part partner to us and a competitor to us. I know many people there, they're super, super smart. But, but I think you're right that there was a period where if they had had kind of, if they had been able to put together the ingredients in the right way, they could have been the only, they could have been the only game in town. And, you know, for whatever reason it didn't, it didn't go that way.

Well, that's, that segues into something else that we were thinking about, you know, this, the idea of talking to you actually came from another podcast we were doing. Where I was mostly talking about the economics of like internet businesses. And then Erik came out with some, some kind of pessimism on the business of AI and, and, you know, was like, you know, questioning whether, you know, how much, how much of a moat AI companies have. Obviously this is incredibly relevant to Anthropic, you know, and, and other companies that we call startups, but they're already pretty big. And yeah, so, so tell us how you think about the, the business moat of AI companies.

Yeah. So I will say I'm going to separate this a little bit into 2 branches. I will say it's a little hard to separate some ideas and questions around the scaling hypothesis from the business questions. So like, let's consider the branch where kind of the scaling hypothesis is true in some very strong form. And then let's consider the version where it, you know, might be somewhat true or not true at all. In the version where it's true in some very strong form, you know, I've Probably laid this out elsewhere in public, but this is the version where you train a billion, you've trained a $100 million model now, and it's about as good as a good college freshman or something, and you train a billion, it's as good as an advanced undergrad, you train a $10 billion model, and it's as good as a top graduate student, and then when you get to your $100 billion model, it's as good as a Nobel Prize winner. And then you just take that model and you put it to work for basically everyone, right? It becomes your coworker, it becomes your personal assistant, it helps with national security, it helps with biology. And I think in that world, like that system and the products built on that system just become a very large fraction of the economy. There's still a question of like, well, where do the returns go, right? Do they go to NVIDIA on one side? Do they go to the AI companies on the other side? Do they go to downstream applications? And there, the pie is so big that to first order, my answer is yes, they go to all of those places.

Right. But, but, but okay, think about solar power. You know, solar is obviously going to be absolutely immense, you know, and the more energy we need, the more solar there will be. And yet, it's hard for me to name you a solar company that makes a bunch of profit. It is a very commoditized product, even though there's plenty of innovation involved. But yet there's no branding, there's no network effects, there's no, you know, lock-ins of anything. It's very hard for any solar company to make a profit on this thing that is radically transforming our entire world in front of our eyes right now. And so, so I'm not 100% sure that, you know, just the fact that like everything will be booming as it is in solar right now, you know, will lead necessarily to profits being captured by, by companies, but I'm certainly open to the idea that it will be. I just, you know, what, what do you think will be the source? Why will AI turn out different than solar?

Yeah. Yeah. So I think, I think 2 points here and I'll get to what you're saying, 'cause I think it's a, I think it's an important question in most worlds. Maybe I'm just making the point that this is like, if, if the scaling hypothesis is correct, this is going to be like really, really big. Right. And, and so big that even if only 10% of the profits go to this part of the supply chain or that part of the supply chain, it's still really, really big. You make the pie big enough, and that becomes the most interesting question. Although, I'm sure that the people who are deciding how the dollar bills get split up, it's going to be really important to them whether $1 trillion goes here and $10 trillion goes there, or $10 trillion goes here and $1 trillion goes there. But let's actually get to your question, because I think It matters in all the worlds. It's just a matter of what is the size of the pie that you're distributing. First of all, I think on the model side, and again, this is dependent on the scaling hypothesis, if we're building $10 or $100 billion models, there is not going to be probably more than 4 or 5 entities, maybe a couple state-run players will get involved as well. who build these. So, we're looking at something that looks probably more oligopolistic than it does monopolistic or fully commoditized. I suppose there's a question of, you know, will someone release a, you know, $10 billion or $100 billion open source model? I'm somewhat skeptical that, you know, the conviction to do that goes that far. And I think actually, even if such a model is released, One thing that's a disanalogy to open-source software is that these big models, they're actually very expensive to run on inference. The majority of the cost is inference, not necessarily the training of the model. So if you have only, I don't know, 10%, 20%, 30% better way to do inference, that can kind of negate the effect. So the economics are kind of strange. Yes, there's this giant fixed cost that you have to amortize, but then there's also the per unit cost of inference and small differences in that can actually, again, assuming the thing is deployed widely enough, make a very big difference. So I don't know quite how that's going to play out.

So that's actually similar to the economics of heavy industry. Like that's kind of how making steel works.

A little bit. Yeah. A little bit. Yeah.

Interesting.

I would, the other thing I would say is, you know, within these few models, something we're already starting to see is that like models have different personalities. And so, commoditization is one possibility. And I do see some worlds where even within the oligopoly, some particular ways of deploying models could get commoditized, although I'm not sure. But one force pushing against that is the idea that, hey, I make a model that's really good at coding. You make a model that's like really good at creative writing. This third person makes a model that's really good at being kind of like engaging and entertaining. These are kind of choices. And once you start to make these choices, you start to build infrastructure around them. And so, to me, it seems like it creates the preconditions for some amount of differentiation. The other thing that I think might lead to differentiation is just the products built on top of it, right? In theory, you can separate the model layer from the product layer. In practice, they're somewhat linked to each other, and it can be somewhat challenging to work across organizations. So, You know, while the models, there's kind of this, I don't want to say single path, but like, there's like a logic to building models. Many of the companies are going in the same directions with the multimodality they're adding onto it, making the models smarter, making the inference faster. There's kind of, in some ways, one direction, but products are so different, right? If you look at this like artifacts thing we made, which is a way to kind of visualize in real time the code that the model writes, right? We do that. OpenAI does their own thing. Google does their own thing. I think that is also kind of the source of some differentiation between companies. And I think we've already found that the economics of selling apps on top of the model, even if they're relatively thin apps and they're starting to become thicker and thicker, is quite different from the economics of just kind of like Handing over the model via an API.

We'll continue our interview in a moment after a word from our sponsors. Hey listeners, it's Erik. If you're looking for more expert-level conversations, but specifically about top negotiation strategies, then I have the perfect podcast recommendation for you. DealCraft: Insights from Great Negotiators, hosted by Jim Sebanias, Harvard Business School professor and renowned negotiation expert. DealCraft features interviews with the world's greatest dealmakers and diplomats. From former Secretary of State Henry Kissinger to Blackstone's Steve Schwarzman on their most challenging negotiations, from which Jim draws lessons for listeners. Not to mention that The Guardian recently called DealCraft a best podcast pick. New episodes release weekly, and I'd recommend starting with their first episode, which reveals how legendary entertainment attorney John Branca was able to successfully negotiate for Michael Jackson's ownership of his master recordings and the Beatles catalog. Almost unprecedented feats at the time, so don't miss out. Follow DealCraft: Insights from Great Negotiators on Apple Podcasts, Spotify, or wherever you get your podcasts. Hey everyone, Erik here. You know, at Turpentine, we're always looking for ways to streamline our work. That's why we use Notion for all our internal documents. It's been a game changer for us, and I want to tell you why. Notion is this amazing all-in-one workspace that combines your notes, docs, and projects. But what's really cool is their new Notion AI. It's like having multiple AI tools built right in. You can search, generate, analyze, and chat, all without leaving Notion. What sets Notion AI apart is how it personalizes responses based on your content. It's not just another generic chatbot; it understands the context of your work. Plus, it can search across thousands of docs in seconds, even connecting to tools like Slack and Google Docs. And get this— Notion is used by over half of the Fortune 500 companies. Teams using it send less emails, cancel more meetings, and save time searching for their work. It's designed to keep everyone on the same page while protecting your privacy. Wanna give it a try? Head over to notion.com/upstreampod. That's all lowercase. Notion.com/upstreampod. You can try Notion for free, and when you use our link, you're supporting our show. Give the powerful, easy-to-use Notion AI a shot today at notion.com/upstreampod.

I'll kind of float a couple of things that I think may be bucking the trend, but I think it's maybe useful to kind of try to separate this into scenarios because all the analysis that you're describing here, if I understand it correctly, it seems to have the implicit assumption that the AI itself is not going to get super powerful or hard to control. It's like, if we assume that it's kind of a normal technology, then you're off to the races on this analysis, and then we can get into the fine points. But I do want to take at least one moment and say, how confident are you on that? Because if it's a totally different kind of technology from other technologies that we've seen, if it's more— you raised the gain of function research example. If it's that sort of technology that has these sort of nonlocal possible impacts or self-reinforcing kind of dynamics, which need not be like an Eliezer-style snap of the fingers foom. But even over, say, a decade, let's imagine that over the next 10 years that AI's kind of multiple architectures develop and they sort of get integrated and we have something that kind of looks like robust silicon-based intelligence, maybe not totally robust, but as robust or more robust than us. And running faster and the kind of thing that can do lots of full jobs or maybe even be tech CEOs, then it kind of feels like a lot of this analysis probably doesn't hold, right? Because we're just in a totally different regime that is just extremely hard to predict. And I guess I wonder, first of all, do you agree with that? There seems to be a big fork in the road there that's like, Just how fast do these AIs become super powerful or do they not? And if they don't, then yeah, I think we're much more into realpolitik type of analysis. But I'm not at all confident in that. To me, it feels like there's a very real chance that AI of 10 years from now is— and by the way, this is like what the leaders are saying, right? I mean, OpenAI is saying this, Anthropic is saying this, Demis and Shane Legg are certainly saying things like this. It seems like they expect that we will have AIs that are more powerful than any individual human, and that becomes the bigger question than anything else. So do you agree with that kind of division of scenarios, first of all? And then maybe you could kind of say how likely you think each one is. And obviously that one where it takes off is super hard to analyse. And I also definitely think it is worth analysing the scenario where it doesn't take off. But I just wanted to flag that it seems like there's a big— if you talk to the AI safety people, any world in which it's like we're suing Indian AI firms in Indian court over IP is like a normal world in their mind, right? And that's not the kind of world that they're most worried about.

I think that there have been some plausible sounding things that have been said, but I want to just kind of talk about a few technical counterarguments, mathematical or physical, that constrain what is possible. Okay. And actually, Martin Casado and Vijay and I are working on a long thing on this where, you know, Vijay did Folding@Home. He's a physicist. Martin sold Nasira for, you know, a billion dollars and knows a lot about how a Stuxnet-like thing could work at the systems level. And I've thought about it from other angles and, you know, um, And some of the math stuff, which I'll get to. So, for example, one thing, and I'm going to give a bunch of different technical arguments and then let's kind of combine them. Okay. One thing that's been talked about is if you have a superintelligence, it can deliberate for a million years and then it can make one move and it's going to outthink you all the time and so on and so forth. Okay. Well, if you're familiar with the math of chaos or the math of turbulence, There are limits to even very simple systems that you can set up where they can become very unpredictable quite quickly. Okay. And so you can, if you want to engineer a system where you have very rapid divergence of predictability so that, I don't know, it's like the heat death of the universe before you can predict out N timestamps. Do you understand what I'm saying? Right.

This is sort of akin to like a Wolfram, like simple, you Even simple rules can generate patterns such that you can't know them without literally computing them.

Yeah, exactly. At least right now, with chaos and turbulence, you can get things that are extremely provably difficult to forecast without actually doing it. I can make that argument quantitative, but that's just something to look at. It's almost like a delta-epsilon challenge from calculus. How hard do you want me to make this to predict? Okay, I can set up a problem that is That is like that, right? It's basically extreme sensitivity to initial conditions lead to extreme divergence in outcomes. So you could design systems to be chaotic that might be AI immune because they can't be forecasted that well. You have to kind of react to them in real time. The ultimate version of this is not even a chaotic system. It's a cryptographic system where I've got a whole slide deck on this, how AI makes everything fake, easy to fake. Crypto makes it hard to fake. Hard to fake again, right? Because crypto, in the broader sense of cryptography, but also in the narrower sense, I think crypto is to cryptography as the internet is to computer science. It's like the primary place where all this stuff is applied, but obviously it's not the equivalent. Okay. An AI can fake an image, but it can't fake a digital signature unless it can break certain math, you know, and so it's sort of like a, you know, solve factorization problem or something like that. So cryptography is another mathematical thing that constrains AI, similar to chaos and turbulence. It constrains how much an AI can infer things. You can't statistically infer it. Okay. You need to actually have the private key to solve that equation. So that is another math. So I'm going to rules of math, right? Math is very powerful because you can make proofs that will work no matter what devices we come up with.

Okay.

You start to put an AI in a cage. It can't predict beyond a certain amount because of chaos and turbulence math. It cannot solve certain equations unless it has a private key is because of what we know about cryptography math. Okay. Again, if somebody proves P equals NP, some of this stuff breaks down, but this is within the bounds of our mathematical knowledge right now. Physics-wise, physical friction exists. A lot of physical friction exists. And A huge amount of the writing on AI assumes by guys like Eliezer, who I like, I don't dislike it, you know, but it is extremely, there's 2 things that really stick out to me about it. First is extremely theoretical and not empirical. And second, extremely Abrahamic rather than Dharmic or Sinic. Okay. Why theoretical and not empirical? It's not trivial. To turn something from the computer into a real-world thing. Okay, one of the biggest gaps in all of this thinking is what are the sensors and actuators? Okay, because like if you actually build, you know, I've built industrial robot systems that you know ten years ago, I you know a genome sequencing lab with robots. That's hard. That's physical friction. Okay, and a lot of the AI scenarios. Seem to basically say, oh, it's going to be a self-programming Stuxnet that's going to escape and live off the land and hypnotize people into doing things. Okay, now each of those is actually really, really difficult steps. First is self-programming Stuxnet. Like, this would have to be a computer virus that can live on any device despite the fact that Apple or Google can push a software update to a billion devices, right? A few executives coordinating almost certainly can. I mean, the off switch exists, right? Like, this is actually like the core thing. Lots of AI safety guys get themselves into the mind state that the off switch doesn't exist. But guess what? There's almost nothing living that we haven't been able to kill, right? Like, can we kill it? This thing exists. And this is getting back to living off the land. Even if you had like something that could solve some of the other technical problems that I'll get to, It exists as an electromagnetic wave kind of thing on on a certain you know on chips and so on and so forth. It's taking it out in the environment is like putting a really smart human into outer space, right? Your body just explodes and you die. Doesn't matter how smart you are. That that strength on this axis, but you're weak on this axis, and you know, so strength on the x-axis, not strength on the y or the z-axis. An AI outside, you know, pour water on it, you know. This is why I mean the 50 IQ, 150 IQ thing. You know, the 150 IQ way of saying it is it's strong on this X and weak on this X. And the 50 IQ way is pour water on it, disconnect it, you know, turn the power off. Okay. Right. Like it'll, it'll be very difficult to build a system where you literally cannot turn it off. The closest thing we have to that is actually not Stuxnet. It's Bitcoin. And Bitcoin only exists because Millions of humans keep it going. So you, you need— so that gets the second point, living off the land. For an AI to live off land, meaning without human cooperation, okay, that's the next Turing threshold. An AI to live without human cooperation, it would need to be able to control robots sufficient to dig ore out of the ground, set up data centers and generators and connect them. And defend that against human attack. Literally a Terminator scenario. Okay. That's a big leap in terms of, I mean, is it completely impossible? I can't say it's completely impossible, but it's not happening tomorrow. No matter what your AI timelines are, you would need to have like a billion or hundreds of millions of internet-connected autonomous robots that this Stuxnet AI could hijack that were sufficient to carve ore out of the earth and, you know, set up data centers and make the AI duplicate. We're not there. That's a huge amount of physical friction. That's AI operating without a human to make itself propagate, right? A human doesn't need the cooperation of a lizard to self-replicate. For an AI to replicate right now, it would need the cooperation of a human in some sense, because otherwise those humans can kill it because there's not that many different pieces of operating systems around the world. I'm just talking about the practical constraints of our current world, right? You know, actually existing reality, not AI safety guys, you know, you know, reality where all these things don't exist. There's just a few operating systems, just a few countries. If everybody's going with torches and searchlights through the internet, it's very hard for a virus to continue. Okay. So A, uh, on the practical difficulties, there's the technical stuff with, with, you know, with, uh, with chaos and turbulence and with cryptography itself where AI can't predict. and it can't solve certain equations. B, on the physical difficulties, it probably— I mean, like, to be a Stuxnet, Microsoft and Google and so on could kill it. The off switch exists. Can it live off the land? No, it cannot, because it doesn't have, you know, drones to mine ore and stuff out of the ground. And, um, can it, like, exist without humans? Can it be this hypnotizing thing? Okay, so the hypnotizing thing, by the way, this is one of the things that's the most hilarious self-fulfilling prophecy in my view. Okay, and no offense to anybody listening to this podcast, but I think the absolutely dumbest kind of tweet that I've seen on AI is, I typed this in and oh my God, it told me this. Like, I asked it how to make sarin gas and it told me X or whatever, right? That's just a search engine. Okay, what, what basically a lot of these people are doing is they're saying, what if there were people out there that were so impressionable That they would type things into an AI and, and follow it as if they were hearing voices. And that's actually not the, the, the model or whatever that's doing it. That's like this AI cult that has evolved around the world, like, uh, Aum Shinrikyo, you know, that, that hears voices and does like the sarin gas. The point is, an AI can't just like hypnotize people. Those people have to like participate in it. They're typing things into the machine or whatever. Okay. Now you might say, all right, let's project out a few years. In a few years, what you have is you have an AI that is not just text, but it appears as Jesus. What would, what would AI Jesus do? What would AI Lee Kuan Yew do? What would AI George Washington do? So it appears as 3D. Okay. So it's generating that. It speaks in your language and in a voice. It knows the history of your whole culture. Okay. That would be very convincing. Absolutely be very convincing. But it still can't exist without human programmers who are like the priests tending this AI god, whether it's AI Jesus or AI Lee Kuan Yew or something like that. The thing about the hypnotization thing that I really want to poke on that, are you familiar with the concept of the principal-agent problem? Basically, in every, every time you've got like a CEO and a, and a, a worker, or you have an LP and a VC, or you have, um, You know, an employer and a contractor, every edge there, there are 4 possibilities in a 2 by 2 matrix: win-win, win-lose, lose-win, lose-lose. Okay. And, uh, so for example, win-win is, you know, when somebody joins a tech startup, the CEO makes a lot of money and so does the worker. Okay. That's win-win.

Lose-lose is they both lose money.

Win-lose is the CEO makes money and the employee doesn't. Lose-win is the company fails, but the employee got paid a very high salary. So what equity does is it aligns people. That's where the concept of alignment comes from. It aligns people to the upper left corner of win-win. That's when you have one CEO and one employee. When you have one CEO and 2 employees, you don't have 2 squared outcomes. You have 2 cubed outcomes because you have win-win-win, win-win-lose, win-lose-lose, et cetera, right? Because All 3 people can be win or lose. Okay. CEO can be win or lose. Employee can be win or lose. Employee number 2 can be win or lose. If you have N people rather than 3 people, you have 2 to the N possible outcomes and you have essentially a 2 by 2 by 2 by 2 by 2 by N hypercube of possibilities. Okay. It's all literally just 2 dimensions on each axis. There's tons of possible defecting kinds of things that happen there. So that's why in a large company, there's lose-win coalitions that happen where M people gang up on the other K people and they win while the other people lose. That's how politics happens.

We'll continue our interview in a moment after a word from our sponsors. Hey everyone, Erik here. In this environment, founders need to become profitable faster and do more with smaller teams, especially when it comes to engineering. That's why Sean Lanahan started Squad, a specialized global talent firm for top engineers that will seamlessly integrate with your org. Squad offers rigorously vetted top 1% talent that will actually work hard for you every day. Their engineers work in your time zone, follow your processes, and use your tools. Squad has frontend engineers excelling in TypeScript, React, and Next.js, ready to onboard to your team today. For backend, Squad engineers are experts at Node.js, Python, Java, and a range of other languages and frameworks. While it may cost more than the freelancer on Upwork billing you for 40 hours but working only 2, Squad offers premium quality at a fraction of the typical cost. Without the headache of assessing for skills and culture fit. Squad takes care of sourcing, legal compliance, and local HR for global talent. Increase your velocity without amping up burn. Visit choosquad.com and mention Turpentine to skip the waitlist.

One critique of the great AI piece you wrote, which could, you know, could apply to some of the conversations we're having here, is some people were saying, hey, deal with the arguments, don't impugn motivations on people. And then separately, we've also in our chats previously talked about theory of mind and reading people's motivations. How do you respond to that critique or how do you make sense of that? Because I know that's been an intellectual interest of yours.

Yeah, so there's different kinds of arguments as it turns out. So start by defining terms. What is an argument? There's an argument in the form of science. The scientific method proposes a way to be able to debate things. Right. And this is sort of empirical argument, right? And so the way that just, you know, the way the scientific method does empirical argument is you generate a hypothesis for how you think the world works. And then the hypothesis must be testable, right? There must be a way to actually like do a, do an empirical real-world test of that hypothesis. And then there must be a way, as you're setting this up, there must be a way for the hypothesis to be what they call falsifiable, right? And so the idea is like to be a valid scientific hypothesis, there has to be a way to disconfirm it.

Right.

There has to be a way to potentially prove that it's not true. And then you run real-world experiments and that hypothesis either validates out or it doesn't. And so that's empirical argument. The problem with human affairs is that if we're debating things that involve chemistry, that method works really well. For a lot of social, political, let's say broadly intellectual arguments, There's nothing like that exists, right? There's no testable hypothesis. There's no falsifiability. There's no empiricism. There's no way to run a test. There's no way to run the scientific method. And it just so happens AI safety is one of those categories. And Tyler Cowen has been pointing this out recently. He read all the literature on AI safety and it's like, okay, there's no peer-reviewed research. There's no model. There's no test. There's no testable hypothesis. There's no empirical anything. There is not an instance of this bad thing happening that's ever happened. It's not an empirical— the AI safety arguments are not framed empirically. They're not framed scientifically. And so they're basically argued intellectually. And so when they're argued intellectually, now you're in the realm of intellectual arguments. And we have hundreds of years of history basically understanding what happens with intellectual arguments. And what happens with intellectual arguments, my interpretation, Uh, uh, is, uh, a lot of other people's is basically intellectual arguments. The problem with them is they, they, they, they feel like they're rational and they feel like they're logical, but at the end of the day, they're mainly just a lot of words. Um, and then you get this kind of lawyer effect where the person who's better with words kind of wins because their words are better. Um, there, there's this great book. Um, there's a bunch of great books on this that I recommend. Thomas Sowell wrote one of the definitive books on this called Intellectuals in Society. Um, you know, where he, he kind of takes his head on and basically says, you know, why are intellectuals so often wrong on like big social topics? Um, uh, you know, and the reason is because they, you know, they, they, they're the best at using words. And so they, they talk themselves and everybody else into believing things that aren't true. Um, there's this other great book that just came out called When Reason Goes on Holiday that specifically, uh, talks about what happens when philosophers enter politics, which is sort of a category of this. Um, and, you know, and the results are catastrophic.

Um, Right.

Um, and so you're, you're, you know, in internet meme land, you know, what you're, what you're talking about here is basically the difference between, you know, word cells and shape rotators. Um, and so I, I do think it's very important to understand when you're having a scientific argument with shape rotators and when you're having an intellectual argument with word cells. And if you are having an intellectual argument with word cells, I think it's like totally necessary to actually figure out the dynamic of the argument that you're having precisely because it's not scientific. And that, and that gets straight to motives.

In your podcast with, with Sam Harris, He was trying to push back a little bit and saying, hey, you're saying this argument is unscientific. Well, he could say that yours is unscientific in the sense that you can't falsify the opposite. And it feels like the question is, where is the burden of proof need to be? But I think you ended that conversation by saying, hey, we're both tautological. One of us is being tautological, or we're both being tautological, and one is wrong. I'm curious how you assess that or reflect on that conversation in terms of like, where are the differences between you and Sam, who agree on so many other things and are both You know, very intelligent. How do you reflect on that?

Yeah. So look, the scientific method as understood by the scientific community for the last 300 years is very clear on this, which is basically the person proposing the theory is responsible for testing the theory, right? If I propose that something is going to happen and if I have no test to confirm that, I am not making a scientific argument and it's not up to anybody else to disconfirm, right, the thing that I myself have not, you know, proposed in an empirical form. Carl Sagan, it turns out Carl Sagan, who was an empiricist, right, and one of our top scientists of the 20th century, he, you know, he talked about this specifically. And, you know, his phrasing on it, which I think is really precise, is that extraordinary claims in the world of science, extraordinary claims require extraordinary evidence. Right. The bigger the leap that you're asking people to make, the more evidence you have to have. Right. That, that, that, that is actually the thing that's going to happen. And so my, my, my view is my— in my argument with Sam and my argument on this topic, my view is clearly I am being the scientific one. one and he is being the non-scientific one. Um, and, and, and again, it's just sort of core scientific method, which is he is proposing that a thing is going to happen that has never happened before, which is an extraordinary claim that requires extraordinary evidence. Um, I, on the other hand, am not proposing that anything happens that hasn't happened before, right? I am not making a prediction on a thing that's going to happen that's never happened before. I am not making an extraordinary claim. In fact, I'm making, scientifically, I'm making no claim at all. I'm not even putting forward it. I'm not even putting forward a hypothesis because I'm, I'm, I'm putting forward the, I'm putting forward what they call the null hypothesis that basically that You know, that doesn't happen. And so anyway, so any scientist, if you take the topics out of this, any scientist would say the one is scientific, the other one isn't. But again, this gets into the thing when you flip it around and you argue it the way Sam was arguing. And I think here, you know, unfortunately, you, you know, you enter word salad territory is what you do is you basically spin— what you do is you basically spin increasingly elaborate word arguments. Right. And so you extrapolate and you'll notice basically that's what Sam did in the discussion, which is every time I tried to talk to him about a specific thing and said, well, that is unlikely or that seems uncertain or there's no way to test that. He would add another, well then on top of that, if, if, if, right? And so he sets up this kind of chain of hypotheticals. None of it has any numbers attached to it. None of it has any experiments attached to it. None of it has anything empirical attached to it. We're venture capitalists. It's science fiction. What?

Right. As venture capitalists, aren't we always extrapolating or predicting things that could emerge based on new technologies?

You may have noticed that we're wrong a lot.

Yes, most of the time. Right? Yeah, like we're wrong most of the time.

Most things that we would like to have happen don't happen. I mean, I wish they did.

Okay, so maybe segueing from science to religion, you write about in the piece how the AI safety community has some religious dynamics. There's also the effective accelerationist community, i.e., EA community, that has some religious components to it. Why don't you sort of give us the lay of the land here, or how do you make sense of that?

Yeah, so maybe to give you a little bit of theory here. So there's this guy, Erik Vogelin, in the 20th century who was a political philosopher who is the sort of clearest thinker on this topic that I've been able to find. He made famous in the mid-20th century, he made famous this term called immanentizing the eschaton. Imminent. Immanentizing the eschaton. It's a phrase you can look up. It was a popular phrase in American culture for about 20 years and then kind of drifted out of sight. But, but basically what it means is trying to, trying to deliver heaven on earth. Right. And so, so basically what Vogelin said was, you know, we had an established understanding culturally of religion and its role in our society, basically from the inception of Homo sapiens, basically up through Darwin. And then, you know, Darwin, you know, and the scientific method kicked the legs out from under that and, you know, sort of invalidated, you know, kind of falsified, you know, falsified science, at least for a lot of people. Um, and then we basically proceeded into the 20th century under the belief that we didn't need, you know, religion. We just needed, you know, we just needed to, you know, dispassionate, dispassionate argument debate. And he basically argues, he argues, C.S. Lewis also argues this. Chesterton, I think, also argued this. He argues like, it just turns out that just purely science, you know, pure science, pure logic, pure rationality is not enough for human existence and human community. Um, C.S. Lewis used the term the God-shaped hole, um, which is sort of this idea that if, if all you have is science and rationality, there's still this missing hole that is basically, is going to be filled by some, you know, kind of view, uh, you know, some sort of vision of the transcendent. Um, and then, and then what Vogelin said basically was in the early to mid-20th century, we, we in the West created basically 3 new, you know, he called them, I think, uh, did he call them political religious movements? Um, and, and they were basically, right, communism, fascism, and then, you know, kind of FDR-style liberal democracy. Um, and then, you know, basically World War II and the Cold War were basically a religious war. And, you know, that settled out basically with 2 of them losing and one of them winning. Right. Political theology, I think, is the term that he used. And so, you know, we ended up with sort of a political theology, you know, sort of progressivism, which is sort of our prevailing religion. Specifically with AI, it turns out that you very— you have had for the last 20 or 30 years, you have had a very classically Vogelin-like, immanentizing the eschaton-like religion in the form of the singularity, sort of transhumanism and the singularity. Um, and so, you know, think Ray Kurzweil, you know, think the Singularity Institute, Vernor Vinge, um, you know, other folks like that. A lot of the early AI people also who basically said we can deliver utopia, you know, heaven on earth through AI. Um, and then, you know, what's the other side of delivering, you know, heaven on earth? The other side of that is delivering hell on earth, right? And so, and on that, on that side of things, in religious terms, you get the apocalypse cult, which in Christianity is known as a millenarian cult. Right. An end of the world cult. Right. Um, and, you know, basically what's happened in AI is we have the, we have the positive utopian singularity cult, which is, you know, quite, which is actually quite small right now. Uh, and mostly, mostly boomers. Um, and then we have the, um, you know, the, the apocalyptic, the apocalypse cult, uh, the millenarian cult, uh, you know, which is primarily, primarily millennials. Um, and, uh, and, you know, that, that one is right now is, um, you know, that one is a very attractive kind of gravity well for a lot of really smart people right now. And again, Vogelin would say, well, obviously it is because these are people who grew up in a secular, rationalist society that didn't give them any sort of transcendent value. And here they are, right, with the claim to transcendence, you know, even if it's the negative claim.

And so EAC is trying to provide the positive claim to transcendence.

Yeah. And so EAC is a little bit— and I did this a little bit. I did this. The title of my essay, right, was AI Will Save the World. I did that. I did that a little bit of a tongue-in-cheek. When I wrote that, I was doing a little bit tongue-in-cheek because I was, I was a little bit— if you read that at face value, Right? It would imply that I'm basically trying to amanitise the eschaton myself. Right? I'm trying to deliver heaven on earth. I'm trying to save the world. AI will save the world. Right? I can deliver you know utopia. And of course, and I outline, and I do believe what I said. But I you know I outline a whole bunch of a whole bunch of ways in which AI is going to make the world a lot better. And so you know I viewed it as sort of a little bit of a tongue in cheek, sort of semi serious tongue in cheek kind of thing of like okay, you know the other path, right? Sort of the the utopian path as opposed to the the the you know the heaven path. as opposed to the hell path. Now, the problem is though, my heart's not really in it to do that. Like, I'm not a, I'm not, I'm not particularly, I'm not particularly religious. I'm not a natural cult leader. Um, uh, I'm not doing what you would do if you wanted to have a full counter movement. Um, uh, you know, I'm not going kind of full whatever, Charles Manson or David Koresh, like whatever you would do.

Yeah.

Um, you know, uh, appointing myself, you know, you know, the other historical figures we can mention who have created religions. So I'm not doing that. And then I would say, one of the reasons I have a lot of affinity for the so-called effective accelerationism is that when I read their stuff, it's sort of in parallel with how I was thinking when I read it. And basically in a very similar way, it's a little bit tongue-in-cheek. I mean, look, you can tell it's a little bit tongue-in-cheek because they call it effective. They borrowed the term effective. They're kind of teasing EA by borrowing the term effective.

Yeah.

You know, but also, you know, and look, you can express IAC as a, as a utopian religion if you want to. And when, when, when the guys who did it, you know, talk about it, they, they will kind of put that shade on it when they're in a good mood. But it's also a very— it's also in my mind a very straightforward, practical expression of how the world actually works. And so, you know, for better or for worse, I, you know, I actually consider it to be quite grounded. And so it's, it's a— at its heart, it's a fairly straightforward defence. of capitalism, of technology, of innovation, of productivity, right, of human ingenuity. So it's something that, you know, a Thomas Sowell or a Milton Friedman or a Friedrich Hayek or whatever would have said, oh yeah, that, you know, that's exactly the same thing that I'm saying.

One of the things I think is really striking about some of your findings is just how small-scale intelligence can be, you know, whether it's a clump of cells, you know, that constitute a xenobot or an anthrobot, or it's even just You know, apparently a, you know, virtual cells in a, in a running a little sorting algorithm, but it starts to get confusing, I guess, when it's like, okay, how small can we get? You know, is there, is there some nature? A lot of these systems, maybe not all, but a lot of them do seem to have like multiple scales. So I'm kind of wondering, does like, is that multiple scale question Critical? Could you have something that just works at a single scale? What would that even look like? And also you said this framework for mind everywhere. Do you have any intuition for how this relates to— you kind of said everything is subjective. Does that also imply that there is some subjective experience on behalf of these systems? Do you think these little sorting cells have any sense of experience? I think these questions are also very undertheorised in AI. people are today just like, oh, well, it's an AI. Of course it doesn't, you know, have any experience or have any moral value. I'm not rushing to say that they do, but I'm also like, you seem very quick. You know, everybody seems so confident in that. And I'm kind of like, not so confident.

Yeah. Yeah. The one thing I can say for sure is that we absolutely cannot be confident because we do not know. I mean, I hear people all the time making these pronouncements that definitely is, or it definitely isn't. No, we, we, we do not have a principled way of answering these questions for the biological world. And that means we have, we are completely out to sea when we're faced with unconventional embodiments. And we know this, you know, science fiction has been at this for well over 100 years. This idea that when something shows, you know, lands on your front yard and sort of trundles out and it's kind of shiny, but also it's given you a poem about how happy it is to meet you. And it's kind of got wheels and you're not quite sure where it came from, but also it's, you know, it's, you're having a great conversation with it. Like, Well, what are you going to use as a criterion for how you're going to treat it and so on? So all of the old categories that we used to have in terms of, well, did you come from a factory or did you come from the process of random mutation and selection, right? Or those kinds of things. These are all terrible categories and they're not good ways of making that distinction. Let's talk about the first question about how low down does it go? I mean, People even, uh, there's, there's, um, minimal active matter research where people can make, uh, very simple systems out of like 3 inorganic chemicals and then they can solve mazes and they can do interesting things. So I think that all of this really becomes disturbing when you insist on a binary category, when you want to know, okay, is it cognitive or isn't it? And then, then, then you've got a real problem because look, Each of us starts out life as a single unfertilized oocyte. It's a thing. It's a little blob of chemistry. Uh, it's a, we, we, we look at it and we say, okay, very clearly this does not have, um, the kinds of cognitive capacities that, that humans have. 9 months and maybe a couple years later, you've got something that obviously does. Developmental biology offers no sharp line during this process where a lightning bolt says, okay, now you've gone from physics to mind.

Okay.

There is nothing like that. The whole process is smooth and continuous. So there is no escape. So, so all, all of this, these continual— first of all, there's the continuum of development, then there's the continuum of evolution. If you think that you as a human have some very, you know, sort of specific competencies, just start walking backwards. So, so which of your hominid ancestors and which of the, you know, sort of mammals before you and before that, and eventually you get to something that's basically, you know, much less complex than a current single-cell organism. Where did those things peter out, right? There is no good story about that. It is imperative that we understand that it's not, are we cognitive or aren't we? It's how much and what kind, right? It has to, it cannot be binary. There's no way to support, not only that, now with bioengineering and robotics, we can make combinations. So if you think machines are not whatever, you know, whatever, and humans are, fine. So right now we have cyborgs walking around. We have people with microchips in their heads and various other, you know, insulin pumps and various other things. That's only going to continue now, right now, now, right now you're, you're dealing with, I mean, talk about non-neurotypical, like right now you've got somebody that's, you know, 98% human, maybe 2% digital. Eventually you're going to get to 50/50 and every other kind of combination. What are you going to do then? Are you just, are you going to do it by weight? Like what percentage of you is, is original parts and what percentage of you is cyborg? Ridiculous, right? So there is absolutely no way to maintain this hard distinction. An interesting question might be, is there anything in this universe that is zero on the cognitive scale? Okay, because I think very simple things already are not zero. Now the question is, is there a zero?

So this is a hard question.

I'll tell you what I think about it right now. Let's ask this. What would the most simple, the most basal, like the most basic versions of agency look like? What, what do you need for that? Now, it's obviously not gonna, you know, people, people say, well, you think that the rocks have hopes and dreams like us? No, the, you have to scale down. The point isn't that it's going to have our level of cognition. What does the most minimal level look like? Right? The smallest possible. Well, I think you need 2 things for that. Um, you need some degree of goal-directedness, and that in William James's definition is the ability to reach the same uh, state by using, um, by different means. Okay, so same goal by different means. So you need some degree of goal-directedness, and you need to be not completely, uh, your actions need to be not completely explainable by current local conditions. So if, if what you're going to do is completely, uh, determined by all the physical forces acting on you right now, then you're probably some sort of billiard ball, and, and that's, you know, that's it. So those 2 things, I think even particles have those. Because least action, least action kind of laws in physics tell you that there's goal directness baked into the bottom levels of the universe. And quantum indeterminacy gives you a really dumb version of, uh, of, of not being predictable by local conditions. It's not great cognition because it's random. I mean, that's not really what we like from, but, but it's something. And so, and so here's, here's what I would say. If, if there is any kind of a definition to life, I think life, we call life those things that are really good at scaling those up. So if you've got a rock, it has no more capabilities than the parts that went in it. Didn't scale. It's sort of lateral and it's bigger, but that's it. If you've got something that's alive, it's cranking up the agency, the indeterminacy, and the goal-directedness across every scale of organization. And its cognitive light cone is increasing, you know, as a function of time. So my friend and colleague, Chris Fields, who's a brilliant physicist among other things, I asked him, so, so can you have a universe in which there is no least action in which like there would be zero? And he said that only, you know, in a universe where nothing ever happened, right? In a completely static universe where nothing ever happened, it would be zero. But as soon as you get, you get interactions already, you know, you've got, you've got the basics of the least action. So, so I believe, uh, in this universe, I don't think there is a zero. I think everything has some capacity, but some things, um, scale it up in an interesting way and some don't. And that means that for some things, the tools of behavioral science are going to be applicable and for some things they're not. Right. And that's the empirical— that's why this is not the same thing as ancient animism where you saw a spirit in every rock because it actually does not pay off to treat rocks with tools from behavioral science, but it does pay off to treat gene regulatory networks that way. And it absolutely pays off to treat various animals that way because that's how humans train dogs and horses knowing zero neurosciences because there's this amazing interface that some animals expose that you can train them. So yeah. And then I guess the last thing you talked about is about inner, like this inner perspective. I try not to say too much yet about consciousness per se, but we can say something about inner perspective and which systems have it and which systems don't. Here's what I think. Imagine, Imagine a landscape, you know, kind of a hilly up and down landscape, and you got a bowling ball on this landscape. In order to know what this bowling ball is going to do and to make it do whatever it is that you want it to do, all you have to do as an external observer is to pay attention to your view of the landscape. Your third-person view of that landscape tells the whole story. You know where the hills and valleys are. That's all you have to know. You don't need to take into account anything else. Now imagine a mouse on that landscape. When you got a mouse on that landscape, your view of that landscape is not really that relevant. What's really relevant is the mouse's view of that landscape because he might have been rewarded and punished in different, uh, at different times in different locations. He might have, he, he's, he's seeing that landscape in a completely different way. There's a, there's a, you know, there's a valence map, uh, sort of superimposed on it that he's got his own opinions about where he's gonna go. And so for all these different systems, you can sort of ask to what degree, and again, it's not gonna be a binary thing. It's never gonna be a binary thing, but you can ask to what degree Do I need to take the perspective of that system and ask what does it see and what does it think about what it's going to do? So to the extent that you have to do that a lot, you're dealing with a high agency system that has an inner perspective. To the degree that you can get everything done by your own model of what's going on, then probably it doesn't. Right. And so, and so that I think is a, is a, is a heuristic by which we can start to say, um, how much inner perspective can I expect from these things? And I, I really, you know, as far as being, being sure that, I mean, look, let's be clear. I am not claiming that today's AI architectures are mimicking human brains. Okay. I don't think they are. I don't think they're mimicking human cognition, although they do have some interesting things in common that people don't realize. But, um, I, I don't think they have to. The point about AI is not that you have to be a human for us to have to be kind to you. There are many, there are many, um, living beings that, that are nowhere near, uh, humans. And the same thing for danger, right? You don't need to be human level or above to be dangerous. There are many really dumb things that are extremely dangerous. So the thing we have to understand is that you cannot pin your hopes for, for being able to distinguish like moral worth and things like that. You cannot pin, pin that on what are you made of, because there's nothing magical about, um, about protoplasm. You cannot pin it on being an evolved biological because there's nothing magical about the random meanderings of the evolutionary process. Which, you know, just, it doesn't optimize for anything that we care about. It doesn't optimize for intelligence or, you know, meaning or anything like that. It optimizes for survival, for copy number. And, and, and so none of those things, none of those things are reliable guides. You, you have, you, you can't, you know, you can't say, uh, just by, um, by knowing what something is made of or how it got here, we have to have principled frameworks for making that decision. And, and we don't have them yet in biology really very much.

So I guess the beginning of maybe a framework for starting to make sense of those, you kind of touched on a little bit in those comments, but just to kind of pull out a couple of definitions, I'll try to, try to do it briefly. You tell me if I'm missing anything. I've heard you define intelligence as the ability to solve problems in some space. I think that kind of notion of in some space is interesting. And then you said kind of also there's the question of like, not are we or aren't we, but how much? And you've, you've used the term the cognitive light cone there, I think. In other words, kind of what is your sort of domain of concern and like possible reach, you know, or how far can you project your influence into the world? And that is sort of a way of thinking about how much intelligence you might have. Does that extend all the way up to sort of nature as a whole? Are you thinking of like the Gaia hypothesis? That's a far out one in many people's minds, but it seems to follow pretty naturally from what you're saying.

Yeah. So, so the key, the key to all of this. So, so first, let me, let me define, um, the cognitive light cone is the size of the largest goals that you can pursue. Okay. So you can think about what are the, what are the things that matter to a bacterium? What are the things that matter to a dog? What are the things that can matter to a human? And, and, you know, and could we have at some point a being that like literally is in the linear range, can care about all the living beings on Earth? Like we, we can't do it as humans, but, but maybe our next, you know, Some next stage of evolution can. So, so it's, it's this idea of, of what are the, what are the, the, what, what are the size of, of goals that you could actually maintain? The important thing about this is that you can't decide that just by thinking about it. You have to do experiments because the way you find goals is by perturbing the system. You cannot infer, uh, intelligence or goal-directedness by pure observations. You have to do perturbative experiments. So what that means is, um, you put barriers between that. You make a hypothesis. What problem space is it operating in? What are the goals? These are all hypotheses and multiple observers could have different hypotheses. What, what, what are the goals? What competencies does it have to reach those goals? And now you get to test that hypothesis by putting in different barriers and seeing what it does, right? Does it have long-term planning? Does it have, does it have learning? Does it have, you know, what, what, what, what can it, what can it do? So with respect to the Gaia hypothesis, some people say, Absolutely not. Some people say, well, I like to think it does, and therefore it does. Neither, neither of those is any good because, because you can't do any of that without doing experiments. Now, can you do, can you do experiments in, uh, so for example, people have said to me, well, you know, that sounds, that sounds like you're going to say the weather, um, you know, the weather has some, some degree of cognitive ability. Well, I'm not going to say that because we don't know yet. But one thing that this framework does for you is it doesn't close off the possibility of testing it to find out. Do I know that if you had the appropriate ways to change the temperature, the pressure of air and whatever, you couldn't use some kind of a habituation or sensitization assay or an associative learning to, uh, to show those things in a, uh, I don't know, in a hurricane or something? Uh, we, we don't know. I mean, that's an empirical question, right? That the, the thing about, the thing about this framework is that, is that it does not let you just sort of pick answers out of thin air. It requires you to do experiments. So, so Gaia, you know, can we show that ecosystems learn? Well, ecosystems have stress. You know, would an ecosystem be able to show different kinds of learning, anticipation? We don't know, but that can be tested. We're actually testing this right now. I have a student who's testing this in predator-prey kinds of models to see if they can actually learn from experience. So we don't know. These are all testable questions. So is it possible that, you know, on the scale of, I mean, people ask, you know, on the scale of galaxies or something that, you know, we're a giant, like we're part of a giant mind. I think it's absolutely possible. I don't think we know, and I don't think you can say yes or no just, you know, based on philosophical commitments.

There has been in Western civilization a fundamental tension between science, engineering, and commerce. The commercial mindset always involves talking up the current business plan, talking up the current approach to things. Science involves undermining it, right? Science involves questioning the current approach. Science involves disproving the current approach. So you can see how in a corporate culture, these 2 can be in tension. And most people, also very intelligent people, if they go to elite universities, by default, they're trained in agreeability. The purpose of the educational system is to train you up in agreeability. We basically destroyed the Prussian Science University. It took a century for the destruction to happen. But when we made the Prussian research university model, when it was adapted, adopted in the United States and adapted to scale to most of the population, it obviously can't scale. So something that was designed to sort of protect your PhD autists ended up being a degree mill and ended up being a necessary career step, not just here, but back in Germany and in the United Kingdom and in France. And actually now more and more so in China. If the talent distribution mechanism of your society is the same thing as the scientific mechanism of your society, that is also a contradiction. And that is a fundamental problem of Western civilization. We do not know what to replace universities with. So why am I talking about universities in the context of AI? Like I said, AI is a good test. You have to go deep. So many of these top AI scientists were poached from universities that had good computer science departments, but these weren't even first-tier universities. Can you name a brilliant AI scientist from Harvard? Has a background background, did a PhD there or something. It's kind of interesting. You see that Canadian universities punch above their weight. MIT, well, that's pretty predictable. British universities, French universities, for some reason French mathematicians seem to have a knack for it. Anyone who is trained in the former Soviet Union for some reason, you know, and by the way, that's not surprising. A lot of the mathematics underpinning modern models was actually developed in a compute-starved environment as academic hobbies for mathematicians who were kind of trying to cope with the lack of good computers and trying to figure out new and more efficient ways to do things and working things out on paper rather than on huge training runs. Now, that tradition has developed immensely in the 1990s and the 2000s after the the Soviet Union, but it's notable that a lot of these academic lineages go back there. So after the launch of ChatGPT, I think we have stripped-mind academia for all the good AI scientists. I don't think you have any truly knowledgeable questioners of the nature of the universe of the Ilya Sutskever tier. Left. I think they've all been strip-mined. There was just too much money. There are mathematicians who, for ideological reasons, were not inherently interested in AI but were scared of AI by basically Eliezer Yudkowsky and basically things around that. Arguably, Terence Tao is in that category to some extent. And those mathematicians, I think, will make modest or decent progress because ultimately if you have a holy terror of AI, which causes you to change your mathematics, what you apply your mathematical expertise to, I think it can make you a very good engineer, but it makes you like kind of a mid-scientist. I think the curiosity, not the terror, is the actual underlying deep motivation, which causes these like very flawed human brains prone to various Interesting cognitive flaws like wishful thinking and so on to actually end up locking onto truth. This means that if someone wanted to, they could compile a list of all the AI scientists that are likely to be paradigm breakers. You wouldn't get 100% of them. You could probably get 90% of them, and you can chart which company has the most. And At the end of the day, you can chart which company is likely to not be constrained by needing to develop a product. Let's remember that this breakthrough of application of ChatGPT came out of a company that people used to make fun of for releasing white papers instead of products. Like, this is important to remember before its memory hold that OpenAI, you know, it wasn't just a corporate drama that came out of structuring it as a nonprofit, they basically gave the best AI researchers they could find tenure. Like they basically were just publishing white papers and they were working very hard and they were motivated by this vision that AI will transform the world. They were recruited by literally Sam Altman and literally Elon Musk working together. Like that's in the prehistory. Including things like giving Tesla gifts to the early employees to encourage them, right, to go do that. And I think that cannot be recreated in the current environment. So it's sort of decaying, right? And if you look at the key people like Dario Maday and like Ilya Sutskever and, you know, maybe others in the future, this dream team has sort of split apart. Now, I think they have likely trained up significant human capital. So anyone who was working with them in that initial period before the entire planet decided OpenAI was a huge success, those people, if they were AI scientists, not software engineers, I think they're underrated. And I think people should be trying to poach those because, you know, Like I said, academia is strip-mined. Which places were still producing and training people in the scientific mindset rather than an entrepreneurial mindset? Let's be real. Like, I feel like generative AI is now as much of a buzzword as Web3 ever was. It's Web4 basically, right? It's like, what is supposed to save the balance sheets of all the investors and all the major corporations and actually BlackRock. And when you add it all together, you know, really the US has no plan for solvency except for an AI revolution, right? Like that's the only way in which the Western world can achieve collectively 4 or 5% economic growth per year. Now, of course, if AI worked in this promised way, we wouldn't have 4 or 5% a year. We might have like 10% growth a year or 20%. Truly singularitarian numbers. But, you know, if it works only the way it's working right now, I don't think there's any chance that this results in a 5 or 6% rate of GDP growth. There's no chance it automates labor fast enough to counteract the global labor crunch. And I don't think we are focusing enough on the science of artificial intelligence.

Yeah.

and asking the deepest questions and investing in the most esoteric mathematics. One way to put it is that imagine a world where it is easy to raise money to build particle accelerators and employ experimental physicists, but it is difficult to employ and protect for longer periods of intellectual development theoretical physicists. That is a world where we would hit a wall in physics very quickly. And I think artificial intelligence, it's much closer to physics than it is to, I don't know, fintech or Moore's Law or something like this. So that's my sort of contrarian thesis. We've undervalued AI science. We've overvalued AI compute. We overvalue raising huge amounts of money to like quickly enter the AI race and undervalued longer-term plays to cultivate top AI talent that is intrinsically motivated by their curiosity to push forward this frontier of understanding. And I think it's just because a single AI product went viral. And here's the spiciest take I have, my spiciest take. Had OpenAI not gotten ChatGPT, Had ChatGPT not been viral for whatever reason, probably 2 or 3 UX decisions could have killed it. To be honest, it would've enthused experts and some top programmers, but the general public would ignore it. I think we'd actually be making more theoretical progress because the dream team would still be together and the intent would still be on the white papers. Ironically, you should interrogate these 5 or 6 heresies because I don't think anyone talks about it this way. I've been like surveying the entire discourse landscape and I see very few people arguing these points. So I'm happy to be challenged on them.

Yeah, I think there's, you've touched on like 10 different things that could be full discussions unto themselves. Some of the ones that I think I agree with, assuming I'm interpreting you correctly, I always say the transformer is not the end of history. Yes. And it's important to remember that, you know, we've had for much of our lives, the sort of end of history vibe. And obviously we're now shaken out of that. I think the AI community, maybe not the top thinkers, not the top scientists that you're talking about, but the community at large has experienced a somewhat similar thing where there's been this sort of, well, we just scale up the transformer and we'll keep doing that forever, and that will deliver the sort of transformative outcome. And I personally don't rule that out, but I expect that there will be more theoretical and like, we'll probably never find out because I think we will see more theoretical advances that will mean we're never gonna run the trillion-dollar transformer experiment because there will just be other things that will be sort of better options.

It's good to remember that diffusion models are what's generating all the images. It's not transformers. So what is it if transformers are fully general intelligence in the bedrock of it? Why are they so much worse at generating images than diffusion? I think that we are seeing the Cambrian explosion of the different definitions of intelligence, right? The Cambrian explosion with this event on prehistoric Earth hundreds of millions of years ago, where suddenly there was a vast diversity of multicellular life that came to be. It's like visible in the fossil record. I think for most of our history, we have had only human and animal intelligence, and we added a tiny bit of computer intelligence, which I think is different in kind than human and animal intelligence. And now we have added even more forms of computer intelligence, right? I think that already the Turing machine was already a form of artificial intelligence. I think we're eventually going to have a periodic table of elements of different things that can be called intelligence that are fundamentally different processes. So I think we'll have qualitatively different processes that can produce outputs that we would call intelligence, but they will be fundamentally different processes with different properties, different strengths, different weaknesses. And I think that'll be surprising to us because we're used to thinking of intelligence as a single thing because the evidence is pretty strong in human beings when you do psychometrics and cognitive science that humans are just kind of doing one thing super special. But, you know, maybe the universe doesn't work that way, right? Maybe human intelligence was like the hydrogen atom, the first atom that comes into existence in the universe, but then very quickly the stars start making helium and iron, and then the supernovas start making uranium and all these heavier elements that mere star fusion can't produce. So I really do think that we are going to be continuously surprised at the strength, unreasonable effectiveness of various forms of intelligence as we discover them, but also surprised and shocked by their weakness, right? Why do AI agents When you put generative AI, when you put transformers on a loop, why do they melt down into incoherence and don't pursue goals and cannot actuate and cannot act on the world? Like, that's a surprising thing. I don't think people expected that. If you told someone 20 or 30 years, the kind of verbal behavior this intelligence is capable of, they would assume it's trivial to use the words to command a robot. And have the robot do things, right? I think they would assume it's trivial, but it turns out it's not. Or they would assume it's trivial to have a representative. Wasn't that the thesis of like a hundred startups five minutes after ChatGPT came out? The personal assistant, and they're just not. You know, you can use them, but you can use them the way you can use a Calendly link. You can't use them the way you can work with an executive assistant, a human executive.

of a system. I really agree with, and I think you're sharp to emphasise, I use the term Cambrian explosion sometimes as well. I also have started to use the term mixture of architectures era to signal sort of the end of the transformers era and the beginning. And I think we're just very much still in the beginning.

Yes.

But folks who listen to my feed know that I have an obsession with state-space models. And the big reason for that that is that I see it as one of the first new core mechanisms that performs basically on par with the attention mechanism, but notably, as you said, does have different strengths and weaknesses, which you sort of have to drill down to a very small level to really distinguish. It's quite interesting that at the high level, when they train like a state-space model versus a transformer model at scale, They look pretty similar. Like qualitatively, you can chat with them. They seem to have reasonable memory. When you drill down into the micro skills, you find that there are kind of two sides of a coin, sort of different strengths and weaknesses. And not surprisingly, based on that, some of the most exciting new things coming out these days are hybrid structures where you have an integration of these things. When I just think about, and I'm no expert in neuroscience, but when I just look at sort of a picture of a brain in an anatomy textbook, book, I sure think, yes, there's like some parts of this that are kind of highly repeated. You've got your kind of cortical column that's like definitely a main piece that is repeated a lot and doing a lot of work. But you also see just a lot of other parts. And so it does seem like we're headed for, I think a periodic table is a pretty good mental model for it. We'll have a lot of these different specialized, not even full architectures, but components that you can kind of snap together in all sorts of Upstream with Erik Torenberg is a show from Turpentine, the podcast network behind Moment of Zen and Cognitive Revolution.

If you liked the episode, please leave a review in the Apple Store. Hey everyone, Erik here. At Turpentine, we're building the first media outlet for tech people by tech people. We're the network behind the show you're listening to right now. We have a slate of hit shows across a range of topics and industries, from our AI and investing cluster of podcasts to shows that drive the conversation in tech with the most interesting thinkers, founders, investors, and influencers, like Econ 102 with Noah Smith. We're launching new shows every week, and we're looking for industry-leading sponsors. If you think that might be you and your company, email me at erikaterpentino@neutralspace.com. Thank you so much for having me. That's erik@turpentine.co, and let's partner together.