Anthropic CEO Dario Amodei Talks Scaling Laws, AI Arms Races, and Radical Abundance

Technological innovation eventually makes business process innovation or business model innovation less important because you can just, you know, button press everything, you know, just do everything.

Yeah.

It's kind of like the interface and business process innovation is almost like a substitute for the intelligence of the model. The more of one you have, the less of the other you need. And so it creates this interesting dynamics, right? Where you have these kind of overlapping overlapping revolutions, each of which is a little faster to market than the one before, but maybe the music will stop at some point. We don't know for sure. What I always say about the scaling laws is like, no one should believe that there's some fundamental physical law that means they're going to continue. It's an empirical observation. They could stop at any time. I've been watching them for 10 years, and my belief that they My guess that they won't is just based on the amount of time I've been watching them so far. But, you know, that's not— that's a 60-40, maybe 70-30 proposition.

The trend is your friend till the bend at the end, as they say.

I've never heard that phrase, but that would indeed be correct.

Hey, Upstream listeners. This is Natalie, one of the producers at Turpentine, the podcast network behind Upstream with Erik Torenberg. We're super excited to be able to use this feed to highlight the absolute best content and most insightful interviews across our entire network. Today, we're kicking it off with this week's exciting episode of Econ 102, Our economics podcast hosted by Erik and Noah Smith of No Opinion, where they were joined by none other than Anthropic's CEO, Dario Amodei. The show is usually a running conversation between Noah and Erik, and a couple weeks ago they mentioned they wanted to have someone from Anthropic come on. As it turns out, Anthropic's co-founder Daniela Amodei was listening and let Dario know. The conversation ahead is wide-ranging. Dario discusses the business moat for AI companies, his views on AI safety and national security, how AI will affect the labor market, US relations with China, and regulation like California's SB 1047 bill. Without further ado, here's the full conversation. We'd love to hear your feedback. You can email me at natalie@turpentine.co or leave a comment to let me know what you'd like to hear more of on Upstream and which guests you're most interested in.

Where's our guest?

They're coming. They're coming in, about to let him in.

Hello.

Hello. Hey, Dario.

Hey, Dario.

Hey, Seth.

How's it going?

It's going all right.

It's been a while. Great to see you again, Noah. I, I think we met many years ago. I don't remember exactly when.

We met a couple times. Yes, before the pandemic. We haven't seen each other since then. So yeah, that's, that was a startling number of years ago.

I, I can't hear you all that clearly. It's kind of coming through.

How about now? Is this, is this a little better?

Oh, that's much better. There, whatever you did, fixed it.

Okay. It's called holding the mic closer to me. Because no, I had to abandon my usual setup because the internet signal was being patchy in that room in my office. And so I have to get a relay, but until I do, I'm just podcasting from my couch.

We have a running joke at Anthropic, actually goes back to other companies I've worked at, that I've had to use on almost every podcast, which is we'll solve AGI before we solve video conferencing. And I think it's going to be literally true. I think it's not a joke.

Maybe the AGI can tell us how to solve the video conferencing.

Yes, but the AGI will come first.

Dario, we haven't met face to face yet, but I'm very close friends with Yasmin from Spark, who's been singing your praises for years and let me know about how big Anthropic was going to be years ago and put me onto you guys. So it's great to chat and have you on.

Yeah, no, Yasmin's great. She's been a very, very supportive investor.

Excited to get started.

My sister and co-founder, heard you, you know, you say on your podcast that it'd be great to have me. And so, you know, you spoke, you spoke my name and you summoned me. So here I am.

We summoned you into existence. And maybe let's start there, Dario, because you have an interesting intellectual evolution. Why don't you briefly, briefly describe your intellectual evolution and why you're on an econ podcast and why it's fitting?

Yes, let's see. So I don't know if I go, if I go all the way back, like, You know, I think early before I got involved with AI, I was just trying to figure out what would be most interesting and impactful to work on. So I did my undergrad in physics. I believe Noah's the same, actually. And then considered a bunch of things, actually considered going to grad school for econ, decided not to do that. Went to grad school for computational neuroscience and biophysics. I was very interested. I considered AI as well, but at that time, I just didn't believe that the AI of that day was all that interesting or had made all that much progress on important problems in intelligence. And so I figured I'd study the one object in the universe we know of that is at least sometimes intelligent, the human brain. And so worked on that, worked on basically networks of biological neurons and trying to understand what was going on in them, which turned out to be a very difficult task because just the biological access and the physical measurement let alone even getting to the analysis, just turns out to be a very, very difficult task. And spent most of my time on that and very little of it making tiny bits of progress on trying to understand the actual algorithmics of what is going on inside the human brain. And then around, I did a postdoc at Stanford, worked on proteomics, and then around 2014 or so, the deep learning revolution was starting. I saw the work on AlexNet and QuocNet. And so I decided to get involved in that field. I worked with Andrew Ng at Baidu for a year, and then I was at Google for a year. Then I joined OpenAI very shortly after it started, was there for about, was there for about 5 years, developed a lot of the ideas around scaling laws, helped build some of the early GPTs, as well as inventing the RL from human feedback method along with others that, you know, would eventually be used to produce some of these these commercial chatbots and then left around the end of 2020 to, to found Anthropic and, you know, have been running Anthropic ever since.

Here's a question in general, you know, this is an econ podcast. We like to talk about a little more of the econ side than the, than the pure tech side. But yeah, so is Google the Bell Labs of the artificial intelligence age? Because they, you know, they produced the research that led to modern deep learning and transformers, et cetera. And they didn't really manage to commercialize it very well, kind of like Bell Labs. And they funded it from their, you know, IT monopoly, kind of like Bell Labs. And then, you know, a bunch of interesting people like, you know, worked there and then sort of went off and started companies, you know, like eventually the Fairchild people did from Bell Labs. Do you think this is an apt analogy?

Yeah. I mean, while nothing is ever a perfect analogy, I certainly think there's something to it, right? When I was at Google, I mean, you know, for many people there, They thought of it as a kind of continuation of their academic careers, which was you know the same the same as the same as you know the same as Bell Labs—an industrial you know an industrial environment that that has many things in common with an academic environment except that it has kind of more more resources to make things happen. And so you know people working on a lot of things, right? The transformer, which is kind of one of the key inventions that you know that drove the field. You know that was one of maybe a hundred things that that were being. that were being worked on. And if you went high enough up, if you went high enough up in the organization, you know, there was no reasonable ability to distinguish between that and the 99 other things that were being invented. It was like, it was let a thousand flowers bloom. And, you know, it was, I think during that time that I first started having ideas around things like the scaling hypothesis that, hey, we needed to take some of these innovations, we needed to really scale them up and we needed to put them together. And in theory, Google was the best place to do that. They have like the biggest clusters in the world. They have, you know, a lot of kind of talented engineering staff. You know, they kind of had all the ingredients, but, but the Google machine was organized in a certain way. It was organized to do search. I don't think it was necessarily organized to kind of put all these pieces together and scale up something radically different from, from what, from what had been done before, from what the business was about.

Right. Like, like Bell Labs wasn't set up to, you know, invent computers and give everyone a computer. It was set up to like wire everyone up.

Yeah, exactly. It was a telephone company. And, and, and so, you know, I mean, you know, I can't speak to Google, obviously. Obviously now they've, you know, in addition to inventing all of this, these, these, you know, these, you know, in addition to inventing all these amazing things, you know, they have, they're one of the 4 companies that has, you know, that has the frontier models. They're, you know, both a partner to us and a competitor to us. I know many people there, they're super, super smart. But, but I think you're right that there was a period where if they had had kind of, if they had been able to put together the ingredients in the right way, they could have been the only, they could have been the only game in town. And, you know, for whatever reason, it didn't, it didn't go that way.

Well, that's, that segues into something else that we were thinking about, you know, this, the idea of talking to you actually came from another podcast we were doing. Where I was mostly talking about the economics of like internet businesses. And then Erik came out with some, some kind of pessimism on the business of AI and, and, you know, was like, you know, questioning whether, you know, how much, how much of a moat AI companies have. Obviously this is incredibly relevant to Anthropic, you know, and, and other companies that we call startups, but they're already pretty big. And yeah, so, so tell us how you think about the, the business moat of AI companies. Yeah.

So I will say, I'm going to separate this a little bit into 2 branches. I will say it's a little hard to separate some ideas and questions around the scaling hypothesis from the business questions. So like, let's consider the branch where kind of the scaling hypothesis is true in some very strong form. And then let's consider the version where it, you know, might be somewhat true or not true at all. In the version where it's true in some very strong form, you know, I've Probably laid this out elsewhere in public, but this is the version where you train a billion, you've trained a $100 million model now, and it's about as good as a good college freshman or something, and you train a billion, it's as good as an advanced undergrad, you train a $10 billion model, and it's as good as a top graduate student, and then when you get to your $100 billion model, it's as good as a Nobel Prize winner. And then you just take that model and you put it to work for basically everyone, right? It becomes your coworker, it becomes your personal assistant, it helps with national security, it helps with biology. And I think in that world, like that system and the products built on that system just become a very large fraction of the economy. There's still a question of like, well, where do the returns go, right? Do they go to NVIDIA on one side? Do they go to the AI companies on the other side? Do they go to downstream applications? And there, the pie is so big that to first order, my answer is yes, they go to all of those places.

Right. But, okay, think about solar power. You know, solar is obviously going to be absolutely immense, you know, and the more energy we need, the more solar there will be. And yet, it's hard for me to name you a solar company that makes a bunch of profit. It is a very commoditized product, even though there's plenty of innovation involved. But yet there's no branding, there's no network effects, there's no, you know, lock-ins of anything. It's very hard for any solar company to make a profit on this thing that is radically transforming our entire world in front of our eyes right now. And so, so I'm not 100% sure that, you know, just the fact that like everything will be booming as it is in solar right now, you know, will lead necessarily to profits being captured by, by companies, but I'm certainly open to the idea that it will be. I just, you know, what, what do you think will be the source? Why will AI turn out different than solar?

Yeah. Yeah. So I think, I think 2 points here and I'll get to what you're saying, 'cause I think it's a, I think it's an important question in most worlds. Maybe I'm just making the point that this is like, if, if the scaling hypothesis is correct, this is going to be like really, really big. Right. And, and so big that even if only 10% of the profits go to this part of the supply chain or that part of the supply chain, it's still really, really big. You make the pie big enough, and that becomes the most interesting question. Although, I'm sure that the people who are deciding how the dollar bills get split up, it's going to be really important to them whether $1 trillion goes here and $10 trillion goes there, or $10 trillion goes here and $1 trillion goes there. But let's actually get to your question, because I think It matters in all the worlds. It's just a matter of what is the size of the pie that you're distributing. First of all, I think on the model side, and again, this is dependent on the scaling hypothesis, if we're building $10 or $100 billion models, there is not going to be probably more than 4 or 5 entities, maybe a couple state-run players will get involved as well. who build these. So, we're looking at something that looks probably more oligopolistic than it does monopolistic or fully commoditised. I suppose there's a question of, you know, will someone release a, you know, $10 billion or $100 billion open-source model? I'm somewhat sceptical that, you know, the conviction to do that goes that far. And I think actually, even if such a model is released, One thing that's a disanalogy to open-source software is that these big models, they're actually very expensive to run on inference. The majority of the cost is inference, not necessarily the training of the model. So if you have only, I don't know, 10%, 20%, 30% better way to do inference, that can kind of negate the effect. So the economics are kind of strange. Yes, there's this giant fixed costs that you have to amortize, but then there's also the per unit cost of inference and small differences in that can actually, again, assuming the thing is deployed widely enough, make a very big difference. So I don't know quite how that's going to play out.

So that's actually similar to the economics of heavy industry. Like that's kind of how making steel works.

A little bit. Yeah. A little bit. Yeah.

Interesting.

I would, the other thing I would say is, you know, within these few models, something we're already starting to see is that like models have different personalities. And so, commoditisation is one possibility, and I do see some worlds where even within the oligopoly, some particular ways of deploying models could get commoditised, although I'm not sure. But one force pushing against that is the idea that, hey, I make a model that's really good at coding, you make a model that's like really good at creative writing, this third person makes a model that's really good at being kind of like engaging and entertaining. These are kind of choices. And once you start to make these choices, you start to build infrastructure around them. And so it, to me, it seems like it creates the preconditions for some amount of differentiation. The other thing that I think might lead to differentiation is just the products built on top of it, right? In theory, you can separate the model layer from the product layer. In practice, they're somewhat linked to each other and it can be somewhat challenging to work across organizations. So, You know, while the models, there's kind of this, I don't want to say single path, but like, there's like a logic to building models. Many of the companies are going in the same directions with the multimodality they're adding onto it, making the models smarter, making the inference faster. There's kind of, in some ways, one direction, but products are so different, right? If you look at this like artifacts thing we made, which is a way to kind of visualize in real time the code that the model writes, right? We do that. OpenAI does their own thing. Google does their own thing. I think that is also kind of the source of some differentiation between companies. And I think we've already found that the economics of selling apps on top of the model, even if they're relatively thin apps and they're starting to become thicker and thicker, is quite different from the economics of just kind of like handing over the model via an API and saying, here, we're opening the tap, buy it.

If the scaling laws hold and things get as big as we think it might get, do you expect these companies to be nationalized at some point? And could that present another moat, or how do you expect—

Yeah, I mean, that's not what people traditionally talk about when they talk about a moat. I think that does get to these things about national security. I mean, Again, we bifurcate into the world where the scaling laws are right and where the scaling laws are wrong and things taper off. If things taper off, then this is just a technology like the internet or like solar or anything else, probably bigger than most, but not unprecedented based on where it has gotten so far. And then I don't think it'll be nationalized. And I think a lot of these questions about who gets the value will be absolutely front and center. If the scaling laws are correct, and we're building models that are like Nobel Prize-winning biologists, top-of-the-industry coders, or better, then I think both the questions about national competition and questions about misuse and the autonomy of the models will become front and centre. And I think I've said elsewhere that I'm not sure of literal nationalisation, but I think The government probably has a big role to play, right? We could get to the point where these models are, you know, one of the most valuable, perhaps the most valuable national defense asset that, you know, the United States and our allies has. We'll want to care a lot about them being stolen by adversaries, about whether adversaries can keep up, about whether we can deploy them as fast as adversaries can. You know, I just think of a, you know, model that can, you know, integrate all the intelligence information across you know, that, that, you know, that the United States or, or, or, or, or, or one of its allies receives, or, you know, coordinate all of our, all of our, you know, military or logistics operations. You know, that sounds really, that sounds really quite powerful.

So did, did you think that, that Leopold put it pretty well in his discussion of, of China versus US in terms of AI?

Yeah, I mean, I think Leopold's, Leopold's essay was, was super interesting. You know, there's, there's a lot of it, there's a lot of it that I agree with. It's maybe a shade further towards kind of the nationalization perspective and the national security perspective than I would necessarily go. But my view is, I would say, not particularly far from it. You know, I just definitely think when a technology becomes powerful enough, this idea of like 4 or 5 companies that are just kind of like autonomously operating it and doing whatever they want. It just doesn't seem like it's going to lead to a good outcome. And I say that as the person in charge of one of the companies. So there are many models of government involvement in industry for national security reasons. There's like the SpaceX model of a public contract. There are public-private partnerships. There are things that look like the national labs. There's literal nationalization. Somewhere in there, there's, there's a model that makes sense at some point. I don't know what it is yet. I don't know when the right time to do it is. I suspect the right time is not like right now, like not this year. But, but, you know, if the scaling laws are right, things can change really fast.

This is, yeah, that, that, that fits very closely with what I was kind of writing about internet companies for years about how they'd eventually, you know, partner with the government. I have a, I have a question. Which is about, I have this big thesis. So, you know, the story of electricity, how basically at first when they got electricity, manufacturers tried to rip out their steam dynamos, you know, add electrical generators. And of course that was much more lossy. So their power was reduced because they kept the same form factor for the factories and they didn't, and it was a loser, a money loser for them. But then somebody figured out you could run electricity in parallel to a bunch of workstations and they changed the way that manufacturing worked instead of from one big assembly line to people taking things, you know, to different little workstations and doing their work there. And that led to enormous productivity gains over a sustained period of decades. And so I've always had this suspicion that AI is similar. Well, I think the internet's similar actually, but, but I think AI is similar in that at first everyone seems to kind of be thinking AI is a person. You know, you get people actually literally talking about numbers of, of AIs related to numbers of human employees, which doesn't make any sense to me because like, that's not It's not divisible into individuals. And, and I mean, you could make an agent-based thing that does act like that, but why anyway? But, but I guess my, I'm seeing everyone thinks about AI in terms of directly replacing human beings. And my thesis is that this is just the first pass. This is kind of like electricity directly replacing steam boilers. Wasn't that great of an idea. And I think that I sort of have this forecast that people are going to be a little disappointed when there's only a few cases in which that actually works, like customer service. You know, some other well-defined things. But, but I think there'll only be a few cases in which that direct replacement of humans actually works. And then we'll have a Gartner hype cycle, you know, sort of bust where people are like, okay, well, that doesn't work. Well, AI is a big, big fat bust. Who needs it? And then some, you know, creative entrepreneurs will say, okay, well, actually, instead of just using AI to act just like a human replacement, we could use it in some creative way. And of course, I can't tell you what that way is because then I could be a billionaire. I don't know. But then, you know, people figure out creative ways to use AI in ways that humans employ— human employees were never used, direct, you know, that take over some human tasks, but that complement other human tasks, and that create whole new business models. And that then we'll see this, like, you know, resurgent boom. This is kind of my, my forecast, my, my Gartner-style prediction. Am I crazy?

Yeah. So it's, I think this is like, this is like a mix of things that I agree with and things that I might disagree with, or maybe like, This is true, but some other things are true as well, which seem like they have a lot of tension with it. So, so first of all, I think I basically agree. Just, just, I'm just thinking this through live. I think I basically agree that if you freeze the quality of the current models in place, everything that you're saying is true. And, and we basically observe, we basically observe something, you know, quite similar in our, in our kind of, you know, in our commercial activities, right? So, you know, we both offer, you know, Claude.ai that you can just talk to, but also, you know, we sell the model via API to a whole bunch of folks. And people are definitely, you know, it's taking them a long time to figure out exactly what is the best way to use the model, right? And, you know, do we make it a chatbot? Questions about reliability of the model abound, which I think are driving some of the concerns you're saying. Like, let's say I have a model, 95% of the time, let's say it wants to offer financial information or analyze a legal contract or something. 95% of the time, it gives a correct answer. Maybe that's even more often than a human gives the correct answer. But people don't really know what to do with either end users or the company itself, the 5% where it doesn't give the correct answer. It doesn't necessarily give the wrong answer in quite the same way a human would. How do you detect those cases? How do you do the error handling? How do you handle that? And, you know, it's very different for something to be useful in theory and useful in practice, right? Like I mentioned before, this artifacts feature we have, right? Early on when we had Claude.ai, it was like, you know, you could have it write some code and then you could paste the code into, you know, into this compiler or interpreter and then it would like make your JavaScript video game or whatever, and then some things would be wrong and you'd go back to the model and say, blah, blah, you know, how can you correct this? Closing that loop has, you know, has just kind of made a big difference. We've also seen like the model, like big models orchestrating small models. So, you know, this is something that's like very different from thinking of the model as a single person, although I don't know, maybe it's, maybe there is a person analogy to it, right? We have like bigger, more powerful models and like faster, cheaper, less smart models. And so a pattern that we found with some customers is the big large model, you know, has in mind some big task and then it like farms out a copy of like 100 of the small models. And using that, you know, they'll go off and do the task and they'll like report back to the large model. And so you have this kind of, instead of a person, you have this kind of like swarm that's doing the task in a very kind of non— non-human-like way, right? It's almost like how a colony of bees would, would, would, would perform the task. So I think there's— this is all to say, like, you know, I think there's— we're still figuring out what the best way to use the models are, and there's lots of different ways to use the models. But I also think that as the models get smarter, they get better at powering through these problems. So as they get smarter, we're going to be better at turning them into agents. They're going to be better at doing tasks end to end. The amount of the human element, the amount that humans are going to have to do is, is going to decrease. And so again, it just all comes back to like, will these scaling laws continue? If they do, it's like, it's like a series of processes like the one you're describing. If they freeze, then the innovation stops and the, the, the research innovation stops and the process you're describing plays out.

You basically think that, that scaling brings back what I consider like the dumbass use case.

Yeah, yeah, yeah, yeah, yeah.

Because like, you know, right, right now, you know, I, the dumbest thing that I could do with ChatGPT is be like, hey, write me an article about blah, blah, blah in the style of previous Noah Smith articles and just comes out with a perfect thing. It can't do that right now. It can't even come anywhere close to doing it. But, but the idea is if we scale, then eventually we get to the point where, you know, all I have to do is say, hey, write me a Noah Smith article in the style of this. And then I get to take over the world because I get to spam infinite high-quality Noah Smith thought into the, or, you know, or even higher than, than natural quality. And so basically everyone will just be reading Noah all day long because I will be, you know, me and my AI will dominate your, your thought sphere.

I already spend, spend a reasonable amount of time reading your articles. I have to admit that if there were an infinite number of them, it would be quite powerful.

Cackle, evil, evil cackle. Anyway, but yes, so, so I, I get the, I get the idea here, but that's basically interesting because it means that as technological innovation eventually makes business model innovation less important because you can just, you know, button press everything, you know, just do everything.

Yeah. Yeah. It's kind of like the, the interface and business process innovation is kind of a, it's almost like a, it's almost like a substitute for the intelligence of the model. The more of one you have, the less of the other you need. And so it creates this interesting dynamics, right? Where you have these kind of overlapping revolutions, each of which is a little faster to market than the one before, but maybe the music will stop at some point. We don't know for sure. What I always say about the scaling laws is like, there's, no one should believe that there's some fundamental physical law that means they're going to continue. It's an empirical observation. They could stop at any time. I've been watching them for 10 years, and my belief that they— my guess that they won't is just based on the amount of time I've been watching them so far. But that's a 60/40, maybe 70/30 proposition.

The trend is your friend till the bend at the end, as they say.

I've never heard that phrase, but that would indeed be correct.

And what would change your perception there? What would change your odds there?

What would change my odds there? So I think, I mean, first of all, if we just, you know, trained a model and, you know, trying the next scale of model and it was just really crappy, and then we tried it a couple times to, you know, to try to unblock it and we still didn't, I'd be like, oh, I guess the trend is stopping. If there were problems with running out of data and, you know, we couldn't generate enough synthetic data to really kind of, you know, continue the process, then, you know, then at some point I'd say, hey, this actually looks like it's hard. You know, at the very least, the trend is going to pause, you know, may or may not stop, but, but, you know, that's what it might happen. It's, it's still my guess that those things won't happen, but, you know, this, this is a profoundly non-trivial question.

We'll continue our interview in a moment after a word from our sponsor. Fast forward to the end of 2024. Think about your goals. What can you do right now to give yourself the best chance of succeeding? If learning a new language is on your list, you absolutely need to check out Babbel. Babbel offers a range of learning tools: self-study app lessons, live classes, and even podcasts, which have always been my favorite way to learn. Studies from Yale, Michigan State University, and others continue to prove Babbel is better. One study found that using Babbel for 15 hours is equivalent to a full semester at college. Babbel isn't just a game to kill time and make you feel like you're learning a language. It's not overly academic or rigid either. It's all about learning language for the real world. Babbel stands out because it's designed by real people using a modern conversational teaching approach. It's not always easy, nothing worth doing ever is, but it's straightforward and designed to help you start speaking in just 3 weeks. With Babbel, I was able to brush up on my intermediate Spanish to ramp up for travel to Argentina last year and was able to set clear goals based on how much time each week I wanted to practice. Join millions of Babbel language learners across all age groups. Here's a special, limited-time offer for our listeners. Right now, get up to 60% off your Babbel subscription, but only at babbel.com/torenberg. That's babbel.com/torenberg, spelled B-A-B-B-E-L.com/torenberg. Rules and restrictions may apply. The tech world turns to the Brave browser for its unbeatable privacy protections. But did you know that Brave also has a private ad platform? Brave Ads offers first-party targeting, and it's been cookieless since day one, so you can relax while third-party tracking cookies disappear from the web. Today, millions of people turn to ad blockers to avoid being tracked and pestered online. But Brave's new ad model aligns incentives for users and advertisers. Users earn rewards for viewing ads, which they can save, spend, or pass along to their favorite creators. And advertisers score points for respecting user privacy, generating ROI without invasive tracking. So whether it's high-impact announcements on the New Tab page or keyword-targeted ads in Brave Search, Brave Search offers diverse, private, future-proof ad formats for all your business goals. Join the future of advertising at brave.com/ads. Mention Turpentine to get 25% off your first campaign.

All right, Erik, I think you're up. Ask a big question.

Darien, how much do you worry about arms races? How much of a contributor are they to your overall AI risk picture? And do you worry more about US versus China or competition between firms?

Yeah, so I don't know. I mean, I, you know, there's, there's kind of no rule that says we have to be in a situation that has like a single right answer or a good answer. So there's, there's different things I'm concerned about, right? You know, I'm, I'm concerned about, you know, what's come to be called safety, although that's kind of like a weird term. I divide it into kind of the autonomous behaviour of AI systems, which I think is not a big issue now, but as soon as we make agents and systems become autonomous and they're also smarter, then I'll be more worried about it. And that might happen fast in line with the scaling laws, and also kind of misuse of the models. And so that may be distributed misuse, proliferation. And so that That points towards, hey, we have to be careful in how we do this. We, you know, we have to make sure that we, you know, have the right checkpoints, measure for the right risks, don't go too fast. And, you know, that's something that's important to us. I can talk about our, like, you know, responsible scaling plan, which is how we continue to scale the models while attending to these risks. And then on the other side of it, there's something which, you know, I think Noah has written a lot about, which is the, you know, the renewed competition between US and China. Like, you know, I read your article on Cold War II, you know, as a descriptive picture of what's happening. You know, I think it's really correct whether we want it to happen or not. I don't know. And it may be irrelevant. It seems to me that it's happening. And if we take seriously this hypothesis about how powerful the AI models are going to be, you know, they're really powerful enough to shift, perhaps single-handedly shift, you know, the balance of power on the international stage. And then we have to ask questions about, you know, after models of this scale are built, is it democracies or autocracies that are going to be, you know, that are going to, you know, win on the world stage? And, you know, I think we shouldn't just worry about the safety risks of AI, although I'm very, very worried about those. You know, we should also worry about, you know, assuming we get it, Assuming we get it all right, that we don't have to worry about, you know, about kind of AIs themselves or about terrorists or kind of small proliferative actors misusing AI, then, you know, we need to make sure, I think it's very important that, you know, that certain values survive or even triumph. Like, I think an AGI-enabled autocracy sounds like a really, really terrifying thing if you think it through, and it's something we don't want. It's something we really don't want. And so, we need to do both. There's often tension between the two. I think there are a number of policies that kind of help with both, and then some where there's a tension between the two. One of the policies I really like is what the US has done with chips and semiconductor equipment. I think holding back the autocracies has been an effective strategy because it does two things. One, it gives us an advantage, but it gives us more time to attend to the risks. It basically gives us some breathing room, right? Where there's otherwise a hard trade-off between these things, it gives us some breathing room. I have a feeling while, you know, in addition to companies competing with each other, or sorry, in addition to countries competing with each other, companies compete with each other. Companies, if the situation gets dire enough, can be brought under, you know, a common legal framework, right? We can argue about what regulation is appropriate. If evidence really started to emerge, where I think there is some now, and there may be much, much more in the future, that these things are dangerous, either in the misuse or autonomy direction, the coordination problem can be solved, right? That's what government is for. That's what regulation can be for. But the international coordination problem is extremely difficult to solve. We live basically, the international world is this Hobbesian anarchy. Hopefully, we could sign disarmament treaties. You know, we should definitely try for cooperation, but there's no mechanism to enforce it. And so, you know, I would favor trying even while believing that, you know, maybe our odds of success are not that great.

All right. So I'd like to shift gears a little bit to talk about, you know, the impact on labor, which is obviously another thing everyone likes to talk about with regards to AI. Erik Brynjolfsson has this thesis, and I pretty much, you know, kind of agree that generative AI, at least so far, compresses skill differentials. So that when we see that with, you know, GitHub Copilot, or, you know, call centres, or college essay writing, or any of these other standard tasks, you know, the first-pass tasks to which AI has been applied, we see that the people who are bad at it do much better. And the people who are good at it do a little, a bit better. And then the people who are great at it don't do better. And so what you end up getting is the skill of the top people becomes less valuable because the bottom people can compete more with the top people. And to me, that was a hopeful finding because it meant that the age where only— a little bit like when factory workers were able to compete with artisans back in the days of garment manufacturing. the early Industrial Revolution. I said, okay, well, it's a machine tool for the mind. It's like, now you can just have some schlub from a village in North England make cloth that's as good as the best artisan, or at least 90% as good for 10% of the price. And so then you, at that point, you compress the skill return distribution, and then maybe that could fight inequality. Do you have any thoughts on this thesis?

Yeah. So I think in terms of, again, I always separate things out into, you know, kind of eras. You know, I think in terms of what we're seeing now, my perspective absolutely matches that. You know, it's just interesting to see, for example, as the coding models get better, that, you know, some of the most skilled programmers that I've worked with in many previous models said, you know, this just doesn't help me very much. Like, this isn't very This isn't very useful to me. Now, finally, occasionally, say with Claude 3.5 Sonnet, future models probably will also be true of other models that come from other companies. They're starting to occasionally find more use of the models. But I definitely think it's the case. If you look at something like GitHub Copilot, it's been a leveler. And I also agree that it's a good thing in that if we look at the era of the internet, it's had— I mean, you guys would know more than I would, but it has this— It's had this kind of aggregating effect where it's like everything becomes global, news, music, writers, and then there become these very large returns to superstars, and that kind of drives growth, but it also drives a lot of inequality. And so I think it's a good effect that there's this leveling. Unfortunately, again, as the scaling laws continue, That may only be one era. There may be another era where the AI models start to do what kind of everyone can do. Actually, then they may be sort of a leveler as well. And actually, I actually agree with something I think you said recently, Noah, which is that comparative advantage is still going to be really important. So I think even if we had our, you know, our like, you know, if all, if AIs were much better than humans at coding, or even better than humans at biology, it's just surprising. Even if some small fraction of the task continues to be done by humans, it's just remarkable how adaptable humans and the economy are at reorienting themselves around the part of the task that humans can still do. Even if 90% of the code is written by AI, humans will get really good at the other 10%. And then even if 100% Write it. Well, you're still speccing out the design. You're still connecting it to everything else. You're still writing the product. You're still— there's surprisingly many things. And so I think comparative advantage is going to endure longer than people think, even in these crazy scaling worlds. I don't think it'll endure forever. Again, if it goes on long enough and if some complementary things are built, there's no reason You know, that it can endure or be a significant factor forever, but I think it'll go on a lot longer than people think.

We'll continue our interview in a moment after a word from our sponsor. Hey everyone, Erik here. In this environment, founders need to become profitable faster and do more with smaller teams, especially when it comes to engineering. That's why Sean Lanahan started Squad, a specialized global talent firm for top engineers that will seamlessly integrate with your org. Squad offers rigorously vetted top 1% talent that will actually work hard for you every day. Their engineers work in your time zone, follow your processes, and use your tools. Squad has frontend engineers excelling in TypeScript, React, and Next.js ready to onboard to your team today. For backend, Squad engineers are experts at Node.js, Python, Java, and a range of other languages and frameworks. While it may cost more than a freelancer on Upwork, Upwork billing you for 40 hours, but working only 2. Squad offers premium quality at a fraction of the typical cost without the headache of assessing for skills and culture fit. Squad takes care of sourcing, legal compliance, and local HR for global talent. Increase your velocity without amping up burn. Visit choosquad.com and mention Turpentine to skip the waitlist.

Also, you know, one thing I probably could have been clear about in my post about comparative advantage that I wrote, It got a lot of attention, but I think one thing I could have made clearer was that in a world where the, the, the sort of upstream constraints on AI are the same as the upstream constraints on humans, then we're in trouble compared to Vannevar. No, we're, we're kind of in trouble because both are fighting for energy. You know, simple explanation, if data centers take away energy from growing food and food gets more expensive, then people get mad. And that's bad for humans. But In the world where the constraint, where the upstream constraints are different. So I think most people think about the capabilities, the downstream, you know, complementarity and substitutability. But I was talking about the, the, the, the complementarity and substitutability of production factors. So for example, if AI, if most of the energy, if most of the effort in making AI, if the biggest bottleneck is not energy, but ability to manufacture sufficient compute, then I think we're okay. Because then it's basically like, and my example was there's one of Marc Andreessen, and then there's one of Marc Andreessen's executive assistant, there's only one of each, there's a different idiosyncratic constraint on those two, you know, the fact that those two factors of production, if you will. And so similarly, I think that comparative advantage is likely to be better for us if the bottleneck in AI resource-wise is more about compute than about energy. And I should have been a little more explicit about that, but do you basically agree with that? Like, what do you think about that?

Yeah, I think that makes sense. You're sort of saying, like, if the, you know, just to use a somewhat ridiculous analogy, if the AIs are Cylons and the process of making and growing them is very similar to that of humans, then we're in trouble. But if it's a server farm somewhere where the inputs are totally different, then we're fine. I think I I haven't thought deeply about that. That sounds pretty plausible. That sounds pretty plausible to me, at least at first blush. Again, if we're in a world where AI is reshaping the world and the whole way that the economy is designed, that's kind of like at the end of the scaling curve, then we could be talking about something different. But no, if the normal rules of economics kind of apply, You know, I think they will for a while, then, then this, I mean, this sounds very sensible.

Thanks. Also, you know, as for the sort of hyperscaling world now, now, by the way, the stuff that we're talking about, I can show you paper where basically it shows all these models and, you know, our intuition basically pretty closely matches what you get when you write down a simple model of this stuff. And yes, there is the world in which scaling capabilities, the world does exist in these models and it kind of works kind of like you think. But my, My other question is, does it make sense to think about a world of incredible abundance where AI is so good that it gives us amazing biology and amazing manufacturing and amazing every single thing that we could want just gets 10x better, 100x better, you know, whatever. And yet human beings themselves are impoverished. What are the cases in which we, we have to worry about a world of radical abundance in which humans are utterly impoverished?

Yeah, let me maybe take the 2 parts of those one by one, because the world of radical abundance, I mean, that's what we're hoping for, right? I talk a lot about the risks, again, both the autonomy and the kind of misuse in national security. And sometimes people get from that, get the impression that it's like, oh, I'm like a doomer, or I think all these bad things are going to happen. No, my perspective is more that like, you know, by default, I think really great things are going to happen. And I'm obsessed with the bad things because they're the only thing standing between us and all these great things happening, right? They're the only thing that can derail it. So I'm like, I'm just totally obsessed with like, you know, finding the blockers and like destroying them. So first of all, yeah, the world of radical abundance, like I think a lot about biology because I used to be a biologist. And I think we're really underestimating what is possible when people look at AI and biology. Certainly, 10 years ago when I was in the field, the attitude was, look, the data we get from biology is of questionable quality. There's only so much of it we can get. The experiments are often confounded. Sure, it's great to have more data analysis and big data and AI, but it plays at most a supportive role. That's maybe changed a little with AlphaFold. But my picture, of course, and I don't know, maybe this This goes a little bit in what you think of as replacement pitfall, but is the AI models kind of serving the role of a biologist or a co-biologist? If we think about what's really advanced biology, like, it's really disproportionately a few technologies that kind of power everything, right? So, like, genome sequencing, right? Just the ability to, like, read the genome. Fundamental to most of modern biology, right? More recently, CRISPR, the ability to edit the genome, right? Fundamental to many experiments where we want to intervene in animal experiments, starting to become important to pharmaceuticals and curing diseases. Although there's a while to go and it needs to be more reliable. There are a lot of other techniques that are needed. I think if we get AI right, it could increase by 10x, maybe 100x, the rate at which we invent these discoveries. So, if you look at CRISPR, it basically, the assemblage involved comes from the bacterial immune system. And this was known for like 30 years, but connecting it to the idea that you could use gene editing, how you could do it, that you needed to add these other elements to it, It, it, you know, there was no reason you couldn't have done it 30 years ago. It took 30 years to invent it. And so I, I think if we can, you know, greatly increase the rate at which these discoveries happen, we'll also greatly increase the rate at which we cure disease. And, you know, the way, the way I think about it is, you know, could we, could we have like a kind of a compressed 21st century, right? Could we make all the progress in biology that we were gonna make in the 21st century using AI by speeding things up 10x. And if you think of all the progress we made in biology in the 20th century, and then compress that into 5 or 10 years, to me, that's the upside. I think that might literally be possible. And diseases that have been with us for millennia, we could cure. And of course, that would do great things to productivity, would increase the size of the pie, extend the human lifespan. So all that's great. That's what I'm hoping for. That's what hopefully we're all hoping for. Humans ending up impoverished. I mean, I guess the easiest bad story would be, okay, well, all this huge wealth is created, right? We get like double-digit GDP growth in the developed world, but it doesn't, the returns disproportionately go to, you know, the companies that are developing it, the employees of the companies, complementary assets that need to be produced. And then, you know, the average person somehow ends up in a situation where they don't share in it. And maybe even more, those in the developing world, which I think is actually a more plausible worry, kind of end up getting left out of it. The extent to which, when you think about people being left out of a growing economy, I think that's happened a lot more between countries than it has happened within countries, although it's also happened within countries. So, I guess that would be the easiest way to do it. Within countries also, countries have a history of redistributing, but we haven't been as good at, say, redistributing to sub-Saharan Africa as we have within the United States, because there's no—

Yeah.

governmental entity that has the jurisdiction to do that.

That's an aspect of the problem I think people don't really think about a lot is effect on other countries. Like, but possibly that, you know, that would take us too far afield. Erik, do you have, do you have more?

I have one question and it's basically on the 2 sides of the risk coin. So some people say that because we aren't able to really understand how the human brain works, to bring it back to the beginning of our conversation, that we shouldn't be worried about AI sort of developing consciousness or agency, because we can't even understand our own programming. How can we, you know, program or understand AI? That's, that's on one side of the safety concern. And then on the flip side of that, sort of the AI-powered surveillance states, I'm curious if we could speculate more on the risks there. It seems to be the opposite risk profile around like, maybe a question asked that is like, what, how do we think AI might develop in China or Russia or Iran?

Yeah. Okay. So I think on the safety side, I mean, it's several, again, several different things. One, I think the fact that we don't understand how human brains work, I don't think that alone should give us that much comfort about AI systems, right? There are some pretty bad humans out there, right? If I look at a 2-year-old child and I'm like, is this going to be the next Gandhi or the next Hitler or something in between, we don't really have any way to predict that, right? I mean, maybe make some guesses, but it's just really hard. So you kind of don't know what you're getting. And I think that's true of AI systems as well. On the other hand, it doesn't mean— it should tell us that it's like, it doesn't mean— it doesn't imply that the systems are necessarily impossible to control, right? Like, You know, we have ways of educating humans. We have ways of creating a balance of power between them. So it's not an entirely reassuring message, but it's not an entirely frightening message either, right? If anything, it suggests that, you know, some of the old problems that, you know, we've had with just humans being aligned, you know, two sides of the coin, we may have the same situation with AI. That said, if we can understand what's going on inside, I think that improves the situation quite a lot. And we've had a team at Anthropic that since the beginning has worked on interpretability, looking inside the model to try and understand why it does what it does. That's something that I know from experience is hard to do from the human brain. It's much easier to do in software. There's still the algorithmic, the computational complexity of it, but we can face that complexity without worrying about how to reach into the goop of a brain without killing the organism involved. So we should do as much of that as we can. And the ideal is if we do that really well, the situation might be better than it is with human alignment and our ability to predict humans. You know, on the other hand, it's like, we know a lot about humans. We know a lot less about AI. So like, if I were to make the, the, the, the bear case, it would be, and we already see this when we deploy AI systems, they're alien. They make mistakes that sometimes are hard to understand. They, they get things right that no human would get right, but sometimes they make mistakes that no humans would make. And I think that's a reason to worry from a safety perspective. So I don't know. I think I and Anthropic often have these kind of balanced perspectives on things. And that's our view on safety. On what's going to happen in Russia and China, I mean, I would go back to the export controls, right? We have no available hard governance mechanism for controlling what happens there. As far as we have all these debates on safety issues here, I'm happy to see that there are some people in China who think about the same issues. But we have no mechanism for making sure that they do it right. There's no mechanism for democratic accountability. Authoritarian governments have a history of being more reckless than democratic governments. So I think it's generally bad. I think it's generally bad news when they're ahead for a number of reasons and from a number of different perspectives. You know, to some extent, we face a tension between speed up to beat them and think about safety. But there are some things we can do that kind of are good from both perspectives. And I think we should try to find more of those things and do them.

Well, we have you for 2 more minutes. SB 1047, Elon just endorsed it. And any quick thoughts? Some people are concerned that it could create a path dependency in how we think about regulation. What are your quick thoughts on it?

Yeah, so, you know, Anthropic wrote 2 letters on this. First was to the original bill where we had some concerns that it was a bit too heavy-handed. You know, the bill sponsors actually addressed many, though not all of those concerns, you know, maybe 60% of them or so. And after the changes, we became substantially more positive. You know, I think we couched our view in terms of, analysis. We have a lot of experience, you know, running safety processes and testing models for safety. And so we felt we could be a more useful actor in the ecosystem by kind of providing information, by informing, than kind of, you know, picking a side and like, you know, trying to beat up the other side or, you know, whatever political coalitions do. But, but, you know, I think we were, I think, after the changes, more positive than negative on the bill. And, you know, I think, I think, I think overall in its current form to our, you know, tying kind of to our our best ability to determine, we like it. Our concern with the original version of the bill was with something called pre-harm enforcement. So the way the bill works, it's very similar to kind of RSPs, the responsible scaling plans, which are this voluntary mechanism that we and OpenAI and Google and others have developed, which says, you know, you have to run these every time you make a new, more powerful model. You have to run these tests, tests for autonomous misbehavior, tests for you know, misuse for biological weapons, for cyberattacks, for nuclear information. And so if you were to turn that into a law, there's 2 ways you could do it. One is you have a government department and the government department is like, you know, these are the tests you have to run. These are the safeguards you have to do if the models are smart enough to pass the tests. And, you know, there's the kind of an administrative state that writes all of this. And our concern there was, hey, a lot of these tests are very new and almost all these catastrophes haven't happened yet. So I think, you know, somewhat in line with those who were concerned, we said, hey, this could really go wrong. Like, you know, could these tests end up being dumb? Could they, you know, in a more sinister way, kind of, you know, be kind of repurposed for political— The other way to do it, which you thought was more elegant and might be a better way to start for this kind of rapidly developing field where we think there are going to be these risks soon, they're coming at us fast, but they haven't happened yet, is what we call deterrents, which says, hey, everyone's got to write out their plan, their safety and security plan. Everyone decides for themselves how they run their tests. But if something bad happens, and that's not just AI taking over the world, but just could be an ordinary cyberattack, something bad happens, then a court goes and they look at your plan and they say, well, was this a good plan? Was it— person reasonably believe that you took all the measures that you could have taken to prevent the catastrophe? And then the hope is that there's this kind of upward race where companies compete not to be the slowest zebra, right? To prevent catastrophes and not to be held, not to be the ones to be held liable for the catastrophes that happen. So opinions differ. You know, many, many people obviously are still against the new bill. And, you know, I can, I can understand where they're coming from. It's a new technology. It hasn't been, you know, we haven't seen these catastrophes. It hasn't been regulated before. But we felt that it kind of struck the right balance. You know, time will tell if it passes or even if it doesn't pass, probably it's not the last we'll see of regulatory ideas like this. And, you know, that was another reason why we thought, you know, we could best contribute to the conversation by saying, hey, here are the good things about it. Here are the bad things about it. We do think that as amended, the good things outweigh the bad things. This is an ongoing conversation.

And this kind of bill wouldn't make you move operations out of California?

Yeah, that was the thing that was most perplexing to me. There were some companies talking about moving operations out of California. In fact, the bill applies to doing business in California or deploying models in California. Moving your corporate headquarters, for better or worse, change your status vis-à-vis the bill. So, you know, honestly, I'm surprised the opponents, you know, didn't say, oh, this is scary because it applies You know, because it, because it applies anywhere. And, you know, that was, that was a reason why we really wanted to make sure that outweigh the costs, which we, which we do feel. Um, but anything about, you know, oh, we're moving our headquarters out of California, this is going to make us— that's just theater. That's just negotiating leverage. Like, it, it bears no relationship to what— to the actual content of the bill.

Got it. Actually, can I ask one more question? Do we have time for me to ask one more quick question?

Yeah, I, I'm, I'm happy to. I'm happy to keep talking. You know, at some point Sasha's gonna gonna interrupt me and tell me to stop, but I'm just happy to keep talking until she does that.

Well, I have one more question, which is in terms of alignment, most people think about how do we make an AI-based world in which humans do well, in which things are good for humans. I think a lot about how do we make an AI-based world in which things are good for rabbits, because I really like bunnies and they've got kind of a raw deal from the universe, but Like, a rabbit in the wild will live only 1.5 years on average, but then a rabbit in captivity will live comfortably for 10 to 14 years. And so how do we, other than making rabbits the official sort of animal of Anthropic, which is a thing I definitely think should be done, Kipley will work with me on this, other than that, how can we create a world where things are good for bunnies in a world of super AI?

Yes. So I don't know, I thought about some similar things. So I have a, I have a horse, and the way I would describe horses is that they're kind of like gigantic rabbits in their behavior and just in their, you know, they're like prey animals, but like, you really want to protect them. Like, they really create this protective instinct in you, and, you know, you want to make sure that, like, they have a good life.

I don't know.

I guess the way I would think about it is if we thought about the AI correctly, like, hopefully, you know, we've created some general principle of, you know, you should be kind to, like, less powerful beings. And so it means that, you know, humans should be kind to rabbits and horses, and AIs should be kind to humans. And hopefully also the AIs would be themselves kind to rabbits and horses as a generalization of that, right? That we would have some— the AI, you know, the combined AI-human ideology or picture of the world would be based on some kind of— some kind of, you know, protection of these, these creatures. I, I kind of suspect that to, to, to, you know, if we ever built benevolent, powerful AIs, they might look at us a little as, as we look at the rabbits and horses, that we're a little, we're a little, we're a little helpless and, and, you know, kind of cute and need to be protected.

All right. Well, I, I, I definitely think more attention needs to be paid to the rabbit alignment problem. Actually, that, that was, that was one of the, the only science fiction short stories I ever wrote was about, you know, a, a future where AI just decided it had to protect bunnies and other small fluffy animals. And would occasionally like blast any humans who had thought might potentially threaten them.

Yeah, no, no, no, no, you're too close to the rabbit. Like, this is your last warning. Space laser's coming for you.

That happened. That did happen. Well, it was great to see you again.

Yeah, really great to see you again. I've really enjoyed your, your essays, particularly on, you know, just, just kind of like the international and national security situation. I basically have have the same view. And in fact, think that these issues are even more important because if I'm right about the scaling laws, there's this enormous technologically disruptive revolution that's happening at the same time. It feels a little bit like the atomic bomb being built during World War II. I mean, maybe that's an overly dramatic analogy, but I think it creates— It highlights those issues even more. and creates some tensions that we have to navigate. So I'm very concerned about it, and your analysis has been helpful to me in thinking the issues through.

I appreciate that so much.

Hey everyone, Erik here. At Turpentine, we're building the first media outlet for tech people by tech people. We're the network behind the show you're listening to right now. We have a slate of hit shows across a range of topics and industries. our AI and investing cluster of podcasts to shows that drive the conversation in tech with the most interesting thinkers, founders, investors, and influencers like Econ 102 with Noah Smith. We're launching new shows every week and we're looking for industry-leading sponsors. If you think that might be you and your company, email me at erik@turpentine.co. That's E-R-I-K at turpentine.co, and let's partner together.