Carl Benedikt Frey, "How Progress Ends: Technology, Innovation, and the Fate of Nations" (Princeton UP, 2025)

This episode is brought to you by Facebook. So you were scrolling on Marketplace, and there it was, the bike you'd been searching for. You sent a message, and it turned out the seller was super chatty, kind of funny, and an avid cyclist. The next thing you know, you're in a cycling crew. Well, a community cycling group. The thing about Facebook, you might find more than what you're looking for. From a browse to a bike ride, this summer find more on Facebook.

Hey, Washington, D.C.! Class it up with Crocs! You know back to school is coming in fast, so why wait to find your new fave footwear? Step into a local Crocs store and step into your new look. Try it. Style it. Make it yours. Because the right pair doesn't just show up, it shows off. First day fits handled. Walk out ready for whatever's next. Visit your nearest Crocs store today.

This episode is brought to you by State Farm. Listening to this podcast instead of doom scrolling? Smart move. Another smart move? Getting help from one of State Farm's 19,000 local agents when you choose to bundle home and auto. Bundling, just another way to save with the Personal Price Plan. Prices are based on rating plans that vary by state. Coverage options are selected by the customer. Availability, amount of discounts and savings, and eligibility vary by state.

Welcome to the New Books Network. Welcome to the New Books Network. I'm Leo Bader, and my guest today is Carl Benedikt Frey. Carl is the Dieter Schwarz Associate Professor of AI and Work at the Oxford Internet Institute and a fellow of Mansfield College, University of Oxford. His new book, How Progress Ends: Technology, Innovation, and the Fate of Nations, traces a sweeping history of technological progress, from ancient China to the current advances in AI, to illuminate how new innovation can be translated into lasting growth. Carl has been working on the relationship between employment and automatization for a long time, and How Progress Ends puts this question into an exhaustive historical context. So Carl is the perfect person to tell us if we can still expect to have a job in the coming years. Carl, thanks for joining me.

My pleasure.

So your book hinges on the observation that that both decentralized and centralized modes of economic organization can drive progress at different points. Why is this and how should we orient how we think about technology and progress knowing this?

So I think for most people engaged in science, technology, and innovation, we know that there is a time to explore and we know there's a time to get things done. And execute. And so if you write a book, for example, you spend a lot of time initially talking to people, reading broadly, trying to get input to figure out what it is that you want to do. But then at some point when that has been crystallized, you need to get your act together and write the thing down and execute. And most tasks, whether in academia or in business, are like that. So no factory could function if every worker questioned every procedure, right? But very few new ideas would emerge without questioning. And so there is this tension between exploration and exploration, between innovation and execution, if you like. And when something, when the objective is clear, it's much easier to execute it from the top down. And so the Manhattan Project delivered the first nuclear bomb. The Apollo Project put a man on the moon. the Rad Lab advanced radar technology. But that's quite different from, let's say, ARPANET, where there was no plan to develop the modern internet. That hinged on a much more unpredictable chain of exploratory efforts. And so that happened in a much more decentralized manner.

So we have to start with decentralization. But once we land upon an idea that we know is worth exploiting, it needs to be more centralized. We can't keep questioning.

That's right. And then we do decentralization again to go on to the next thing.

So what are the implications of this for how we should think about the state's relationship to progress, which I assume needs to be more flexible than, than many politicians or pundits might Conceptualize it?

So the state can take on a role both when it comes to exploration and exploitation. So as you may have noted in my earlier answer, the Rad Lab, Manhattan Project, the Apollo Project, these were state initiatives, right? And they were more of centralized nature and were more about executing based on existing findings and existing technology. And so when a project can be clearly defined, the state can play a reasonably prominent role in executing from the top down. ARPANET was also state-funded, but it depended on a decentralized network of universities to draw talent from. And obviously when you give people more autonomy, you need to be sure that you're hiring the right people. And the pool of talent that the United States has had access to through its decentralized network of universities was a prerequisite for that. And so much of the university system was also in part seeded through state grants back in the 19th century. So the state needs to take on very different roles when it comes to supporting innovation and technological development depending on what it is, what it is it's trying to achieve.

And do you see consistencies across the historical record and what kinds of approaches or what kinds of instruments states have to promote this exploratory phase? And maybe conversely, instances in which states do one or two very significant things that, that prevent this?

So clearly having an open space for scientists to interact and debate and disseminate their findings is important. And that's something we do see throughout history, that that is quite important to frontier discovery. But again, it depends a little bit on context as well, right? And so if you go back to the 19th century, competition between states played a very major role in pushing for reform that encouraged exploration. And that, for example, consisted of getting rid of gilded regulations, And so the Prussian defeat to Napoleon in 1806 paved the way for the Stein-Hardenberg reforms that made that possible. And Russian losses at the Crimean War paved way for the abolishment of serfdom. And there are many other examples. So Competition, I think, has often paved the way for the kind of liberalizing reforms we've seen in the past. And at times it's also come from a mobilized society. And so there was a push for a more meritocratic civil service in the United States. In the late 19th century, that more meritocratic civil service then went on to institute the Sherman Antitrust Act and a range of other initiatives that gave the state tools to enforce competition that in turn enabled the US later on, for example, to file lawsuits against IBM forcing it to unbundle hardware and software, creating space for new firms to enter like Microsoft. Similarly, the breakup of AT&T, I think, was very critical to the development of the modern internet because it meant that when the National Science Foundation released ARPANET, made it publicly available, It wouldn't be handed over to one carrier service, which could have bottlenecked it in its boardroom if it saw no immediate economic benefit to it. And the economic benefits of the internet were far from clear at that point. So I think those are some lessons that we can draw.

And when we draw this into the present day where a lot of economic growth is concentrated in big multinationals, does that change sort of the nature of competition and does that create any more of an opposition, do you think, between states and companies? Because companies are no longer so bound within one state, even if they're headquartered in one.

So there's nothing inherently bad with large corporations. And so you need large corporations to leverage technologies like factories, bring down costs and make sure that new products are widely accessible to the population. And so there is no people's car, there's no T-Ford without the Ford Motor Company and the assembly line and some other technologies that gave rise to mass production. That said, consolidation can inhibit innovation, and we've seen with various antitrust initiatives that subsequently you tend to see an upsurge in patenting, you tend to see an upsurge in new job creation, you tend to see an upsurge in entry. But on the downside, you may also jeopardize, uh, the return to being a first mover. And so you may, uh, cap the expected return entrepreneurs have to pursuing, uh, productive, uh, innovation. So a trade-off does exist there, uh, and, uh, I think, you know, part of the reason, uh, that That we've seen a lot of innovation happening after antitrust cases is that for the most part, first movers are not that concerned about a situation where there might be a monopolist, which is usually very far off when you set up. a business. Now, if you go back just, you know, a couple of years and you ask yourself, you look at the tech giants we see today and you ask yourself, do they have a monopoly? Well, they look a lot safer back then than they do now since the release of ChatGPT. And that was indeed acknowledged in the recent lawsuit. against Google. And so unpredictable changes in technology matters as much, if not even more, than antitrust, and no corporation can be safe. At the same time, it's not a very good system if you're dependent on incumbents making mistakes and missing new technologies. And so we have been seeing in recent years an upsurge in outright killer acquisitions whereby firms buy up competitors just to close them down. We have seen a revolving door between the US Patent and Trademark Office and incumbents whereby examiners grant low-quality patents and then go on to work at some of these large technology companies, which raises barriers to entry for new smaller firms entering the market. So yes, I do think broadly speaking that there are concerns about the rising market concentration that we have been seeing over the past 2 decades or so.

I'm curious about whether you see a, so to say, correct turning point for moving from exploration to exploitation, if there's any sort of consistency across history in when saying, okay, let's, let's stop looking at all these different ideas and let's, let's go all in on this one or these 2 ideas is the right decision or in other cases the wrong decision.

I think it's very hard to get that timing right. And I think in the 1970s, the mass production system essentially peters out, growth stagnates all across Europe, the United States, and behind the Iron Curtain as well. And so it's clearly the case that We didn't just have something to replace that growth with at that point. That eventually became the computer. And as I argue in the book, a key reason that that happened in the United States rather than in Japan, which many predicted would overtake the United States in the 1980s, had to do in part with competition policy, in part with the federal system permitting differences in regulation. And so in California, famously, non-compete clauses are void. And that meant that when engineers left Shockley Semiconductor to set up Fairchild, they could do that. And when 2 of them left Fairchild to set up Intel, they could do that. And that's contributed to the constant renewal. of Silicon Valley in this period, but there was nothing planned. And clearly when it comes to antitrust initiatives, you don't want to apply antitrust arbitrarily in the hope that by hurting some incumbent, you might get some entry. You do want it to check. Uh, uh, anti-competitive, uh, behavior. And so I think it's very hard to get the timing right. And so I think what you need to be focused on is to make sure that incumbents are not competing unfairly and then let the best technologies win in the marketplace.

This episode is brought to you by Facebook. So you were scrolling on Marketplace and there it was, the bike you'd been searching Or you sent a message and it turned out the seller was super chatty, kind of funny, and an avid cyclist. The next thing you know, you're in a cycling crew. Well, a community cycling group. The thing about Facebook, you might find more than what you're looking for. From a browse to a bike ride, this summer find more on Facebook.

Hey, Washington, D.C., class it up with Crocs. You know back to school is coming in fast. So why wait to find your new fave footwear? Step into a local Crocs store and step into your new look. Try it, style it, make it yours. Because the right pair doesn't just show up, it shows off. First day fits handled. Walk out ready for whatever's next. Visit your nearest Crocs store today.

This episode is brought to you by State Farm. Listening to this podcast instead of doom scrolling?

Smart move.

Another smart move? Getting help from one of State Farm's 19,000 local agents when you choose to bundle home and auto. Bundling, just another way to save with the Personal Price Plan. Prices are based on rating plans that vary by state. Coverage options are selected by the customer. Availability, amount of discounts and savings, and eligibility vary by state.

You mentioned antitrust as one big one, but what other institutional structures can be built to ensure that in, in one state or in one society, there's space for both exploration of some technologies and some simultaneously exploitation of other technologies that are further along and to give both of them the space and the resources that they need to continue.

So I think it's just as you said, it's about creating that space and then you can plan what, you know, all these localities or firms or whatever it is that you're thinking about do with that space, right? And so One of the key reasons that Soviet Union failed to adjust to the computer revolution is that it didn't have any space for exploration. So the entire economy was more or less managed from Moscow, which managed steel and railroads and everything centrally. It's quite different from China where provincial leaders had much more autonomy. to pursue different strategies to attract foreign investment, to spur growth after the opening in '78. And so I think it's more about having that space, having that diversity in regulation, enabled and then leaving it to local authorities and firms to figure out what's the best path forward within that space.

If we think about developing economies that maybe are not as interested in being at the forefront of the next great technological innovation, but more in catching up on, on what there already is, should they then Would it make sense to adopt a more centralized economy that can then spur very quick adoption of already existing technologies?

It might do, but let me qualify that in a number of ways. So first of all, industrial policy has been pursued all around the world with very mixed results, right? So Latin America pursued industrial policy. It didn't work out for the most part particularly well. And you can look at Korea and Japan and to some degree China, perhaps to a lesser degree, and say that here, those cases, industrial policy worked quite well. And I think key difference there is that those places have a long history of having meritocratic states and relatively strong states that can enforce, for example, export discipline and make sure that firms that— only firms that are actually competitive in global markets are allowed continued support. They also have more capacity to actually get things done, roll out infrastructure, provide public services, and They have a capacity, quite importantly, to check vested interests. And so if you have some of those 3 conditions, then I think you're in a better place to pursue industrial policy. But, you know, if this was easy to achieve, most places would've done it. And the fact that That's mostly a few places in East Asia and to some degree Europe in the post-war period and in the 19th century catching up to Britain that have done this suggests that it's not that easy. And in many cases those preconditions are lacking.

What relationship do you see between the public and private sector across the historical record, both in the exploration and exploitation phase? Mostly a productive cooperative one, or can there be conflicts that hinder innovation?

Well, there can absolutely be conflicts and collusion, and the more you support one firm, the more invested you are in one company, the stronger the incentive there is to undermine competitive forces that might displace that firm. And that's why it's so worrying some of the tendencies that we're now seeing in the US and the government taking shares in Intel and US Steel and planning to do so in a range of industries. And even in China, most progress, innovation, productivity has become from private foreign invested companies. And In the cases where it's done well, industrial policy has been beneficial. It's been mostly local initiatives and local authorities having firms competing against each other within this vast market in China. But if you take the United States in the 19th century, It didn't have a lot of state capacity. It opted for small government, and so it needed to depend much more on private enterprise, on private initiatives to roll out railroads, for example. And so that turned out to work reasonably well in terms of letting private firms develop the railroad network. But it is also true that as a consequence of that, there were a number of scandals, there were a number of corruption cases, and that eventually paved the way for a more meritocratic civil service through the Pendleton Act. Which then began to take on more tasks itself. But yes, that collaboration between private enterprise and firms does still exist in the United States and exists in many other economies to varying degree. And so it's easy to understand why the state wants to outsource some activities, but it's also then hard to sometimes monitor performance and make sure that competitive forces remain intact. And so in Italy, for example, in the late 19th century, early 20th century, when we wanted to develop not just its own railroads, but also its own railroad industry developing locomotives, for example, it had procurement practices which actually permitted also foreign firms to compete for those contracts, which put pressure on Italian firms as well, and I think contributed to keeping them in check.

What implications do you think your argument has for debates around what kinds of political systems are best for generating economic growth? If we think about sort of the caricatures, democracies would be less decisive and authoritarian systems presumably would be able to better exploit innovations once they have been produced somewhere. But should we maybe be a bit more careful in drawing immediate conclusions here, given that, as you say, there are these 2 distinct phases of growth?

Well, so I think first of all, there's compelling evidence that on average democracies grow faster than non-democracies. But if you look at autocracies, there are also different kinds of autocracies. So Singapore and Zimbabwe there is a lot of shows that there is a lot of variation within that category. And so as a general rule, more institutionalized autocracies where you have, for example, a state bureaucracy where meritocratically recruited civil servants is more able to get things done than a personal autocracy where everything is down to one person who arbitrarily promotes and selects loyalists to keep the regime floating. And so a meritocratic autocracy, or institutionalized autocracy can roll out often infrastructure more quickly, for example, by overriding vested interests than a place that, or let's say, private property rights are stronger and more secure. But also in an autocracy, priorities can change and they can change quite quickly. And so if you're a firm in an autocracy, you ideally want a seat at the table and you want to invest more in building up political capital to shape the priorities and have some say. And that means that you are diverting resources away from innovation, exploration towards lobbying. And political capital. Now, you have that to some degree in democracies as well, to be sure. And spending on lobbying in the United States has basically doubled since 1990. But you do have the courts, you do have reasonably secure private property rights, and that means that your less dependent on changing political priorities than you are, for example, in an autocracy. So I think it's fair to say that the democracies for the most part have some pretty strong advantages. We can sometimes look to some autocracies that are doing very well in building quickly and catching up and fool ourselves into thinking that we will soon be overtaken. But the task at the frontier is quite different to the task of catching up.

Absolutely. Let's end by talking about AI for a little bit. How does it compare with other historical transformative technologies like the steam engine or the printing press? And does the fact that it maybe has some potential to kind of intrinsically generate knowledge mean that it could potentially start to bridge this divide between decentralized exploration and centralized exploitation?

Well, potentially could is doing some work there. And so, yes, it potentially could, but it's not doing that right. And so, yes, AI can be useful tool for science and innovation. But I struggle to see how it's currently at least anywhere near as transformative as the internet and the personal computer, right? Those were the ultimate research tools connecting the best scientists and inventors around the world, streamlining the research process, Giving most people access to the world's stored knowledge in their pockets. And what we get is a sort of decade streak of productivity growth, which then eventually in the mid-2000s peter out. And so I don't think AI is on par with that yet. It could be, But it's by no means clear, and it's by no means clear that the models we have now, which are basically large language models that are unable to be out in the real world and learn from experience and learn continually and basically are rewarded by imitating human behavior rather than learning will get us there. So I think that's very much an open question. I don't know the answer to whether, you know, which is the right path forward. I don't think anybody actually really knows, although some people think they know. But, you know, in the end of the day, you need somebody to show that something works. And that's the beauty of decentralized technology. exploration. And so I don't really think that AI is changing the equation. We still need exploration. We still need human ingenuity, at least for now. And I'm open to the possibility that that might change in the future, but we're definitely not there yet.

So there's a sense in which it might be sensational as something that's very cool and powerful to see, but in terms of the actual growth and innovation that it's generating at the moment, it's not quite where other technologies have been.

Yeah, I mean, to be clear, suggesting that it needs to push frontier discovery is also quite a high bar. So I think it can be very useful in medical research and in other domains, which, you know, and, you know, useful in screening the literature, et cetera. But, you know, if you had trained an LLM on data available up until 1900 and you would've asked it, will humans ever be able to fly? It would've looked back at the long list of failed attempts. It would Possibly have looked at data on birds to suggest that flight in principle is possible, but it would've concluded that there's no bird that weighs more than 30 pounds that's able to get off the ground. Larger birds like ostriches can't fly. And so just before the Wright brothers took off, it would've no chance to predict that. But clearly it's a useful tool for many other things. And so, you know, I don't want to diminish the technology, but I don't think it's, you know, where the personal computer and the internet is. And it clearly hasn't created this sort of new industries on the scale that the automotive did or electrical industries. Or, uh, even if you sort of take automotive, then, you know, it wasn't just automotive industry, it was a range of supplier industries. Uh, uh, um, um, uh, creating components and a host of industries producing machine tools to make the components, um, et cetera. And so, uh, AI right now is more like a productivity tool, and it may eventually change. It may eventually create new industries and more sustained growth as a result of that. Uh, but we'll have to wait and see.

Well, we should wrap it up there. Carl's new book, How Progress Ends: Technology, Innovation, and the Fate of Nations, can be purchased through the link on the New Books Network page. If you're interested in more, Carl has another book, The Technology Trap: Capital, Labor, and Power in the Age of Automation. Carl, thanks again for joining me.

Thank you very much for having me.

Labor Day savings are happening now at The Home Depot with select appliances starting at $399. Plus, save up to an extra $1,000 and get free delivery on appliance purchases of $998 or more. Get a Whirlpool laundry tower featuring industry-first UV Clean technology designed to reduce bacteria in the wash without fading fabrics. Plus, with great prices at The Home Depot, At The Home Depot, you can save on select appliances designed to make laundry day easier. Shop Labor Day savings at The Home Depot today. Offer valid August 27th through September 16th. Visit your local Lowe's store or online for details.