How Progress Ends with Carl Benedikt Frey

If all we'd done since 1800 was automation, we would have productive agriculture, we would have cheap textiles, but not much else, right? We would not have antibiotics, vaccines, computers, rockets. And so we need to begin to find use cases for AI that are not just about automation if productivity growth from AI is going to be sustained.

Welcome to the Thinkers and Ideas podcast, where we discuss important new books and ideas in business. I am Adam Jobb, Senior Director at the BCG Henderson Institute, and I lead our research on strategy. I'm sitting in for our usual host, Nicky Lang, this week. And it's a great week for me to be here because joining me today is Carl Frey. He is the Dieter Schwarz Associate Professor of AI and Work at the University of Oxford. And in 2019, he authored The Technology Trap, which was selected as a Financial Times Best Book of the Year. Now, Carl has a new book coming out called How Progress Ends: Technology, Innovation, and the Fate of Nations. In the book, he argues that progress throughout history has not just depended on technological innovations, but also on institutions ranging as widely as the Catholic Church, American saloons, and socialist central planning. And hopefully we'll get a chance today to touch on some of that. I really look forward to our discussion, Carl. Thank you so much for joining me.

Thank you very much for having me. It's a real pleasure to be with you.

Great. So let's get into it. And I'd love to start at the title of your book, How Progress Ends, which might be interpreted to imply that you have a rather pessimistic view of our future. Is that so? And if so, why? Why might progress be ending?

So I don't necessarily have a pessimistic view about the future, but a key theme of the book is that we shouldn't take technological progress for granted. If technological progress was inevitable, the first industrial revolution would have happened much earlier in human history. If progress was inevitable, most places and countries around the world would be rich and prosperous today. And if we look at individual nations that have succeeded in technology historically, generally, they weren't technology leaders for very long, whether it's the Song Dynasty in China, whether it's Britain in the Victorian era, whether it's the Dutch Republic before then. And today, the United States remains the undisputed technology leader, But it's also true that productivity in the United States has been disappointing for the past 2 decades or so. It's not just Europe that has a productivity problem. And in fact, if you look to China, which is the country competing with the United States for technological leadership, productivity has been disappointing for the past decade. As well. So to answer your question, it is by no means inevitable that we will face a prolonged period of stagnation. But right now, the technology leader, as well as its peer competitor, are not doing particularly well in productivity terms. And so I think we need to take a step back and ask ourselves why.

Well, you're asking the question for me. Why is that? I mean, I introduced it a little bit. I guess it depends on the relationship between technological innovation and institutions. Could you elaborate on that a little bit?

Sure. So, you know, the competing theories for wealth and poverty around the world can broadly be fit into 3 buckets. One has to do with geography, one has to do with culture, One has to do with institutions. Now, if you take the Soviet Union as an example, the most centralised economy the world has ever seen, it actually performed in pure output per capita terms quite well for a number of decades, particularly through the 1950s and then early 1960s. But as we all know, it eventually stagnated and even collapsed. Now, the geography of the Soviet Union was essentially the same when it grew rapidly as when it collapsed. The culture was certainly different. And the institutions, if anything, actually drifted more towards openness, which you would think should be good for growth. But overall, institutions were relatively stable too. And so what I argue in the book is that in order to understand both the growth spurt and the collapse, You need to understand the interaction between institutions and technology. And the fact that technological progress entails 2 very different pursuits. On the one hand, you have exploration, where you try to come up with new ideas, new technologies, new prototypes. And then you have a scaling phase, where you essentially try to produce what you've conceived at a sufficiently low cost for that to become available. to the bulk of the population. And the institutions that support these 2 processes can be very different. And some countries can be very good at scaling, but not at exploration. And to remain the technology leader for longer periods of time, you need institutional flexibility to move between these 2 phases. That's what the United States has done successfully historically, I argue in the book, but there's no iron law of history that suggests that that will continue to be the case.

So the natural question to me is, what are the systems that are conducive to these 2 modes of exploration and scaling up as you describe them? Like, what institutions are needed to make each of those happen?

So for exploration, you need decentralization. The reason for that is You cannot know upfront which technological pathways would be successful. So you need different people to explore different pathways. So if you take the Soviet Union as one extreme, right? If you were an aircraft engineer and you conceived a new engine, you could go to the Red Army to ask for funding. If they declined, you maybe had or 3 other options. If they declined too, you wouldn't be funded. And of course, if what you did didn't align with national priorities to begin with, you wouldn't be funded at all, right? That's quite different in the United States, where, for example, Bessemer Ventures declined to invest in Google, which was at the time not a sure bet. AltaVista Internet Explorer were key competitors. But other venture capital firms took the risk to invest in Google, and they subsequently learned that it turned out to be a good bet. Katalin Karikó, who laid the foundations for the mRNA vaccines that allowed us to escape the pandemic, she was born in socialist Hungary. She moved to the United States, Even in a decentralised university system with a decentralised grant system, it turned out to be exceedingly difficult for her even there to get funding for her project. She even was denied a promotion or even denied tenure by her own faculty. But she nonetheless was able to take the risk. And if you look around the world, there's no single planned economy that has conceived any of the commercial key inventions in history. The Soviet Union came up with some good technologies for military applications, but even Sputnik didn't spark a revolution in telecommunications inside the Soviet Union. So, for exploration, you need this competition, and you need institutions that support competition. A key advantage of the United States is its federal system. The fact that industries were very concentrated and production was focused on large hierarchical firms in New England didn't prevent Silicon Valley from emerging, where, for example, non-competes were void. People could move horizontally from one startup to another and set up a concurrent set of experiments. which drove innovation and subsequently the expansion of the cluster.

So I'm thinking a bit about what the root cause of the issue is of getting stuck, let's say, and struggling to make further progress. And it seems to me that there is a kind of natural progression that you get to in the book where you start out in a more decentralized mode like Google or Ford in the early 20th century. These firms make Innovations in a decentralized system, then they scale up and they become the corporate giants, and they of course focus on scale as a source of competitive advantage. Is this always the way things go, and is that the issue of why we end up in a more stagnating centralized system over time?

I think you describe it well for successful economies, right? But in many economies, the problem is that this doesn't And so, one of the reasons, as I point out in the book, that the First Industrial Revolution took so long to materialize was that the craft guilds controlled much of production in Europe, in China, and in many places around the world. And they vehemently opposed any new technologies or industries that threatened their incomes and their skills. Britain was the first country to begin to undermine the craft guilds, partly because Parliament realized that to succeed in global competition with other European powers, they needed new technologies that reduced the cost of production because British wages were relatively high. And secondly, as markets in Britain became more integrated with an expanding road network, the craft guilds, whose power didn't extend beyond the city in which they were active, they were eventually undermined as they faced competition from outside. That process unfolded around 200 years later in China, which explains why it took a long time to industrialize. And similar forces with incumbent interests blocking technological change are still prevalent around the world today. So yes, you're right that in an ideal case, you see this sort of cycle, technology lifecycle, that you described. And eventually, at some point, incumbents have reaped most of the gains from a technology. A market becomes increasingly saturated. There is less scope for improving production processes to cut costs. And then many incumbents shift towards investing more in protecting their own rents rather than investing in new innovation. And that's often where stagnation happens.

Then perhaps going into solution mode and trying to ensure that further progress remains possible. I could imagine 2 kinds of solutions. One is to find a way to revert back from a stagnating centralized mode into a decentralized mode. The other one would be to find some kind of happy medium or peanut butter and jelly system that combines the best elements of both worlds. One such example that comes to my mind is the post-World War II United States, which is characterized by a lot of technological breakthroughs, the internet, GPS, space travel, but also massive scaling of these technologies and increased prosperity. Would you agree that that's like a happy medium system, or are there other examples of this?

So, I think the post-war US is an interesting case. And I would add, I think that the post-war miracle very much depended on technological exploration in the late 19th century. So, the electrical revolution, for example, powered much productivity growth in the post-war era. Those foundations were laid by people like Nikola Tesla and Thomas Edison in the 19th century. You see that in the US, productivity growth begins to take off in the '20s and '30s before World War II. So, some of these technological foundations were already established and already appearing in the productivity statistics. Yes, it's certainly true that the foundations for the modern internet were laid in the post-war period, but they were rather accidents that later boosted productivity in the 1990s, not so much contemporary America. So, I see the post-war period as one with relatively centralized institutions, but certainly also some decentralized exploration and new inventions emerging, but still very different from the very liquid, fluid market for technology that existed in the late 19th century when essentially all patents were taken out by individual inventors. Essentially, all of those patents were traded and later bought by growing technology companies. So yes, I think there is something that could be said like a sweet spot. But even the US, I think, needed to adjust its institutions, or at least employ more vigorously some of the tools that it already had developed to then unleash the computer revolution. And I think the lawsuit against IBM was such a moment, which essentially forced IBM to unbundle hardware and software and open up space for firms or startups, at the time, like Microsoft, to enter the market. That's very different from, let's say, Japan, which was widely predicted to overtake the United States as a technology leader in the 1980s, much like you're going to see the predictions around China today. But Japan did not have anything like the vigorous competition policy that the United States had. And so production continued to center around Keiretsu conglomerates, and entry of new firms did not materialize in software in the same way. Secondly, in the United States, you have the breakup of AT&T, which meant that when the National Science Foundation essentially releases ARPANET, the precursor to the modern internet, it's not stifled in the boardroom of AT&T. It evolves organically and involves a lot of different players.

So whose responsibility would you say is it to initiate these transitions from a centralized into a more decentralized society? It seems like this is something that is happening at such a high level that it's only the responsibility of regulators and lawmakers, like the lawsuits, antitrust that you mentioned with AT&T and IBM?

So I don't think that there is a single person or agency that is responsible for this. So as I mentioned earlier, you want, you know, an advantage with the federal system is that you have a laboratory exploration. And that means that there is no single person that is responsible for it. And I think that is one of the key insights from the book, is that you don't want a handful of people to be responsible for a process that needs to happen more or less organically. You don't want AT&T to have a veto right over the proliferation of the modern internet, for example. And so, That responsibility is, for lack of a better word, a collective one. If you go back to the 19th century and the emergence of the first really large corporations, you had a political system in the United States which was very much characterized by cash for favors. And it really took the public, the American public, to notice that. To demand change, which gave rise to the Pendleton Act, which gave rise to the first meritocratic bureaucracy, and eventually to the Sherman Antitrust Act. And so you need to have a mobilised society that is not so mobilised that you run into expropriation of all firms and capital in the country, but you need to have a society that is sufficiently mobilized to keep actors in check when they overreach and engage in anti-competitive behavior. And that's a very thin balancing act. And that's also one of the reasons why progress is fragile.

Yeah. So turning to the technology of our times, AI, which is obviously one that has significant potential for unlocking progress, What do you think are the main hurdles in the way of actually reaching or realizing that progress? And I assume they might be different in different parts of the world.

So right now, if you look at foundation model providers, for example, competition is undeniably pretty strong. And the key question for us is, is this just a temporary blip, or is this likely to be the case for some time as the technology matures, as new use cases are being explored. I think it's probably likely that the partnership between Microsoft and OpenAI would have been an acquisition if not antitrust had yet again become a concern. It is certainly the case that they still control a significant share of the market for LLMs, but they are by no means the only game in town. If you look at the firms then using the technology, though, something that we do see is that markets have become increasingly concentrated in AI-using industries. And that potentially matters because, you know, large and smaller firms tend to use technology differently. Large firms that have scale tend to use AI for automation and to lower costs in what they're already doing, whereas smaller firms are more likely to use it to develop new products, new activities, new industries. And it's those activities that are really key to long-run economic growth. Think about it this way. If all we'd done since 1800 was automation, we would have productive agriculture, we would have cheap textiles, but not much else. We would not have antibiotics, vaccines, computers, rockets, etc. Most of the things that define the modern world weren't about automation. And so we need to begin to find use cases for AI that are not just about automation if productivity growth from AI is going to be sustained.

And that depends, I guess, following your argument, on giving widespread access to these foundation models, the hardware, and so on to smaller companies or even individual innovators.

Yeah. So I do think going back to the 19th century example, the late 19th century was probably the most inventive period in human history. And so In principle, AI agents, AI tools more generally could help individuals explore ideas, develop new products, become more inventive. But that also requires them to be relatively widely accessible for further refinement. And so the decisions to what degree we move towards open source or some degree of relative openness versus the technology being more or less monopolized is certainly going to determine to what degree we will see the potential benefits of AI materializing.

Now I want to switch gears a little bit and talk about business implications. We have a lot of business leaders listening to the podcast, and it strikes me that this central conundrum that you discuss of exploring but also scaling up is one that is very familiar to business. They all have to run and reinvent to execute on what makes them successful today, but to also find ideas that might help them be successful in the future.

So one question is, from your historical work on this problem at a more macro-institutional level, are there any insights that might be relevant to business leaders as they try to navigate the same conundrum in their So if you go back to the computer revolution of the 1990s and you look at which firms and which countries did actually see an uptake in productivity growth, you find that it's relatively confined to the United States, with some exceptions in the Scandinavian countries. What firms in those places have in common is that they are relatively decentralised. That means that people at the lower levels of the organisation have decision-making rights and that they are rewarded Also for experimentation, right? I suspect that most people listening to this podcast, if they know something about generative AI, it will come from experimenting with a tool rather than just reading about it. And so the people that actually do this experimentation, they are the ones that are closest to it, know the applications. And if they can't make decisions, well, then, you know, you slow progress down.

Right.

Within the organizations. It's also a question of what you reward inside the company. If you reward people for performance in relatively narrow tasks, people will obviously focus on that. And then experimenting diverts attention and resources from that, and then people are not going to experiment. And so that's why, you know, decentralization wins in periods of technological turbulence. That's why I suspect what we already see, that firms using AI tend to be more decentralized. And so I would suspect that, you know, the lessons from the computer revolution will also hold in the age of AI. Another lesson though, I think, is that in periods of rapid technological change, challengers, startups often do have an advantage. If you look around the world today and you take the top 5 companies by market capitalization in the United States, they're around 39 years old. They're all technology companies. If you look at Germany, France, Spain, Italy, Japan, The UK, they're around 100 years old or 100 years plus. In France, they mainly produce luxury goods. In Germany, they are companies that stem from the Second Industrial Revolution. And so, the American dynamism in this case is actually really exceptional. And it also underlines that we shouldn't take technological progress for granted. And it's only a handful of countries that have been able to sustain this dynamism for any significant period of time. And it suggests that for companies today, for larger firms, you might be better off going out and scouting for technologies developed outside your own organization. And then there's obviously a question of still leaving that organization a sufficient amount of autonomy to actually being able to realize the technology it is developing.

Great. I think those are some very actionable insights. So I'll go to our final question, and it's essentially going back to the title of the book. So Francis Fukuyama obviously talked about the end of history. That hasn't happened yet. You're talking about the end of progress. I'm wondering what makes you hopeful that there may be further progress ahead.

Well, so a key theme of my book is that there is no end of history, right? So progress is constant work in progress, right? So you don't— it's not that you arrive on a particular set of institutions and then you're done, you can relax, everything will sort itself out. The message of this book is that you constantly need to adjust your institutions as technology progresses. And that's sort of an interactive progress. What makes me optimistic? Look, I'm Swedish, I'm an economist, I'm academic, right? So, you know, people usually don't invite me for my optimism. But, you know, I am generally excited about what's happening in artificial intelligence and what has happened in artificial intelligence over the past few years. You know, I cannot deny that what we've seen with developments in technology over the past 200 years and the resulting human flourishing should make you somewhat skeptical to somebody proclaiming the end of progress. The key message of my book is that actually, in a way, a positive one, but it's sort of pointing out the fact that there's nothing automatic about it. We need to get our act together. And if we get our act together, we can have innovation, we can have growth, we can have prosperity. But if we don't get our act together, this doesn't happen automatically. And even if you live in the UK like I do, there's a lot of gloom these days, right? But even if we don't develop these technologies over here, somebody else might, and we can bring them into the UK and use them. And that's still then A positive story even for us.

Great. Progress is a work in progress.

I love that sentence.

Thank you so much, Carl, for this conversation.

It's been a great pleasure. Thank you.

I've been discussing How Progress Ends: Technology, Innovation, and the Fate of Nations, which comes out via Princeton University Press in September of 2025, with Carl Frey, the Dieter Schwarz Associate Professor of AI and Work at the University of Oxford. If you liked our conversation, please make sure that you're subscribed on your favorite podcast platform. And also, we always welcome feedback, so do feel free to send me a comment or reach out to the BCG Henderson Institute directly. Thank you for listening to the ThinkersNDS podcast, and we'll be back with a new episode in 2 weeks. Thank you.