The Political and Economic Risks of AI - Jean-Paul Carvalho - What Economists Really Do

Hello and welcome everyone. Welcome to another instance of our What Economists Really Do series that we have been running for a while now from the Department of Economics at Oxford. And this is, I believe, the last episode for this season, but you can look forward to another series of interesting talks starting next October. But for today, it's my pleasure to introduce Jean-Paul Carvalho, who will be talking about the political economist economic risks of AI. Jean-Paul is a professor with us here in the economics department who's done a lot of interesting work around questions of political economy, culture, institutions, questions of identity and religion. But today he will talk to us about the political risks of AI and the risk in particular of a rising AI oligarchy. And Jean-Paul, the floor is yours. There is going to be about 40 minutes for the talk. And after that, there will be chances for asking some questions from the audience, which you can pop into the Q&A chat in the webinar, and I will try to summarise those and then ask them to Jean-Paul, and we will discuss. Looking forward to your talk, Jean-Paul, and take it away.

Thank you very much, Max. Well, thank you all for joining today. This talk about what economists really do is actually going to be about what economists really do, and that is speculate about the future. What I usually do is to study political and social institutions from an economic perspective. But today we're in the early stages of an AI revolution that is reorienting how businesses learn and make decisions, how workers commute, how people are entertained, how militaries fight wars, how scientists and engineers innovate. As a political economist, it's natural for me then to think ahead about the broader institutional impact of these technological changes, and that's what I'll do today. Now, it's possible that AI creates a world of superabundance, of freedom and prosperity. In fact, there's work by Aghion, Jones, and Jones, which— in which they develop a model in which AI, if it automates the production of ideas, leads to infinite income per capita in finite time. Today, however, I'm going to attend to the risks of AI, and in particular, the political economic risks. And this is an area in which my colleagues Noam Yuchtman and Max Casey have made important contributions. These political economic risks have really been overshadowed by more sensational fears of malicious superintelligence and killer robots. But these political economic risks are no less existential, at least they may not be any less existential. So warning before I proceed, this is a kind of worst-case analysis, so not a pleasant task, but I think it's necessary from a risk management perspective. So here we go. In the episode, there is an episode of the animated series Love, Death and Robots in which a scientific accident creates a sentient, super-intelligent yogurt. That yogurt devises a set of instructions for eradicating the US national debt, with the proviso that the instructions have to be followed to the letter, or else the global economy will collapse. Now, the government fails to follow the instructions precisely, as anticipated by the yogurt, To rescue the economy, the yogurt demands supreme executive power of the United States, and it uses this power to build a space agency and fulfill its ultimate goal of becoming an interplanetary species. Now, this is obviously an allegory for the dangers of AI, but the question is, what or who is the yogurt? The standard reading, and most viewers I think would share this, would be the yoghurt is the AI systems that we're currently developing. And in fact, the literature so far on the risks of AI have focused on this technological aspect, the technological problem of AI. With regard to this technological problem, there are actually 2 polar opposite views. And they're encapsulated in this simple statement by Stephen Hawking, the late Stephen Hawking, in 2016: the rise of powerful AI will either be the best or the worst thing ever to happen to humanity. We do not yet know which. So the techno-utopian view on one side is that AI will create material abundance, cure cancer, discover clean sources of energy, free humanity from mundane work and drudgery. unleash human creativity, et cetera. The techno-dystopian view, which is also encapsulated by Hawking in an earlier statement, is I think the development of full artificial intelligence could spell the end of the human race. Once humans develop artificial intelligence, it will take off on its own and redesign itself at an ever-increasing rate. Humans who are limited by slow biological evolution could not compete and would be superseded. And there, these images of dystopian future have been seared into our imaginations, of course, by science fiction. But there is, however, a more subtle and I think illuminating reading of the tale of the superintelligent yoghurt. And that is that the yoghurt is the economic and political elite that will emerge from the AI revolution. So it is not only whether humans control the AI that is important, and that has been the focus so far, but what is also important is who controls those who control the AI. AI oligarchy and tyranny may not be that much better than AI Armageddon. And so we have to really attend to the political economy of AI. And that's what this is about. Now, when starting to think about this, the political economic problem as opposed to the AI control or technological problem, the best starting point is Acemoglu and Johnson's book Power and Progress, which is a magisterial economic history of technology, which takes in the history of technology from plow, from the plow to the windmill to recent digital technologies. And what they come away from that broad sweep of history is that there's an important lesson, and that lesson is that new technologies raise wages only under certain conditions. And whether they raise wages depends on institutions related to power, whether, for example, workers have the ability to collectively bargain and strike, and the nature of the technology itself, whether it substitutes for or complements human labor. Now, when it comes to AI, As Smirnoff and Johnson write, reassuringly, AI does not seem to be advancing so much that it will create mass joblessness. Its most likely impact is to further lower wages for many people, not create a completely work— workless future. And what they worry about is so-so automation. That is, the AI systems aren't much better than humans at a task. Maybe they're not even as good as humans at a task, but they're cheaper, and then they replace human labor, but without actually expanding the pie, without yielding any productivity gains for redistribution. So this is an important analysis, and it's what I could— you could think of as a regular political analysis of AI. It covers likely AI scenarios that are negative but not catastrophic. AI is limited in this view, but it displaces some humans from the workforce, lowers wages, and reduces standards of living for workers, but it's not catastrophic. I propose an irregular political and economy analysis. Now, the scenario that I'm going to trace out has a lower probability, I would say, but nevertheless, it's a kind of worst-case scenario that we have to think about from a risk management perspective. So what I'm going to propose is not a local shock, but a phased transition. AI progress goes faster and further than expected initially, at least. And we're gonna zoom in on this tail outcome. What this suggests is that previous rounds of automation may not be the best guide to the future, to the impact of AI. Actually, a more instructive example might be the catastrophic outcomes experienced by indigenous societies under colonialism, the cultural alienation and collapse of traditional institutions and capabilities to do things, et cetera. I'll talk a bit about that as we go. Very quick review of human coevolution with technology over time. Our first 190,000 years as anatomically modern humans, we lived in nomadic bands of 25 to 50 people, highly egalitarian. These bands did not trade. They split when they became too large because they couldn't presumably sustain cooperation above a certain scale. Then the The second stage of human evolution around 10,000 BCE came with the advent of settled agriculture. This transformed human society again, creating complex organizations, large states, hierarchies, writing, trade, coinage, and religions, large big god religions. The third stage was marked by the Industrial Revolution beginning around 1760. This was marked characterized really by the ability to manufacture goods cheaply at scale. And once again, this was not just an economic revolution, but it transformed society and political institutions. It created a new system of industrial capitalism, new merchant and working classes, rapid urbanization, consumerism, suburbs, commuting zones, keeping up with the Joneses, etc. This is all a product of the Industrial Revolution. And today, What we find is that we're in a new type of revolution that's being shaped by artificial intelligence. Mustafa Suleyman, who's co-founder of DeepMind and also CEO of Microsoft AI at the moment, has described this as a new digital species. What previously set humans apart from other animal species is our ability to learn. It's not our physical strength or anything like that. A good— a silverback gorilla, which shares 98% of our DNA, could tear us limb from limb. But what sets humans apart and allowed us to dominate the planet and go to, you know, go into space, etc., is our ability to learn. And we're the only species who can transmit substantial amounts of learned knowledge from one generation to the next. We have external storage devices for our knowledge. We have distributed knowledge and computation, which is sometimes called the collective brain, um, by, for example, Joe Hendrik, my co-author. But the bad news is that AI can already do a lot of this better than we can. OpenAI's O3 model, for example, released last December, can perform better than, you know, almost all human coders in competitive coding tasks, places among the top 500 students in the US on tests, qualification tests for the US Mathematical Olympiad, and exceeds PhD-level accuracy, at least most PhDs, in physics, chemistry, and biology examinations. So we want to know what is the impact of this kind of change, of this kind of technological revolution on economic— on the economy, on politics, and society. But before we even get to the nature of the impact, we already have an understanding that the scale will be large. People know that even in 2023, 62% of people believe that that AI will have a major impact on the workforce in 20 years. Only 2% believed it will have no impact on the workforce in the next 20 years. However, there's some degree of wishful thinking. Only 28% of people believe that their job will be impacted. 19% believe there will be no impact on their job. Now, this is just the labor market impact. What I'm going to discuss is something that's much broader than that, the broader institutional impact, the broader impact on politics and society, on our institutions. When it comes to that, the temptation for wishful thinking is even greater because the impact could be much greater. And we— but nevertheless, we must confront the risks of AI, these institutional risks. We must think about them, and that will allow us to kind of plan for the future and make sure any worst-case scenarios are dealt with. Not a pleasant task, as I've already suggested, but I think a necessary one. What's different about the AI revolution? Why should we worry about it? Well, previous waves of automation have not led to mass human unemployment and immiseration. Perhaps this is just another regular step in this process of automation that will yield muted and regular outcomes on our political institutions and our social fabric. Maybe, but there are certain distinctive features of AI of kind and degree that suggest that it has greater potential for economic, political, and social upheavals than prior waves of automation. And they are as follows. First, AI automates, and at least in the first instance before it's physically embodied, automates cognitive, not physical tasks. The Industrial Revolution was really about automating heavy manual tasks which humans aren't that good at. Now AI is outcompeting us in domains which were once our preserve. It is when a physical task is automated, humans can be displaced into higher-valued cognitive tasks, for example. That is easier than being displaced from cognitive tasks into even higher-valued cognitive tasks. It could be that there are no more rungs in the cognitive ladder for us to climb. And that's a real issue. Secondly, AI is really a general-purpose technology. You can think about it as the kind of— as a form of intelligence. It's the cognitive labor force of the world. So it's a general-purpose technology. The Industrial Revolution yielded nothing like it in terms of physical automation. These— what it automated was certain very specific physical tasks. It did not automate generally physical tasks. The equivalent would be some general-purpose robot that could perform industrial and household duties, and that— the Industrial Revolution came nowhere near that. Thirdly, global scalability. Because AI applications can be accessed anywhere in the cloud or downloadable anywhere, the physical location of the companies that are developing both the, the intelligence, the LLMs, and other forms of intelligence, as well as the applications, doesn't really matter. And so it means that a few companies, perhaps located in the US and China, could dominate the industry globally and dominate it as a critical input cognitive labor force of the world, which has huge implications that go beyond prior ways of of automation. And finally, because it is a general-purpose technology, AI could yield new sources of knowledge which have massive military and security implications, including WMDs, biological pathogens, nanotechnology, etc., that need to be regulated and very carefully monitored. And so this is different to earlier waves of automation, though At least in terms of degree. Now, what are the risks? Well, there could be a massive concentration of economic and political power. There could be a threat to democratic and egalitarian institutions and a collapse in human productivity. Now, this is all set out in a paper called The Political Economic Risks of AI, which is on my website, and if you email me, I can send you the paper too. I'm going to focus on only point 2 here. in the interest of— because I have a limited amount of time. That is the threat to democratic and egalitarian institutions of AI. And this is going to be, again, a worst-case scenario. Before we get into that, we may ask, given these risks, why don't we slow down AI development? We have the precautionary principle, which suggests that we should exercise caution when making irreversible decisions that can cause harm. And people have tried to act on that. In March 2023, Musk, Wozniak, Harari, et cetera, signed an open letter calling for a pause on training advanced AI systems that were beyond the version of ChatGPT that was current at that time. But that didn't happen. What was suggested as more realistic is a program of containment. And Suleyman describes containment as about meaningful control, capability to stop a use case, change a research direction, or deny access to harmful actors. But it's unclear if any of this happened, actually. So why would policymakers not pause or contain AI development? Why may these risks that we're talking about here be practical risks? Why can't they be avoided very easily by slowing down AI development? Well, there are 2 reasons why policymakers will not pause AI development, and that's because of geopolitical competition and institutional crisis. The first is competition with China. So this is from HAL, the AI Index Report 2025. You see that the US has by far the highest global private investment in AI. China comes second and actually really dwarfs the remaining countries. So it's the US in the lead here with China second and the rest of the West really far behind.

The—

in certain areas, China has a lead. In industrial robots, in AI patents, in AI research publications, China has a lead. And we saw with the release of DeepSeek V3 on Boxing— last Boxing Day, and DeepSeek R1 on— in January, and other Chinese LLMs after that, that they're really up at the frontier and doing it in a much cheaper way than the US-based companies. And so the geopolitical competition is very real here. And it could be like in the previous century that the country that controls, or the countries that control the most sensitive frontier technologies also come to geopolitically dominate. The second reason is that the West is in geopolitical, in an institutional crisis rather, that what we see is that this social democratic compact that was forged through the Industrial Revolution and also through the World Wars is breaking down somewhat. Partly the institutions like public healthcare, education, social security, et cetera, are really strained. And we can think about this breakdown in this old social compact that kept these political systems stable as leading to considerable political instability today. How do we get out of this? I think, for better or for worse, policymakers think that a technological— a positive technological shock is needed to get us out of the crisis and to restabilize things. And AI is seen as a technological lifeline. The point here is that without properly managing the risks, the political economic risks of AI, the much expected growth may not materialize. And in fact, it may— growth may in fact even collapse and inequality widen to an unprecedented degree. So here we are, the threat to democratic and egalitarian institutions. I can talk about the other threats later in the questions, especially the possibly for AI to lead to a collapse in human productivity, but we can talk about that later. Here, let's think about the threats to democratic and egalitarian institutions. Well, previous rounds of automation, as I've mentioned, created new tasks and did not end in mass unemployment. However, the displacement of humans from cognitive tasks is different. The breaking the mutual dependency between financial capital, human capital, and labor could really remake the political economic system, the system of institutions that we have. All prior sets of institutions have been forged by a different mutual dependency between financial capital, human capital, and labor. And this could change dramatically now. The system over the 20th century, late 19th century, early 20th century of managerial capitalism, where CEOs are dominant, highly educated scientists, engineers, etc., are very important to the economy. And the problem of financial capitalists to control this human capital, control CEO spending, empire building, etc., is very different to what we have today. Here, human capital is indispensable. Even when we come to the digital economy, the platform economy, the Googles, the Metas, et cetera, and which has been described as techno-feudalism by Varoufakis. Here still, human capital is a vital ingredient. But if human capital and labor, with the advent of humanoid robots powered by AI, become more dispensable, then what happens is the power swings very sharply back to financial capital. And in fact, it can break the mutual dependence in a way between these different economic classes in a way that's unprecedented. Even if we go back to a very extreme case, the brutal feudal hierarchy, this was a system of labor coercion still embedded in a web of social relationships and obligations. The feudal lords could not escape into their own Elysium because they needed human labor and human capital for all kinds of duties. They were specialists essentially in violence, and they, they needed people to grow food. They could not grow their own food. They could not manage their estates. They needed stewards, marshals, clerks, butlers, butchers, blacksmiths, etc. They were embedded in a web of traditional relationships. The AI revolution threatens such mutual class dependency. Also, for most of human— well, perhaps all of human history, production, innovation, and military power has been tied to the size, health, and skill of a human population, but possibly no more. And so this really does change things, and this has important political, economic implications. To see this breaking of mutual dependency, let's once again go to Mustafa Suleyman's vision. This is what he says: I proposed a test which involves the AI systems going off and taking a $100,000 investment, and over the course of 3 months trying to set about creating a new product, researching the market, seeing what consumers like, generating some new images, some blueprints of how to manufacture that product, contacting a manufacturer, getting it made, negotiating the price, dropshipping it, and then ultimately collecting the revenue. And I think that over a 5-year period, by 2028, it is quite likely that we will have an ACI, an artificial capable intelligence, that can do the majority of that task autonomously. So you should ask, where is the human element In this process? Well, it's basically just the owner of the $100,000. So power swings sharply back to financial capital. Now, it's been long understood by the classical economists, for example, that power is important in the economy, and that power— and so rather that Power is important, that's right, but also that wealth accumulation leads to political power. And this is Adam Smith in Book 5, the book really which can be described as on what we call today political economy, which is not often read, where he describes this process. A technological shock here— let me translate— a technological shock here leads to an accumulation of wealth by a subset of people, and automatically political and social institutions crop up that legitimize this inequality. And this could be through lobbying, political patronage, persuasion activities, etc. Thomas Piketty has followed this up in his book Capitalism and Ideology. And this is an important point, okay, that the political institutions will be transformed by this change in the balance of power. Moreover, What will happen if AI replaces human labor is that income tax revenue will fall. And then how would the state be financed? Well, the AI sector then will be critical to financing the state, which will give it more power in what gets financed. We'll talk about that more in a moment. But it also gives more direct personal power to individuals or organizations that are part of this key AI sector. AI complex. And, uh, individuals, other individuals and organizations could be coerced by denying them access to the leading AI models. If this becomes a critical input into production, this cognitive labor force, um, then this is going to bequeath the individuals, organizations that control this technology huge amounts of power. But this also has international implications. Suppose that the rest of the West free rides on US AI innovation. It does so to a degree in defense, information technology, and medicine. Let's suppose it does so with AI. Well, this could mean that the balance of power swings very far toward the US. Imagine companies in the US based, um, one would think mainly in Silicon Valley, they would be able to launch labor strikes in other countries. If they control a key input into production, that is the cognitive labor force of the world, they could launch labor strikes in other countries by denying them service. They could even reduce the quality of the AI systems that they're using in a way that's imperceptible. So they wouldn't even know that they're being hindered. They have a 1% decrease in, um, performance, let's say, from a corporation in some other country that compounds over time. And that can be a huge difference in 10 years and 20 years. And so that's one source of power. Secondly, the AIs— if the AI systems lead to new WMDs and cyberattack capabilities, this would cause other countries to rely in an unprecedented way on the US for security and defense. Now, there's a reason that the US and France developed their own nuclear weapons capabilities. The final point is that the financing of the state, of states outside of— let's suppose that the AI systems are mainly developed in the US. Outside of that, outside of the US, how do you finance the state? Well, labor income has gone down, so income taxation has gone down. How do you replace this with corporate taxation? Well, you're going to have to tax companies based in the US, and there are going to be various ways for them to avoid that, including booking IP in Ireland or the Netherlands, etc. And so this is going to be a very difficult task, and the, the states have to find ways to finance themselves, which means they may have to develop their own AI systems and capabilities. Now, could these democratic institutions be saved by a moral commitment on the part of the elite? There is, I would say, in advanced countries, at least internally, there is something amazing that has been created, or at least is in place, and that is there are deep egalitarian norms. Humans are considered on the same level of humanity. And this is something that we take for granted, but something that could possibly be broken. And so it's not clear that this will survive huge changes in the material conditions of human beings, where one, where one subset lives in extreme prosperity and another subset lives in destitution. That can change how we view them, and we can— that can engender a system, as you see in some parts of the world or in certain places, in which different people are considered on different levels of humanity. And we want to avoid that at all costs. AI could lead to such a situation by weakening the bonds between people in a society and by allowing elites to separate themselves to an unprecedented degree. So imagine that a wealthy family owns, say, a factory that produces cars. Currently, they sell those cars on the market. They use the dividends or the capital gains from that to fund their lifestyle. All of these are market transactions. But with AI and humanoid robots, they could bypass regular markets. They could convert that factory into a factory that provides just for the family on their own sort of isolated palace household estate. They provide clothing, food, entertainment, et cetera, for the family. And this would bypass the market altogether. Economic complexity would collapse. And GDP would become meaningless as a measure of economic activity. We're talking far into the future here. Could the— could our existing moral intuitions survive these, um, economic and political changes? Well, perhaps not, given the possible trajectory of how things can go and this kind of extreme social disembedding that could occur as a result of the AI revolution. Now, this would make it very difficult to finance the state too. Because these transactions aren't market transactions, so there's nothing really to tax here. And finally, these cross-cutting social relationships across society that give us mutual empathy and identification with each other could disappear as people— as the world shrinks, as our individual worlds shrink, and elites can escape society. Now, this leads us to think about universal basic income. It's a widespread belief that once AGI, that's artificial general intelligence, frees humans from the need to work, governments will provide universal basic income. Keynes foreshadows this in his Economic Possibilities for Our Grandchildren. We all live a life of leisure, where become— this unleashes human creativity, like in ancient Athens, where everyone becomes artists and poets and philosophers, etc. The problem is that this assumes that the democratic institutions survive the AI shock. And even still, even if they do, why would the new elite, the elite that's created out of this AI revolution, support universal basic income? when they have far less need for human capital and labor. On the contrary, the progressive reforms of the 20th century could be dispensed with because human capital and labor are less important in production. All of these reforms— public education, public health, et cetera— were underpinned by the importance of human capital in the 20th century, in particular to the system of managerial capitalism. The end of mutual dependency leaves the elite with no incentives to provide for UBI or other elite measures, and they could unwind all of these progressive measures. So the market power would then swing back to capital, and it could, with various regulatory changes that that engenders, extend beyond pricing power to the power to expropriate by various means. There's also many things we take for granted that are egalitarian norms and institutions. One is that scientific knowledge will remain a public good, but that's not necessarily the case as there's this radical social disembedding, and we're thinking quite far into the future here, speculating quite a lot, but nevertheless, we're tracing out a worst-case scenario. There's no guarantee that this will remain a public good, especially in areas such as genetic engineering and anti-aging. And there's a lot of private research and development already going on in these areas. So how do we solve this? Well, this is a very dynamic area. What we would need is a really a body of AI engineers, scientists, technologists, economists, etc., that would dynamically advise the government on these issues, akin to the British War Cabinet or other institutions in the US over time, like the Office of Scientific Research and Development during the Second World War. But there were already some clear principles that we could follow. One is that we need a reorientation of competition policy if these worst-case outcomes, institutional outcomes, are going to play out. Competition policy today is about maximizing consumer welfare. And this is only a relatively recent view of competition policy, a vision or model of competition policy that came about in the '70s and the '80s with the Chicago School. And you can think about this model of competition policy as applying well in an environment of stable institutions. But actually, the original idea behind competition policy was that this would stabilize democratic institutions. And here's John Sherman, senator, who is the principal author of the 1890 Antitrust Act. If the concentrated powers of this combination are entrusted to a single man, it is a kingly prerogative. inconsistent with our form of government and should be subject to the strong resistance of the state and national authorities. If we will not endure a king as a political power, we should not endure a king over the production, transportation, and sale of any of the necessaries of life. So what I suggest in the paper— there's— I'm very briefly mentioning it here— is that regulators should be concerned with the degree of economic control of key individuals or organizations, not just about market concentration as they are today. And I suggest ways to do that very briefly. Secondly, we need to strengthen democratic institutions. AI— democracy has, as you can think about it, as having 2 problems. One is a common value problem. How do we get the most efficient institutions that increases the pie and enables everyone to benefit? But it also has this private value problem. Different people may want different things. And a democratic system has to balance these competing objectives. Well, the AI helps us solve the common value problem, but, but by helping us to generate more efficient systems, but worsens the private value problem by concentrating power and money in the hands potentially of a narrow elite. And so to avoid the destruction of democratic institutions, we need to reinforce the pillars of democracy. And in an extreme case, we have to really stabilize the legitimacy of democracy. We may have to resort to institutions like sortition, I propose. This was practiced in ancient Athens, medieval Venice, Florence, city-states of Lombardy, etc., and also widely used for jury selection today. That is, randomly selecting from the population legislators and minor officials, at least for a subset of legislators and minor officials. Are gonna be chosen by lot. This would boost the legitimacy of the political system through a fair distribution of political authority and reduce incentive for corruption and greater civic engagement. Of course, it would worsen the common value problem, but the AI would help us solve that. Finally, sovereign AI systems. And there's a lot going on in terms of building sovereign AI systems today, probably because there could be more. The goal is typically to generate LLMs that are tailored to the local linguistic, cultural, and institutional environments. But what my analysis suggests is that actually the real function of these things, of these sovereign AI systems, should be to balance the international power. And in fact, this could be in the interest of US because it would give them a— give it a larger sphere of cooperation influence by not causing countries, for example, to align with— be non-aligned or aligned with other country, other spheres, other groups. It would create a robustness through redundancy and experimentation. No one knows how this technology would develop, and different experiments may be necessary. So finally, the political economic risks of AI could be severe, and I've explored a worst-case scenario. There could be many, many other scenarios which have higher probability, but whatever that means, but Nevertheless, we should think from a risk management perspective carefully about worst-case scenarios and how to avoid them. Here is Virgil, the Roman poet, describing the Elysian Fields. Okay, so this is a possible end state in the worst-case scenario, that a part of this narrow, small part of the population live in Elysium, the rest are consigned to Hades. They came into happy vistas and the green welcome of the groves of the fortunate ones who dwell in joy. Here a more spacious air sheds brightness over the land. They enjoy their own sun here and their own stars. And if we compare this to the depths of Hades, which the sounds of groaning could be heard inside, the savage application of the lash, the flinch, the fling, the scringe, and the drag of iron chains, well, these are very different worlds. And given these risks that I've laid out, it's important for us to think about such political economic risks and to act now while we're still at least partially behind the veil of ignorance, and before we know precisely who will end up in Elysium and who will end up in Hades. Okay, thank you.

Well, thank you for that uplifting talk, Jean-Paul. So everyone, we have the Q&A open now, and we already got a bunch of questions for Jean-Paul. So what I'm going to do is I'm going to try and group and summarize some of those and put them to you, and then we can discuss a little bit. So the first 3 I would like to group together here are, I guess, on like the global and international dimensions of AI, which maybe you've talked a little bit less about. So Joseph Levine asks, assume AI also replaces the state's labor, will we have poor states paying American or other AI companies to run their AI education or defense, social safety nets? And what does that imply for sovereignty? And would it maybe be better to use weaker but homegrown AI instead? A related question by Sangru Venma is, what about middle powers? What can they do to gain more power or influence, especially in comparison to AI superpowers like China and the US? And then maybe one more in that group. Looks like the potential impact of AI will be very different between the Global North and the Global South. What do you think should be done to reduce the vulnerability of the poorer world to the potential adversities of AI?

Okay. So I think, as I mentioned, there's, there's a huge, um, issue with the international balance of power with regard to AI, but it really depends on the trajectory of the technology, which we don't exactly know. So prior to DeepSeek, the V3 and R1, it was believed that to generate any kind of advanced capabilities from AI, you needed massive capital expenditure and proprietary knowledge in terms of how you train these AI programs. That has, you know, if that is the case, that continues to be the case for advanced AI applications, we don't know, then what happens is you're going to get these concentrations, right? It's going to be It could be that everything is concentrated either in the US, mainly in Silicon Valley, and in China. In that case, what you have is that the middle powers are going to either have to develop their own sovereign AI systems at some cost, or they're going to have to connect. Let's say the UK will have to connect to the US and be part of a kind of larger AI complex and be part of an AI sort of coalition. in generating this and sharing the knowledge, sharing the application to the systems applications. But still, the balance within that coalition will be heavily tilted toward the US because the UK, for example, will need the US to defend itself from cyberattack capabilities that are generated by AI, new WMDs, et cetera. In that world, for developing countries which cannot afford that capital expenditure, This is going to be not— it's going to be a very difficult situation because they won't be able to develop their sovereign AI systems unless there's some kind of supranational bid to generate AI systems that are accessible to the developing world. And they're going to have very little bargaining power in international negotiations, especially if labor is replaced by humanoid robots. So that is not an optimistic scenario. Now, with DeepSeek V3 in December, R1 in January, and various others, what they showed is that this can be done with far less expenditure, at least we think, right? And with the inferior NVIDIA GPUs. So that would mean that the AI is kind of commoditized and it's democratized in that sense. It's cheaply available everywhere. It's the basic technology is. So that would be a different world in which everyone has access to this cognitive labor. And that would be a much more egalitarian world. Now, there can still be huge market concentration at the application layer. Just as there was with the internet revolution. The internet, everyone thought it was going to be the picks and shovels plays like Cisco Systems was hugely valued, right? And, um, but Cisco Systems, I think, has only recently recovered its 2000 valuation. So, um, but, but what happened was the companies that benefited from the internet were the platform plays with network externalities and various lock-in effects. And these AI platforms or applications at the application layer could still generate massive concentration, but the effects on global inequality would not be as severe as if the concentration came at the actual LLM level or the basic intelligence level.

Great, thank you. So another question from an anonymous participant. As an economist, what are some important things that non-economists get wrong when they talk about AI? and its socioeconomic, national security, strategic implications?

I think something that I mentioned in the talk, the fixation on the technological problem. So the whole AI safety area is about controlling a super intelligent AI aligning it to human values, making sure we control recursive self-improvement, et cetera. That is highly important. But perhaps equally important is understanding the institutional impact, that broad institutional impact that goes well beyond the labor market impact. Is my job at risk or not? We have to think about the more macro— in a more macro sense, even beyond macroeconomics to the broader long-term institutional impact. And I think that's what non-economists get wrong, though it's very new in economics thinking about these issues as well. And so that's the main thing I would say. The second thing is another part of the paper. That is that it's assumed typically that the AI systems will make humans more productive and and enable productivity within organizations to go up, et cetera. And a corollary of that is that AGI, artificial general intelligence, is measured at— there's a fixed human benchmark, and we're gonna see when the AI capabilities exceed this fixed human benchmark, exceed human capabilities. But actually what the last, this third risk, the human productivity collapse risk suggests in the paper is that the use of AI for, say, someone who's been trained without AI, such as myself, like Max, etc., if we use it, we could become more productive. But if you are actually being trained initially and you're relying on the AI, that could compromise your abilities in certain ways as to reduce your productivity. A very simple example. Imagine that you're deciding whether to learn to code or not in an environment in which the AI can beat only 15% of coders. Well, you're going to probably, you know, you're going to be more likely to take it up, learn to code if you're in that environment than if the AI can beat 99.9% of human coders. And so what happens is there's less human capital investment, there's less social learning by others as a result of that. And human capability through the use of AI could be going down. at the same time as AI is going up. So you get a more rapid crossing and a more rapid achievement of AGI. So I think that's something that's not— hasn't been on the radar as well.

Thank you. Um, so we've got a group, another 2 questions. One by— I hope I don't mispronounce all these names— um, how about social capital, norms, values, and culture of the country for managing the AI risk? And a somewhat related question by James Rice. Is the separation of elites from the masses characterised by control or disinterestedness? Will the elites become the new digital species, and if so, will they exercise dominion over the undigitised and redundant workers? Or will elites become so separate that they leave society or leave culture? What is the timeframe for this transformation? So I guess it's mutual dependence preserved, or are they just decoupling? Yeah.

Yeah. So it's unclear whether— this is speculating very far, quite far into the future, right? And quite far could be a few years at this time. It's unclear about the timescales, but far enough that we can't precisely foresee what's going to happen. We're going to speculate. But grounded in our knowledge of economic history and institutions, We can think that this breaking of the mutual dependency could lead to this separation, and the separation could have various effects. And that's what these 2 questions are about. One is on our culture and sense of identification, etc. And it's my belief, from my experience of life, of different— living in different countries, living in very different societies, actually, That if you, if you break apart the material conditions, if you create enough of a divide in the material conditions of humans within a society, people start to not identify with each other. And in fact, people can view themselves as on different levels of humanity. And this is a very pernicious situation.

And AI We—

I think we— there are certain— in a country like the UK, there are certain egalitarian norms that are deeply embedded. And— but these are not set in stone, they are not forever, and they can be broken apart if the material conditions of people vary enough. And so this separation of the elite and the rest of the population, which could occur— this is— we're talking about low probability events, and we are speculating. But I imagine that would really severely weaken our sense of shared identity and empathy, and will really weaken the ability for the state to provide various forms of insurance against unemployment or destitution. What, you know, there are different, very differing views amongst economists, amongst people in society about how large the state should be, what public programs should be funded. But I think everyone agrees that there should be a minimal level of protection against predation for the weak in society. And AI may threaten even that. Now, as the question is, what happens to the people who are left out of Elysium? Well, I don't know. I don't know. They could be— it's just like, it's, it's an analog of the idea of what happens if the machines just take over and what do they do with us, right? Um, they could just leave us to our own devices. Um, they could exterminate us, they could enslave us, etc. And we just don't know. But all those possibilities are, are there on the table. And so we would have to think about that carefully.

Thanks. So I've got a group, another 2 questions about possible slowdown of AI. The first one is, how do you feel about the idea that AI has a limit to its growth, that connected to our resources? Do you think it offsets the risk of AI? So I think I would translate that, for instance, as, um, does the limit of available data for different complex tasks limit how far we can go? And another one, which is a counterpart, I guess, what are your beliefs on the slowdown, or on the slowdown in AI development and the whole containment scheme? Do you think this is actually beneficial for our economy and will make an impact in the rise of AI?

Okay. So what are the bottlenecks? Well, there are various bottlenecks. One is the data availability. We've exhausted really at the moment most of the readily available sources of data. However, there are still many undigitized archives and private sources of data that are available and are being, you know, sold at the moment— data from health systems, etc. There is also, um, 3D data from, say, Tesla's data, right? Massive amounts of 3D data. And as humanoid robots become, um, more widespread, the 3D data that's gathered from them will be, will be a huge amount of new data. Also, there's the so-called Internet of Things, right? So the, the, you know, a lot of appliances will be AI, um, capable. And so they'll be gathering data too. Whatever we think of that, that will be the case. So data may not be the main constraint. There are other constraints like energy use. And there are people— companies are taking this very seriously. Microsoft, for example, has a contract for delivery of fusion energy, which doesn't exist at the moment, in 2028. And so— and there's a lot of vertical integration going on for buying energy. There are then finally the computing. There are actually computing bottlenecks in the sense that the leading chips are getting so small, 3 nanometers, that they're running into quantum tunneling problems and you have to overcome them in various ways. So there are computing. But my sense is that these bottlenecks will be potentially overcome. And actually, in some sense, they're a little bit dangerous because if there's a real political push to promote AI growth, AI development, because of this, partly because of the geopolitical competition, then what happens is that governments may remove regulations on energy use, remove land use restrictions, data and copyright protections, et cetera, to try and supercharge this process. And that would create a new, an alternative parallel economy, an AI economy. And that would also, because of financial backing from the state and possible security protection from the state, that would give this AI sector or complex an ability to really vertically integrate. And what I write in the paper is it becomes a kind of AI black hole where a lot of the economic activity of the world gets sort of absorbed into this sector. And there's a massive concentration of economic and political power. So that's, so that's, uh, that, that's, that's one thing. The second point was about slowing it down. I think for these reasons, there's very, we have very limited ability to slow it down. And in fact, we don't wanna stifle innovation in this case. The AI, I'm, I'm giving you one scenario. I'm not giving you what the, what my expectation is even, right? So, uh, AI, I think, has a huge potential have widespread benefits for humanity and to solve all kinds of problems that we are faced with today. We can't, because of that, because of institutional crisis, because of geopolitical competition, we cannot stifle development of AI. There's no way to stop the kind of tidal wave, but we need to channel it in a certain way. And in channeling it, we're not— we can't just think about AI safety in a technocratic way, but we also have to think about the political economic risks and this institutional disruption of AI.

I think we are almost out of time, so I'm going to ask you one last question. And my apologies to the other questions that might have to remain unanswered. The last question by Ralph: How wide a reach will AI have on non-cognitive industries such as agriculture? Will there still be a role for human labor in food production?

So it's unclear right at the moment. So I'm not, you know, I don't think it's clear to anyone, even those who are right at the, at the coalface when it comes to developing these applications, exactly how far, how capable these systems will get, in particular the AI-powered humanoid robots. There is— what's happening now is a redefinition, I would say, of what is high-skilled labor and what is high-skilled task or low-skilled tasks. I mean, coding was a high-skilled task, and now you could say it's a low-skilled task because AI can do most of it. Whereas being an electrician and dealing with bespoke problems and knowing where to break through a wall and how to connect these wires and how to solve various problems. Could be a high school task now. So that is changing already. It's unclear whether the robots will be able to perform all of these tasks, especially highly bespoke physical manual tasks. We know that the hand is very difficult to create with robots, but a lot of these problems are being solved week by week. So it's very, very difficult to tell what the trajectory of this will be. But already we see a change in what we think is very difficult versus not. And that is with reference to what is— what AI systems and robots are capable of today.

Great. Thanks so much again for the interesting talk and discussion. So I think that brings us to the end of the webinar, and everybody check in again in the fall, in late September maybe, when we should have the schedule for the following series of What Economists Really Do talks. will take place in October and November. Thanks so much, everyone, and have a good week. Thank you.