How can we attain AI sovereignty? with Sarah Hunter and Tyler Cowen

Good afternoon and welcome to what is very much not the graveyard shift of Civic Future Day 2, but perhaps the most important conversation that we might have today. And we're going to be discussing AI sovereignty, what that means for the UK and the various ways in which that might be achieved. We're going to have a bit of a conversation and then branch out into questions. So do get thinking. And joined for the conversation by Sarah Hunter, and Sarah's worked in government in the UK, led public policy in Google X, and has various important non-exec positions in institutions that are important in the UK. And then Tyler Cowen, see Tyler's a professor of economics and one of the world's leading thinkers on AI and its impacts on society and the economy. And we talk about AI sovereignty, and I suppose a sensible place to start with this is, thinking about what the term means and how the two of you understand the term is a conversation that people have, often disagreeing about what it means. So Tyler, maybe I'd start with you. What do you think AI sovereignty actually means?

I don't think it's a well-defined—

and you need to use this one or the people can't hear you.

I don't think it's a well-defined concept. I'm not sure such a thing exists. If you assume the normal definition that somehow the UK controls its own AI, there was a recent Substack by Anton Leicht And he tried to estimate what AI sovereignty would cost. First of all, it requires a coalition of mid-sized nations. You were just in a coalition of mid-sized nations called the EU. You felt you didn't have sovereignty in that, or at least some of you felt that way. So is that right? Right off the bat, is that even sovereignty? But if you had such a coalition, Anton estimates it costs, what, like $500 billion over 4 years? I mean, is that going to happen? And there's still no guarantee that it works. And he says the personnel costs could cost $65 billion, again over 4 years. And are you in the position where your procurement will be rapid and your public sector will tolerate the notion of people showing up, you know, not as CEOs, just as researchers, and pulling in £10 million a year in salary plus options or other rewards? It just doesn't seem like a realistic scenario to me. So I think the best most mid-sized countries can do is to be useful to the world in other ways and increase their bargaining power. They made decisions a long time ago that if something like AI were to come along, they would not be sovereign. I'm not even sure America and China will enjoy AI sovereignty. The companies, the leading labs in both places now have a lot of bargaining power. So, you know, what does sovereignty mean? Like, who's the "we" that's supposed to be in charge anyway? So I think often the discussion does better by sidelining that concept and just trying to figure out what are some good concrete steps we could make that would be useful for the UK or the US or for AI or for users, whatever. And just marginally hope you keep on getting into better and better positions. UK in particular, again, you did this thing called Brexit. You had all these plans for whatever you would change. Some of you may agree, disagree. But as far as I can tell, you didn't do most of it. So like, AI aside, you didn't prove that much sovereignty to begin with. I'm sure Dominic Cummings is a fraught figure in many circles, but I read his Substack. What he says about UK government, as far as I can tell from my distant vantage point, but it's basically true. So I'll just say like, start with sovereignty and get a bunch of good things done. And then like 2 weeks later, start worrying about the AI. But first, show some sovereignty.

Sarah, there's so much to say.

I—

the starting point is I really hope our AI strategy is not like our Brexit one. I really hope we have more success there than we did on, on that. Um, I basically agree with you. I think, A, it's massively misused as a term. It seems to mean a lot of different things to a lot of different people without any specificity, uh, and it's not clear for what purpose, you know, what, what does sovereignty mean and why does it matter? No one's really like nailed that. So I think my starting point is let's redefine define the term in a way that is meaningful and useful. And for me, uh, yes, I think there is a value in having a stronger sense of optionality and agency over the AI that is used in this country. And by AI, I don't mean just Frontier Lab access. I think, you know, I worked at Google X before the Frontier Labs came along. Everything we worked on was AI plus robots, basically. Uh, and they're things that really will change the world. I mean, I worked on self-driving cars for many years. I genuinely think there's some transformative things that are not dependent on frontier labs and that we should have some sort of national control over. Um, but I don't think it's about full-stack sovereignty. I think that's unrealistic and frankly a waste of money. There's lots of other things we can do, but it's not that.

I suppose one of the things that people mean when they talk about sovereignty is the ability to not be shut down by someone else or controlled by other actors? And do you think there is a route to that that is different to just participating normally in a free market and hoping that things come through such that you have sufficient market power to be able to kind of withstand, you know, situations where you are in conflict and people might want to shut you down?

I think there's a lot you can do and should do and in large part are doing so that you have lower-tiered systems that are open source and yes, Chinese, though you're not visiting a Chinese website to get it done, but you create and control your own version of it. And if the U.S. does something nutty, which occasionally we do, believe it or not, you have some recourse, but it's probably going to be an inferior product. But U.S. companies do that already. I mean, I think Palantir has some version of that behind the scenes. Israel is highly dependent on most of all the United States in different ways. And we're all vulnerable. And a country the size of the United Kingdom has so many dependent vulnerabilities anyway, including intelligence sharing from the United States. You're down to a small number of aircraft carriers, or is it carrier now?

We have two.

You have two. Okay, so it's still carriers. And the AI dependency is just another piece of that mix. It's not the end of the world. If you had a US president who wanted to zap you, he or she could do it. But I think practically speaking, you're likely to be last in line as a target.

Uh, oh, oh good.

Been one of our best friends, whether you always enjoy that role or not. So a lot of other countries should worry about it more than all of you. But to let companies build backup systems in accord with the dictates of normal cybersecurity and not being too dependent on any one particular supply chain and needing resiliency. All that makes sense. The government in different ways can help. I'm all for that. But in a way, that is all fairly routine, and much of it's happening now.

Yeah, I think that— I mean, I think that's right, apart from the fact there are some really important core bits of our national security that you don't want to have sitting in Google Cloud servers in, you know, the Netherlands. Um, so I do think there has to be some relatively small amount of, you know, air-gapped cloud capability. You know, we have the Isambard supercomputer, which does do UK research and could be utilized for other, you know, national security purposes, and data center where, you know, defense data can sit. And, and energy, to be honest, I mean, it feels to me that You know, if you look at the stack of things that make up an AI system, um, abundant and cheap energy— and so many conversations today have come down to this as well— abundant and cheap energy is the most important thing we should be building. And that gives us sovereignty in all kinds of different ways, not just in AI. But that feels to me where I would prioritize actual, you know, onshore, robust, strong, sovereign assets. But the rest of it, as I said, I think is about optionality and capabilities. And there I think there are assets which we can invest, which aren't about chips or data centers, but about people, which again give you that sort of sovereign capability in this new concept, um, that actually can be applied whatever the situation. You're right, we talk about if, you know, oddly enough the US became crazy— that might not ever happen again, but you know, maybe it will. But there's also things like COVID, or there's, you know, scenarios where you want to be able to know that you can pivot, change, rebuild systems. And that, for me, it's about the talent. That's about having people here that can turn their mind to it. Like, like during COVID you know, as far as I remember, half of the Google team went to go and try and build, you know, uh, contract mapping systems in, in the White House. I mean, those— that sort of asset is real and we should invest in.

Strong agree on all that. Since we're at a Civic Future event, just paying your civil servants more. And getting better people is one of the top, after energy. Have a better civil service that's better rewarded and less brain drain is one of the best things you can do for all sorts of problems.

Yeah, I mean, I have to say, I think that the previous government and this government have done rather well. There's the IAI, the incubator for AI in Number 10. They are doing quite a lot of that actually, and amazingly have managed to pay civil servants with technical skills, a lot more than you'd imagine. But I think, I mean, I think this country, one of the things that makes us different from, you know, the Valley is I think the diversity of talent you have in one place. So when I lived in Palo Alto for 5 years, it was amazingly monocultural. And I think a lot of the sort of hypotheses about where AI is gonna end up is because those hypotheses, those predictions are made by engineers who don't actually engage with the real world very often. You know, the idea that we're gonna have sort of, you know, in the next 18 months, you know, sort of robots who are taking all your jobs is just not how the world works. And yet the engineers see curves and go, "Oh, that's what's gonna happen." You know, therefore we're all gonna be out of a job. And I think that the talent that you get in this country and a few others, but not that many, is you can go to a dinner party and have an artist and a politician and an engineer, all world-class, together. And I think if, if AI is going to be the sort of intelligence layer for all of us and all of our industries going forward, you want it to be built by everyone. You don't just want it to be built by, you know, engineers.

Tyler, you talked about building bargaining power, and there's a version of an argument, and I think Aled made a version of this this morning, that One of the ways that you build sovereignty is by building bargaining power, is by owning things that are scarce, owning choke points in a supply chain and being able to use that as leverage. I mean, do you think there is any scope for the UK trying to be strategic, identify a point and say, we're going to own that choke point, we're going to own that thing that is scarce, double down on that as a way of securing bargaining power? Would you write off that argument?

Well, you have what might be the world's greatest city, right? That's something. And two of the world's greatest universities and a bunch more which are very, very good. My concern about London in particular is it will face competition from AI-intensive startups that might eat away at its advantages in business consulting, law, banking, you know, life sciences and southern England more broadly. And whether you're already with startups of your own that are manned by what I call AI maniacs, people who are maybe outside the establishment, not always credentialed, not the typical kind of Oxbridge person who becomes some sort of partner and lives in Belgravia in London, whatever, goes to restaurants in Mayfair. And that part of London which is central to what the city is now, I think, is highly vulnerable. And you wanna make sure that the groups competing with those services are sufficiently British and well-rooted here, So the surplus of London is not being sent abroad. And that's not mainly a governmental thing. I think it's largely cultural. Like, do AI maniacs here feel at home? Do they have high status? Are young people encouraged to become AI maniacs? Is the whole status hierarchy a bit too rigid and snooty? And there's too much status from being negative to people rather than positive? All that could evolve pretty quickly. And maybe it's better than it used to be. But that, I think, is a central question for keeping London great and keeping it as a kind of supply chain asset, which now is super valuable, could be even more valuable. It just has to adapt a bit to the new world of AI.

Sarah, should we grab a choke point? Is London the choke point?

I think it's too early to know. I get a bit frustrated about the idea that the frontier models, that this is, this is what it's going to be like. This is the AI future. And I just think we're at the start of something. I just don't think we know what's gonna happen next. Are the frontier models gonna be like the utilities and actually all the value comes at the app layer at the top? We just don't know. And so I think trying to spend a lot of money, 'cause everything is massively overpriced at the moment, whether it be talent or chips or whatever, a lot of money now making that bet feels to me not the right answer. I, I like the, you know, radical innovation view where you basically put a lot of bets, small bets, on lots of different things and allow that, you know, to happen through the talent that, that exists. Um, so I, I'm skeptical about whether now's the time I'd put money into a lot of that stuff. I mean, it's a shame we don't have a lot of, you know, natural minerals. I mean, that would be great, wouldn't it? If we were Australia, we'd be— we'd know what we'd be focusing on. But right now it's not obvious to me what the UK choke point might be.

To what extent do you think that ownership matters in sovereignty? So we talk about DeepMind as a great sort of British success story, obviously not owned in Britain. Mark was here talking about Faculty this morning, obviously sold to Accenture. To what extent should we as a nation be bothered by the things that are successful being bought out by Americans? I'll ask the American.

You've had to get used to that a long time ago. I mean, Google DeepMind is the most obvious example, but in dollar terms, not even the largest. But I think what ownership means even will be different in this future. So if you think about the question, does the United States have AI sovereignty? We've actually found we're in a position where our government finds it hard to regulate the leading AI companies because it needs the products. Regulation would slow them down too much. Making everyone go through a security clearance, well, that would get rid of all the Chinese researchers who are a pretty high percentage. Like locking down the labs, maybe you ruin the culture. There's remarkably little we feel we can do without killing the golden goose. So we have this weird bargaining game with the major AI companies and the government. And we cannot apply rule of law to that bargaining game. Rule of law is too slow. Congress is too ill-informed. AI changes too rapidly. So we're doing a kind of regulation by threat or by back and forth and by personality. That sounds terrible. I guess it is terrible. But I don't really see what else we're supposed to do. I don't want to set up an FDA to have to approve every new AI release. And the Trump administration rightfully doesn't want to do that either. So all the old paradigms we have about regulating things or owning things, I think they're quite out of date for the major AI labs. And we're in this new mode where we're just making stuff up as we go along. It's a bit like World War II in a way. Like, you couldn't fight the war by regulatory agency. You just did things You hoped for the best. My goodness, you hoped you had the right leaders here. You had Churchill. And we're living that now, but I don't think many citizens realize it. But with super high stakes, so far I think it's actually gone pretty well for US certainly, but even for UK. And we're just going to keep on living that, and its pace might even accelerate, and people are not even noticing.

Yeah, I think when an industry is moving as fast as this is, I don't see what the alternative is, to be honest. Like, even if you wanted to try and control it, which you shouldn't, it's, it's impossible. I, I think the, the ownership question is, is interesting for people in this country, particularly as there is more public money going into private companies trying to work in this field. Like, I think we, we talked about this a lot at ARIA when we were setting up, um, the board and the organization about, you know, when we give grants— and we can give grants to any type of organization anywhere in the world Would we ask for anything in return? Now, obviously, it's to build a bit of, you know, one of the programs, but we decided not to have any IP restrictions or requirements at all. No equity, nothing. There's a small clause that if you launch and make a successful business in another country, we can— there's a license fee, but basically, it was as free a grant as possible because we took the view that the goal is to create things that will bring benefit to the world, and any restrictions in the way we disincentivize people to do that. Um, DSIT have launched a Sovereign AI Fund. You probably know about it. It's half a billion pounds. Um, I don't know if they have any equity or, or restrictions or requirements on, on the grants, but they're quite big grants to quite large companies. You know, Kraken, which is very successful, has been given a grant. Um, so I do think it's an important question when we know the failure mode in this country is that most startups go overseas to raise their next round. So I think we need to think about this properly. I don't think we should just assume that the startup model is the way to bring public benefit, particularly in a country where that has not worked so far in its entirety. I'm not saying, you know, DeepMind being here is a great asset, don't get me wrong. I, I have many friends work at DeepMind, and I love them being here. But, uh, I think things like, you know, golden shares, or like, if you're going to put money into a private company that becomes very successful and you want them to stay here, what other assets do we have to try and incentivize that to happen? Um, I'd also say— it's an aside, but I think it's related— um, we used to be really good in this country at setting up organizations that were neither private nor public. So my example is the BBC. You know, it's a very odd structure, the BBC. The money comes from the license fee, it doesn't come from the Treasury, and it's a public good. And I think in this field, and actually probably in a few very big spaces where there are big problems to be solved, actually you don't want to have startups or the government running the vehicles for solving it. You might want to have a new sort of entity where there's a different capital stack and a different set of incentives on top of it. And that for me is where there's an interesting ownership conversation to be had.

Tyler Cowen, in your last answer you said the Trump administration doesn't want to be kind of approving model release by model release. But of course, we have just had a situation where, you know, they were approving model release by model release. Do you think that's a, like, stable equilibrium, an unstable equilibrium? Where do you think we will end up with that?

I don't think there's any stable equilibrium with the regulation of AI. The Trump administration does not want to do it by regulatory agency, but the Trump administration wants to do it with simply its people at the table making decisions. That makes me extremely nervous. It goes against all my instincts, but I nonetheless see, I think they've had the best regulatory policy toward AI of any country. It's the one thing they've more or less gotten right. And so many other things they've screwed up, like corruption, immigration policy, very long list. But US AI companies have done great and they're still going great guns. And the notion that you have to sit down with a bunch of people across the table It can't last that way forever, but I actually don't want to change that. And there's no way to do away with the table. It gives very high priority to views from the national security establishment. In the USG, I think they're the people who understand this issue the best. And again, something like the World War II analogy to me is not crazy. Of course, a lot of those decisions are discretionary and should be concentrated, and not in the normal sense either democratic or rule of law. You just have to do them and kind of push, push, push.

Can— so I hear you, and as someone who used to work at a big tech company, I agree that the sort of don't regulate until you absolutely have to is the right approach. But do you think the sort of nascent, and I think growing, backlash against AI in the US could have been mitigated by some more interventionist policies on part of the federal government?

I don't think so. I think you said before we're early in this. To me, it's like the first inning. And the actual problem is AI does mean most lives will have to change a lot. And I think the net benefit is quite large, but people don't like change. And especially educated and professional classes are realizing that. And that's going to happen anyway. So my preferred way of thinking about it is you just want as much positive change as possible before the opposition is too mobilized. And the Trump people have done a good job at that, even if you disagree with many, many other aspects of that administration. So I hand it to them, and they're trying to keep all the plates balanced, and that will at some point prove impossible. But that's the sense in which I'm an accelerationist.

I think it's really interesting observing the backlash against— I mean, I have 3 teenagers, and 2.5 of them are fairly opposed to AI for different reasons. And I think it's a really interesting misjudgment on the part of the big AI companies, frankly, to not understand that social license is a really important part of any consumer technology to actually have the scale you want it to have. And it's hard to do. It's not just a sort of, you know, great products. Who's not going to love that? No, it's what you say and how you do it really matters. Um, when we started doing self-driving cars, we were really conscious of the fact that, uh, there was a significant risk that people in Palo Alto would have blacked-out SUVs with no one behind the wheel going 80 miles past their kids' school, and it would not have the desired effect. Uh, and so we worked really hard to make the cars really small and cute and, like, work with disabled communities and blind communities to talk about the— like, it was really thoughtful about that social license thing. And I do, I'm curious to know where this is gonna go.

Even Waymo, which I love, Washington, D.C. won't do it. New York City may be turning its back on it. The incoming government seems to be getting cold feet for London. And it looks and feels as innocuous as possible. If there's a problem with current Waymo, it's that it's too safe. It refuses to break the law when a slight amount of discretion might, in a funny way, be safer for everyone. So I think a lot of the opposition is inevitable. And again, Waymo has done so well, and the places where you have it— Austin, you know, Scottsdale, San Francisco— the record is so good, it hasn't mattered nearly as much as it ought to have.

Pick up a little bit on this. And when I speak to people who aren't in an AI world about what they're worried about and what they're thinking about, one of the big backlashes you get is about the sense of importing values or AI coming with values that maybe they don't agree with. And we seem in this conversation to be pretty relaxed about being kind of consumers of models developed elsewhere, but AI does have values that it comes with. And I mean, Tyler, you've written before, I think I'm paraphrasing, but you know what, maybe the greatest achievement of Western civilization was producing most of the text that happened to be on the internet when we were training large language models. That's because it's, you know, therefore being built into those models is a sense of values and philosophy. And should we be more worried about a kind of cognitive layer of society being something that is outside of control within our society, whether that's under like democratic control or other?

I think the disorientation over time will be extreme. So just one simple example that people hardly ever talk about, I think even now, forget about advances, you could just create like your own branch of Hinduism with sacred texts, thousands of pages, which you would do through probably GPT-5.6 now, or maybe Fable-5. And maybe only half a percent of the human race wants to build out their own religion or part of a religion. But if you have half a percent of people doing that at close to zero cost over time, as token costs fall, religion all of a sudden looks very different. And people don't like weirdness. To them, Waymo is weird. GPT is weird. This is weird. That is weird. To the people who grow up with it, it will seem quite normal. Just like our differences with, say, medieval society to medieval individuals would have been super weird. But we're used to it. And I think that's where a lot of the opposition comes from. Weirdness and not liking change and not liking the fact that both your job will change very much and you cannot exactly foresee what your children should be doing. You cannot tell your kid, "Here's the world you will grow up in. Here's what it will look like." And when I was growing up, I had this expectation what being a professor would be like. It's in fact more or less exactly what I got. That's the world people want, and that's not the world we're giving them right now. And there's not any good way around that. I'm all for it. I'm like, "Press the button. Live with stress." I think a lot of the AI doomers, they're actually people who just are not good at living with stress. And they're in this monoculture, they're not personally good at dealing with stress, and they come up with all these weird stories to manage their own stress. But I say, let's do it.

I hate to suggest that you might not be usual in that regard.

I mean, I'm with you, I like change, so I don't find it stressful. But I mean, it's interesting that, 'cause not only is AI providing this level of insecurity in the world, but the world is kind of insecure. I mean, I think we discount the extent to which a changing climate has actually made people for the last 15 years actually quite scared about the— and not able to— no, with no agency to affect it. I think there's a lot of things that are happening in the world that bring that level of unpredictability. And I know at least two people who live in Silicon Valley who have AI as their god, as their religion. And so actually, you know, in times— I know— so in times of stress and uncertainty, you know, why are conspiracy theories so popular? Because we're stressed and we need some answers. We need to make sense of the world. That's what religion does. And I, I maybe suggest that it's not that AI is going to unpick religion, but it's going to become a religion to give us some answers to what is otherwise a very scary world out there.

Sarah, how do we apply thinking about this to children and the regulation or protections that we do or don't need for kids who are growing up in this world and having this technology and struggling to deal with that stress dislocation?

Well, I, I've, I've sort of come full circle on this. When I worked at Google, I was definitely in the— there's no actual evidence that this is— I was working on YouTube at the time— there's no evidence this is a problem for young people. My children happen to be very young at the time. Uh, 15 years later, my children are a lot older, and I'm all in on a ban, I have to say. Uh, not because I think, uh, YouTube is a problem of all the social media platforms. I think it's— I don't actually think of it as social media. I think of it more in the sort of broadcast category. Um, but I think that there is enough correlative evidence, and anyone in this room who's a parent knows the daily struggle of, of persuading children not to be on phones. Um, so I think we're in the scenario where actually politicians are, which is, you know, you don't have perfect information, you're not 100% sure what the effects of something are,. But when 9 out of 10 of your parents, voters, tell you it should be banned, it is your responsibility to do it. Um, I also think, and this is a sort of a more of a sociological impact of a ban, it's not going to be perfect. Um, and I'm sure of all the tools that we might use, it's the least, uh, it's the least useful version. Like, this is the first version we're going to have. Um, but it's an opportunity for parents to talk to their children about the issue. It's a forcing function for a greater social, uh, educative process, uh, whereby actually we're all forced to recognize that it's our responsibility as parents to, to exercise some power. And that sort of collective responsibility within a community, within a school, is easier to— it's easy to pull that— but pull that lever, if you like, with a ban as the starting point.

On this we very much agree. I'm strongly opposed to a ban. If you put the Bill of Rights up for a majority vote, I'm not sure it would get 1 vote out of 10. A ban for under 16, it's the first step toward not having free speech in your country. I was speaking recently with a 15-year-old in Australia, which instituted the ban. He said, all my friends were still on it. And the next step, if you want to enforce the ban, is to make everyone register, everyone verify their identity. And to do this in the United Kingdom in particular, which is a country where, as I read, 30 times a day, like, a policeman shows up at someone's house because of something they posted on social media, and that anonymous posting will no longer be possible because we wanted to ban social media for young people. I think you have given up on what is the greatest British heritage that you've produced, which is John Milton and a culture of free speech. And the power of the state will get larger and larger and larger. And the alternative of relying on norms to evolve, social media to evolve— In funny ways, AI may help solve a lot of the problems of social media. "Oh, I don't feel like texting today. You know, I'll have Claude manage my account and give me a summary 2 days later." Whatever people resort to, or maybe Instagram will become all slop because it's AI-created 80% or something, and people go off it. So we're also in like the first inning and a half of social media. And to pass a law we're not gonna reverse that will irreparably damage free speech when norms are still evolving, and we should rely on parents to make these decisions and not turn that power over to the state. A democratic state where most voters in either of our countries are not really fully on board with free speech, I think is one of the greatest, greatest mistakes we could make, and I dearly hope the United Kingdom comes to its senses. Fortunately, in the US, the way free speech law is interpreted by our Supreme Court, such a ban violates free speech, and we won't do it. So I hope here you make the same decision.

Last thing from—

Wow.

I sense a debate that's been missing from today. And I mean, last thing for me, if we just both answer quickly and we'll go over to the floor. I mean, if you— if I were able to give you £10 billion to spend on improving AI sovereignty in the UK, interpreted however you want, how would you— how would you spend your money? Sarah?

Um, I would buy Springer the publisher of Nature magazine, uh, and create— and any other sort of associated sources of science material— and create a small language model, uh, that is AI for science, uniquely British dataset, uh, that any British-hosted startup could utilize at a small fee, uh, and create a sort of national advantage in AI science. In this country?

I'm not opposed to Britain doing that, but I think the key issue is military sovereignty. You need to think national defence. If you had a real military again, an actual preparedness, some of that is going to mean drones, but whatever is the best way to spend that money to revitalise your military, I think that's going to boost your sovereignty in all scenarios, AI or otherwise. So I would say get on that. Simply going up to what, 3.5% 100% spending it on domestic companies doing things maybe like tanks or heavy equipment that may not be useful. That's not the way to go. But whatever is the futuristic way to be ready for the next wars, Britain needs to be there. Britain has been awesomely good at that in the past for centuries. It's hard for me to believe that cultural capability has left this country entirely. And that's what I would do either for AI reasons or just for reason reasons.

Any questions from the floor? Great. Okay, let's take a couple at a time. Seb, your hand went up first. Oh, there are just two Sebs. Great. Both Sebs. The Seb with a microphone and then the Seb without a microphone. Just like the streams colliding, two in one place. Just to go to your point, Tyler, about Brexit and the UK working with other mid-sized nations. I mean, if you look at the EU's AI policy, it is truly awful. In the sense that they've reverted to instincts which you could argue had the UK remained in the EU, it might have influenced them in a different direction. But all that regulation has been to try and limit, to try and restrict, and do everything possible to stop any kind of innovation on that. Whereas I think the UK has gone in a different direction because it's not in the EU on this particular basis. So if you were thinking of other mid-sized countries that have got the AI regulation, right, Who would they be? Who should the UK work with? Because when you look at Europe, it doesn't strike me there's anyone obvious we can form sovereignty with there. And can I just take— let's just take two together at a time. And so could the microphone turn Seb without a microphone into Seb with a microphone? It's Harvey.

It's Harvey.

As not Seb, my question is about So with the— basically with Fable-5, we have this big hype for it and then the Trump administration employs an export ban and then just like that, the model is no longer available. 3 weeks later, my Max plan is out. Thank you very much, Anthropic. So I'm curious about how much authority the state actually has in— with the state, kind of like state-company relationship. How much authority does the state have over these technologies and how much leverage could the companies actually utilize? Because I don't think that they really do have much capacity to really— if the government wants to ban it, then they will ban it from being opened.

First, you know, the EU regulation is terrible, uh, and that's the strongest argument for Brexit in my opinion, actually, is AI. But there's a tendency in the UK I find to not distinguish between when coming in third is highly worthwhile and when coming in third is almost meaningless. And there's some parts of the AI stack where right now you're coming in third, but it means very little. But there's different statistics you can produce on startups and the like where you're clearly coming in third, but it may not amount to anything. I think the key thing is just non-AI-defined businesses in the UK. How good a job do they do of incorporating AI to boost their productivity? And it's not clear where you're coming in in that regard. You're not yet coming in third. You might come in third. There's a very good chance you'll come in third. And if you can do that, coming in third is worth a lot, even if you come in some other place for pure AI. You asked about mid-sized countries. The places that really get it are not mid-sized. For me, it's Singapore and UAE. Canada seems to be waking up, but I don't know many mid-sized countries that are doing a good job on AI policy. Like simply not doing anything wrong is not worth that much at the moment. It's like having some coherent strategy, and Singapore and UAE definitely do. Now you might think they're making some mistakes, but they're also willing to respond to feedback if they're making mistakes. So I would praise those places. Like how much power do the companies have? Well, it's only a few weeks after the ban. 5.6. Came out yesterday. It's simply out. You can debate, you know, how good different models are as hacking tools, but Fable 5 is back. People interpret that episode differently. I was opposed to, you know, the export directive from the Trump people. But I think on net, if you look at it realistically, the whole episode has gone relatively well compared to how U.S. government usually does things. And we got something out of our system. We accepted a certain degree of risk. We're living with it. We are going to get zapped on a bunch of issues. But in the meantime, some of our institutions are shoring up their own defenses with these tools, which is incredibly important. And we have a sense of urgency because it became a headline. And the U— national security state in the U.S. can use these tools to hack foreign adversaries. I think it's going really well. So the companies have a lot of leverage. That's good. Over time, might they get too much leverage? That's possible. But for now, I think the balancing act is going okay.

Don't you think the ban— I mean, this conversation, AI sovereignty, would have not been on the agenda were it not for the ban? Like, people who don't understand AI at all are all of a sudden panicked about sovereignty capacity and are looking at open-source models to— like, I sort of feel there's a number of things the Trump administration have done that in the long run will backfire. Banning chips from export to China, guess what, will make more efficient models as a result. Like, you're, you're forcing innovation in places that you were market leaders in, and it doesn't have to be that way. You know, everyone was very happy having the US being the world superpower apart from China, and now everyone's looking for someone else to do middle power deals with. Um, in terms of middle powers to pick, you know, who doesn't love Canada? What a great place. A lot of cultural and also great academic skills in, in this context. Iceland, for its enormous amounts of cheap energy. Uh, I, I think there's some geographically very close countries we should be working with outside of the traditional EU countries that actually could give us much more leverage in places we don't have it.

I'll take a couple of questions again. We've got one down here and then Juliette. Oh, sorry, just here.

Okay.

Yeah, I think with the Fable situation, like, we were in the absurd position for a while where even people who worked Anthropic and weren't US citizens couldn't work on Frontier models. And, um, you know, people who weren't in the US couldn't access the model as well. Um, and like, I think Yeah, being frozen out of, uh, front-end model access, like, does pose, like, a strategic risk for the UK. And I guess, like, what steps do you think the UK could take to make sure it's not outside the tent, uh, going forward and, like, has— can rely on having continual access to front-end models?

And then go back there.

I was struck recently, uh, seeing a conversation about AI sovereignty happening within Silicon Valley, and the, the meaning applied to it was not national sovereignty but corporate sovereignty, as to Anthropic stealing your data and then replicating your business model and basically, basically driving you out of business, like a kind of modern-day Microsoft or Google in the early days. And I wondered if, you know, the, the UK's position on sovereignty surely relates to what companies we have and what choke points they're currently using. And, you know, maybe those won't be choke points in the future, as you were saying earlier. So how can we— it seems, it seems the idea of sort of cooperating with other mid-sized powers to to create AI sovereignty is slightly meaningless unless— until you know where in the industry or the stack that you sit and, and what, what assets you have are under threat.

Sarah, do you want to start?

Yeah, I mean, it's hard, and particularly right now. I don't think there's any— I don't know any country who's got a strategy for this period we're in that is solid. Everyone's making a bet on, on it going in a certain way. Um, and let's be honest, we don't know it's going to become the sort of frontier model future. We just don't know that yet. There's a lot of stuff that could still go wrong. Um, I do think this issue about unique data is quite an important asset that we still have access to that has not been lost. You know, the models are based on data that they can access, and there's quite a lot that isn't in there. Um, so asking ourselves some honest questions about where are there small models that can be built that can be unique to this country, where are there assets that actually are not already being sort of decimated as, as the frontier models grow. I think that's an important question. To the question about how do we get access to, to ensure access to frontier models, I mean, I think this current U.S. administration is an unusual one, shall I say. And, and in other times, you would expect there to be some sort of negotiation capacity, if not to get access to the most current model, but to the, the, to the, to the generation below. That feels to me like something that is a very reasonable position for even the most, uh, sort of ambitious and non-friendly US administration to have, is say, we're going to keep the most current one, but you can have the one that's 6 months old.

Alex Karp at Palantir, some of the biomedical companies, they raise the concern you mentioned. I think it's great news that they're worried. What they're really saying is they're afraid Anthropic, OpenAI, whichever other companies, you know, Google, can outcompete them. They're convinced AI is really powerful. And if they're worried, I mean, I'm celebrating. I have nothing against those companies at all. I don't think we're headed for a world where Anthropic owns every company because they've created the IP for anything humans can do. I just don't think we'll come close to that happening. But when other companies get worried, I'm getting excited. That's how I would put it.

We'll take a couple more. There's a gentleman on the aisle here, and then, and then we'll go to Sam.

Hey, uh, thanks both. Uh, so to what extent, uh, do you think this AI sovereignty concern is a moral panic. Two reasons to ask this question that you both observed throughout. So one on the open source model, Sarah, as you've spoken to, every 5 to 6 months it looks like they're catching up with the frontier models to 80 to 90% of their capability. If that's true, then you can just host it and like the marginal cost is just sort of, you know, compute and electricity. And Tyler, as you've described before on the defense national security side, there's a whole range of assets that the US by default does not export over interested partners, which are all of those governed by ITAR. There's also a huge amount of data that the US doesn't share with even its most trusted partners, i.e., the UK, which is no foreign data. What makes AI unique given these two things, and to what extent do you think some of these concerns are moral panic?

Let's just take, take one more. Right, right next to you.

Um, is one way to attain a form of AI sovereignty to be just a really, really big and important customer? I can imagine a scenario where the NHS could be the largest purchaser of tokens in the world because it's a very, very, very large company. Lots of British businesses could adopt AI at a much faster rate than foreign companies, and we could just be really, really important. I know your president is a big fan of exports and is willing to give access to quite important AI infrastructure purely for like soybeans. So is this a potential route of becoming sovereign?

Yeah, you have incredible biomedical data here through the NHS, and that's a potential choke point that you can use to British advantage. I don't know how far you've gotten with that so far, but I think you're one of the countries in the best position to just come up with a lot of new innovations, as you've done in the past, you know, during COVID times. It was incredible what came out of the UK. So the, the value of that is much, much higher. Than it used to be. You know, open source, people say it's 9 months behind. I think that's a little misleading. I don't think it means in 9 months it will, you know, surpass. It'll always be 9 months behind, perhaps, because there's something derivative about it. And 9 months is a long time in AI land. And being 9 months behind is actually being very behind in any area where reliability matters. So I guess I've become over time a little more skeptical about open source. And I know all the evaluations and what comes out and the new meta evaluations. I look at all that. But at the end of the day, I think for the important things, people want the very best models for at least a while to come. They won't be open source. They'll be from 2, maybe 3 companies. And that's the lay of the land. And that's actually really good news for the West. But that's what I expect for a while. Is it a moral panic? Yes, it's a moral panic, but moral panics can be good. Like, I think the whole Fable thing was a godsend for the UK, that now you're not asleep. Yes, the US did something stupid. I was against it. But like in business, sometimes times come where the main thing you need to do is just educate your board of directors or maybe your customers. And in AI land, in a way, we're there now, and everyone in the world is waking up and figuring out what they're gonna do because the Fable thing happened. And it's a blessing in disguise, even though I think it was a stupid, ill-conceived policy. So I think even the US is better off when you're not so dependent on us. And I want the UK, even from a purely American point of view, to have a lot more AI autonomy to the extent that's possible. It's not good for us when everyone depends on us, whether it's military AI or whatever else.

I think we can add— we'll add the NHS data into my £10 billion, uh, AI for science small model. I think that's a good one. Um, I don't think we should rely upon being big customers as a way to— as a route out of, of where we are now. I think we want to be builders. We just— also, we're never going to write checks big enough to compare to other checks that are going to be written to Anthropic. That's just— that's just never going to be the, the one asset that we can put on the table. But we do have great people building great things. Um, actually, we've— at RA, we've got a program which is, um, in chips and semiconductors. And to your point, like, I'm not sure it would have got the, the attention and the novelty it does— it has— had it not been for the sudden sort of like massive uptake in, in GPU, uh, cost and, and the shortage of them as a result. I think the innovation that has come from the recognition of where we are going in terms of energy costs and supply chain shortages. You can see in the UK in the last year how much innovation's come around. So I, I do think it— you do need to kick up the ass.

To summarize, uh, we'll take a last couple of questions. So there's a hand on the aisle there and one over from you, and then we'll come back to Julia and call it a day. Thanks, uh, very much. Um, I love the phrase AI maniacs, and I share your enthusiasm creating a sovereign supply of AI maniacs. When it comes to looking at our education system, I'm interested in what you think an ideal system would look like. Do we say buy all the secondary school kids a pro subscription, just let them vibe code or, you know, give them a sandbox they can play around in? Do we take them away for a week and teach them how to focus on ideas that can scale, come up with good ideas in the first place, be ethical? I'm just interested in your thoughts as to how we do it in the future. Although I do think I'm against testing and having exams for this. Thank you.

I don't have any grand overall—

Can we just go— we'll grab a couple. So there's just to your right.

So talking about Mavericks again, 5 of my friends have moved to San Francisco in the past 6 months, all of them tech or founders. And how do we stop the brain drain?

And we'll just do the last one. If the microphone can just back down to the front. Thank you. Sorry, I'm giving you a workout.

I have lots of friends who are AI doomers. Some of them are— do have anxiety disorders. I also know some— I also know some people who are anti-AI for for social justice reasons or because they don't like what it's going to do to the fabric of society. These two groups are very different, but both of them I think are converging around solutions from a safety perspective or from a society perspective that look to me quite authoritarian or anti-freedom. Is this something that you have also seen? Is this something that you worry about? What is the likelihood that we have a free and plural society in an age of strong AI?

So 3 things we've got: school, brain drain, and opponents and authoritarian safetyism. Um, Sarah, do you want to start?

Um, I think all 3 of them are about young people, actually, you know, future career opportunities, you know, how they, how they view, uh, the future in general. Um, I, I actually think our— I mean, there was another session earlier about our education system not focusing enough on history. I actually think we've got a great education system in this country. I'm a big fan. I've got 3 children who've been through the state system, and they are taught a lot of history, uh, and they're also taught actually how to think critically and challenge, uh, sources that they don't trust, and to know the difference between fake and true news. Um, I think it's worked for many different, uh, applications, technology in the past. I don't see the case for everyone becoming coders, and let's be honest, it's the first career that seems to be under threat from AI anyway. So I'm rather confident, certainly about the British curriculum being able to flex to solve for the future. And I don't think moving to San Francisco is a bad thing. I spent 5 years and came back, and I prefer it here. So what can I say?

Education, I think we need a lot more experimentation and feedback. My intuition is to think you need to lock kids in a room more without AI and just have them write. I think you do more of that already than we do in the US. So that's great. But we probably need to do more of that. And then you need to teach everyone how to use AI. The schools should do that. The teachers, for the most part, don't know. The students know more than the teachers. So I don't know how to get to that end. But ideally, that's where we would want to be. Brain drain. I think young people in the UK need higher status. Than what they get. And culturally, there's more seniority here, and there's greater negativity to attitudes, and less of a can-do or tolerance of failure. And that could all change quite quickly, and changing that is a free lunch, it doesn't cost you anything, it'll bring more revenue into the till, and more of that can happen. And then, you know, free society, I think our notions of freedom will change. I think any society with strong AI will have more surveillance than what I personally want to see. And this to me is one of the downsides of strong AI. There's already a lot of surveillance in the UK, closed-circuit television. But the notion that all the surveillance we have in any society is now so easily searchable, to me is scary and a negative. I just don't see how we avoid that. I don't think voters really mind. I think voters mostly like surveillance done selectively or maybe invisibly. And I think we're headed for more of that. That's one of my big worries about the future. The stopping AI involves a lot more surveillance yet. Basically, you have to control everyone's computer in different ways. And, you know, shut down VPNs or anything like a VPN, that's not even possible. But just trying to do it would wreck liberty. So we're going to be putting up with more surveillance. I don't like that. So, but be ready.

On that extremely positive and uplifting note, Sarah Tyler, thank you very much.