Claude Myhtos, UBI, Superintelligence, Alignment, Energy and Nukes

Welcome to another AI Wisdom Pod. We have Lindley Lenthati with us today. Lindley's a PhD, he's a director of Cambridge Inference, uh, in Cambridge of course. Uh, the guy is very interesting. I spoke with him this week and was really intrigued in his background in astrophysics, computer science of course, was the chief data scientist for Cambridge Machines Asset Management. So was a quant as well. Member of Founders Pledge. I interviewed Andy McLachlan and I decided to reach out because Andy owns, uh, FP, uh, and yeah, uh, Lindley, welcome to the pod. Tell us a bit more about yourself and, uh, what you're up to.

Yeah, so, uh, as you said, so I'm co-founder of this tech startup in Cambridge, UK, uh, called Cambridge Imprints. We are building AI safety and monitoring tools for regulated industries like banking, looking to get into defense Uh, and as of Wednesday this week, I've also founded a nonprofit called Tertium AI that is looking to help build AI in middle powers.

In middle what?

Sorry, middle powers.

And what does that mean?

Uh, so geopolitical middle powers, as in, uh, the UK, Europe, uh, Canada, Australia, these sorts of places.

Won't that become not so relevant when Trump goes out of power?

Well, I mean, this assumes that Trump goes out of power. And, but no, I think that even with the next president, I mean, who knows who we're going to get, right? And I think independent of who the president is, I would be very surprised if at some point in the next few years, the US government doesn't decide that sharing AI beyond some capability level is not just a huge strategic error and just stops doing that completely.

Yeah, it's like when you have all the power in the world, what are your incentives to share it, right? Like, the, the cards are not stacked for you to want to be charitable. And but the counterargument to that is that if we unite— I mean, if we say the West, right, which is silly— I've always been a guy that thinks and acts globally, right? But the West, if we unite, we could build something even bigger and better. I think the current president right now— and why the emphasis is on that is that it's basic win-lose, his, his mindset. And while the best businesses on this planet are— they, they do some win-lose, but they also do a lot of win-wins, which means uniting with these countries, building data centers there, protecting them. And so forth. So I think it's just like a basic mindset misunderstanding. I think most smart people understand that uniting our forces means bigger pie and bigger share of the pie for everyone. Don't you think that?

Um, well, I mean, I think a reasonable historical analog to this, uh, is when we were developing nuclear weapons, right? I mean, here UK scientists helped with the Manhattan Project We helped, you know, we helped the US build nuclear weapons. And then Truman et al turned around and just cut everyone off, including us, basically refused to share the technology. And they did this because not sharing nuclear weapons and hoping that they could amass enough to kind of, you know, have a massive military advantage over the rest of the world was just for America, the right strategic choice. And I kind of think that the same logic just applies to recursively self-improving AI.

I love making analogies between nukes and superintelligence, but is it the right analogy? Is it the best analogy we can do? Because nukes were physical, they were, yeah, built in a lab, and the consequence was destructive. Like superintelligence, we love speculating that Yeah, there might be a Terminator side of things. I think you and I spoke about the less than 30% odds of shit going very bad, which doesn't mean like full extermination of humanity, but may— it may certainly mean like a bunch of people losing their jobs, protests, humans killing each other as a result. That would be way more probable. Yeah. Do you feel it's the right analogy, the nukes and superintelligence?

Yeah, so I think the The analogy isn't through the destructive power per se, it's just through the political leverage that it allows you to exert. Um, if superintelligence— if superintelligence allows the US to just accelerate away technologically from the rest of the world and do this kind of, you know, William MacAskill century-in-a-decade style progress, then this just means that they will be so much more advanced than any other country in the world, that they can basically dictate terms to whomever they want.

Yeah, but at that point, if we can't contain the superintelligence, I don't think superintelligence would be biased. I think only for a short period of time we can technically hold it until it decides to not be biased towards any country, certainly not a quote-unquote bully country.

Yeah, I guess this depends on how, how alignment works, right? I mean, in principle, we could end up in futures where AI just does the things that we ask, and it's super capable. But, you know, it just— someone just asked it to do X, and it'll just tell them how to do X. And it's not about that thing having views on, you know, how the world should be. I mean, I agree that a better future is one where this AI has those views, as long as those views are obviously aligned with our own interests, but it's not obvious that that's the future we'll get.

Isn't alignment about energy also fundamentally? Because my brain was like, okay, yes, it wants to be useful. If it's useful, the human should give it something back, or the best humans, right, should. But if what we're asking is going to be a net taker, it's not going to be valuable for the full ecosystem, it's going to understand the full ecosystem better and it takes its energy, I don't think it will comply with the, the ask. What do you think about that?

Uh, so when you say energy, do you just mean like the energy requirements to run more?

Yes, and also more broadly, I think my number one trait as a founder and an investor is energy efficiency. Anything I'm going to invest my time and energy in, I expect some kind of return, right? And everything is computed so that I get something, I get a kick out of the time I invest in something. And I think AI, superintelligence, would also reason in some way. Of course, yes, the, the— it's stuck to the grid, uh, to, to chips and all of that. So what do you think about that?

Yeah, so certainly there are models where, yeah, this are like cooperative, uh, alignments where we kind of promise things to the AI and then we do those things. Certainly superintelligence, I think, will want to have more power, want to have more compute. I guess it's just less— I'm less certain that it— what it will be willing to trade off for those things. You know, could, could someone in the US government say, when it has access to this, basically come up with a plan that on the face of it seems, you know, broadly peaceful But results in US culture overriding all other culture in the world.

Well, it's already overriding all other culture in the world.

Um, I think that's quite a strong claim. I think a lot of countries might, uh, push back on this.

I think it's pretty scientific. Like, if you check in every pre— any country in the world, they at least watched a Hollywood movie or a series. They were— they were Nike t-shirt, fake Nike t-shirts, right?

Oh, right. Yeah, yeah. But I mean, you know, there is still you know, other languages than English or American, right?

Oh no, no, for sure. I'm just saying like, yeah, right now it's the superpower. The fact, by the way, I think that's a strong reason why they're superpower, uh, it's culture, it's Hollywood. That's one of my thesis. The second thesis, the brain drain, right? Like, it's, it's like, it's a never— it— they never stop sucking in the best talent, which is great for the US. If I'd be president of that country, I'd be like, wow, I'm sitting on a moat right here. Um, but again, the, the orange man is like pushing away the, the best forms of talent, which is immigrants, right? So he's like playing all of the, the cards wrong.

Um, yeah, I guess with things like the culture, it's the culture thing. It's kind of— it's always a spectrum, right? It's like, at the moment, yes, American culture, uh, is very pervasive around the world, but there's a still a very long tail where, uh, that becomes more forcefully applied to other countries.

Coming back to energy, do you feel energy is the real bottleneck right now? Do you feel the models will keep on scaling forever in their current state, or we need something groundbreaking like, I don't know, neuromorphic chips and just a new approach to frontier AI?

Um, yeah, so right now energy probably is the bottleneck. I think that probably though, I think there's a very reasonable chance that transformer-based architectures with the power requirements that they have will just get us all the way to AGI slash whatever else. And then that system will help build the next generation of architecture that's far more energy efficient, far more data efficient.

I fully agree with that. Your nonprofit, why nonprofit, right? Like, you've seen what happened with OpenAI, and yeah, let's start with that.

Uh, yeah, so broadly speaking, I, I don't think that for-profit companies should be the ones to build superintelligence. Uh, I think that this by and large leads to worse— will probably lead to worse futures than if a nonprofit that isn't being directed by, uh, you know, like shareholders, right? Uh, are the ones that is the thing that's actually building AGI/ASI.

So just like I agree with your thesis that yeah, the current structure is going to lead to a brand new paradigm, um, I think that us continuing on the for-profit structure will yeah, lead will lead to a new paradigm, which is the end of capitalism eventually. So I do feel we need cash and fuel, uh, to finance these breakthroughs until, um, yeah, superintelligence is achieved and eventually UBI. I think that money is just like fuel right now. If you remove like all— I'm not saying you have stigmas, right? I'm saying most people have stigmas with money and it's like, oh, money is bad. And, um, the best the best— well, I mean, yeah, I'm not sure if this is true, but I feel in the next year the most impactful enterprises are going to be social startups, right? Like that are for money but also use that money for good. Can you double-click on why you think the nonprofit is the right structure?

Yeah, so firstly, obviously money is still going to be very important to this nonprofit. The idea though is that that funding would come from a mix of middle power governments So increasingly governments are switching on to the fact that having control over their— the AI that their country uses is going to be increasingly important. I mean, thankfully, uh, and then this is worth something, right? I mean, to come back to the, the nuclear analogy, companies are willing to spend quite a lot every year to ensure that they have nuclear deterrence. And in that same way, I think that countries will just start switching on to the fact that having access to frontier-capable AI is just a national security requirement and that they would be willing to contribute to an organization that's helping them get that.

Do you even think that countries will be a thing in the future? Don't you feel sovereign states make way more sense? And we've— it's just so archaic that we're, we're born into a country and we need to pay taxes to the big man, to someone that we don't even agree with. We do share some values, but not like 95% of values. With that person. And I, I don't see a future in which, uh, governments can hold their shit together if 90% of people lose their jobs and banks, banks fall apart because they depend on salaries and banks are funded by governments.

Yeah, so I, I definitely think that there is a big space of possibility for this, like, post-money, post-border future. Uh, you know, as technology, as technology increases, it's going to be easier and easier for people to move around the world. And, and cheaper and cheaper to do this, right? And so at that point, the, the idea that you even live in a specific place when it's just so easy to travel to the other side of the planet, um, these kind of ideas just start to fade away. I guess I'm worried about the transition to that future. I think that there's a lot of ways that we just don't end up there. Um, some of these are due to loss of control from AI, but just a lot of them are geopolitical you know, power lock-in and country— one country ensuring that we just never reach that state.

I've always— I mean, sort of naive question, and maybe my brain already knows the answer to that, but governments can sort of print money forever and can reinvent the system. Of course, it becomes complex once they trade with other countries, right? Because other countries need to agree with that fact, but Don't you feel that like if everyone loses their jobs, we'll need like a big reset? We'll need like banks to jump in and governments to just be like, oh, here are the new standards. Some form of UBI. Have you thought about— because my form of UBI would be corporations fueled, or as much as a dystopia as this sounds like, I think corporations would be the best place to redistribute value if if they, they have the right payment systems, right? Like OpenAI's monthly thing, not sure how much it's gonna scale, but, uh, yeah, any thoughts on this?

Yeah, I mean, I think UBI is our best— the best idea that we have at the moment. But as you say, it depends on, um, governments and companies working together and having some tax infrastructure that actually works. I mean, Tax at the moment is kind of notoriously broken all around the world, uh, and is also often just governed by the people who have the most money. Um, so yeah, it's not, it's not obvious to me. Like, let's— Elon Musk, right? Uh, he clearly— I think he, he understands this triad of build AI, get in space, have robots. And, you know, I could easily see him being the person that is the first person to start mining asteroids and, you know, gathering resources in space. So there's a huge amount of wealth available in the solar system, right, for someone to capture. But there's no space governance. So what happens to all that wealth when the first person goes out there and is like, okay, the asteroid belt, that's mine? How do we— how do we even begin to redistribute that? I just don't think anyone is really thinking about these things on the sorts of scales that they need to be thought about.

Yeah, and I hope Elon's not the guy in charge of UBR, whatever. I don't think this guy's trustable from his last move. Um, yeah, and then, yeah, like, uh, but I get what you're saying. Like, space is limitless, and if you're the first space colonizer, that gives you a huge moat. Um, which would place me and other guys direct competition. That guy's like, oh, I'm gonna launch my own rocket, get the fuck out the way, probably kill your robots that try to take my rocket down or something. Um, which I don't— I think this guy would be capable of that, you know. It's like, oh, I own this space now. And anyway, um, what— where does that take your, your nonprofit? Like, what's the next step for it, right? Like, what's your first achievable goal right in front of you?

Right, yeah. So the first, first goal, um, and what we're pitching at the moment, is to basically get the, the UK to lead a program to just be the best open-weight provider in the world, right? This, I think, is completely achievable for the UK to do on its own. Uh, it's not that expensive, it doesn't require that many GPUs, And this then is the launching point for the kind of middle powers coalition. You bring in Canada, you bring in Australia, you bring in, uh, France, Germany, these sorts of countries, and collectively work to match the US closed frontier. Um, and from there things get a bit harder to determine, uh, exactly what, what happens from that point.

Yeah, I'd love to see that coalitions AI, like I said, I think superintelligence is going to have its own values. But till we can hold AI, I think it's definitely going to have better values than the US says AI. And yeah, maybe even these sets of values would have more scalability over the long term. Who knows? Maybe the— what the US is building is not scalable. Maybe it will fold on itself to some degree. So it's like you onboard governments, they pay, they pay you like multi-millions per year. What do you deliver in exchange?

Uh, the weights. So the idea is that this starts off being with the UK phase like an open weight program, um, up until the point that, you know, like UKAC or other AI security institutes determine that the capabilities of these models are getting too large to continue to release them open weight. And then the weights are shared with, you know, the partners that are in the program. So they contribute capital, they contribute compute or expertise or data, and in return, they get the weights. And then they can just do what they want with those, right? Downstream companies within those countries can provide the inference. National security infrastructure or other critical industries can just run the models themselves. And there's kind of just, you just break this dependency link on the US.

For not technical, for not technical people, what do you mean when you say weights? You mean the key algorithm that would transform that data into the end product? Useful.

So, so the weights as in the, the, the, the numbers that make up the AI model, right? The thing that you run on the GPUs. And when you say, hi Claude, that big black box spits out a, "Hi, how are you doing?" So you would guard these weights with your life.

How would you make sure that these are not being copied?

Right. So this is where, I mean, the kind of the partners that I mentioned there, so Canada, Australia, France, Germany, I mean, these are the sorts of countries that have the kind of security infrastructure to be able to receive the weights and receive them securely.

Do you mean like hardware? Like, isn't it like a code base that's not on GitHub, that's on your computer, and that's it?

Or, uh, as in, so cyber security and hardware security.

Got it.

However, yeah, I mean, but to be clear, I mean, I think this is just a really hard problem in general, right? I mean, actually making sure that your systems are robust to state-level hacking is just an incredibly hard problem. I mean, I remember there was this 80K podcast, um, I can't remember who it was with. He was basically saying that he expected that, uh, the big AI companies in the US had already been backdoored by, um, you know, state-level Chinese actors.

Oh yeah, for sure. Um, and that's what you're into, right? Like, you're a cybersecurity guy at the very deep level, so Tell us a little bit more about that.

Yeah, so the Cambridge Inference side, it's less cybersecurity and more making sure—

Pen test? Pen test?

Sorry, so no, so less cybersecurity and more just about making sure that when a bank runs AI, that it actually does the things that they've asked to do. In the short term, this means that it's not putting backdoors in their code, or that it's making sure that users can't jailbreak it to get information they shouldn't have. And in the future means that it guards against the AI that is now, you know, self-aware that it's in a bank, and it's aware that the money is useful, and it tries to do its best to turn everyone's cash into Bitcoin and spin up a data center somewhere.

Claude Meathos, how groundbreaking was that moment and how did you interpret that?

I think, yeah, I think it's a really interesting release. I mean, not really, it's something you probably already knew before it even happened. Well, I think that what it does is it just confirms that Transformers are just not plateauing. I think that there are a lot of people, especially in Europe, who kind of maybe had their fingers crossed that Transformers were starting to, to, you know, hit the flat bit of the S-curve and that they had some opportunity to come up with the next technology. And I think Mythos just, I don't know, just completely put that theory to bed. Um, I think there's, there's no sign whatsoever that Transformers are starting to slow down.

Coming back to the nonprofits, isn't there someone already doing this? There's definitely interest and talks, right? But is there someone putting something together to unify the West but non-US governments together in the AI race?

Um, right. So I think exactly the problem is that there's a lot of talk, uh, and not a lot of action. Um, I mean, it's been interesting talking to people and actually, you know, putting a concrete proposal on the table and saying, you know, this is a thing that we could do, is actually like surprisingly well received because it's not just the UK should spend money on AI, or we should all work together. But it's, um, and so one of the reasons that I started the organization was precisely to have an actual entity doing the thing.

Yeah, I remember Mark Carney's, uh, speech at, uh, I think was Davos, right? The World Economic Forum. Is so good, so realistic and non-woke, and just like, hey, here's what's happening, let's deal with it in realistic terms and not like peace and love and, uh, would like to be, uh, terms, right? So yeah, I love that. Um, yeah, is he the only guy thinking like that, or is there— it's also like the thinking in UK, Germany, etc.?

Um, I think Mark Carney is probably the person that's put it best. Um, and I think I said that, I mean, the name of the company is referenced to his third path to building Frontier AI.

Oh really? Huh, interesting. Um, that is a Canadian— I still, still love some many values upheld by, uh, by Canada. What would you get, uh, Demis Hassabis on board for that one? Um, or is he too US?

Uh, well, I mean, if he was interested, uh, if he wants to jump ship. So part of the, part of the proposal is that you have a few big-ticket salaries precisely to try and get people out of DeepMind, out of Anthropic. And I think that if you put a credible alternative on the table where you could say you can, you can work to make sure that the UK ends up, or the middle powers end up in a good place, and not just work to make sure that the US does, I think that a lot of people would probably—

yes, I love that play, right? With the last guy we were talking about, Turing, right before you, and We're talking about bonuses of $100 million for top AI researchers. You know, there, there's another way to money. You just have to be patriotic and be like, hey dude, help your country. And a bunch of them would jump ship.

Yeah, indeed. Uh, I mean, we're not, we're not planning $100 million bonuses, but you know, like £500K to £1 million a year for the top people, plus they actually get to ensure that we don't end up with a US hegemony. I mean, this sounds like—

Dude, I love that. I'm behind that. But last question as a VC, right? Like, why are you the right guy to head that and who's on the team?

Well, so the team is forming at the moment. I'm talking to people at the moment to see who will be the first people on board. Why me? I think this is a good question. Why has no one else done it? I mean, I think this is why I'm a good choice. Yeah, there are people out there who are better connected than me with more money than I have who could have done this and didn't.

I'm sold. Where can people find out more about you and the nonprofit? I guess the landing page is building right now, but till then, where can people stay in touch?

Yeah, so you connect with me on LinkedIn. So terriumai.org actually does now exist. It's just a one-page like there's nothing interesting there, but it exists as a thing. And I will be posting more about that hopefully over the next few weeks as I apply for funding and start, start doing stuff with it.

I was Lindley and Charles for