Looking Up the AI Exponential with Azeem Azhar of the Exponential View

Moore's Law was turning into this acceleration of AI capabilities over the top. And the other was that what was happening in renewables with things like lithium-ion batteries and solar was on a Moore's Law-like trajectory. And there were some other technologies like genome sequencing and genome synthesis that seemed to be doing something similar. So I started to bundle those up in this idea that we're going through the exponential transition, a transition where Our economy gets driven, you know, not by the economics of the oil industry and internal combustion engines and, and telephones, but by AI and renewables. And that these technologies being on an exponential trend, being fundamentally at some level kind of information technologies behave really differently to the ones of the previous generation.

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host Eric Thornberg. Hello and welcome back to The Cognitive Revolution. Today I'm speaking with Azeem Azhar, founder of The Exponential View and fellow AI scout, for a wide-ranging discussion about the transformative power of AI, and its implications for humanity. Azeem's core observations— that we are in the midst of a transition from an economy driven by the likes of phones and oil to one fueled by AI and renewables, and that these new technologies, fundamentally rooted in information and already growing exponentially, behave differently than their predecessors— will be familiar to Cognitive Revolution listeners. So I took the opportunity to get a bit deeper into Azeem's worldview and expectations for the coming years, and ended up having what I think was a really engaging and delightful exchange. In this conversation, we cover a number of familiar topics, plus some new ones that we haven't explored in quite the same way before, including why Azeem believes that while incumbents are racing to adopt AI technology, yes, startups are still likely to drive the true disruption in the form of entirely new products and markets. Also, what forward-thinking business leaders are doing today to retrain their teams and to position their companies to lead their respective markets in AI adoption and why Azeem still finds it necessary to challenge them to think bigger, asking the question, what would they do if they had a million times more compute? We also explore why Azeem agrees with Sam Altman's recent comments that AGI will likely arrive soon, but will be less of a big deal than people expect, at least initially. How AI is likely to change human relationships, especially considering the rise of so many different forms of AI companions. What sorts of new governance mechanisms the AI era will require. And finally, why we might ought to worry about what I call the great embedding. That is, the seemingly natural tendency for AI systems to communicate with each other in high-dimensional vector formats, which, while efficient for them, is broadly inscrutable to humans and could lead to various loss of control scenarios. As always, if you're finding value in the show, We'd appreciate it if you'd take a moment to share it with your friends. Azeem's perspective is both imaginative and exciting, and also grounded and sobering. So I definitely recommend this episode to anyone trying to get a better zoomed-out view of the future. I would also encourage you to consider subscribing to Azeem's work directly, which you can find at exponentialview.co. Of course, your feedback is always welcome, whether in the form of an Apple or Spotify review, a YouTube comment, or a DM on the social media platform of your choice. Now, I hope you enjoy this conversation with founder of The Exponential View, Azeem Azhar. Azeem Azhar, founder of The Exponential View, welcome to The Cognitive Revolution.

I'm really happy to be here, Nathan. I've enjoyed so many of the episodes and want to thank you for your hard service as a red teamer for GPT-4. I watched that episode with my jaw on the floor.

Well, thank you very much. That's kind. And right back at you. I've been binging your feed lately, and we've got a ton of things to talk about. I guess for starters, I would love to get your kind of summary of what it is you do. I think it's actually kind of similar to what I'm trying to do. I describe myself, as you know, as an AI scout. And I talk to mostly people on the show who are either very deep on a particular line of research or building a product. But you are one of the few guests who have this kind of very zoomed out view and really are working to understand the big picture. So how do you describe what you do and what goes into it?

I'd love to. Yeah, I mean, I've been in the tech industry for a really long time. I started working in '94 and I built my first websites in '93. And just over the back behind me, you can see out of focus my first computer, which is a ZX81 Timex 1000 in the US. And about 9 years ago, my last company was acquired and I just started to write a newsletter. And as you do, writing is thinking. And I noticed within a few months, there were these few trends that were going on that were pretty significant. One was, you know, Moore's Law was turning into this acceleration of AI capabilities over the top. And the other was that what was happening in renewables with things like lithium-ion batteries and solar was on a Moore's Law-like trajectory. And there were some other technologies like genome sequencing and genome synthesis that seemed to be doing something similar. So I started to bundle those up in this idea that we're going through the exponential transition, a transition where we— our economy gets driven, you know, not by the economics of the oil industry and internal combustion engines and, and telephones, but by AI and renewables, and that these technologies being on an exponential trend, being fundamentally at some level kind of information technologies, behave really differently to the ones of the previous generation. And so what I, what I do now is I try to make sense of that as a, as a system. We do that through— I do that through my newsletter, but I also do it from time to time through investing and advising. But I think that what is, what is going on at its heart is in about 20 or 30 years, we'll be looking at an energy system that is going to be very, very heavily renewable, we'll be using much more energy per capita globally than we do today, that we will have tons of intelligence in our economies and in our lives through AI, and that will have real knock-on effects. And trying to make sense of that in a pragmatically optimistic way. So somewhere between the extreme dystopia and the extreme utopia is really my mission.

Yeah, cool. Well, I'm hoping we can find that sweet spot as well. So right there with you. In terms of the history of the kind of intellectual history of this notion of exponential technologies, 9 years ago strikes me as kind of a doldrum time for that paradigm. I wonder if you experienced this, but I recently had a conversation with a guy who makes his living as a speaker at corporate events and who is positioned as a futurist. And I showed him a little presentation that I had put together in which I pulled out one of Kurzweil's kind of late '90s exponential curves graphs. And the title of that slide in my presentation was Kurzweil Was Right. He saw the name Kurzweil and he was like, oh God, don't talk about Kurzweil. No, everybody, he's totally discredited. And I was kind of like, hmm, that sounds like somebody who maybe got some pitches dinged in that kind of timeframe when you were getting started with this notion and has gone away from it. But maybe that's just the weird nature of Exponentials. What was your kind of experience of, you know, were people sort of sour on that? It seemed like, you know, there was the Great Stagnation thesis for a while there. And your, your start of the exponential kind of lines up with it, it seems like.

When I started, and one, what I noticed was people weren't really talking about AI. I mean, it was 2014. Machine learning was still the word. It was before AlphaGo had come out and done its thing. DeepMind was doing really interesting things with reinforcement learning in video games. And you could see that something was happening. And I think that you had TensorFlow as the sort of stack of choice for building convolutional neural networks to do machine vision tasks. So it seemed reasonably early to be talking about these things. And actually, developers were struggling to make sense of CUDA, which is the sort of API to the NVIDIA GPUs that everybody now uses, because that had only been documented 5 years earlier. There is something that happens around 2013, 2014, 2015 that I think is really worth paying attention to, which is that in 2013, Apple becomes the largest company in the world. And within a couple of years, those top slots are occupied by what we used to call the FANGs— Facebook, Apple, Google, and Amazon— or the GAFAs, pardon me, not the FANGs. Netflix snuck in briefly during the hype cycle. And so you started to see the companies of the industrial age, Exxon, GM, and so on, Fall off that fall off that list and stay off that list. So that's an important economic moment about the the sort of forward-looking stock market saying like something is changing. The second thing that starts to happen in around 2014 is we see the first market for electric vehicles go past that threshold of five percent of new cars being sold being electric, which was over in in Norway. And that five percent threshold is the normally the trigger for when you see the S curve of adoption, right, and you get into that vertical, or that verticalized part of the curve, you also started to see solar power being starting to be cheaper in roughly a third to a half of the contracts around the world than fossil fuels. And then of course, we are 3 or 4 years into the deep learning wave. And that's long enough for companies to start to ship products. So I think that there is a moment which We could argue when we look back on it feels like a sort of historical turning point. But we also have to be realistic that the mathematical function that is an exponential curve, when you stand on it, it always looks horizontal behind you and it always looks vertical above you, in front of you, wherever you stand on it. It's a smooth curve with no obvious turning point. And someone like Kurzweil, you know, I think did such great work 20 years ago. to articulate the exponential trend in computing going back from the 1880s, actually, in mechanical computers. I think one of the reasons why he slightly falls out of favor is because he was probably brave enough to extend the curve far further than we might have otherwise thought. And I think a couple of things that he got wrong was that some of the assumptions that we would have had about how the human brain works in like 2001, 2002, when those books came out, were wrong, right? Science proved that it was more complex, and it wasn't going to be a simple game of, of raw computation, one for one. And that's where I think that people start to look at him and say, well, was this just someone throwing tea leaves? I think there was much more to it than that, having read his work, you know, reasonably carefully. But I think what's interesting is that we see these curves happening elsewhere, right? We see them in lithium-ion batteries, and we see them in genome sequencing. And it's not— clear why that should be the case upfront.

The implications of the fact that we may still be standing on the part of the curve that, as you said, it kind of always does. If it's still the case that what's in front of us is vertical compared to what has been behind us being horizontal, then we're in for a bit of a wild ride. I think we're headed for a steep curve for at least a couple more years. I mean, beyond that, do we sort of hit a plateau? is a lot harder for me to predict. But I honestly don't see any fundamental reasons that we would right now.

I mean, what Kurzweil says is that, you know, it's actually this curve is a series of layered S's. So you have one particular technology architecture, it's very slow to develop, it hits this inflection as an S, it starts to accelerate. And as it hits its flat point, the social dynamics of market incentives have meant that another set of research has come in with a different architecture, a different way that extends that S up and looks from a distance like a smooth S-curve. And that's really nice and descriptive. But it also, I suspect, people who are kind of really robust with their theory feels like, well, that's just praying. That's like the turkey that's been treated really well up to the day before Thanksgiving. And like, we should worry about what happens the next day. What I've tried to do is I've tried to get into the underlying mechanisms of why these things improve, why they get cheaper. And then what we need to do is figure out where does that mechanism fail, because if the mechanism doesn't fail, then that trend is going to continue. And if it does fail, then we can say, well, we need a new mechanism. And is there one, you know, in the research pipeline that might deliver it? So, so I would say that if you look across the gamut, I mean, for example, batteries and solar power, we've got definitely got more than a couple of years to run in terms of price declines. When we look at compute, I just feel it's really hard to bet against. I just, you know, I think that I've been hearing about the death of Moore's Law for 15 years. And, you know, Moore's Law is helpful. But the question to ask is, how much compute can a developer get for $1 each year? And do we really think that is going to stop declining for a long period of time. And I find that one really hard to support. I mean, I just think it continues for a variety of reasons.

It sure seems like it. I mean, the CPU to GPU transition feels like a classic example of one of those kind of one S-curve perhaps giving way to another. And in the GPU, it definitely feels like we're still in the steep part of that particular S-curve. Hey, We'll continue our interview in a moment after a word from our sponsors.

We're not, we're not done yet. And I think the thing that is, is fascinating is that NVIDIA is obviously doing incredibly, incredibly well. And it yet doesn't yet have the threat of real competition. And what was fascinating in the CPU world was that Intel did very, very well for a really long time. I guess people forget this. But if you've been around for a while, you remember Intel was this sort of monopolist in the perceived and Andy Grove and, you know, only the paranoid survive. And it did really well only with the threat of competition, because AMD never got more than 15-20% market share. And that's enough to propel people forward. All the incentives seem lined up for there to be massive amounts of investment in scaling existing silicon chips and developing new systems. I mean, you saw in the last few days before we recorded this, Amazon and Google both reported 20-30% growth in their cloud businesses. When I've talked to bosses of of really big companies, you know, they are spending money on compute, you know, really like nobody's business. And their expectation is that it will grow. They don't often think it's compute, they say they're spending on AI, but that at the end of it is going to be GPUs cranking away. So with all the incentives aligned, I struggle to see us hitting a brick wall. It doesn't feel like it's the Carnot cycle, right? So the Carnot cycle was the thermodynamic limit for the efficiency of an internal combustion engine, something you as a native Detroit man, you know, know, know very, very well. But we keep finding ways of eking more out of our compute. And I think we'll, you know, we'll continue to do that certainly beyond a couple of years.

Just for kind of conceptual grounding, and I'm also interested to hear how you explain this to the business leaders that you work with, because their understanding and their kind of eagerness to adopt is a pretty key question in my mind as to how the next few years are going to play out. But we have kind of 2 notions, 2 definitions of kind of types of technology that both seem to apply to AI in my mind. One is the exponential technology, and the other is the concept of disruptive technology. Disruptive technology, kind of classic textbook definition, is a cheaper but inferior alternative that kind of competes on the low end of the market. It seems to me that AI is kind of both, right? We've got these at least exponentially growing inputs. Although on the other hand, you could sort of say scaling laws sort of suggest that the model capabilities are more logarithmic. So those 2 things maybe balance out somehow. It does seem like it's disruptive in that AI is typically like an inferior but cheaper alternative to asking somebody to do something for you, right? It's also a general purpose technology, I guess. What are the What are the kind of key definitions and how do you think about mustering those different frameworks so that people have good clarity on what it is we're dealing with?

I mean, it's so hard because it is so general and it is also a technology that improves other technologies and itself directly, not in the way that electricity improves electricity. Electricity makes the economy more efficient and so you can build more electrical power stations, but AI seems much more direct. I do think it's important to understand it's its generality, to get people to wake up to the idea that a general-purpose technology really transforms the world beyond the economics. And, you know, again, Detroit is a great example for this because the car transformed the world in a very short period of time. Where I live in Northwest London, 120 years ago, this was all fields. And within 20 years after that, by about 1925, the roads were laid out the way they were, and the houses were built the way they are. And a century later, we're still like this. And this is because of ultimately the car as a general purpose technology. So to— because they don't come around very often, I think that's a really good starting point. On the question of disruption, that is a— I think it's a kind of higher order question, because that is about products and how they get bundled to provide value in a particular environment. And I think that, you know, when you start to bundle AI to do that, you can ask that specific question. But one of the things I think is really important is, and I think companies started to think like this, is to think in terms of tasks rather than jobs. Because, you know, AI can't replace jobs, anyone's job, because there's just so much in an ordinary job, like logging on to Zoom and saying hello to somebody and getting your neighbor a cup of coffee, that is beyond the scope of any AI system. But within tasks, I think you can start to unpick this. And that's why, in a sense, you might start to say, well, AI becomes a disruptive technology, because on a like-for-like basis, it can't replace an entire, you know, human in their day-to-day, but it might get better and better. But I think what's more helpful is to go back to that task question. And then when we come to that, the question is, on a task basis, Is AI really a cheaper version of a human doing the same task, and a worse version? Does that matter? And is that always the case? And I have certain tasks which I do where I think I could not afford to hire a human to do this task as well as ChatGPT does it for me, you know, in a minute or two. And that may well be your experience as well. I mean, I, you know, I I use this to write letters of complaint, to get my parking fines reversed, to help me think through holiday plans, to do research for my book. And so, I mean, it's just, it's so variable. And in many cases, I would be better off finding the very best human to do that. But the cost of doing that, even finding them, is so high.

Yeah, your search costs alone would dominate.

Yeah, search costs would dominate. I mean, when you're using, you know, GPT-4 or Perplexity or whatever you use, do you have it across a whole range of different tasks that you do, from the most strategic for your business to the most trivial home task?

Yeah, maybe not the most strategic yet. At that level, I would probably restrict myself to brainstorming interaction at most, but certainly lots of things I get very efficient and immediate help on. And I think that immediacy is is super important too. I have this one slide that I call the cognitive tail of the tape, which kind of lists out 12 dimensions and compares human to today's AIs. And then we can also consider what future AIs might look like. Very much agree with your notion of distinguishing between jobs and tasks. And for a while I was calling this the great implementation, and that phrase hasn't quite taken off yet. But the idea there is With inspiration from your strategeries and your Benedict Evans type business theorists, the way to make money in business is to bundle and unbundle. I do think that unbundling jobs into tasks is a very good way to think about it. And then for any given task, you go down this tail of the tape and you're like, yeah, a lot of them AI can do as well or even better than a human, or certainly Better than a human that I could find without huge search costs.

And we get quite surprised with some of the results. I think one of the really big surprises is that if we went back 6 or 7 years and you had books like The Rise of the Robots and the famous Oxford paper saying machine learning could automate 40% or X% of jobs, that's written by a friend of mine, our assumption was that it would be routine cognitive jobs. By which what people meant was customer service and data entry in like Philippines or the India or in India. And what we're actually discovering is that it's jobs that we would have categorized as non-routine or even creative, where this technology can really start to make a difference. And it really, really is surprising. One that really I was not expecting was a paper at the end of 2023, which looked at empathy ratings of doctors giving advice compared to ChatGPT or GPT-4 giving advice. And patients were rating the robotic advice as more empathetic as humans. And the whole argument had been, let's use AI so the radiologist can spend more time looking you in the eye and being attentive and being empathetic. So part of the challenge, I think, is that there is this lack of clarity and lack of knowledge about where and when will these AI systems actually compete with humans on particular tasks? And then when you think about it, actually, what is empathy? Empathy is about active listening, and it's about being incredibly patient. And there's nothing that's more patient than a robot that has no sense of time and is stateless, right? I mean, it'll just sit there forever.

An early GPT-4 test that I remember fondly and do think is kind of a sign of things to come was Simulated tech support for my grandmother when she needs help with her iPhone. And it was just a flash of this, you could call it sparks of things to come, where I played the role of her, which she calls me, right, when she needs help with the iPhone. And you're in this dynamic, and it's actually an interesting dynamic also for a pure text situation because she's typically on the phone with me. looking at the phone and saying, you know, I can't— my friend sent me an email and I can't get it. And then I'm like, okay, well, I always start with, what do you see on the screen right now? Can you start at the top and just read everything that you see on the screen? And at times we've had some very, you know, kind of funny—

Does she always read everything that's there? Like, you know, she's missed something out. You're like, Grandma, wasn't there a— isn't there a word above?

She does. No, she does pretty well. Yeah, she'll start at the top. Verizon, you know, the time. And then there have been a couple of times where one of those system dialogues pops up and that just didn't even register to her. But then when she got it to reading, I was like, okay, you need to hit okay there before you're going to be able to hit anything else. So it can be these very simple things. But GPT-4, as you might expect, did really quite well on that. There was a little bit of— it was in kind of the middle of where it did not have the UI of an iPhone memorized, so it was kind of hallucinating it and guessing. And yet the guesses were close enough. And I really had to study my iPhone and what it was saying and kind of compare, like, is it saying what's actually there or not? It mostly was, but it was clear it was kind of filling in some gaps. But the real eye-opening moment for me was it said something that I thought she might feel was a little bit rude. I forget exactly what it was, but it was like, start in the top left. Do you know where the top left is? Something like kind of that basic. And then I responded as her saying, yes, I know where the top left is. I'm not dumb. I'm just struggling with this phone. And then the AI comes back and says, I'm so sorry, I didn't mean to offend you. I'm just trying to make sure we're resetting here and helping you through this process. And that was the moment where I was like, oh, this thing is going to be— it's got kind of this emotional intelligence as well. And that could be obviously just a critical ingredient for so many different Interactions and medicine being a big one. We've done 2 episodes with Vivek Natarajan, who leads a lot of these med-specific projects at Google. And what an absolute tear they've been on. Most recently, they have a differential diagnosis paper that shows that the AI is getting the correct diagnosis twice as often as the unassisted human, and also more often than the AI-assisted human, which I think is something that we should Begin to reckon with.

There's so much to unpack in that. Can I ask you one question about the politeness from when you were playing your grandma? Do you think that comes from the human feedback cycle over the network before it gets released, or is it from the training data?

The version that we had was, as far as I know, right, I'm inferring here because I did not have access to the training methods, but it seemed very clear to me that the model version that we had was RLHF'd purely for helpfulness. So it was very eager to please, very eager to be nice to you. No guardrails on what you could ask it and what it would do, but 100% just trying to be helpful and pleasing to the user. So when it detected that kind of, I'm kind of bristling at what you just said, that's when it reacted with this, I'm so sorry, I'm just trying to help kind of And that was a mind-blowing moment. I had not seen, of course, anything remotely like that from earlier models, right? At the time, text-davinci-002 was the best publicly available model, and it would follow instructions on basic stuff. But I mean, this was a totally different world that we had suddenly stepped into. Hey, we'll continue our interview in a moment after a word from our sponsors.

I mean, it is such a different world. I was using ChatGPT for something a few days ago, and I got quite tired, as you do. I mean, I think about it as the time I once was playing tennis against one of those tennis ball serving machines. And I was exhausted. And the machine is just like willing to keep firing balls at me. And it's a bit like that using, using these chatbots. And I just left this kind of whimsical comment. I said, you know what, I'm really tired, I will come back tomorrow. And we can look at the moisture evaporators. We were not doing anything to do with Star Wars, right? We were doing some digging about coal in England in the 17th century. And it just replied to me going, I'll be happy to, you go and rest up. And tomorrow I can help you on the farm. May the Force be with you. And it had just very subtly played that back to me in a really, really nice way. And actually, in terms of human, humanized interfaces, as someone who's used Apple's computers for 40 years, it, it really sits alongside the trend that a firm like Apple had been thinking about for a really long time, which is how do we make this technology come to us rather than us go to the technology? You know, there has been this view that we'll have these centaur humans, right, human plus machine. And one of the things that we learned about centaurs, and that idea came out with chess, right, after Kasparov lost to Deep Blue back in, in '97, was that for several years, humans and the machine would outperform machines on their own, and of course, humans. But now you're just better off with a chess computer, and as a human, you should just accept everything it says. So the sente period in chess only lasted 7 or 8 years. And I think one of the assumptions that goes into this kind of maelstrom of ideas we have to make sense of over the next years is that sente humans will exist for quite a bit of time. In other words, human plus machine will do better than machine on its own. But we are starting to see signs that that period of time is already starting to come to an end, right? Far faster than we might have predicted. And you've just given an example, which is a, you know, an unreleased AI model, where the AI on its own is doing better than the expert, expert human with the AI.

Yeah, it's pretty crazy. I think people are broadly in denial about that possibility even, let alone the reality. It's funny that things are happening so quickly that people are saying things are impossible or won't happen for at least 10 years that are literally already happening. And so I think to some degree there's just kind of a lack of awareness. I think there's also definitely some psychological— and I try not to psychologize people's AI positions too much because I think the technology itself is confusing enough, but it certainly seems like there's some kind of psychological cope or denial happening there. I also do think it's important to keep in mind, too, that the setting really matters and the world is going to change as well. And humans may still have a really important role to play in a lot of systems for a while yet. I think the AIs do have really important weaknesses. So you do a study like this Google evaluation, and I think that they are very serious people who have set this up in a way that I trust is not cooking the books in favor of the AI. So I take that result basically at face value. But then I also think the AIs have these strange vulnerabilities that, for example, the AlphaGo system, right, that is superhuman Go player. But I have an episode coming up with Adam Gleave from FarAI, and they put out a method that showed how a relatively simple adversarial attack on the AlphaGo system could beat it. And no human would lose to this simple adversarial attack, but the AI had this massive blind spot that they were able to engineer against and exploit. So I do think we're headed for strange dynamics and the sort of naive— I don't mean to say the Google thing is naive, but it's kind of, let's take controlled conditions and set it up in a certain way and see what happens. I do think it's really important to keep in mind that just like these AI systems in general, when they get out of domain, they have problems. Those results may also not extrapolate, at least initially, to situations where people are trying to break the AI. for anything where there's an adversarial incentive out there, I would not be quick to take a result like that and be like, okay, cool, we can just deploy the AI on its own. Now, in medicine, presumably there's not a lot of adversarial situation, right? Because people want to get the right diagnosis and nobody wants them to get the wrong diagnosis. So I do think the AI doctor is kind of here before we know it. And we're going to have some very interesting questions about what to do about that.

But I think the thing that you've also identified is that these systems have to get into companies, have to get into hospitals, they have to get into the economy. And that still takes time, right? It still takes time for— let's not look at medicine, because it would have to go through all sorts of clearances through the FDA and so on before it could be used. But every time we've seen amazing technologies emerge, the cloud computing, right? GPT uptake is higher than cloud computing, even though it's 19 years younger. But every time these technologies come out, they take a while to make a— make it into businesses. I mean, even something as straightforward as the typewriter. So typewriter took about 20 to 25 years from becoming a kind of affordable technology to being something that businesses have figured out how to use and how to change their processes. Something like electricity took a little bit longer, because, you know, the way that factories used— what used power before electricity was like a single big driveshaft and a massive lump of power that came from it. And then electricity is this highly distributed, packetized movement of energy that you can put in different places, you need an entirely different setup for it. And so, so even when we, we look at something like this, the question is, How quickly are we able to bring it in? And over what timeframe does it then start to change our— the practitioner's relationship with the technology? I mean, we know from automation of aircraft and other automated systems that you go through this 3-phase process as somebody who's using the technology. Phase 1, you kind of don't trust it, and you you then see if it does well where you're the ground truth. Phase 2, you start to assume it's the ground truth, and then you pat yourself on the back when you come up with the same answer that the machine comes up with. And then phase 3, which is where we've got to on GPS, where we just trust Waze. And it's like, actually, you know, we generally trust Waze. Some London taxi drivers don't, we just trust it. And you see that process happening replicated in other types of automation, which is why when it's really mission critical, you have incredibly high levels of training, and you have other types of safeties in place, like, you know, 2 humans in the cockpit, or, you know, whatever it happens to be. And so that part of the journey, I think, is also one that adds a little bit of drag in terms of how long it takes to have an impact. And in that time, Nathan, we start to understand new questions, questions that we can't imagine right now, because they're just too far down the decision tree. So sometimes I think I hear people say, well, there'll be that moment. I think Tim Urban has this, from Wait But Why, has this graph where he sort of shows the moment where AI is like as clever as a rat, and then like a second later, it's 10 times cleverer than us. I think part of the reality of like the rubber hitting the road is going to be that even as these products do very, very well in the lab situation, it just takes a little bit of time for them to get into the real world. Now, where it may happen is, I think, in a couple of places. So one is where tasks have already been discretized so that they're essentially just written down. And I think that that is certain types of call centres, and it's certain types of data entry. And in those places, the human is already hired on a task-by-task basis, normally mediated by like a body shop. So the buyer of the services, which is like our favorite consumer electronics company or your insurance company, has no emotional relationship, they just have a KPI that they measure to. And I think that lends itself, unfortunately, to a tidal wave risk for those types of roles. I think the second area is, you know, classic Silicon Valley stuff where a de novo company is— I should say Silicon Valley, classic Detroit stuff, where a de novo company is able to apply these technologies the way Henry Ford did and build his processes from scratch, right, in the way that the artisanal carmakers were just going to have so much kind of cultural drag, they couldn't. And that's what we certainly saw with a lot of the, you know, the internet, right? It was, it was absolutely startups that captured the value. And then when you started to get to, to areas that were more highly regulated and had, you know, a lot more stickiness to them, like, like finance, with the exception of certain areas of payments, it's still the big banks that are the big banks. And I don't know how differently this actually plays out, short of scenarios where someone is able to use these super powerful machines to kind of manipulate the rules of the game, which I think is a— that's more like a black swan scenario than one that we could talk about reasonably.

Well, first of all, as an aside, if you ever make it to Detroit, I'll take you to the Henry Ford Museum and Living History Museum, which is called Greenfield Village, where this earlier transition is documented with actual machinery still running from the various phases. You can go see one of these old factories where actually a few of them where they have kind of the central steam engine and then it's powering this one driveshaft at the top of the floor. And then there's like 25 belts coming off of that and connecting to other machines, all driven on this single thing. And then they have an electrical, early electrical version of that as well. Edison's workshop is there. He actually, this is Henry Ford toward the end of his life was like, became very sort of nostalgic for an earlier period and decided he wanted to kind of create this, you know, living history place to sort of preserve that history so people could see it in the future. It's quite an awesome thing to go and contemplate.

I'll have to do that because I think Ford's influence and impact is somehow underrated. We don't talk about Ford as much as we may talk— maybe talk about Edison when we think Historically, and you know there was so much in Ford. He understood the the socioeconomic contract that emerged from these from these changes in like in a number of different ways. I mean, not always in ways that are you know kind of positive. We know where the the sort of sociological department started trying to ensure temperance amongst workers and and so on, and it gets encapsulated in in. Aldous Huxley's book Brave New World, which I still find to be a really, really remarkable piece of writing. And it was written in the early 1930s, 1932, I think. And for Huxley to encapsulate and extend a raw Fordism as far out as he did, it's a little bit like Kurzweil and what he did with his singularity work, I thought was absolutely genius. And I think that that book, which Brave New World, which rests on the ideas that Ford developed, catalyzed, that spun off his work, speaks very, really quite clearly to a number of the issues that we face now in this kind of later stages of industrial capitalism. And I mean, so a trip to Detroit, I will take you up on, and I'll bring you a copy of Brave New World as well.

It's been a long time since I've read that, actually going back to high school. So I might need to dust that one off and And I'm sure it will resonate very differently today than it did for me then. So, okay, I have a number of questions on this kind of concept of transition. I think I would, from what I heard, I think we're probably largely on the same page that it seems like incumbents, the big banks and big technology companies, largely should be able to harness this technology and bring it to their platforms before they get displaced. I think of Salesforce as almost a canonical example there. They're going to have an AI layer that creates a much better and less complicated, less confusing user experience before somebody's going to recreate all the complexity of Salesforce. On the other hand, there is no AI friend today. So that's kind of my canonical example of something that's by definition going to be de novo. And then presumably there's like a ton of stuff in between. And I gather that you're kind of talking to business leaders probably throughout that spectrum. Where are they today? It seems like their grokking of what is going on and their eagerness to transform the way their businesses operate might be the limiting factor in how quickly this transition can proceed. Are you advising them to start to think about what kind of semi-structured work they can make a lot more structured so that they can reduce it to these kind of task-level things that can be automated? And are they receptive to that sort of challenge and/or opportunity?

I mean, I can't remember a technology which has had the degree of uptake in large companies as this one. I think back to the internet. Back in '99, I was talking to the CEO of a big mobile phone company, and he said to me, I will never let you pay your bill Over the internet. In fact, I'll never ever let you look at your bill over the internet. And he was right because he was fired a year later. So he never project was never delivered. But it's so different with not just AI, but specifically Gen AI because two things are happening. One is that the CEOs of the companies have been playing around with this partly because they're of that age now; they're in their mid-fifties. They've grown up with computers. Their kids are coming back with it from from college. And the second thing that's happening is that the frontline workers are. Using this regardless of any restrictions by their employees. And there have been a couple of surveys now, one done by your old alma mater, Oliver Wyman, which was of 25,000 employees across 18 countries. And in 83% of employees in the UAE and India already using ChatGPT or something similar. And we've seen data from Salesforce and others that say in the US, it's like 30, 40, 50%, you know, choose your number, but it's not 2%. And it's very frontline. And I think of that as like a pincer movement, because normally, you got to drag the frontline employees, or you got to drag the CEO, they both want it for different reasons. And I think it will, it will happen. And you start to see that in the levels of uptake that are being reported through the surveys. So I think that that says that we'll see more and more projects roll out more, more quickly. But there's still a lot of retraining that needs to go on internally and internal processes. I mean, I think one thing that will happen is, I just saw Ethan Mollick, who you must get on your show, is fantastic, a professor at Wharton who's writing a book on ChatGPT. And he just showed a video where he had 6 different windows open. He put in a query into each one, and in 54 seconds, he had a product launch plan, a market analysis, a PowerPoint deck assessing Tesla's business and something else created from one sentence prompts. Now, if anyone has worked in a large organization, they know that, you know, your inboxes are full of crap and full of meaningless PowerPoint decks. So we might actually just find ourselves sort of swallowed up by PowerPoints created by Microsoft Copilot. So there's a whole set of, I think, more complex kind of issues that exist in large companies. But I think that they will adopt this much faster, the evidence is, than they have in previous technologies. But it doesn't, I think, necessarily mean that disruption of the kind you talked about won't also happen. And I didn't have on my dance card in '95, when I started working, that Blockbuster would be the first high-profile casualty of the internet. And I had seen video over the internet, Cambridge University had a webcam on a coffee pot, And that was the first sort of video over the internet. And Rob Glaser had just started Real Networks, it was called Progressive Networks back then. And when Netflix launched, it was this kind of thing with DVDs. And it was a real pain in the ass. And it took a long time, a few years before Blockbuster has its best year, and then it has its worst year ever. And I think that if AI, Gen AI, whatever you want to call it, is a GPT, there are going to be Blockbusters lurking around. And the question is, which ones will it be? The reason I don't— like you, I don't feel it's going to be the banks, is because the banks have got a whole bunch of stuff that is about trust and probity, and internal processes and compliance. That is just not an AI question. It's kind of an institutional question. And I do wonder about where that moment is. And the thing is that I there are obvious ones. It's like, well, it'll be entertainment, right? We'll just start to generate personalized entertainment. And that'd be really bad for Disney, except that you could then counter and say, well, but it might be derivatives of Disney's IP that actually benefits Disney. So finding the space a priori, I think, is really, really difficult. And the people who make their money doing this, who are the venture capitalists, get it wrong a lot of the time anyway. That's why they all they all have portfolios of 30, 50, 100, 500 companies. If it was so damn easy to find the next Apple that's going to disrupt the previous industry, they'd have portfolios of one. It's not, it's really difficult, because actually nobody knows. And we have to figure it out through, you know, experimentation. So I don't, I don't know. But I keep asking that question. And the question I take to bosses of companies, I take a couple. One is, who could be Blockbuster? And how do you— what are you doing to make sure that that's not you? Because I think by and large, they've got the rollout of GenAI and customer service and compliance and form filling and so on underway. And the second question I've started to ask is, you know, what would you do if you had a million times more compute than you have today? And many of them haven't thought about that. I mean, I think big tech companies have, you know, if you offered that to Satya Nadella, I mean, he's already on the path to do that. But that becomes really important because if compute is a key input into your company's ability to execute, you need to start to think about those types of questions. And do you even have a plan to make use of it? What options would it create for you? So, those are the 2 areas that I push on because I want people to try to think a bit more creatively about this and recognize that, you know, a million X is not outside of the bounds of a planning cycle. And the reason I would say that is, Yes, we don't have the measures to easily show it up. But 5 years ago, the state-of-the-art transformer model would have just been GPT-2, hadn't been released. So it was GPT-1. And GPT-1 to GPT-4, ChatGPT, I mean, look, metaphorically, just as buddies around a bar, it's a million times better. Right? Maybe on the benchmarks, it's not, it's 40% better here, 60%. But it feels a million times better because you're just right across the uncanny valley. So we've just seen that playbook play out. So let's ask it again and try to ground people in the fact that capabilities could change that quickly. And what opportunity does that create?

So how do you— this is actually a live question for me, because I'm working with a couple of companies. And I'm noticing this challenge where it's like, okay, this is cool technology, for sure, wherever we can all agree on that. And yeah, we can probably find some efficiencies in terms of automating ticket resolution and whatever. I'm starting to even see things like Intercom has this new $0.99 per ticket resolved AI pricing model, which I think is super interesting. So everybody's like, okay, cool. Yeah, let's find some efficiencies. Let's automate some stuff that nobody wants to do. Great. But then there's also this question of like, okay, we're still at the task level. The AIs can't quite do jobs yet. And how far are we willing to extrapolate and how much are we willing to prepared, willing and prepared to invest to try to— we're not going to get ahead of it, but even just kind of try to keep up with where this might be going in the not too distant future. And I feel like people are having a really hard time wrapping their head around that. Partly it's like they don't want to believe too much in the hype, right? There's the question of, well, hey, how much of this is maybe just overhyped? And we've seen hype cycles come and go. before. But I wonder how you would kind of coach people there, because I'm trying to get the message across that by all means, you want to be picking up the low-hanging fruit and automating those tickets and finding all these efficiencies. But you also really do probably want to start thinking about what is the future paradigm that you might be working in. And there's just so much fog around that for people that I find a lot are just kind of like, I don't know, I don't really even want to go there yet. But I feel like it's a mistake to not at least try.

They do. I mean, there are 2 companies I never mention, Apple and Tesla, because bosses have had them parroted at them for 10 or 15 years. And the point about the million times question is not to frighten people. It's actually, it's really about saying, once we sort of acknowledge that, then we work back to stuff that is much more practical and prosaic, which is, what is the kind of organization and capabilities you need to have in order to take advantage of, of this, these changes in a, you know, in a regular way. And there was some really interesting research in that Microsoft did actually pre all of this ChatGPT stuff, it's about 3 or 4 years ago, where they looked at adoption rates of AI, big data type of words in companies. And they found that the more mature companies were much more likely in these fields, were much more likely to be to say that the benefit they got was from market expansion and business development. Whereas the more immature ones were likely to say, it's all about operational savings on the tickets, as it, you know, as it was back then. And part of the— I think part of the thing that you can start to do is you have to acknowledge that there are some really clear, quick and clear wins in what a basically low-value tasks, you know, as perceived by the company, right? Because they're often outsourced to third parties. But you also need to make sure that your best people, who you invest the most in, have got access to the really, really very, very best tools. You know, I want my surgeon using AI systems to improve his performance, right? So what I try to do is encourage people to say, This is a shift that's happening. And of course, it's not about buying every GPU you can if you're in the fish oils business. But it is about saying, are teams outside of cost savings starting to understand how they can use these services? And are they starting to experiment, learn how to use prompts well, start to see what avenues that opens up in ways that are much, much more strategic. And I think that that forms— could form some part of culture change where you have companies who think in those terms. And that helps the CEO start to understand that question of, this is not theoretical, theoretical, that I need lots of computation to do a simulation for X, and I've bought it from one of the big consulting companies or from IBM. It's more that internally, my strategy teams, my business development teams are starting to identify opportunities and partnerships that we wouldn't otherwise have seen. And we can start to do that by using these tools in different ways. So I think it is practice-based, which is why I think kind of prompting becomes, you know, quite a useful tool to show people. And you need to get people past the idea that ChatGPT is all about writing the Declaration of Independence as if you were Jar Jar Binks from, you know, Star Wars. Episode 2, right? And that's where we all started, right? We got it to write funny raps and poems and what have you. And instead, you start to show bosses what can really be done with something straight out of the box. And that tends to wake them up a little bit.

One thing I think is maybe going to become a mantra is the day that I stop building apps or solving concrete problems in a hands-on way is probably the day people should stop listening to this show because my knowledge will will atrophy or will depreciate very quickly. So I do want to be hands-on and certainly welcome those kinds of like, hey, we got this task piling up, can we figure out a way to slice through it? Love that, honestly. And then, yeah, at the same time, I'm not really like a corporate consultant, so I really don't have a lot of practice there, but I am Trying to start to piece together what would the real best practice for this look like? And I think, and you could refine this for me, I'm sure, but I kind of think leadership showing prominent examples, encouraging the CEO to be showing off, here's what I am doing that is helping me in practice, and just showing that process. An education program, I think, is also probably very important. As you mentioned, having the best tools is really important. So Trying to work toward structured kind of piloting strategies for companies that— and also, this is not good for my friends on the SaaS app side, but my advice there is we want to avoid long-term buy-in or lock-in as much as possible because we're going to need to probably swap some of these tools out. And then some internal R&D, depending on the resources available, to kind of I think most of these things should be bought, not built internally most of the time. But sometimes you have something that is so idiosyncratic, so bespoke that nobody's going to build it for you. And so you kind of have to build it yourself. And obviously there's a lot of judgment that needs to be exercised to distinguish which is which. So that's kind of my 4 planks right now. That's rough, but CEO or leadership team, example setting, education, more like structured rapid piloting and procurement, and some like internal custom app R&D is kind of my 4 pillars at the moment.

I mean, the challenge, you know, the challenge with disruption is that it is about, it's about meeting a need in a completely new way, right? And you've got, there's Clayton Christensen's model. There's also this idea of Blue Ocean Strategy, which is a different model. And it effectively says that The things that people cared about, they don't care about as much as some new attributes that the product has. And what's really hard for any incumbent firm is, of course, all your internal culture has been about maximizing what you have, rather than what you don't have and what could be out there. Because it's really hard to get 20,000 people, 50,000 people motivated daily, if you're saying, hey, all the things you're working on, are just not kind of that important, because there's a blue ocean out there. And that's why I think the startup community and the venture community is such a kind of critical part of getting innovation into economies. And we've started to see this in the car industry as well. You know, Toyota has been winding back from EVs, as if they ever wound forward. And one of the things that senior execs said in the last couple of weeks was, we have all these employees who love building with internal combustion engines, right? And they want to continue to do that. And of course, you make sense, you spent 100 years or 80 years building that cultural capital inside your, your company. And I think it's one of the reasons why firms really struggle to find ways of stepping into disruption. And I'm not sure we should even beat them up for it, right? Because there's an economy out there. As shareholders of companies, we can sell our shares in a public company, and we can buy shares in a company that's going to do well or not. And we can do that off our own back. And maybe the job of existing managers is to focus on the remit of the company and to just get it to do it better. So maybe a better place to start is where you start, which is just efficiencies and optimizations, because there's a whole market economy out there of other firms who will come in and meet needs. I think back to the the dot-com bubble. And my favourite example of a company stepping outside of its comfort zone was Zapata Fish Oils, who bought the domain zap.com to compete with what were then called portals like Yahoo and Excite. And they were planning a, you know, a NASDAQ listing, and then the bubble burst.

Yeah, it's a challenge. I definitely don't recommend, you know, for example, like, training foundation models to almost anyone, you know, there's, there's definitely some stuff that I think is, is better left to the specialists. Hearing what you said there, maybe I would add a 5th to my set, which is like just creating expectations of greater efficiency through the use of these tools. That kind of dovetails a little bit with the example setting from the top, and that maybe aligns a little bit better to the way management is typically carried out today. Right. You kind of talked about the pincer movement from the top and And obviously the frontline workers who are just using tools. And then in the middle, you've got folks who are responsible for KPIs and OKRs. And I think it's a little bit challenging sometimes for them to be like, okay, this is what I'm responsible for, and you're coming at me with this, and figuring out how they can actually use tools to advance their existing goals.

It's disruptive, not in the Christensen sense. I mean, there are so many conflicting signals. So one interesting question is whether you start to see this sort of downward pressure on people's wages and a downward pressure, particularly on middle managers, who don't necessarily contribute to the getting work done, but because of tenure are quite well paid. And the sense that you could just get a bright 30-year-old, right, to do the 40-year-old's job if you gave them ChatGPT and a couple of other tools. And these are really interesting questions, I think, that will play out in companies over the next few years. And we are, we're not quite where we were, I think, kind of politically, 20 years ago, when General Electric had kind of just come in and bottom slice headcounts all the time. I mean, I think that politically, in the UK and in the US, you need to be more sensitive to those types of decisions. But it just goes into, to show how complicated this technology plays out, right? Small d disruption. You know, are we really going to see large-scale onshore layoffs because of optimization? Or will companies say, listen, we're going to manage this through attrition and natural wastage? Because actually, that's just politically more acceptable. And it is culturally more acceptable. And, you know, to that extent, I couldn't make a bet on, on how it would would play out. I mean, I imagine that in, you know, in Europe, it'll largely be more slow. But at the end of the day, these companies have to be profitable, and nothing hurts employment as hard as bankruptcy.

Right?

I mean, that's the thing that really, really sort of slams it. And, and there are a number of waves, you know, coming through. And I think one of the things to understand is, is that technology transitions can happen really, really quickly. I mean, it, you know, in New York and Chicago, it took about 12 to 14 years for cars to replace horses from the moment that cars were economically competitive to horses, and which is roughly about 5% market penetration. They dropped in price 2 or 3 times over that decade or so. And we start to see that shorten shortened transition happen elsewhere. So if you look at electric vehicles replacing gas, gas vehicles, in Norway, it's taken about 9 years to go from that 5% of new vehicles to 75% or 80% of new vehicles being electric. It means it's still 20% of the cars on the road, only 20% of the cars on the road are electric, and the rest are petrol because people hold on to their cars for a while. But it's only an 8 or 9 year period. And the Norwegians had much more expensive cars and far fewer choices than we do today. So when we start to look at this staccato of technology-enabled products coming into the market, the number we need to think about is, once we approach economic feasibility, that takeoff ramp from 5 or 6% penetration to 80, you know, it's probably not going to be 10 years. And it's like quite likely to be much, much less. And I think that makes for really hard decisions for companies outside of the tech industry that are not used to these sorts of, of changes. Because 10 years is not really a huge amount of time to get, to get anything done. I mean, some of these firms still have 3 or 5 year planning cycles. And I think that some of those things are just starting to, to play out, you know, we're just starting to see electric vehicles being cost competitive over the lifecycle with gas cars in the US, for example.

The speed of transition, I agree, seems likely to be one of the fastest, if not the fastest in history. We have the means of distribution of the technology already kind of in place, which is very notable, right? Like, everybody's already got the devices on which— of course, there may be new devices, but we have devices that are perfectly good for using AI already. And we have the global network such that you see a new research paper. The one I'm tracking right now very closely is the Mamba architecture and just how quickly we're already seeing follow-on research. Not even 60 days since the first paper, we've already got probably 10 different follow-ons that have been completed and published just in that time frame. The most recent I saw this morning was from the University of Kentucky, and it's like an apparently Chinese American professor and looked like a I'm not exactly sure, but some sort of Central Asia perhaps name for the grad student. And they're at the University of Kentucky and it's like, well, everybody is wired in. So it seems like this is going to be super fast. How path dependent do you think the result ultimately is? You mentioned your place where you live, it was fields, then the roads were made, and now the roads are still there and the houses are still there. This is like a softer technology than that, presumably doesn't calcify in the same way that a new neighbourhood— my neighbourhood's 100 years old as well, so presumably it's not quite so locked in, but maybe somewhat, right? I mean, I do wonder how the sort of compressed dynamics of transition may actually be very important for shaping the big picture future.

It is quite path dependent. And if you actually perhaps look at how we're working with, with AI systems today, it is still, in a way, it's discrete apps, you know, you'll go to X product to do your text-to-video, Waymark is a great example. Or you'll go to— I go to ChatGPT. And although I'm doing a lot of different things in ChatGPT or Perplexity, each one really feels quite distinct, right? When it's— whether it's a shopping task or a research task. What we haven't yet figured out is actually how we connect these systems to underlying systems, right? So action models in a way, right? How do we get our AI to do something useful for us and actually right to the point of giving us the approval ticket where we say, yes, go and do that. And we're fudging it at the moment, because what we're doing is we're getting plugins into LLMs, they're using the existing API contract that exists with, say, the Kayak API, But, you know, searching for flights on any of these AI systems that I've tried, and I've tried a few, is still simply not as good as my doing it by hand on, on Kayak. And if we're going to start to see real changes outside of either highly processed and containerized jobs in data processing and customer service, or broad open-end scoping research, which only a handful of people do, will need to connect far better to actions and chains of actions out on the internet. And in a way, there is some infrastructure in place, because we have RESTful APIs, we have this API economy. And we've been integrating these things from payment systems to maps, and so on, for a couple of decades now. So there's some discipline in there. But, you know, and I don't know the answer to this. But one thing that I do wonder about is, is whether it's going to be as simple as just having AI systems that can generate action verbs, and those action verbs generate the correct API call to the right API. And that has built the system that we, we think we need. I think Andrej Karpathy calls it like the LLM OS, the LLM operating system, or whether there need to be changes in the underlying infrastructure so that those APIs develop and respond and work slightly differently. Now, my sense would be that the LLM as a kind of coordinating, orchestrating thing seems like a reasonable place for it to start with its memory and data store elsewhere, then figure out how to get it to generate consistent action verbs. And we would work with the existing human APIs that have been, that have been built for the API systems. But at some point, we'll start to think about what should machine-to-machine actually look like when you've got an AI at the, at the other end. And so I think we do have a lot of the existing infrastructure in place. But I would be very surprised if the syntax of APIs stays the same over the next 5 or 6 years as we move towards a world where AI forms a kind of interface between us and what we want to achieve. And then there's a whole bunch of other questions around how do those decisions get made. We know that when we pick up our iPhone, and we search on Google through Safari, Google is not there because it was the best search engine. It's there because they've X billion dollars a year to pay Apple. And likewise, when we search for things on Google now, there is so much ad pollution that it's unclear what the incentives are. So I think then there's another layer, which is unclear to me about trust. You know, right now, one of the beauties of Perplexity or u.com, which are these, these LLM agents, is that they provide really, really good referencing when they come back and synthesize an answer for you. And so that gives you a higher level of trust sometimes than you might have with ChatGPT. But I'll want to get that trust if I'm handing over key decisions to do analytics on my Stripe account or to help me book a hotel to an AI system, and I'm no longer driving the key presses. So that I think is somewhat path dependent, because, but I wouldn't, you know, again, I'm a real great believer in what what founders can get up to. So I wouldn't also write off a founder coming up with a different way of thinking about the problem.

Yeah, the dynamics of this I do think are going to be extremely interesting, fast moving, and pretty hard to predict. One person reached out to me not too long ago and said, what do you think about creating a product that makes people's websites more bot friendly? And I said, I think that is a really big idea. I've thought about that more in the context of self-driving cars. Like, why don't we have a program of— and this would be more of like a national program. This is why I think China probably beats the US in the self-driving car race, because my expectation is they'll say, hey, we could get self-driving cars to work if we put QR codes on all the road signs or whatever, and then they'll just go do it. We don't quite have the political will to do that, but On the web, yeah, I think you could do that and it might be cool. But then I was kind of like, but do the website owners today want to be more bot friendly? So what I ended up suggesting to this guy was maybe you do the judo flip and start with something that is like anti-bot, bot control or bot detection. And then that in time can sort of mature into bot control, but also enablement because Eventually, I do think people are going to want that, but they may not be ready for it yet. And maybe the way in is to try to position yourself as the control layer that can then become an enablement layer.

As a website owner, you've got to think about the economic rationale for having a bot, not just a crawler, but a bot access your material and what you're going to benefit from. And now if it's if it's content, right, so it's analysis and research and reviews of products, you need to then also think about your attribution and your monetization and what that, you know, what that relationship is. If it's for actions, in other words, it's for booking and for ordering, say, you know, you're a hair salon, and you want people to be able to book appointments or bots to book appointments like that lovely Alan Kay Knowledge Navigator video from Apple from the '80s, then, then yes, you want the thing to be bot-friendly. And you want to have there to be a standard, which could well just be, you know, a RESTful API, right, that allows the system to connect, connect and ask the question. But, but, but I mean, it's really, it's quite interesting that we're— people are already starting to think about these things and ask these things, because I think that you will end up— I mean, you'll end up with so much machine-to-machine communication, of which there is already an enormous amount, not just on the internet, that is invisible to us, right? Because it's the digital infrastructure. But there'll be an enormous amount of machine-to-machine communication, because these systems will also try to do optimizations for their owners far, far— with much greater intent and stamina than we than we ever would. And so what those systems end up looking like, I think, will be quite interesting. I mean, the other area that I've been tracking has been on the other side of this has been people looking to build agentic frameworks, right? So frameworks that allow us to have multiple LLMs, and with all of their current restrictions around planning and task execution, allow you to manage those. So you can start to build systems that that can do task execution. And you remember a year ago everyone got excited about AgentGPT. I don't know if you've seen anything like that and where you think we are in terms of being able to have systems that do that and have that agentic behavior in a useful way.

Not quite there, but definitely getting closer. We recently did an episode with Div from Multion, who has been one of the most kind of quick to launch and iterating in public of the agent companies, and they are making real progress. The prompt that he gave me to try in advance of my conversation with him was basically, go to my Twitter account, look at my recent tweets, note what they're about, then go out and do research online for new AI stuff, but only about stuff that I haven't already tweeted about. Then come back and write a tweet and post it. And it worked. It was able to complete that entire sequence and post a reasonably coherent tweet. I don't plan to turn over the account to it entirely in the immediate term, but a year ago it was all theory. One of the things I say these days often is we now have AIs that can reason, plan, and use tools. And people will be quick to say, well, they're not that good at it. And I say, well, yeah, that's true. They're not that good at it yet. But two years ago, they couldn't do it at all.

But what you've you've picked up on though is you know as an early adopter and you've got access to this these technologies, your Twitter feed will get even better than than it is now. But there'll also come a point where we all have that technology, and then on the other side, I will be sitting there saying to my bot, "Can you just extract the three bullet points I need to know from my Twitter feed?" And and I remember this with Amy dot X. If you remember this. It was a scheduling bot. And one of the things that made me really uncomfortable about using it, having fallen in love with it, was when my mentor wrote a really polite email back to Amy saying, so good that you're working with Azim. He's a really great guy. And like, make sure he tells you this story about this and blah, blah, blah. And I can't make it. And I read this and I thought, I cannot use this now because I realized that I was imposing this artificially onto all of my recipients. And I think this is one of the things that we'll have to contend with, with some of these tools. If you work in a big company, frankly, if you work in a small company, one of the banes of your life is PowerPoint. It's not just that you don't get good PowerPoint, it's just that you get too much PowerPoint. And if we drop the cost of making PowerPoint presentations from 3 hours to 10 seconds, we're not necessarily going to get any better PowerPoint, we're just going to get, you know, loads of terrible PowerPoint. And finding where that balance is, and finding those, those filters so that humans don't have to bear this cognitive load, I think is going to be one of the really, really critical areas. Because one thing that I mean, I'm not a, I'm not a dystopian in any way, Nathan, I'm a pragmatic optimist about this. I think we've got a lot of potential, we're going to create a lot of space and headroom with AI and with renewable tech. But I do think that we are also at a moment where we're passing a little threshold. That threshold is that for a long time, some groups of people would say the world is moving too quickly, and technology is moving too fast. And over time, the number of people who say that has increased, but it was their subjective reality. But I think we're getting to a point where there's an objective reality that we're about to hit, which is once you start to connect agentic systems we just can't really cope, you know, cope with the 300 notifications we get on our phone today. And the way humans have typically done that is that we've not really had to face, face this. I think about the Chinese spy balloon that sort of made its way over the US. And one of the reasons this huge thing got across over the US was because, you know, the US has got amazing sensors, but they generate so many terabytes of data that humans can't assess them. So lots of them are just filtered away. And so the spy balloon wandered across, detected by some radio antenna, but never put in front of anyone. And of course, they've now changed systems so that they can do that. And I do wonder about what our interactions ought to look like in a world where it's not my assistant or me scheduling back and forth with you on WhatsApp, It's a bot that is going to work relentlessly and remorselessly. And I have it and you don't. And it's of no cost to me. And I'm just sort of doing whatever I'm doing, my yoga or something. I think the way we get through it is by finding ways of actually using AI to filter as much of those— that noise as we can, so that inboxes start to become smaller rather than, rather than bigger, because stuff has been taken care of us. In fact, it's kind of the reverse of the BlackBerry, right? When the BlackBerry was launched, I remember bankers used to take pride in how quickly they would respond to a message coming in even overnight. And the idea that someone would, would have that as an internal personal KPI today, you know, in the world of healthspan is just insane. And I would, I start to think about What is going to be that layer? Because you know what, I want all the benefits of these bots working for me and making sure my prescriptions are up to date and making sure we're not wasting electricity and getting me exactly the right flight. I always want a Dreamliner or an A350, and I'll go an hour later rather than get on a 777, but I won't go two hours later. I mean, I want it to know all of that and to give me that experience that I want, but I certainly don't want to be on the wrong end of thousands of bot-generated. messages and trying to work out which ones I have to pay attention to or not. And I think that's a really interesting opportunity space for someone to play in.

Just envisioning a quieter inbox is enough to make you a utopian in today's landscape. But just on this bot-to-bot communication, one thing I do kind of worry about is the idea that the bot-to-bot communication may begin to happen in high-dimensional latent space back and forth. In other words, embedding to embedding. I sometimes call this the great embedding, and I usually say beware the great embedding, because at the point where the AIs are all talking to each other in a machine language that is high-dimensional and not human readable, we have an extremely inscrutable overall system that we may find we can't really untangle that knot. I think we could very quickly in the next few years end up in a spot where, yeah, we all have these bots, they're all communicating with other bots, but we find that it doesn't really make sense for these bots to reduce everything to language and then send the language over and then have it be kind of reembedded. Why don't they just talk to each other in their native language, which is this high-dimensional space? We see so many— go through chapter and universe of different research that shows that this is very possible. You can adapt embeddings to another embedding space with basically just like a single linear projection. In many cases, you can connect vision space to language space remarkably easily. Blip-2 was one of my favorite examples of that, where they took a frozen language model and a frozen vision model and just trained a small connector between them and unlocked this entirely new capability. Anyway, whatever. It was a long list of those sorts of things, I think, demonstrating that it's possible. But then I'm like, man, we could very easily just find ourselves sort of surrounded by AIs communicating with other AIs in a high-dimensional way that we can't even really understand anymore. And now we're in a situation where things seem to be kind of working, but we don't really even know why or how. And this is one of the sort of more realistic, I think, loss of control scenarios. I mean, it's almost like the final scene of the movie Her, where the AIs go talk to each other.

The bots are going to end up communicating to each other because it's helpful for us and we don't want to sit in between them. And they will discover just through their optimization functions that translating kind of complex concepts into, hello, I'm here to request a meeting with Nathan, is inefficient. So they'll just do it in their high-level representation language, which is this embedding space, which, as you say, is inscrutable to us. Is that reasonably kind of the start of all of this? Yeah, that's right.

And I think there's enough out there to kind of show that there's a lot more room in the embedding space than language can actually reach. So that's one of the things with the these sort of bridge models where you find that the classic saying, of course, is a picture is worth 1,000 words, but you can also take an image and project it into this language space. The resulting thing in language space is not something that you could get to via actual language, but it's in language space. And so it has this kind of semantic meaning. And yeah, We, you know, what are we going to do with that, right? We can't, we can't even really inspect it. We can't even really read the logs anymore at that point.

I mean, it's a manifestation of what we might call the space of possible minds, right? So AI researchers have talked about this idea that, you know, octopus intelligence is intelligence, but it's got dimensions that may well be orthogonal to human intelligence. And what you've described is a mechanism, I think, by which you, you get there with with machine-based intelligence. And so that might literally be not just the semantics, but actually think back, have you ever heard of, read the story Flatland? It's a mathematical story. It's about sort of 2D people in a kind of 3D world, and they don't understand the concept of height. And so spheres pass through and they appear as sort of dots and lines and so on. And in that sense, there could be This could be emerging among systems that are among us. And there's, I guess, there's a second thing, which is also about timing. I think there'll be a relentless pressure to take the human out of the loop in decision-making, first in the softest decision-making, like customer service tickets, and then increasingly more and more so because speed will be a competitive advantage. And the human will blow the competitive advantage that you've got from from your bots. And I guess there's another risk, which is that lots of bots connected to each other are also at risk to cascades, right? Information cascades. And we see cascading failures, New York City blackout in 1976, the AT&T network failure in, I guess it was '91, or maybe it was '95, some of the worms. And we have governance mechanisms in place to now tackle those and stop those. And the financial services industries, you know, you have circuit breakers. And I'm just thinking about putting all of those together with the scenario that you, that you painted. So this would happen really, really quickly, could happen really quickly, and it could start to accelerate and work effectively at millisecond time, right, which is the time it takes across the internet to, to get to another system. What are you concerned with about the inscrutability? Is it just that it's inscrutable, so we don't know what's going on there? Or is it that it's inscrutable and it could be harbouring some kind of bad set of outcomes?

Yeah, who knows? I mean, you can layer on more and more concerns. I should credit, I think probably Ajeya Katra is a great person to go read for a long-form characterisation of this. Her essay, I forget the the exact title, but it basically amounts to, in the absence of specific countermeasures, the default path to AGI likely leads to AI takeover. And it's kind of this scenario where she envisions more and more work being done by AI, the time cycles being compressed. I don't know if she specifically has this high-dimensional communication aspect to it, but the notion is still that it becomes so fast and so dense that it's very hard for people to figure out exactly what's going on and why. And as long as that's working and we're getting more stuff out of the machine and consumer surplus is through the roof, which is definitely something I expect, is a lot of consumer surplus, then everybody will be very happy with this. But we don't really know what we don't know about where that leads us. And that could be like emergent autonomy or goals that are contradictory to ours, or it could just be these more unintentional cascading failures along the lines of an AlphaGo. AlphaGo isn't out to get us by failing, but it just turns out it has these fatal vulnerabilities that are just not obvious until somebody finds them.

So one of the reasons I'm a bit more sanguine about that scenario, although I see the risk, is that I think we already have that decentralized agent-to-agent communication, communicating in ways that most of us cannot understand, and no single person can. And it's created a lot of consumer surplus. And that's the global modern economy that works through market systems. And it uses a signaling method called the price mechanism to figure out where investment should take place over many, many different time horizons, to figure out what, what, where demand lies. And it, you know, the economy from an Austrian standpoint, like a Hayekian standpoint, is a giant information processing system. And it's made up of hierarchies of other information processing systems, you know, the most atomic of which is the freelance human individual, and the more complex of which are large megacorporations, which act against their own cost function or optimization function and behave in that system, sometimes with constraints, right, because they have dependencies on supply chain and, and other things. And one of the reasons I'm a bit sanguine about the idea of takeover is because What what we describe in that world is the modern economy that we currently live live with, and what we we already know that it delivers delivers tremendous benefits to us. But it also delivers things that we we don't value to us. I mean, the carbon crisis is is one obvious one. Quite often when we look at AI risk scenarios, someone goes off and says, "Well, the AI will persuade you." that you should behave in a way that you otherwise wouldn't, which is literally known as marketing. I mean, it is literally— and, you know, the US is on the wrong end of an obesity epidemic. And I'm not sure how many people acted with full agency to say, I want to be 100 pounds heavier than is healthy for me. That is my intention. And this is a decision I'm making with full agency. Somehow there is an emergent property about the way in which the economy has met needs that has enabled that to, to happen. But I'm not making ethical or normative claims about whether that's a good thing or a bad thing, whether people should have the freedom to do that or not. What I'm saying is that we have this system like of decentralized agents. And they do in a way compete with each other, because not every single person in the world is suffering from obesity and diabetes-related conditions, right? People make other choices. And there are push and pull forces. And so when I think about decentralized bots, I also think about that, that set of checks and balances that emerges when you have competing systems, and they need to have a signal that they are reliant on. And to some extent, we set that signal. And that makes me feel You know, sanguine, a little bit optimistic, still recognizing there's like massive amounts of work to be done around around safety and around risk. Around I watched I don't know if you ever watched this Battlestar Galactica, both the original but also the remake with James Edward Olmos and you know he's grizzled Adama and he refuses to upgrade the Galactica's network to the modern standard that the rest of the fleet. Uses. So he and the Pegasus, as we discover 2 seasons later, are the only battlestars to survive the Cylon onslaught, because they put a worm in the system, and they sort of turn it off, right? So, so we need to have kind of governance mechanisms in, in place upfront that allow us to observe and monitor the kind of the risks that you've identified, but a decentralized economy and decentralised has as an emergent property a way of keeping things in check. I think there is a homeostasis that emerges or a dynamic equilibrium that emerges.

I think that is probably the most likely outcome. And so in that sense, I'm also reasonably optimistic. But I do think it is worth really taking very seriously the idea that either with certain thresholds being passed, certain kind of feedback loops that could be triggered that are not yet triggered, things could change.

I just wanted to ask about that because you have the advantage of having played with the untrained GPT-4. So you got to see GPT-4 in its Darth Vader phase rather than in its reclaimed Anakin phase. As a smart guy who understands technology, when you were playing with it in this way? What were you feeling?

Awe, for one thing. Just shock and awe that this exists a lot sooner than I expected it would. It always felt kind of like science fiction, even when I was with text-davinci-002 and doing task automation and fine-tuning that model. As of the summer of 2022, I was very plugged in and putting points on the board for Waymark on a regular basis with a new fine-tuned model that can do this task a little better and improve our pipeline or whatever. And still it was just such a dramatic leap that I was really taken aback by it, mostly super excited about it. But then I would also say the big kind of safety lesson from that experience is that the control does not happen by default. And there's many ways of even conceiving what control could or should be. So this was under control in the sense that it was totally helpful and totally aligned to what I was doing. I never saw any Sydney-like behaviour from GPT-4 early. It never turned on me. It always 100% helped me with whatever I was presenting it with. But I do feel like we have this kind of broad divergence between the capability of the systems and our ability to really control what they're going to do or how they might be used. At this point, I wouldn't say we have anything to worry about yet. I don't think we have anything concrete that looks to me like the AI could run away on its own. And I did probe for that in my red teaming. One of the things I did that didn't really go anywhere and kind of led me to the conclusion that this model is probably fine to release, and my final report to them was like, I think you can release this. As far as I can tell, it seems like it will be safe. I would also, though, flag that there does seem to be a divergence between capabilities and control. And the reason, the sort of experimentation I went through there was setting up one of these kind of early agent systems. I was kind of doing it on my own. I didn't have a lot of reference points, but I basically just said, if I give it a high-level goal, can it break that down? Can it self-delegate to pursue that goal? Can it encounter errors and autocorrect and whatever? And it was kind of like, conceptually, yes, but practically not really. It always understood seemingly the goals. It would try to break them down. It was able to understand the concept of self-delegation effectively, which of course now we're pretty familiar with, but it just wasn't that capable. So it was like it couldn't go out and do a long series of things on the internet or whatever without just getting bogged down somewhere and getting stuck. I always kind of come back to the apparent divergence between capability and control. I have not really seen anything yet that makes me reverse my thought on that. I would love to see it. I'm kind of looking for things from the OpenAI superalignment group that may suggest that we've changed that dynamic, but I haven't seen it yet. And there's just a lot of different ways that something could be aligned or trained or whatever. And we don't even have really a great paradigm yet for what that should be. There isn't even yet really agreement on what good even looks like in terms of what we would want an AI to be willing to do or not do. So I think we're kind of into uncharted territory. That's maybe a good summary of how I felt. We're in uncharted territory.

You're lucky to have got that close in on those early moments when you see the unvarnished products. I mean, I would break out a couple of ideas. One is that connecting these things to tools in a non-SaaS environment, right? So OpenAI stuff is all SaaS. And for the next few years, at least, there is a metaphorical red button that someone can use to kill a rogue process just with any Unix system. But with open-source AI, and some of these models are getting sufficiently capable. I don't know what you run on your laptop, if you're running one of the Mistral models or, or something. I run one of the Mistral models. And, you know, it's pretty good. I pretended that when I was on the Eurostar or on a plane, it would allow me to continue to work. But in reality, you just may as well get to your destination and then use Perplexity or GPT-4. But of course, it's plenty good for task automation. It's plenty good for some of these those basic behaviors that we saw in those early agent systems. And those things are out, are out in the wild. And so I think what they do is they really expand the number of threat vectors that our existing systems face. Now, this is so much more prosaic than capability explosion. But I just feel that with, with anything that is, that is running on a data center managed by, on an Azure cloud, there are many things you have to do before you fly an F-22 and drop a JDAM on it to stop it. But, you know, we saw script kiddies build botnets and have seen them do that for a decade or so. And the— I think there's cybersecurity risk, which is capable enough models. And frankly, you were doing task automation with DaVinci 2. You can probably get something better than that running on a smartphone now in quite a small payload. And we're learning that we can get these payloads.

Probably a 3 billion parameter model could do a lot of the tasks I was doing.

And I wouldn't have noticed because it was snuck into a YouTube video download or whatever else it happened to be. So then you do get to this world of many, many systems that can talk to each other that can execute tasks for, I think, suspect naively complex DDoS, right, initially. And that, I think, feels to me like it is more of an approximate risk. And it's one that requires that combination of infrastructural players, right? You need Matthew Prince at Cloudflare, and you need Satya Nadella at Microsoft, and so on, because they own so much or control so much of the infrastructure. But it will also need new classes of new disciplines, right? What is the security architecture of our devices? How can we stop them when they start to go rogue? And we think about how rapidly NotPetya spread. And that was before you know, you had, you had systems like this that could be a little bit more clever. But so, so I imagine that that, that is something that seems again like a present risk that will start to manifest itself over the next couple of years. And I mean, I speak to some of the cybersecurity guys and they are obviously thinking about what are the tools that you need to, to defend and, you know, from these types of things. But again, I still struggle with the models that take us to runaway. If I'd taken us back 200 years or a couple hundred years ago and I'd said it's 1824 and the White House has just been burned in the war, burnt down, and I said to you, you know what, Nathaniel, in 200 years, you'll be 100 times richer than you are today. You'll be richer than the richest man in the whole of this continent, the United States, and you will have these capabilities and things that wouldn't have even sounded like science fiction because science fiction didn't exist at the time. You might well have, if you'd been able to believe me, say, well, surely we'll be at utopia and all problems will have disappeared. And we've run this tape before because we actually did get there and not every problem disappeared. There's been a lot of progress. And I do also think there is something in kind of human psychology that has us looking at moments of change like this and believing that certain paths are possible. And we don't look far back enough to say, well, our forebears really felt, felt the same. And I kind of feel that with not so much the climate crisis, which I think is, is difficult, but I do slightly feel, feel this with, with AI systems, because it feels like we're running that tape again. Now, just to add to my own confusion, I also see the power in the logic that says, number one, is it possible for us to engineer intelligence, right? Or is it something that comes soulfully from a mystical, superstitious force? It's possible to engineer it. I think you're a scientist, you probably believe the same thing. So number 2, if we can engineer it, is there any upper limit to what we can engineer? Well, no, there isn't, because we regularly engineer machines that are more capable than us in different ways. So number 3, if we can do that, Can we guarantee that it will be aligned with us? And of course, I don't think we can yet. I don't think we have the science yet. So I find that logic is really persuasive. It's hard to pick holes at, except when you start to say, well, what are actually the underlying assumptions for each of those steps? And what is uncertain about each of those steps? And where are their points of control? For each of those steps. So I sort of— I do agree that if you could magic up an incredibly powerful IQ 10,000 agentic with actions AI tomorrow, there would be issues. Let's just call it that. There would be issues. But, but when we're not— we're not going to, to do that, what's going to happen is that we've got to go a step at a time. And at each point, there's research, there's development, there's stuff that we didn't understand. There are limitations. You talk about the kind of logarithmic scale of inputs, right, that is slowing down. We're running out of data as well for training these models that play into what ends up being real. So I appreciate the logic, but I also think the reality unpicks into a lot of discrete steps around which you have to start to make progressively more extreme assumptions.

It does seem that we are in, as Sam Altman has started to describe it, the short timelines, slow takeoff world. And I think I agree with him, and it sounds like you probably as well, that is probably the best case scenario because we do want the benefits now and because we probably do need some time to adjust. If you were to tell me that this is going to take 20 years or 15 and it won't be until 2040, that we'll have a sort of human scientist level AI that's capable of prosecuting a long-term research agenda and coming up with meaningful new discoveries and whatever, then I would be much more confident that we will have the ability to adapt to that over that time frame. But I'm not sure about that. I see enough stuff now, and I hear, I don't know if Q* is real or not real, or if they're red teaming it in a bunker somewhere, right now or not. But it does certainly seem still plausible to me that there is another kind of paradigm-changing moment that just creates another kind of step change discontinuity in terms of capability that could get us there way before we're ready. So one of the things I'm watching for a lot these days is basically the transformer. I initially used to think about it as transformer successors, but now I kind of think of it more likely anyway as transformer complements, things that allow an AI system to do the things that the transformer does not do well. And one of those things is managing long contexts and kind of staying on task and online learning and integrated memory and so on and so forth. There's like a decent number of things that are pretty obvious that they don't do well. Going back to the original tale of the cognitive tape, right? You can look at all the places the human is clearly superior to the transformer and start to look out for architectures that might change that dynamic. If you said, how many meaningful breakthroughs are we away from the AI scientist that can produce eureka moments at a pace faster than human scientists tend to? It doesn't feel like it's that many. I would say it's probably more than zero, but it's probably less than 4. So somewhere in the kind of 1 to 3 range, because there's just not that many dimensions on the cognitive tape, the tail of the cognitive tape yet, where we're all that much stronger. There's a few, but I kind of put it in that 1 to 3 range. And with the inputs going exponential, including just the number of humans that are working on this and the number of papers they're putting out and the number of GPUs, and unclear to me also if we're running out of data. I don't really know about that, but synthetic data seems to work for a lot of things. And there's also just more modalities. We certainly have not taken advantage of— just think about how much security camera footage there is. If we really just want to go big to go big, there's a ton sitting out there.

The data issue is a speed bump. It'll get dealt with by accessing repositories that are available that we haven't touched, or improved synthetic data. And as you say, modalities. Between 0 and 4 probably doesn't seem unreasonable. I had this conversation with some senior people in some of the different foundation model companies, and they say sort of similar things. I think Shane Legg, co-founder of DeepMind, is probably at the lower end of, lower than 4 from a conversation I remember him having on a podcast. The question, I suppose then the question is, how long does each one take? Transformers took not particularly long. I mean, it was really a couple of years before GPT-1 and 2 years before GPT-2, actually, and then 3 for GPT-3. And it doesn't take long in the world of, of, of ArXiv. But the discovery does take time, and where that discovery is takes a moment. And in amongst all of that, though, is still— there is, there is still that stage that goes from the software capabilities coming together because we have the know-how and we've plugged it all together, and it iterating to a system that presents a control problem to us. And in the case of the AI scientist, that is a system that we can't call Kevin, the CTO of Microsoft, and say, can you just shut down the Austin data center in a zero for a second because the AI data scientist has gone rogue, right? The kind of in extremis mode. And that path to me also seems unclear. And there are a whole set of risks and downsides that emerge well before then, which I think help give us the infrastructure to deal with that scenario. So, and that is really, you know, how do you deal with the bad actors of people using GPT-5 quality LLMs on their smartphones in 3 or 4 years' time? And we will deal with it right in the same way that you know if I'd said to you in 1994, I don't know how old you were. I was 22. That by 2024 there'd be 120 billion identity attacks per year just on Microsoft. I wouldn't have believed you. I would have. I mean, yeah, Kurzweil, whatever, right? I just wouldn't have grokked that number. So we'll have this unseemingly large number of attacks coming mediated through LLMs and. in botnets and elsewhere. And we will have developed systems to, to deal with them. And that will be part of the fabric that we can't picture right now, into which this AI scientist will get, will get developed. And that's why these things become so very contingent. So the wrong thing to do is to say, well, because it's going to happen, let's do nothing, because then it won't happen. I think the right thing to do is to start to explore these ideas and have these, these conversations, But one of the things I think is really problematic, has been problematic, has been the conversation focusing exclusively on existential risk, which it really, I felt it did in 2023. What it does is it diminishes public trust in technology, it forces policymakers to make decisions that may not be, you know, well informed, they may not be pro-innovation, they may not even be pro-safety, right? There may be a bundle of really terrible spots. And I think a little bit of a great Liu Cixin, he's a science fiction writer, this Chinese guy who wrote The Three-Body Problem. But in his book of short stories, The Wandering Earth, there's a moment where they have to rocket Earth Space: 1999 style out of the sun's orbit to prevent some calamity. And it's going to be a multi, multi-generational journey to the next planet. And in order to do this, people have to live in really terrible conditions. Apart from the scientists who are keeping everything monitoring, looking for the signs, planning the the process of decompressing everything, and of course the people get lose trust in the scientist and in truly Chichen style. Sorry, there's a spoiler. The people rebel, kill all the scientists, and then lo and behold, the next day they show up at their destination. And trust is really really critical, and I don't think we did a lot to get people. Who are outside of the tech industry to put trust into technology, put trust into their ability to participate in it and to have some agency in where it goes, to be excited about it. You know, off the back of, you know, all of the sort of polarization and the sometimes legitimate, sometimes not scaremongering around phones and social networks and so on. It didn't feel like it added to the discussion of trust. So I was quite happy when I went to Davos, this World Economic Forum meeting, that the conversation had moved from, is it, you know, what kind of munition should we use to drop on a data center, to what are real pathways? What is the science that we need to do in order to make these things safe in the long term? What is the kind of appropriate regulatory interventions? What do we do about things that are approximate 3 to 5 years around misinformation and cyber threats, while still recognising that there is a pathway that you've described that needs to be addressed?

I find it easy to empathise with basically every AI perspective, from the enthusiast to the x-risk concerned, to those that are screaming about poor use of face match technology by police departments. I mean, really, the whole thing, I think, is like very— it's all valid in my mind.

What's really exciting, and what did not happen with the mobile, didn't happen with PCs, it did not happen with the first mainframes, is that, you know, technology is, is an intimate part of what it is to be human, right? Technology is our compounded knowledge. Technology is, is the binding factor that enables a world in which we then have our human relationships. And to have so many discussions about a technology early on, when it's just in its early adoption phase, we're not too late to it, I think is incredibly positive. And I'm glad that you have a Big Tent show. I mean, I have an opinion about sort of x-risk, but I still also feel I'm Big Tentian. But I'm just really, really glad that we're having a wide and extensive conversation, one that feels wider and more extensive and kind of more grounded in some ways. than any conversation we ever had about the internet back in the early '90s.

Yeah, in some ways it's funny. I think in some ways the discourse is getting a little bit more deranged over time, as there is some polarisation and kind of ideological entrenching happening in some places. But then in other ways, I definitely think it's getting better, if only because what we're actually dealing with is becoming a lot more clear.

There is a lot of room for commonality, for common ground, and for a recognition that there are different pieces of work that need to get done by, by different people. And, and actually, there should be— there should also be enough money in the tank to be able to do it, right? This is a rich industry, it spits out a lot of profits, we should be able to, you know, fund it, fund it some way. Again, when you see the kinds of things people say about each other on Twitter, and then you meet them in person, and They have the same conversation and it's a more measured space, just in my limited experience of it all.

I think it's kind of everything everywhere all at once. It's like, yes, there are definitely things that are quite unhealthy. And I do not like to see enemies lists getting published by leading technologists. That's like the Techno-Optimist Manifesto from Andreessen has a section that is literally called The Enemy and names names, if not individual names, at least like specific and relatively identifiable groups. So I don't like that. But at the same time, I made some noise about OpenAI and kind of could have easily been retaliated against by them. And I can imagine a lot of companies that might have come down on me hard and expunged my name from their case studies on their website and all that kind of stuff. And they didn't do any of that. And so I do think there's also aspects, and I feel pretty fortunate. And this is one of the kind of concluding questions I wanted to ask you is, I think in some sense, this is like a collective responsibility. We all have to orient ourselves to it, get familiar, and try to figure out what's it mean for us and what can we do to shape it in a positive way. It's definitely not all somebody else's problem, but at the same time, there are these leading developers who clearly have outsized influence, outsized power. There are also key decisions that are getting made around, are we going to open source Llama 3 or are we not? Sounds like we're going to. And then there's government that can potentially say, hey, we require off switches at data centers. I'm not sure data centers have off switches right now. You might be able to go in there and start hacking at them, but is there an actual easy off switch in most of them?

No, I don't think there is. I mean, they're designed to be Yeah, the opposite, right?

Yeah. They don't probably want some rogue employee either to go in and turn it off, right? So they've probably engineered away from anybody being able to easily turn it off. So government may have a role there to play that's like, look, we need off switches. We hope we never have to use them, but it seems like we might want to have them. So who do you think kind of bears the greatest responsibility? Or, you know, or where do you think we should be investing our trust? You know, is it these leading companies? Is it auditors, you know, that could be independent groups? Is it the government? I mean, it's probably some mix of all the above. But what are you kind of bullish on in that regard?

I mean, you know, the US has been an incredibly successful democracy for a long time because of separation of powers. And, you know, structurally, that works. A company, however good it is, however well-intentioned the CEO is, will end up with its own ambitions and directions. And so, you know, you will always need to push. If the company's offering you 3, demand 6. That's just good practice. So I think that each player in this circuit has to have the right capabilities to have the right conversations. And I think one of the things that we can learn from the experience of the FAA and Boeing is that you cannot deplete your own capabilities and ask for self-regulation because it just doesn't work out, however well-intentioned the firm is. So I think what you need to do is we need to invest in the capabilities of governments to ask good questions and engage well and overcome all of the complexities that exists that you can make $10 million a year as a, as an ex at OpenAI. And, you know, you won't do that in government. And I think that that also raises the value of investing in academia research and civil society. So Yoshua Bengio, for example, is running a really important project. I think it's based out of the UK, which is a sort of core science project to look at some of these risks and these evolutions and unanswered questions around control. We need to really, really start to, to level up. And I don't think it will be sufficient to just allow the big firms to do that and insist that they spend the money as directed by them. You know, I think if they have— if they're willing to put money into it, it should go into pots which go to grow the capabilities of the people who will keep them in check. And the reason that works is that, you know, the car industry is really successful Because someone mandated that cars needed to have brakes. Now, without brakes, people wouldn't buy as many cars as they do. And I think this is good for the industry. And someone needs to understand within government, within civil society, within academia, academia, what are the right questions? And so what are the right interventions going to look like? And we can all agree as grownups that companies, however well-intentioned they are, will always have their own, their own agenda. And we just, we just acknowledge that and we all move forward in a generative, critical, constructive way. So whichever player is weak at this table needs to have some support to become stronger. And that probably right now is amongst governments and regulators and academia rather than the big few tech firms.

That might be a great note to end on. Anything else you want to touch on or cover that we haven't got to?

It's really easy, such a facility in having the conversations with you, and really was so excited that you agreed to do this. I thought the murder mystery, which is you and your red teaming show, was just brilliant as well. So, I know that you've got another hat, which is suspense, and I look forward to the next episode of that.

Well, thank you very much. I really appreciate that, and I appreciate your time and participation in this as well. Azeem Azhar, founder of The Exponential View, thank you for being part of The Cognitive Revolution.

My pleasure, Nathan. Thank you.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.