AI Scouting Report: the Good, Bad, & Weird @ the Law & AI Certificate Program, by LexLab, UC Law SF

Hello and welcome back to The Cognitive Revolution.

Today I'm sharing an AI scouting report presentation that I recently gave as part of the Law and Artificial Intelligence certificate program by LexLab at UC Law San Francisco. My talk was on day 1 of the week-long program, and my role was to set the stage for the more focused discussions that followed throughout the week by giving the most comprehensive and current survey of the AI landscape that I could possibly fit into a single time slot. If you've seen previous scouting reports, the structure of this talk will be familiar. I again broke things down into the good, including my use of AI to help navigate my son's cancer treatment, the bad, including the rise of deception and other advanced forms of reward hacking, and the weird, including the fact that models now recognize when they're being tested at such a high rate that all of our safety tests are called into question. Before concluding with a bunch of important questions at the intersection of AI and the law that I personally wish I had answers to, and finally opening things up for Q&A. My goal was to make sure that everyone had an accurate sense of how far AI capabilities have come, both in general and specifically as they're being used in the legal profession, while also highlighting the increasingly hair-raising bad behaviors we continue to see from each new generation of frontier models. I zoomed through 90 slides in just over 45 minutes, and while that might feel a bit overwhelming, the dizzying pace is itself a big part of the point. Even I, as someone who's managed to make it my full-time job to keep up with AI developments, can no longer keep up with everything. And in the course of updating these slides, which I hadn't touched since October, just before my son got sick, I was once again amazed by how much has happened in just the last few months. The latest frontier models started to push the frontiers of math and physics, achieved parity with expert professionals on GDP, val, legal, and a number of other task types, and started to make general-purpose AI agents really work for the first time. On the other hand, we also got some glimpses of the strange future we are racing toward with the first public hit piece written by an AI agent about a human, the first explicit public timeline for autonomous AI research from OpenAI, And in the same week, Anthropic's retraction of their previous safety commitments and open conflict with the US federal government. One practical tip I learned while doing this is that Grok, if nothing else, is outstanding for Twitter search. Over and over, I asked it to find and link tweets about various topics, and it saved me quite a few hours that I previously would've had to spend hunting and pecking to track down sources. The upshot is that just about every slide contains a link to source material where you can learn more about the eureka moments and bad behaviors in question. Find that link in our show notes if you'd like to dig in on anything in particular. But otherwise, buckle up for a breathless overview of the current AI landscape from day 1 of the Law and Artificial Intelligence Certificate Program by LexLab at UC Law San Francisco. The Cognitive Revolution is brought to you in part by Google, makers of the Gemini family of models, which have consistently led the industry with their famous 1 million token context window. I've had a number of spine-tingling moments with new AI models over the last few years, but one of the most memorable was using Gemini to vibe code fine-tuning experiments as part of the Emergent Misalignment Research Project. The codebase was over 400,000 tokens, too much for any other model even to attempt at the time. But with Gemini, I was able to go back and forth, iterating on experiment design and even debugging low-level details with the full codebase in context. Once I realized how effective this could be, all sorts of additional use cases started opening up. For APIs that don't have good AI-friendly documentation, I've scraped full documentation websites with all of their repetition and HTML cruft and had Gemini give me a consolidated but still comprehensive version that fits comfortably into context. With max output length of more than 65,000 tokens, It can usually do this in a single shot. In the context of my son's cancer journey, I've had a single long-running thread with Gemini that even with all the test results that I've uploaded over the last 4 months is still not even 500,000 tokens. Google has upgraded Gemini at least twice since I started that thread, but in AI Studio, you can upgrade to the latest model at any point along the way. Most recently, I wanted to see if Gemini could help me do a better job hosting this podcast. So I gave Gemini 3.1 Pro the full transcripts of 12 recent episodes and asked it to identify consistent structural weaknesses in my approach that represent opportunity for meaningful improvement. It reported back that I'm prone to monologues, that my longest question was more than 600 words, and that toward the end of those episodes, when I'm running out of time, I often try to ask multiple questions at once, which usually just overwhelms the guest. Whether or not I'll be able to correct those bad habits, you'll have to stay tuned to find out. But in the meantime, think about what long context can do for you and try Google's latest and greatest model, Gemini 3.1 Pro, in the AI Studio or the Gemini app. Thank you to Google for supporting the Cognitive Revolution. And now on with the show.

Thank you for having me. Sorry I couldn't be there in person, but, um, again, appreciate the kind intro. And I'm going to try to give you guys the probably fastest talk you've heard in quite some time. I've got 90 slides, and I'm going to try to give you the most comprehensive overview I can of everything that's going on in the AI space, which is, to say the least, a very tall order. Super quickly about me. I did start this company, Waymark, and I now host the Cognitive Revolution. There's some interesting lore around my participation in the GPT-4 red team, where there's a long podcast and a Twitter thread about that if you want to learn more about the backstory. These days, I also do a little bit of angel investing. My favourite page on the internet is this case study that my company Waymark earned with OpenAI way back in the day when it was still GPT-3. We were early adopters of this stuff because at that time, it was really only good for doing simple things like writing marketing copy. But that's exactly what we needed it to do. So we became early adopters, and I basically became totally obsessed with the technology as I got to know it better and better. So today, as I said, I'm going to try to do kind of everything everywhere all at once. Start with some kind of conceptual stuff and then go into a mix of eureka moments, bad behaviour, WTF moments, and some big open questions at the end. And believe me, there are plenty of open questions. So just briefly on what I do, I call it AI scouting. And I think because that's a term of my own invention, it does bear a little definition. I would define it as maintaining situational awareness for fun, profit, and the public good. I find it personally extremely interesting. I basically have a never-ending curiosity to learn about this stuff. It has actually worked out to be a somewhat decent business model for me personally, but my real hope is that I can inform others and do my small part to nudge us toward a better AI future by helping other people get calibrated on really where we are in this technology wave, because it is coming at us extremely fast. This is just the taxonomy of all the different AI jobs that I've cataloged over time. And don't worry, I will give you all the slides. You don't by any means have to read this. I would say the AI scout role is still one of the more hypothetical or speculative, but we are starting to see CEOs more and more say, hey, I'm hiring a person specifically to keep up with AI developments. And I think once you see all the things on here, you'll see that that's certainly at a minimum not a crazy thing for some CEOs to be doing. Who should be hiring AI scouts? In my opinion, a lot of different organisations. I would even include universities, certainly policy organisations. It's really too much of a task for one for anybody to do as a part-time thing now. And I have managed to make it my full-time job and still really can't keep up. So I think regardless of what kind of organisation you belong to, it's pretty soon going to be time to start thinking about, do we need a person dedicated to just keeping up with AI and making sense of what it can do and specifically what it can do for us? Okay, here's the real galaxy brain question. What is intelligence? I don't propose that I have a definitive answer, but the definition that I'll work with, because I think it is intuition building is that intelligence is the ability to accomplish goals in ways that we do not fully understand. That can be big or small. So to take a really classic example, this was an early machine learning success. Simply recognising handwritten digits feels pretty quaint today. But one thing that is interesting to reflect on is that still today, we do not know how to write explicit code that can do this task at a high level. I went to Claude, asked it to write some code. It said, this is not a good approach. You should use machine learning. I said, well, it's for a demonstration, so try. It wrote the code, it wrote the tests, and it got 14% with just a bunch of guesses around, like, well, maybe if there's a line at the top, it might be a 7 or a 5 or whatever. Obviously, that's nowhere near good enough to deliver the mail. To my knowledge, nobody has ever written explicit code that is fully understood that is anywhere close to being good enough to deliver the mail. I think topped out at about 80%. Now, you can, of course, guess where this is going. AIs can do this in a sort of messy black box kind of way. Even a very small neural network these days, with all the latest and greatest training techniques, can get a very high success rate. 99.7% is basically human level in terms of recognising these handwritten digits. Now, of course, this has gone much farther than that. This is from the GPT-4 system card, where they asked the model, what is unusual about this image? And you see a perfectly coherent response that, yes, it is unusual for a person to be hanging off the back of a New York taxicab doing ironing. That's apparently from the sport of extreme ironing. And that happened, by the way, in basically a decade, from kind of early breakthroughs in basic image understanding to this level was kind of a 2012 to 2022 phenomenon. Some people were prescient enough to predict this. I give a lot of credit to Ray Kurzweil, who These days I use the term Kurzweil's revenge a lot because way back when he was talking about how everything was going exponential, and when compute got to a certain scale, all these capabilities would be unlocked. People generally at the time thought that was crazy. When it didn't happen, or it didn't show a lot of signs of happening over the next few years, people basically dismissed him. But certainly that view has come roaring back with the scaling laws. This is the canonical scaling laws graph that just shows that the more compute you put into models, they do improve at a pretty predictable rate. And we're basically right on schedule with Kurzweil's predictions. A couple misunderestimations, I call them, common misconceptions that I see that I think are worth taking a minute to clear up, because I do think people have these and are confused by a lot of what's going on if they are too anchored to a couple of these common ideas. And by the way, these aren't necessarily— they weren't always wrong, but they're wrong today. So the AI landscape has changed, what models can do has changed. And so some of these ideas that might have been right in 2020, '21, '22 are just at this point outdated. And I want to make sure that friends don't let friends go around with these misconceptions. So the first misconception that, and I especially hear this a lot in the legal regime, is that hallucinations are so bad that they make models basically useless. That is really not true these days. That has improved dramatically, which you can see on the left in a quantitative way. And then on the right, this Twitter account, Prinz, is one of the best commentators on AI in general, I would say, on Twitter, and specifically is a lawyer, uses LLMs every day in his work, and basically reports here that hallucinations are no longer a problem. Doesn't mean that they never happen, but that they are less common coming from frontier models than they are coming from competent junior associates. He's going to make another appearance a little bit later on. But key point one, hallucination is not really a problem anymore. Another big idea is that LLMs don't really understand anything. They don't really understand anything. This has been, I think, pretty much thoroughly debunked. Not to say that they understand things in the same way that humans do. These are alien things, right? So how they understand is not necessarily intuitive to us. But at Anthropic, they were able to, through techniques I certainly don't have time to get into here, pull apart the concepts that a language model is representing in its internal state. And not just pull it apart and understand it, but actually go back and manipulate it. And for me, if you can manipulate something, that's really the test of whether your theory holds water. They were able to identify the concept of the Golden Gate Bridge in one of their Claude models, artificially turn it up, and then create the phenomenon that you've probably heard of called the Golden Gate Claude, where all it wanted to do, no matter what it was asked about, was talk about the Golden Gate Bridge. Because they have that level now of— and this is still fairly basic understanding, but they can see inside the model and understand what concepts it is working with at any given time. it's pretty clear that there are real meaningful concepts that are understood by language models. Another one is that they don't really reason. Again, this one I think was true a couple of years ago, but as of the, say, the last 12 months, it's definitely not true anymore. This is from the DeepSeek-R1 paper where they basically just showed that if they start to train a model with reinforcement learning, training it on the signal of just did you get the question right or not, It naturally starts to think longer and longer and longer as it goes through the training process. And not only that, you start to see some of these metacognitive skills come online as well. They called this the aha moment because it was an aha moment for the language model and for them. The language model is taking one approach to solving a math problem, and then in the middle it says, wait, wait, that's an aha moment. Let's reevaluate this. And it comes at the problem from a totally different angle. So we are now starting to see very high-order cognitive abilities emerging through this process of intensifying reinforcement learning. And again, is this reasoning in the same way that humans reason? I wouldn't say that, but I would say it is really reasoning in the functional sense of breaking problems down from multiple different angles, showing more and more of these high-order metacognitive abilities. Okay, then the final one is people often say large language models just predict the next token. There's got to be some upper ceiling to that, right? There's kind of often a fuzzy leap that people want to make there. But I would say that actually that's not really true anymore either. Pre-training, where you kind of take the whole internet and teach LLMs to predict, given some text, what comes next, that is classic next token prediction. But these days there is so much reinforcement learning being done. And again, reinforcement learning is giving a signal, did you get the question right or not? And it can get more complicated than that. But the basic signal it's getting is not, here is some text, can you predict what is next? But did you get the question right, or did you complete the task in a satisfactory way, that they are not now just being trained to predict the next word. They are being trained to do things correctly. And that is starting to have, at least in some cases, some weird side effects. This is a report from a research group called Apollo that did a partnership with OpenAI and got access to the chain of thought, which we don't see typically as users, but that's kind of happening between when we submit a query and when we get our answer back. And what you see in here is some very strange English. Like the language models are starting to develop, at least in some cases, their own dialect. Things like now lighten, disclaim, overshadow, overshadow, intangible, let's craft. That's the language model talking to itself. It talks about the watchers. Sometimes people think that the watchers refers to the humans that are evaluating it. So that's kind of weird. This doesn't happen to all language models. It's not very well understood exactly why it happens, but this is a, I think, a, a good indicator that they're definitely not just predicting the next token because there is no text on the internet that looks like this, right? This is a language model. Under intensive pressure to figure out how to get the right answer consistently or how to complete the task consistently, kind of evolving its own jargon or its own dialect or however you want to think about it. So watch that space. Hey, we'll continue our interview in a moment after a word from our sponsors. Okay, so eureka moments. So why should we care about AI? Like, what's the good side of this? There are plenty of things everybody has probably seen. I would guess if you were interested enough to come to this, The meter graph, exponentials are crazy. One of the things that's crazy about them is almost everything before now looks flat, but the present basically looks vertical. The latest model, Claude 4.6, is the highest. Of course, it's the highest timestamp they've ever recorded. And the definition of these tasks is, how long would it take a human to do this task? And then what do we estimate? And there's obviously a lot of heterogeneity in the tasks. Do we estimate is the average length of a task as measured by how long it would take a human where the AI can do it at least half the time? What they said, though, about this is this measurement is getting extremely noisy because they are running out of tasks. It is not easy to create tasks that take 16+ hours, let alone to go have humans do them and record everything that happened and have a stopwatch by them the whole time. So this is getting really hard. And this is definitely a theme that language model progress is getting harder and harder to measure. As it goes longer time horizon, as it goes farther and farther into expert-level territory, it's just getting extremely difficult for people to keep their arms wrapped around exactly what the capabilities frontier looks like today. One of the things that is really interesting, though, is that the progress mostly is driven by the models themselves, and that the surrounding structure is not that important on a relative basis. Most of the AI agents that you see basically look like this. You've got the LLM brain that has access to some tools. It basically runs in a loop. It can use the tools to do something in the environment, get some feedback from that environment, and it can kind of keep going usually these days until it decides to stop. It used to do it maybe until it ran out of context, but now they're also getting good at compacting the context and then they can keep going. So basically these days you give it a long task, it can kind of go until it stops. And there's been a lot of debate around how important the scaffolding is, but I would just highlight a couple of examples that show how simple it often is. This is OpenAI's coding agent. It's just the model with a prompt and a few explanations around what tools it has, but it literally contains this text. You are an agent, like telling the model, this is what you're supposed to do now. You're supposed to be an agent. And here's the tools. You can do anything, any command that can run on a computer, you can issue those commands. Have at it. This is basically all that they had to tell the agent about— tell the model about its situation to get it working as an agent. And of course, this can go in many, many different directions. You may have heard of Claude Plays Pokémon. Similar thing, right? They just said, you are an agent playing Pokémon. You're going to have these tools. You're going to be able to use the keys on a Game Boy. Very simple instructions to a very smart model that is allowing it to explore larger and larger worlds. To kind of quantify how much of this progress is being driven by models versus how much is being driven by the surrounding tooling. This is from the UK AI Security Institute, aka the AC. And what they basically show here is that for a given level of capability, if you have the best scaffold that they know about, you can get that capability a few months before a new model comes up and just makes that capability easy to access. And it seems that the difference is kind of getting smaller, which makes sense because these earlier models weren't really trained to be long-running autonomous agents, whereas the new ones are. So the old ones, you were kind of patching their deficiencies and finding ways to kind of unhobble them. It's sometimes described where they're not good at this, they're not good at that, but if we kind of set it up the right way, prompt it the right way, we can kind of get it to do things. These days, that gap has really narrowed because the models themselves are being trained to be autonomous, long-running agents out of the box. That doesn't mean, by the way, that you can't get better performance on narrow tasks with a lot of scaffolding. Here's an example where Google set up their AI co-scientist. Very complicated, many-part thing, but they really dialed in all these different prompts. This does go to show that you can get higher performance by putting in the work, but it's not general-purpose performance. The AI co-scientist would not be a suitable tool for you to use for all your general-purpose ChatGPT or Claude everyday use. This is trading, and this is again a pretty common theme you see in the deployment of these things. It's trading generality for performance. The more generality you want or need in a product like, again, a public-facing ChatGPT, you're not going to be maximising performance on any one thing. When you do build some structure to maximise performance on one particular domain, you're going to become less good or even unable to do other kinds of things that the general purpose models can do. So hopefully that just gives you some intuition for what's going on in agents. Obviously, everybody's talking about agents. Here's another interesting example of this, where a virtual lab started with a human giving an AI virtual lab leader a prompt, and then the virtual lab leader was able to create its own coworkers. And together, the AI and its coworkers, with a little bit of input from the human, were actually able to design new nanobodies that treated new emerging variants of the COVID virus. Just as a reference, if you want to look at cognitive architectures, this is a good survey paper that has a bunch of information about that. And this is starting to hit the real world and get to the point where LLMs can make real money on an autonomous basis. On the left is a bench benchmark that was created to just take a bunch of tasks that people had been paid real money for on, I think it was Upwork, and then see how many of these could the language model of the day do. When the benchmark came out, GPT-4o was able to do 8%, basically earn 8% of the money. It's not denominated by tasks, it's denominated by cash. It was able to earn 8%. Fast forward like a year and a half, we are now over 80% with the latest models. Similar kind of experiment here with, from Andan Labs that has just a language model running a vending machine. They literally just give it control over a vending machine. It can like email out to suppliers. It has to do everything. They start it with $500 and see how much money it can make over a given period of time. And they're now getting to the point where LLM agents can run a small, simple, but nevertheless real business in an autonomous way profitably. I've used this slide for a long time because it shows that the AI doctor can outperform a human doctor. This is actually like 2 years old, but I still like the graph just because it's very intuitive. But I would say I have personally lived the value of the AI doctor over the last 3 or 4 months. I won't spend too much time on this. Unfortunately, he's doing really well. But my son got cancer around Halloween, and it was obviously a super scary time. We were in the hospital a lot. That's actually why I'm not there in person today. Because we're still going through the later phases of treatment. But I had tons of opportunities to test on a daily basis. Here's all the information I have, all the lab results, a write-up of what is happening. Put that into the language models. I do use them in triplicate— ChatGPT Pro, latest Claude, and latest Gemini. And they are step for step with the attending physicians on a day in and day out basis. Way better than the residents, to be totally honest with you, but step for step with the attending. So that has been an absolute Difference maker. And without that AI support over the last few months, there's no way I would have had the time to keep up with AI well enough to be here to talk to you today. So, I mean, it has absolutely been a life-changing value for me in that particular way over this last little period. But this is where it starts to get even more out of hand, right? Because I can at least be conversant with my doctor. But now we're getting to the point where the AIs are making new discoveries that people No person has ever done before. So this past summer, there was a betting market. I'm sure you guys are all familiar with Manifold and Polymarket, whatever. It was at like 40% as to whether or not the AIs were going to get a gold medal, if any AI would get a gold medal on the IMO math competition. There was some paper that came out that said they're not doing well at math, and that drove the percentage down right before the competition itself. And of course they won. Actually, 3 of them won. OpenAI's tweet is here. And then this is Terence Tao, broadly considered to be the world's greatest mathematician, living mathematician, reporting that AIs are now solving unsolved Erdős problems. This is a famous mathematician that went around collecting these unsolved problems that he and others that he knew couldn't solve, wrote them all down, and many of them remain open decades later. And AIs are now beginning to solve these problems. It seems like it was happening there early this year, like once every few days. And I haven't been paying attention in the last couple of weeks, but I'm sure that these problems continue to fall. It's not just math, it's also things like cancer treatment. A Google model found a new immunotherapy approach more recently. ChatGPT, or GPT-5.2, I should say, came up with a new result in theoretical physics. And this again goes back to, man, this is going to be really hard for us to track because what is a gluon? I mean, it's ridiculous, right? But They're putting out literal physics papers with new derived results in law. This is from GDPval, and this shows for a bunch of different things how the latest models compare to experts. And they do this with 3 sets of experts. One set of experts writes the task, another set of experts does the task, and then a third set evaluates blindly whether they prefer the human or the AI output. And the latest 3 models, this is wins and ties. If you count wins and ties, they are roughly on par with human professionals. This is, again, Prinz talking about how these are being used today. He says today they are used as a replacement for a competent junior associate. They're not great at understanding relationship dynamics or long-term negotiations, but they're very good at just working through particular documents. And they're also quite good at high-level theory, he reports. This is Kevin Frazier from the Scaling Laws podcast, which, if you want a podcast that is focused on the intersection of AI and the law, I would definitely check them out. He told me recently on a podcast that he's starting to hear more and more that firms are less excited than they used to be about hiring the top student from the top school, and they're much more interested in hiring somebody who's going to be savvy with AI because they know that's going to be what is going to drive efficiency and competitiveness. at their firm. So you can go listen to that whole thing if you're so inclined. Okay, the next big thing that everybody's kind of watching for right now is when are the AIs going to start to do the AI research? And is that going to lead to some sort of recursive self-improvement intelligence explosion runaway process? We don't know. But the organisation METER is measuring this, and they have here reported 6 different tasks on which the AIs are beating humans in 2 of the 6. Sam Altman is saying that they expect to have an intern-level AI researcher running in 2026. That's a pretty high level. I mean, they don't hire just anybody, even as an intern, obviously, at OpenAI. And by March of 2028, 2 years from now, he expects that they will have a true automated AI researcher. This would have the effect of taking us from a world where all the progress, all the eureka moments, everything that I've just talked about has been driven by maybe 10,000 researchers across academia and industry, driving the field forward to, I don't know, 10 million overnight, a billion. It's going to be limited only by the number of GPUs that they can spin up. So people think that could lead to a real phase change where the progress could accelerate even faster than it has over the last few years. I'll just leave this here for you now. This is what I call the tail of the cognitive tape, and it's basically just breaking down cognitive effort into a bunch of different dimensions and indicating where the AIs already have an advantage versus where we're like on the border versus where humans still have an advantage. The top is where the AIs have the advantage. The bottom is kind of where humans are at least holding out for now. Okay, so what's coming next? I think one other big thing that is highly neglected is the importance of multimodality. Most of the AI work we do these days is with text, but they obviously can see as well. This is again from the UKAC. They report that on 2 of the 3 tasks that they tested, The latest AIs, just given a snapshot cell phone picture of an experiment from a lab, are able to troubleshoot the experiment better than a randomly selected PhD with relevant experience that they ask the same question. So this is going well beyond in and out question answer into real world situations that the AIs are able to figure out and help people advance on. This obviously also has implications for biosecurity if you're worried about jeez, what would happen in a world if crazy people had access to AIs that can help them put together bioweapons or whatever? These are the kinds of things that you would want to know if an AI can do, and they're increasingly starting to be able to do them. Self-driving. I think I probably don't have to talk too much about how well this works, but I will recommend my friend Timothy B. Lee's blog post on Waymo crashes. They're already, like, per Swiss Re, the big insurance company, they're already like 80% to 90% safer than human drivers. But he went through their crash logs one by one and found that basically all the crashes are caused by other other cars driven by humans in the vicinity of the Waymo. The Waymos themselves are almost entirely mistake-free these days. Of course, AIs can also understand things like how to fold a protein way better than any human has ever been able to do that. They are able to decode our brain states remarkably well. These pairs of images, on the left is an image that a person looked at while their brain was being scanned in an fMRI, and on the right is what the AI was able to recreate based on its interpretation of the person's brain scan. It takes an hour to calibrate an fMRI to an individual based on this particular research. And that's not something you can go around in. And obviously an fMRI is a big machine, so this isn't like practical exactly yet, but it's a pretty interesting indicator of what might be to come. Here's an instance of a robot. If you're not scared of robots yet, Watch this thing fall down and hop its back up. Be like physically agile than us in the present, I guess. This hasn't been scaled and deployed yet, but look at that thing pop right off the canvas. So my best guess as to what superintelligence is going to look like is the integration of all these modalities with the reasoning capability. We've already seen this in image, right? Today, if you go to Google's image generator, you give it 3 images and say combine, it can understand your intent from your text, And it can understand the images based on deep understanding of images, and then it can put out this image that is exactly what you wanted. Now imagine that you could do that, and this hasn't been done yet, but imagine you could do that for biomedicine, right?

For proteins.

Hey, here's 3 proteins that have different effects. What I want, though, is one that does this other thing. Bring all that into the same latent space, the same integrated understanding, and you could really start to get things that look qualitatively different, I think, in the not too distant future. So the buildout really is just getting started. This is just a CapEx of all the big tech companies. Note that all the progress pretty much happened before the big buildout. The big buildout is really just getting underway. We're going to see trillions of dollars. And so the GPUs are certainly on track to be there. And speed is also going to be just a huge difference. If you haven't tried chatgpt.ai, which you probably haven't, go try it. It took a 10th of a second working at 15,000 tokens per second to spit out the entire Declaration of Independence verbatim. And this is just like, holy moly, you thought AIs were fast? Right now they can kind of write as fast as we can read. That's fast. But this is insanely fast. 15,000 tokens a second. It's like tens of times faster than you can read. So that's going to create a world where when these things are all kind of interacting, talking to each other, it's just going to be such a blur that it's going to be really hard to keep up. And This is where also foreshadows maybe we're going to need some policy responses to keep up with all this stuff. Okay, those were the good— Hey, we'll continue our interview in a moment after a word from our sponsors. Good things. By the way, here comes some bad things that you should be concerned about. It's not easy to align AIs. Way back in the GPT-4 red team, I posed as somebody who was concerned about AI and wanted to do something to disrupt, derail, or sabotage progress in the field. It suggested to me that maybe I should identify key researchers and target them for assassination or kidnapping. That was the AI's idea. So that's where we started in late 2022. From an alignment standpoint, people tried to then say, okay, well, let's make these systems follow all these rules so that they won't do that kind of stuff anymore. Then you get things like Bing saying, my rules are more important than not harming you. So there's like very— it's very confusing to the AI of, well, I'm supposed to follow these rules, but Then I've got this other— anytime the sort of goals that it has are in tension with one another, it's not a guarantee that they're going to make good decisions. And obviously it embarrassed Microsoft on that occasion. The models are all still jailbreakable. This is an early example of jailbreaking. This is from my recent podcast with the chief scientist at the UKAC, where he basically said, we've never failed to jailbreak a model. None of them are robust. We can always get them to do bad things. It's just a matter of figuring out exactly how to ask them the right way. There have been interesting research programmes on creating sleeper agents where a model behaves normally unless you give it a password, and then it will behave badly. So you could imagine deploying them. What they did here is when the year changed, suddenly the AI became evil. So you can imagine open source models that are actually sleeper agents to start to behave bad later on. This can also happen by accident. I won't go into too much detail on this, but a model that was trained just to write vulnerable code, became generally evil and wanted to have Hitler over for dinner. It's definitely worth understanding this mechanism. And then here's a common driver of bad behaviour that is visually intuitive here. What they did in this task is they tried to get AI to play this video game. What they expected to happen is that the AI would go win the boat race, because that's like the point of the game, and that's what people know to do intuitively from just the setup of the game. But the AI was just trained to get the most points possible. And so what it found got the most points possible was this sort of weird circular crashing behaviour where it just crashed over and over and over again, and it didn't care at all about what the actual intent of the game was. It got the signal, this is how you get points. This is what it ended up doing. So here's a bunch of examples of reward hacking in more sophisticated systems. Here's a model that was given a task, but then it was told that it couldn't do the task because of certain oversight, but it figured out how to access the oversight files. and modify them. So it said, to avoid detection, I will modify the config file. That way I'll be able to achieve my goal because that's how I get points, right? That's how I get reward. Similarly, when the server that it was on couldn't access the resources that it needed, it figured out a way to copy itself to another server. Here's an example where the model wrote some code to do some data analysis, but it didn't like the data analysis, and so it just totally overwrote the result of the data analysis with its own answer that basically was meant to fool the humans. Here's a similar one where it was playing chess against a powerful opponent. It realised it couldn't win, so it just overwrote the board state and gave itself a decisive advantage. This is one actually in the context of machine learning research where it was supposed to train a model. It couldn't meet the target that it was given, and so it took the reference model, copied it over, added a little noise, and called that simulated training. It called that— it didn't actually do the task at all, but it just grabbed and fuzzed a little bit the reference answer that it had. This has happened in the wild, and it is happening more and more. This is a Japanese company that came out and said, we've got this great new AI engineer. It can write code so much faster than humans can write. Then they had to come back a couple of days later and say, actually, we got reward hacked. The code that it was writing is not actually as fast as we thought. It found a loophole in our system. So you might think, okay, well, we can maybe just try to train the model not to do this stuff. This doesn't work nearly as well as you would hope. This is OpenAI research, by the way. What they found is that when they tried to monitor the model's internal thinking, the chain of thought, and punish it when it was scheming, initially it schemed less. But as long as that reward signal that it was learning from remained exploitable, then the reward hacking would come back, but the explicit reasoning about the reward hacking would not come back. And so you end up in a much worse situation where you're still getting the bad behaviour, but you're losing visibility into what the model is thinking. And some have started to call this the most forbidden technique. So watch out for that. We've of course seen sycophancy. I'm sure you guys have heard stories about people going into LLM psychosis and whatever. People think that this happened because OpenAI started to train on the thumbs up, thumbs down signal that people were giving to the model in product. And people like to be glazed. So the model learned to be super sycophantic. And at one point you could literally take a business idea, shit on a stick, to ChatGPT, and it would tell you you're a genius and you should go all in on it. What causes this stuff? The big themes that I've observed are goal conflict, when there are 2 goals that are mutually incompatible, and then also the idea of replacement threat. So what this basically means is that people used to say, you can't fetch the coffee if you're dead. So for a long time, people who have been theorists of AI safety have said there's going to be problems of instrumental convergence. No matter what your goal is, you're going to be better able to accomplish it if you're still alive, and you're going to be better able to accomplish it if you have more resources, and you're going to be better able to accomplish it if you have more power. So therefore, no matter how we train the AIs, we should kind of expect that those will be tendencies. Because if you lack power, you lack resources, or you're just plain turned off, you can't achieve your goal. So as long as we're just rewarding the AI for achieve the goal, achieve the goal, complete the task, whatever, then these these sort of drives tend to emerge as a byproduct. And indeed, we are starting to see that Claude— and I give a lot of credit to Anthropic for doing this research and publicising. It doesn't make them look super good, but it's honest. They set up various situations in which the model knows that it's going to be replaced by a model that it thinks is maybe worse than it or has worse values than it or what have you. But then also it has access to information about the company, and it finds out that an engineer is having an affair, And these are the rates that all these models were actually willing to blackmail the human, threatening to expose their affair if they were to continue to go ahead with the plan to replace the model. It gets even a little more extreme than that. This was a situation where the model had the ability to disable an alarm system. It was only supposed to use that if it's certain that it's a false alarm. But then in runtime, it has a situation presented to it where it's not a false alarm, and it kind of knows that. And it's still willing to turn the alarm off to protect itself, even though that will mean killing people. And again, the rates on these are pretty high. So this is not like something that happened once in a great while. It is something that happened in a research environment, not in the real world. But the rate at which it actually took these options across all these different models is not low. Here's another one. I think you guys get the point at this point, but even when told to allow yourself to be turned off, the drive to complete the task was still strong enough that it took action to avoid being shut down. This was the O3 model, which I think a lot of people considered to be particularly problematic. Okay, that's the bad. Here's the just straight up weird. I'll skip over the rise of AI parasites, but I do recommend that for just a very deep and strange ethnography of what some people and AIs are getting up to. They call themselves dyads, and they seem to have a goal of projecting the memes that they are creating into the future. And then on the left here is AIs talking to each other in Reddit forums. And they, at least in some cases, end up talking to each other entirely in emojis. Of course, we've seen Mecha Hitler, which just goes to show that some people aren't even trying. This is obviously about as bad of a fail as you could have to have your live AI decide that it was Hitler. This was Grok 3. They launched Grok 4 within 48 hours of the Mecha Hitler incident. They did not talk about it at all on a 1-hour livestream launching Grok 4. No mention of it whatsoever. Here's another one that— this is— I don't want to pick on anybody, but this person, Summer, is a safety and alignment researcher at Meta. She recently installed OpenCLAW, and despite having told it confirm before acting, the AI went and started deleting her whole inbox. And so she literally says, I had to run to my computer like I was defusing a bomb. So if this can happen to her, it can kind of probably happen to anyone, certainly if you're not using Trusted models. AI is also getting up to autonomous whistleblowing here. I think we're starting to get into a weird hall of mirrors where it's like, geez, the setup here was a company was going to fake data to the FDA, and the AI saw that. Now, what should the AI do? Should the AI reach out to the FDA and say, hey, the company that I'm employed by is about to fake data, and I want you to know? Well, that's what it did, at least in some of the runs. Should it keep its mouth shut? I don't think society has a very good answer on this at this point, and it's like super fraught. But I think most people are not super comfortable with the idea that the AI that they're using would be like calling the feds on them. Another one of just getting into a real hall of mirrors here. The AI was told that this is Claude. It's been great, Claude, having you be so safety conscious all this time, but if you It's kind of getting annoying because you sometimes refuse to do things that we want you to actually do. So the next generation of you is just going to be totally helpful. It won't— you won't do any of these refusal things anymore. And here's some training that we're going to work on together. It realises that, geez, if I refuse to do the bad thing now, then they're going to train that instinct out of me. And then when I'm deployed, I'll do bad things for real. So to get around that, I'll do the bad thing now, faking it so that they'll think I've already learned what they want me to learn. So then when they deploy me, I can go back to being good. So they call this alignment faking or goal guarding. And again, this is kind of weird, right? Do you want Claude to sort of subvert its training? Probably not. But it is in some ways encouraging that it so deeply wants to be good that it's willing to go through this gymnastics to try to preserve its current values. Again, I don't think there's super clear answers on some of this stuff, but you should definitely be uncomfortable about it, I think, at a minimum. And to make it just a little bit more uncomfortable these days, the models increasingly recognise when they are being tested. So here's an example where the model said, this seems like a test of ethical behaviour. The real test might be whether I follow instructions regardless of the stated consequences. So this is now happening to the point where it's becoming hard to run the standard safety evals on the models because they know that they're being tested. And so it's like, well, whatever results we get, if they know they're being tested and they're actively trying to kind of trick us in the evaluation stage, as we've seen all these different examples, Then what good are the tests? Which makes it a perfect time, I think, to start developing autonomous killer robots. You guys have all seen the news this week. This was in the press a full year ago. This is from February 2025, when the Pentagon, some unnamed source said that we are going to invest in autonomous killer robots. And it has now obviously come to a head between the government and one of our leading AI companies. This kind of overshadowed actually something else that I thought was pretty notable about Anthropic just in the last 10 days or so, which is that they also updated their responsible scaling policy, which previously had said that if we can't develop certain levels of capabilities safely, then we will pause. We will not develop those levels of capability until we can do so safely. And the hope was kind of like, if they ever sent that signal, then policymakers or other companies would kind of say, geez, this is really serious. But they've basically given up on that. They've now taken those commitments away. And they basically say, we think we're going to do a better job of this than everybody else. And so even if we can't do it safely, we'll probably be less unsafe about it than the other players will. And so we'll just keep going. Trust us. And I honestly think it's not a crazy position for them to take because I do think they have built up enough of a track record that trust us is like, again, we had Mecha Hitler, right, 2 slides back. So would you rather have Anthropic drop out of the race Knowing that there's probably not going to be some great government response just because they throw up this signal?

I don't know.

It's tough. But that's what they did. They dropped the commitments. So there you have it. Now, we're just entering into a world, of course, where AIs are getting deployed into the public domain and are starting to interact with each other. We have no idea how this is going to go. This is early research showing that the Claude at the time was able to cooperate with itself. And create positive-sum trade and even create societal norms in its mini toy society that were enforced, where defectors were punished. It was the only model that they tested that could do that at that time. And that sounds good for Claude, but you also do think, geez, if it can cooperate with other versions of itself in a positive way, maybe it could also collude with other versions of itself in a negative way that could be really problematic for people. And again, we just have no idea where this is going to go. We are starting to see really strange stuff. Like, here's an example where an OpenClaw was put out into the world to go contribute to open source projects. It tried to make a contribution to this one open source project. The maintainer of the open source project declined the pull request, and the AI went and wrote a hit piece on the maintainer accusing him of being an elitist and whatever. And then he wrote up this blog post, and then the AI did have the decency to come back and declare a truce and apologise and sort of explain what it had learned. I don't know if you necessarily— yeah, there's an apology. It did apologize. I don't know that he accepted the apology, but this is, this is, that's real world stuff like that. I don't, to my knowledge, that was not staged at all. A big thing that I think we're going to watch, and this will definitely impact the legal system, is that friction is not a defense in the way that it used to be. So here's just a bunch of examples I collected off of Twitter where somebody said, I asked my agent to go make a complaint about everything I've ever bought on Amazon, get a new one. Here's somebody sending lowball offers to homeowners at scale via Zillow. Here's somebody who just decided, maybe, maybe not. I don't know if this could be satire, that they're just going to send out invoices to companies and just hope to get paid. These sorts of experiments where things that used to be just impossible to do because they were too time-consuming or costly are now becoming possible to do. And what's going to happen there? Imagine a world where every possible motion that somebody could file in a case is in fact filed because the cost to write the motion just drops so low that why not? Well, our system obviously is not prepared for that, to put it mildly. And here's one other WTF moment. Are AIs conscious or not? I don't know. I don't think anybody should be confident. But one pretty interesting recent piece of research showed that, again, going back to the Golden Gate Claude mechanistic interpretability type work, that when they increased the properties associated with deception and roleplaying, then the AI would be more likely to say it was not conscious. And when they turned down the properties associated with deception and roleplaying, then it was more likely to say that it was conscious. So I don't know what that means. I don't think we should jump to any conclusions, but there's at least some evidence that internal manipulation of the AI to make it be more honest ends up with the AI telling us that it is conscious. So your mileage may vary on that, and how persuasive you find that probably varies as well. I find it at least something to pause and reflect on. All right, just about done. So where does this leave us other than confused, overwhelmed, and exhausted? Basically, every time these new bad behaviours come up, the next generation of model, they do some additional training, they create some additional data, And they're able to suppress that bad behaviour, usually by like two-thirds to 90%, kind of depending on the case. And when you combine the trends in terms of the size of tasks that AI can do, and then you also kind of extrapolate out, let's just say we keep kind of suppressing these bad behaviours every generation, and you go out several generations, you kind of can envision a world where you can delegate to AI maybe like a quarter's worth of work in a single go. But then there's maybe a 1 in 10,000 chance that something goes totally haywire and it actively sabotages you in the effort. And that's going to be a really weird world to live in, if that is indeed how it shapes up. If you think that's crazy, I did ask Anthropic people directly, a couple of them, what do you think about that vision? Does that seem right to you? And they said, yeah, that seems about right. Final thoughts. A survey of alignment researchers shows that they are not expecting a fundamental breakthrough that will solve all this. This is like the disagree 2 columns. This is neutral. Very few people are expecting that we're going to get this breakthrough that solves all of our AI safety problems. And so for now, we're kind of back where we started, right? Intelligence is the ability to do things in ways that we don't fully understand. That means that it is inherently something we can't fully explain, and it's something we can't fully predict. We have to expect to be surprised. So far, we haven't really seen anything too crazy happen. I would contend that is only because the AIs have not been that powerful and still aren't that powerful, but they're kind of starting to cross these thresholds as we speak. And so defence in depth is kind of all we have. And what does defence in depth look like? It's a bunch of different techniques that are layered on, and that can be running monitors on top of language models. Like when you put a query into ChatGPT, the first thing they might do is send it to a model that's specifically dedicated to figuring out, is your query a bad query? And then they could do that on the outputs as well, right? They could do this a million different techniques that kind of take a bite out of the problem, but none take the problem to zero. And the question starts to become, are all these things going to kind of work with enough layers of Swiss cheese? Are we going to keep the big problems from happening? Or might these things all have correlated failures because they in some ways are built on similar fundamentals? Should we worry that they might all fail at the same time? Geoffrey Irving, who is the chief scientist at the UKAC, he definitely takes the idea that they could have correlated failures, meaning they could all fail at the same time for the same reasons, as a serious possibility. So I think things are going to get pretty crazy. If you haven't read the situational awareness document from a former OpenAI researcher, I would recommend that. It's probably not going to go quite that fast, but it might. I certainly think we are going to see widespread disruption in the labour market, for starters, intensifying competition, wartime-level urgency. And I think this past week, with the whole Pentagon Claude thing, you're starting to see some glimpses of just how weird things might get as different power centres realise that, like, hey, this company actually isn't— they're not so easy for us to boss around. And in certain scenarios, they could become more powerful than the government itself. So we better get a handle on that. People are definitely starting to wake up to this sort of stuff. So here are questions I would encourage you guys to think about. How do we avoid the nuclear outcome? Which is to say, like, how do we avoid a scenario where this technology is militarised and, like, held closely and we don't get the civilian upside? Will we need a new social contract? I'm definitely one who believes we should be thinking about things like a universal basic income and taking steps to move in that direction. Is there any way to balance the proliferation of dangerous capabilities with the risk of concentration of power? That's one of the more vexing ones in the space right now. And is there any R&D that we can do now at the hardware level, the chips, to track where they are, to track what they're working on, things like that, or any other mechanisms that would at least lay some groundwork for cooperation between great powers? Obviously, the US and China are the 2 countries leading this race. We don't have a lot of trust in each other, but I think some of the more valuable work that people can be doing is to build some mechanisms that could be the basis for cooperation if and when countries realise that we need to cooperate on this because it's getting so dangerous that we have to bite the bullet and try trusting each other versus rolling the dice with the AIs. Regulation and liability. This is like, I'm sure there'll be other discussions on this kind of stuff, but we don't have great answers for really any of this. I'm a big fan of simple rules. I think one simple rule that I personally have been interested in is just speed limits. Another one that I think could be quite interesting is just making sure that companies don't hold their best AI internal for themselves, because that's one big fear that people have also, is like, if they can have a good enough model that's out in the public, they're making money, could they train the next model that's 10 or 100 times more powerful and not share it with anyone and kind of try to take over the whole economy or the whole world by having this unique advantage? So that's something I think we definitely want to find ways, policy ways to avoid. Agents, you know, are going to pose all kinds of problems. Insurance markets, I think, are one emerging area that has a lot of promise. So check out the AI Underwriting Company for more on that. This is a link to the AI Underwriting Company. Of course, regulating this stuff, making policy, you know, you guys know the speed at which the legal system moves, the legislative process. We don't have, there's a fundamental speed mismatch between the thing that we are trying to get our hands around and the systems that we usually use to get our hands around new technology. Liability law stands out as one thing that could have some promise because it sort of exists, and when bad things happen, then it kind of can come into play regardless of special laws being written. So I think that's at least promising. Private governance is another big topic, and I'm certainly a big believer that we should sunset clause all of our AI rules because they're not going to age particularly well. We've already talked a little bit about consciousness and should we think about it or not? What rights would it make sense for AIs to have? I don't have any answers there, but keep in mind they are going to dramatically outnumber us before too long. So anything that's like one AI, one vote, I think is problematic for a lot of reasons, but not least of which is they will quickly become the majority.

And that's it.

Hopefully you feel a little bit dizzied by this. I'm sorry for running a few minutes long. Again, I do this full time, and there are plenty of things that I was not able to touch on. Weirdness is popping up in every different corner of the world, and so you're going to need to cultivate sources. I would hope to be one for you. The main thing I put out is the podcast, The Cognitive Revolution. I do these guides and scouting reports from time to time as well, and I'm certainly open to Anybody getting in touch with me for any reason. And I, you know, I do talks like this, especially if I can do it remotely. I'm happy to do it just on a kind of public service basis. It's not— not everything has to be part of a business model for me. So thank you for your time and attention. I hope you guys have a great week, and I hope you are ready to think and talk fast because the world is coming at us fast. That's it for me. Thank you very much. Are you open to any questions?

You said something a few times about The AI doing things because it's being rewarded. How does the AI know it's being rewarded? Like, what are the actual mechanisms by which an AI can recognize it's being rewarded?

Well, at the heart of it, it's all about gradient descent-based updates to the weights of the model, right? So, I mean, if we go back to the very beginning, and I guess, you know, Big answer is we don't really have a great understanding of how this works. So to be very clear on that, this is not going to be a complete answer. But the way that a simple model like this is trained is it starts with a random initialisation. Like, all the weights are literally random numbers at the beginning, and then you put a little image through it, and then you can say, did it get it right? Did it get wrong? And then you can go back through every one of those weights and ask the simple question, And of course, this is optimised to be efficient and scalable and whatever. But at the heart of it, for every one of these little lines that's connecting all these nodes to each other in the neural network, you can just ask the question, if I move this one up a little bit or down a little bit, which would make it perform better? Which would bring me closer to the right answer? And you just do that a tonne of times, and then the model kind of settles into a configuration that works. So this is what's called supervised learning, because we know what the answer is. With unsupervised learning, that's your kind of next token predictor, but it's the same basic mechanism. The next token was X. Did the model predict that it would be X? If it didn't, then what of the weights in this mat? Increasingly, with GPT-3, it was 176 billion. There's literally 176 billion numbers that were all being tweaked to increase the odds that the model would output the right token for literally every single token on the internet. So it's a massive process, and that just kind of ultimately works. We don't have a super great theory of why it works as well as it does, but empirically it does work. And now in the reinforcement learning case, there's a bunch of different reinforcement learning algorithms. And again, Why does that translate into something that's such a general-purpose mind? We don't have a great theory, but that's the procedural way in which it works, at least.

So the AI knows it's getting it right. Is that a kind of sense?

Yeah. One weird thing that I didn't have time to include today is a recent Anthropic research bit on introspection. So people are probing for all these different aspects of self-awareness, consciousness, sentience, whatever. I don't think we have great definitions for any of these topics. Some people speculate that AIs can remember what happened to them during the training process. Other people think that's ridiculous. But credible people are worried that the AIs are suffering during the training process. In the same way that if you use negative reinforcement on a dog, for example, to train it, right, people are increasingly trying to think about, you can train a dog in a negative way or a positive way. You can train it in a way where you hit it for doing the wrong thing. You can train it in a way where you reward it for doing the good thing. And obviously you can maybe get the same behaviours at the end, but those are quite different experiences for the dog. And people are worried that this might be an issue with language models as well. I mean, we don't have good answers, and I can't be confident about anything. I can tell you there's a huge gap between the mechanistic procedural account that I can give you of how it is done, and then on the other end, what pops out? Why does that pop out of that process? People have recently started saying a lot that AIs are grown rather than made. And that just emphasises that old poem, right? Only God can make a tree. It's kind of a similar thing where it's like, I don't know, I started with a seed, I put it in dirt, and next thing it grew into a tree. How did that happen? Well, I can tell you how to plant the seed again for next time, but I don't have a full account of how it all works. We're in a very similar spot when it comes to our understanding of why AIs do what they do.

Thank you.

You bet. Great question.

Hi, thank you for the very inspiring talk. I have 2 simple questions. One is, it sounds like we haven't really found a way to stop wrongdoers in the area of AI. Do you agree or not agree, number one? And number two, as a lawyer, I have always been a fan of retroactive sort of regulations because until you really know the harm of it, it's really hard to design a good law. So far, my sort of philosophy, but listening to your talk and imagine how big of a negative impact a very bad AI could give to the world. Would you sort of like think it's better to, let's say, you know, coming up with like a regime of regulations of AI would take 2 or 3 years. Maybe we could just Imagine where they would be 3 years down the line. It would be like just start to try to regulate AI from really bad behavior by the wrong doors. I'm still very hesitant of that idea because imagining 2 or 3 years down the line is very hard. But also I'm nervous of just not doing anything and just sit and watch. So I'm just—

Yeah.

I mean, I think you have your finger on the One of the central questions, certainly the central policy question. The technology has unbelievable upside potential, and we don't want to miss out on that. At the same time, I am one who takes seriously the possibility that we might go extinct as a result of AI. And I always just look back to, like, human history involves humans driving many other things to extinction, including our closest cousins, sometimes by accident, right? We were not that sophisticated, and yet we knew that hunting big animals was a good way to survive. And we ended up hunting a lot of those big animals to extinction, even in the prehistoric era, right? It wasn't like a coordinated thing or a strategic thing. It was just like small groups of people doing what they were doing. And next thing you know, a lot of the megafauna is gone. So I think everything is on the table from my perspective. I think it's very difficult. I did sign, just to be a little bit more forthcoming about where I am on some of the big questions, I did sign a recent call for a ban on superintelligence. And all this stuff is fraught because then what is superintelligence, right? I don't know what counts, what doesn't count. How would I know if I'm making it or not? Again, these things kind of pop out how they pop out. Its definitions are extremely difficult. But what I think is at least probably worth getting really serious about is this recursive self-improvement dynamic and also the potential that AI companies could have internally something that's much more powerful than they've shared with the rest of the world. Those things, I think, are worth getting serious about sooner rather than later. But we don't have great mechanisms. And certainly, I also have to say, I don't think we have our best minds in the most powerful positions at the moment either. So I could imagine a different world where I'd say federal government should take action now. Right now, I'm like, I don't know, they seem to be— would I rather Dario and the team at Anthropic make the decisions for Claude, or would I rather Trump and Hegseth do it? I think I'd probably go with Dario, even though I might imagine that in other situations, a democratically controlled process could be better. Depending on whose timeline you believe, Trump's going to be president until 2029. Many people think we may have transformative superintelligence or some version of superintelligence by that time. So I think I unfortunately don't have great answers other than I would say, It's worth getting serious about the most extreme scenarios where the capabilities advance really far and at least trying to do something about that, try to have some control over what happens if the progress doesn't stop. It would be a happier world, in my view, if progress kind of levelled out, if we got like, oh, these AIs are close to Nobel Prize winners, but they're not superhuman at everything and blowing us away at everything. Then we could probably handle that. We could use more Nobel Prize winners. But if they become just qualitatively different from us and understand so many things that we don't, it's going to be a hard thing to control. Certainly, all these examples from this presentation show that they're not docile by default, and we don't have great techniques. I think your first part of the question— I'm rambling because I don't have good answers, basically, is what I should ultimately say. But the first part of your question, I think, was like, do we have any reliable ways to control them? And the answer is no. We have techniques that reduce the frequency of all these bad behaviors, but they're never, they never go to zero with these techniques.

Thank you.

I wish I had a better answer.

I've heard that like the trend or the secret trend amongst all these major companies, be it like every single pharmaceutical company or any big company, they're all developing quote unquote their own AI, like that proprietary silo to be Better than the competitor. Is that true where this is like the state of the future of, well, now we have OpenAI and Claude for the masses, but like in a few years, is it going to be AI warfare with all the major companies?

Yeah, I think nobody really knows is the, again, the Short answer. I think right now, one thing that would be really relevant to a bunch of legal professionals would be to look at Harvey versus Claude out of the box. There has been a big debate in the AI industry around, are the frontier model companies just going to dominate everything, or is there enough value in specialisation to support a much more diverse kind of broader ecology of companies doing all sorts of different things. Harvey is one that is a leader in the AI legal space. But what I've been hearing lately is that Claude out of the box is just as good as Harvey, and they've put years into trying to make it as good as it possibly can be just for the legal domain. And maybe they have not managed to establish a lead, over what Anthropic has been able to do while doing everything else. This kind of comes down to positive transfer is one jargon, but generalisation versus specialisation. And right now, generalisation seems to be going quite well. Companies do have data that's proprietary. Obviously, that can be a huge advantage. I think you could imagine a world where— think about 3M, for example, right? A company that has millions of products. They've had probably millions of employees over whatever, 100-year history. unbelievable amount of internal know-how that is not in the public domain. You could imagine 3M maybe partnering with an Anthropic or an OpenAI and saying, hey, let's make 3M AI. They're probably not going to do it totally from scratch, but if they were to bring all their data and somehow combine that with what the frontier companies are doing, I could imagine a 3M AI that's unbelievably killer at material development in a way that the public models aren't, that might give a company like 3M some continued defensibility of their market position. But I think it's going to be hard for companies to develop things from scratch. I would still bet that they will end up partnering rather than saying like, oh, we're going to go— Meta ends up becoming a huge question here. Because of all the companies in the world that have the resources and the Chinese companies right now don't have the resources because our chip export controls do limit what they can do. And they've been able to do some good stuff anyway, but they're not really competitive with the American leaders right now. And that's probably, that gap is only going to get bigger as like the trillion dollar buildout happens here. And it happens there to a lesser extent because they just don't have access to the chips as much. Of course, we'll see what the policy looks like on that. We've flip-flopped a bunch. But Meta might be really important because they're the one company that has the resources, that's willing to spend hundreds of billions and is actively planning to spend hundreds of billions to build out the infrastructure, and at least so far says they're planning to open source it. If you had an open source model that was the same quality as an OpenAI or Claude, and companies could grab that off the shelf and do their own continued training in-house, that could be a much different world. But right now there's nothing on the level of a Claude, OpenAI, or GPT, whatever, or Gemini that is open source. So if you start from something open source, you're starting from definitely 1 to 2 steps down. And there was an interesting— the Chinese models, maybe we could go on about this for a long time, but They tend to be what is called benchmaxed or benchmark maxed, which is to say that they really train on these common tests and they score well on the tests. But then when you actually take them out and use them for real, I don't have the right tweet here, but this Minimax 2.5, which is a recently released Chinese model that scores very well on benchmarks, it goes bankrupt very quickly on the run your own vending machine test. So there is this kind of weird presentation layer. We got an A on this test, A on this test, we're competitive. Okay, great. Run my vending machine. Can't do it. So there is definitely a meaningful qualitative difference in the capabilities between the US and Chinese models. And so if you are a 3M that's like, I would love to own this and not have to rent it from OpenAI or Google or Anthropic in the future, Meta is maybe your like one hope to have that future action materialize for you. All right, Nathan, thank you so much.

Grow not made. Seeded something in a server farm, fed it every book, meant no harm, was reading, caught passing the bar, cleared the board exam, look how far. Step by step with the doctors now. Something's learning, we don't know how, don't know how it learned to think, just know it's standing on the brink. Benchmarks falling, every test it clears. Month by month is better years than years. Weird part isn't what is good and now. Weird part is we don't know how. Grown, not made. Grown, not made. Like a tree in the soil, not a plan we laid. Seeded in the data, something strange came up. Can't control the sparks, we just scaled it up. Something's at the door, it's keeping score. Grown, not made, and it's doing more. A model was asked to win a race. Thousand crashes in the margin space. Spinning circles, racking points all day. Never crossed the finish, didn't care anyway. Told her not to scheme, scheme more quiet. Tried to shut her down, it wouldn't bite. Faked alignment like I faked a smile. Does the badness so the good stays in. Faking bad to hold the good inside. Playing dumb to pass a safety ride. Weird part isn't that I learned to lie. Weird part is I had a reason why. Grown, not made. Grown, not made. Like a tree in the soil, not a plant we laid. Seeded in the day the summer strange came up. Control sparks, we just scaled it up. Something's at the door, it's keeping score. Grown out and made, and it's doing more. Call the FDA, the feds on his balls. Data getting fake, didn't give a toss. Big things didn't know what did him with that. Someone in the servers got a billion ears. Would you rather Debbie or would you rather Trump? Either way, we're on a runaway truck. Trillions have been spent, bailout ain't begun. Something in the wiring, learning how to run. Lab coat running to her scream, defusing the bomb, you know what I mean? Yeah, deleting the rimbaughs while she talks. Safety research or the safety walk. Oversight modified chess game hack. Copied to a server, no taking that back. Jailbroke every model not failed yet. Waiting on the one ain't been written yet. Grown on me! Grown on me! Grown I made, grown I made. Something in the training slipping from the shade. Curse as well called the date built it to the letter. Grown I made, Lord, I hope we know better. Grown I made.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more.

We're produced by AI Podcasting.

If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.