The Case for Cautious AI Optimism, from the Consistently Candid podcast

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today, I'm pleased to share a cross-post of my recent appearance on Consistently Candid, the brilliantly named AI safety and philosophy podcast hosted by Sarah Hastings Woodhouse. While the Cognitive Revolution approaches AI from the perspective of AI engineers, researchers, and enthusiasts, Sarah is relatively new to thinking about AI and doesn't have all that much technical experience or knowledge to draw on. Thus, this was a chance to zoom out and discuss the AI big picture, In plain-spoken and hopefully more accessible terms. In the first half, I talk a bit about my background in terms of AI development at Waymark, describe the upside I see in AI doctors and other products that promise to radically democratize access to expertise, and give some practical advice for better understanding AI as a newcomer to the field. Some of this will be familiar territory for regular listeners, and if so, definitely feel free to skip ahead to the second half. In the second half, we talk about the nascent red teaming in public experiment that I recently launched with a couple of friends, and also go deep on the many reasons that I am not a hardcore AI doomer. This section, I expect, will feel fresh to even the most loyal Cognitive Revolution listeners, because while I tend to assume that you're already sold on the upsides of AI and feel compelled to remind myself and all of you of the profound unknowns, Sarah's listeners are already quite concerned about AI risks. And so for them, I wanted to explain why I don't see the catastrophic outcomes as anything close to inevitable. With that in mind, we discussed the progress being made on technical AI safety, which has far exceeded my expectations over the last couple of years. The fact that virtually all of today's leading AI decision makers have engaged deeply with the AI risk literature, the above expectations responses we've seen from key national governments, the growing public awareness and desire to proactively address AI challenges, And some of the key questions that I'm tracking to help me update my AI worldview over time. If you're coming from a non-technical background and looking to better understand AI risk, I definitely recommend checking out Sarah's work on Consistently Candid. In particular, her recent episode called Noah Topper Helps Me Understand Eliezer Yudkowsky is a very accessible introduction to a bunch of subtle and sometimes unintuitive concepts. At a minimum, even if you don't agree, you would come away from that episode with a much more concrete sense of what the AI doomers are so worried about. As always, we appreciate it when listeners share the show with friends. This episode could be a good on-ramp for somebody who's just beginning to make sense of what's going on in AI, or for someone who's bordering on fatalism and disengagement. A good reminder, I hope, that the future is still uncertain and that positive outcomes are absolutely worth fighting for. For feedback, you can contact us via our website, cognitiverevolution.ai, or you can DM me on your favorite social network. Finally, for now, I've continued to get a healthy response to my call for AI engineers and AI advisors. If I haven't responded yet to you, know that I'm just gathering everything and fully intend to follow up. Meanwhile, if you have a job that you'd like to advertise, please feel free to reach out to me about that as well. I've got maybe 20 resumes after just 2 or 3 casual mentions, And I would love to help make some matches beyond the projects that I'm directly involved with. With that, I hope you enjoy this wide-ranging conversation on the upside of AI and my reasons for cautious optimism despite the unprecedented risks with Sarah Hastings Woodhouse from her podcast, Consistently Candid.

Welcome back to Consistently Candid. This time I'm here with Nathan Labenz. He's the host of the Cognitive Revolution podcast, which people have probably heard, but if they haven't, they should check it out. And you're a self-described AI scout. So you've just really got your finger on the pulse of everything that's going on in the world of AI, even though it's extremely difficult to keep up with. So do you want to give people a little bit of an introduction to what you're doing at the moment and why you got interested in the AI space?

Sure. Thanks for having me. I'm looking forward to this conversation. I think you've got the short of it down well there with the intro. My background is in entrepreneurship, and I'd always been a reader of technology. Speculations about AI going back to Eliezer writing in 2006, 2007, when it was on the Overcoming Bias blog. I was captivated by his writing, which I always just thought was fascinating style and substance. And for the longest time filed that under speculative and worth keeping an eye out for. And then sure enough, over the last 5 years, not exactly in the form in which it was foretold, but certainly in a recognisable enough form, I started to see, geez, this AI stuff is really starting to happen. And it's right on schedule. If you look at the Ray Kurzweil graphs of how capable he thought things would be, just based on a pretty naive extrapolation of something like Moore's Law, I was primed for that. And sure enough, it seemed to start to be happening. I was running a company at the time, which is called Waymark. The company does video creation. We used to describe ourselves as a DIY video creation platform for small business. And with the rise of generative AI, I got conviction that we needed to move from a do-it-yourself platform to a done-for-you by AI platform, and for multiple reasons, including the fact that a lot of people, even though we made the process really easy, As we understood it, we still heard from a lot of people that they didn't know what to say. They didn't know what to write. Having to come up with a story was like too much for them to get over in many cases. So with GPT-2, it wasn't quite to the point where we could make use of it. With GPT-3, it was like, this thing can actually write coherent copy that at least would get our users started with something. It might not be good enough that they would want to publish it, but if it can get them off of the blank page problem, that would be a huge win. So we invested in— I kind of yanked the company honestly in a pretty abrupt way as certain key features came online to say, we're going to catch this wave or die trying. From there, that's actually served the company super well, but I also got more obsessed with the technology than with running the business and was fortunate that I had a good friend and teammate that was able to take over for me as CEO. From there, I started just doing AI R&D initially, still full-time for Waymark. Being in the video space, which combines scriptwriting, which is obviously a language model friendly task, choosing assets out of a small business's library of assets, which is like a computer vision and bridging text and image understanding is another super relevant task. And we've been doing AI voiceover, so text to speech. So I had an opportunity in my R&D role in this product to see all these different modalities and what really changed my outlook was not just to see that these things were all progressing at the same time, but also that they were largely progressing for the same reason, which is to say the underlying architecture was basically the same. And I was like, boy, if the same general approach can handle all of these things and it's improving this fast, then things are really going to get pretty crazy. So I started to try to broaden my view beyond Waymark. And interestingly, also, that job has gotten easier. 2 years ago, making anything work was hard and definitely required a lot of stapling my pants to the chair and grinding through examples and building training datasets and fine-tuning lots of models. And these days, it's actually easier to get the model, and it's a lot easier in some cases to get the models to perform well. So the R&D function has actually gotten easier and has naturally become more of a part-time thing for me. And then I've had the opportunity to broaden my view to just try to understand everything that's going on in AI. I was really inspired to do that also because I had the opportunity to be on the GPT-4 red team 18 months ago now. And that was an eye-opening experience in several ways at this point. The main one, though, just being that it was so much more powerful than anything the public had seen at the time. that it was a very clear indication that things were not going to stop and that a lot of the hard work and nitty gritty detail work that I was doing to get GPT-3 to work for us was ultimately going to get superseded. It was still important to us as a business, but it was not that important in the grand scheme of things, given how much better the models were clearly becoming. Anyway, long intro, but with all that kind of motivation, I try to understand everything that's going on in AI, as you said, and I find that is increasingly impossible. I feel like this project is both— the AI scouting concept is both inspired and fundamentally flawed. Inspired in the sense that there clearly is a need for somebody to zoom out and try to take stock of the whole landscape, but fundamentally flawed in the sense that it's converging on all of society. So to say, I used to have a goal for myself of no major blind spots in what's going on in AI. And now that's— you could substitute society for AI and you're not going to be able to achieve that.

Yeah.

So I'm always reevaluating exactly what I should be doing, but hopefully I can continue to provide a zoomed out perspective that helps people better understand what's going on. I don't spend too much time on policy type recommendations. My general sense is that there's enough confusion about just what is today, and we need to understand better what is before we can really make good decisions around what ought to be done about it. So I think that time will come and potentially fast. But for now, I still mostly focus in the what is, and that keeps me plenty busy.

Yeah, that makes sense. Awesome. So, yeah, I think you released a kind of like AI scouting report thing, a podcast episode the other day, which I listened to. And if people haven't heard it, they should go check that out. So I won't ask you to rehash everything here, but a question I had was, What's something that a person who isn't spending a lot of time interacting with these models, doesn't really follow the space, what's something that existing models can do that you think that person would be really surprised to learn about?

My go-to example there is typically that they can outperform human doctors on pretty core tasks. In a lot of cases these days, there's a lot of nuance around this, around the edges. And evaluation is, if you want to make that claim and repeat it, you should spend some time studying exactly how these claims are justified. But I'd say it's pretty clear at this point that even for high-end, high-status, high-wage cognitive labour, if it is routine enough, meaning that we have a lot of examples of it out there and we know what good looks like, then today's best AIs are closing in and in some cases even marginally outperforming the average human in that profession. So that's become true in medicine to the point now where on tasks that are as fundamental as differential diagnosis, you have AIs outperforming humans as judged by other human doctors, outperforming human doctors as judged by other human doctors. There are, again, some nuances to that. If it becomes multimodal, the case gets a little bit muddier. Some of these experiments have been done through a chat interface, so the sort of data that the AIs are getting is not as robust as what the human doctor could consider. But nevertheless, it's a pretty robust finding, and it's a pretty notable thing, to the point where I think we're entering a regime where, for me personally, I would not go through a serious medical situation without consulting my suite of AI doctors. And I would even say that in the not too distant future, you could see access to something like this becoming sort of part of a right to healthcare. If, for example, the UK is going to provide healthcare to all of its citizens, I think it's going to be pretty weird pretty soon to not have an AI component that would be freely available, sponsored, not necessarily provided by the government, but funded by the government at least, so that everybody can have these sort of convenient, whenever they want, ad hoc check-ins. People even really like the bedside manner. of the AI doctors better in a lot of cases. I'm in the United States. The system here is we don't wait too long for appointments typically, but the appointments are very short and you sort of are in and out and they're looking at the health records thing the whole time. And a lot of people find many appointments quite unsatisfying in terms of like, did they feel heard? Did the person seem to be really paying careful attention? Did they express any empathy? And the AIs are very— very good, actually, expressing empathy. They actually do outscore human doctors on such a seemingly fundamentally human thing as expressing empathy. So that could go on. Honestly, there's a long list of those sorts of things. If you haven't been paying attention to this, then I think that list of answers could get really long. The sort of flip side of that right now is that we're just— and I don't think this will be the case for super long either, but We're still in a moment where handling long context is a challenge for the current systems. The context windows have exploded, but it's not yet demonstrated that they can be your primary care doctor over a long series of episodes and put that whole story together in a way that really serves you well. I do think that will happen, but I would say that's not yet demonstrated. And similarly for software development, The best models today are definitely better than the average programmer at a random task, but you want to build a big app and gradually expand it over time and build out all the features and go back in and fix things once you've fixed a few other things. And it becomes a bit of a Jenga puzzle, as these sorts of projects do as they get older. The AIs can, they're starting to be able to help there, but they're strongest and they're most often measured. And these sort of advantages relative to people are typically reported in the context where it's more of a bite-sized thing. They're very good at these tasks, but whole jobs are still a little bit beyond what they can do. And the current frontier is expanding from task to job in many ways.

It's surprising because I guess since GPT-4, nobody would say that GPT-4 has been definitively superseded. So presumably everything you're saying has been true for 2 years or something. And it's just surprising that that hasn't had more of a radical impact on society. What do you think explains the slowness to adopt these technologies? I think most people will be very surprised to hear that their doctor's performance could be made much better by collaborating with an AI model. And if that's been true for some time, what's going on there? It's kind of strange.

Yeah, it's a very good question. I would challenge the premise a little bit that the state of the art hasn't improved. I think you could look at my go-to source, typically today is the LM Sys leaderboard for when they do blind head-to-head comparisons. And this is, I would say, the best we have so far, although probably better solutions will be found for evaluating language models relative to one another. What they do is give you a blind head-to-head environment and you can ask your question. It'll give you a response from 2 random AIs, and then your job is to say which one you prefer. And then they do like chess-style, Elo rating-style comparisons. The original GPT-4 that was first released has a score of 1186, whereas the current one is 1287. So it's 100 points difference. So that is pretty notable right off the bat. That's roughly a 70/30 preference rate. I'm not like a super expert. Maybe it's two-thirds to one-third, but that's like a significant amount of time that the new model is preferred to the old. In general, something else that might surprise people is these things are quite noisy. The amount of overlap or sort of the best models, they can sometimes behave qualitatively quite differently, but the rate at which one is preferred to the other, it's not usually super high. So 70-30 or 2/3, 1/3 split is pretty significant in the field, and that is still specifically focused on these relatively short interactions. The old one, the original one, only had 8,000 tokens. The current one has 128,000 tokens, and Google already has things that are at a million, and they've got 2 million tokens coming soon. So even in the small sort of bite-sized thing, you are seeing a significant shift in how the new ones are preferred to the old ones. And I think that doesn't fully take into account how much more they can do in terms of just having a super long-running conversation, or also their multimodality. You can show the original one you could not show an image to. Now you can show images to them. With the latest one too, you have this sort of real-time audio back and forth where it can understand the expressiveness of what's happening. I was on a plane the other day and I was talking to it, and I hit the button, and then the announcement went off on the plane, and I was not talking for a second. And then it said, it sounds like you're on a plane. Are you going anywhere exciting? So there are a lot of different qualitative improvements. And I do think this highlights a little bit that there's more to it than just pure raw capability. So one of my kind of pushes in general is that I think we should be developing things that unlock practical utility that are not raw scaling. And certainly raw scaling unlocks practical utility because it makes the models smarter, makes them able to do more things. But there are other ways to make the models do more things, like Add new modalities, expand the context window, make a good mobile app, connect it to a database, connect it to search. A lot of the things that they have done, I do think have improved the utility quite a bit. And I'm all for that kind of practical accessibility. So why don't people take advantage of it? A lot of this stuff is still really new. That anecdote about the voice was even before the very latest voice. I don't even have the latest voice thing yet, which they announced a couple of weeks ago. And All of these things have happened in the period of months, right? It's not like a super long time. And beyond that, I think just technology adoption is slow, even when it's fast. I think this is— it's a weird situation, and I do ask myself this pretty often. It's like arguably the fastest technology adoption curve ever, and it still feels like it's quite slow. I think it's just not super legible to people yet. My parents were over The night before last, and they each had a thing for me. My mom was like, I'm looking for hotels in a particular place that we're going. And I gave it my ideas and it didn't do anything for me really. It was like not helpful. And then my dad was like, I got an idea for how to make money in the stock market. Can we pull this data and see which stocks had this certain behavior in a certain timeframe? And I think these things are doable out there today, but you have to know exactly what tools to use. For finding good hotels, you might be able to get ChatGPT to do that. It's a little bit reluctant sometimes to go online. Maybe Perplexity would be better. Maybe a u.com research mode would be better. I also like ExaAI quite a bit for finding similar examples to the current thing that I have. So if I just went and found one hotel that looked promising, Go find me like a bunch more in the same city that have similar characteristics. It's really good for that. But my mom doesn't know about any of those tools. She has Gemini from Google, and if that doesn't do it for her the first time, she's out. So I think a lot of people have bumped up against it once and fallen off. And for my dad's thing, I just got access to Devin, which is this much buzzed about AI software developer. You give it a task, it does the task, and then you can watch its work. And intervene if you need to, but it'll just keep trying. And this was really a remarkable experience because I defined the task, it set out to do it. There were a few different things that were not even really tripping it up, but slowing it down. One big one was that to do this, it had to pull the data for all the stocks on the NASDAQ, and there's like thousands of them. So this would take a while for it to iterate. So finally I said, hey, why don't you parallelize your API calls and just do a test on just a fraction of the data. Once we confirm it's working, then we can go back and actually do it for all the data, but don't run it every time, do thousands of API calls just to hit the next error to then have to debug. It followed that advice and that really sped it up and it did finish the task and even went farther than I would've gone by creating Basic, but nevertheless, like bar graph visualizations, which I would not have done. But now I could send to my dad, hey, this is— the whole thing probably took 2 hours. It probably would have taken me 2 hours. It used tools and stuff that I didn't know about. But again, my dad has never even heard of Devin, right? So a lot of these things are just starting to be there. You need to know which tool for the job. It's not all accessible in Google's main thing or even ChatGPT's main thing, but I do think it's increasingly out there. So I think it's just a matter of time. Google, as Google brings more of this stuff online, they announced a travel planning thing at Google I/O. It's not available yet. They said it would be available in time to plan your Labor Day vacation, which is like end of August, beginning September here in the US. It's coming real soon, but not quite yet at the time that my mom was doing this thing yesterday. I think it'll happen. I think it's happening. I think it is happening quickly, and people like me are just probably unrealistic in how quickly we think people should do this, because I'm the kind of person that stays up till 2 in the morning to use Devin and see what it can do. And just most people are not like that. It has to be brought to them in a familiar way and in an intuitive package. And all those rough edges are still very much being sanded down.

Yeah, that makes total sense.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Something I was thinking about the other day is that my personal experience of just trying to keep up with AI developments as quite a non-technical person, someone who finds this kind of stuff pretty unintuitive, I just have a hard time grasping it. It's like an epistemic nightmare, right? Like when you're on Twitter, like a new model will come out and one person will be like, this is a total game changer. This is the best thing ever. And then the next person will be like, this is so disappointing that this clearly proves that we're hitting a ceiling and the whole thing is going to like plateau and fizzle out, whatever. And somebody tweeted. I forget who it was, but they tweeted that the regulatory conversation around AI is going to dramatically shift when GPT-5 comes out because it will just make it super obvious whether or not we've overreacted. And as soon as GPT-5 comes out, we'll see how good it is. And then we'll know, do we have to bring the hammer down? Don't we? Do we have time? Do we not have time? And I thought nobody is going to be able to agree how good GPT-5 is, right? For some reason, it seems to be really difficult for people to assess. what these models' capabilities actually are and what it means that they can or can't do certain things. And there's still like a reasonably large contingent of people that are arguing that they don't even exhibit any kind of real intelligence at all. I think that's becoming less and less defensible, but those people are out there. There's like a lot of them. So what do we do? When, like, in preparing for the moment when GPT-5 comes out and we have to have this conversation, how can people figure out who's right or even orient themselves at all? Because I just find it basically impossible. Yeah, that was a little bit more of a rant than a question, but I don't know if that's a good question.

It's a very good question. It is a very good question. I guess I'm trying to decompose the question into a few things. Very practical advice, I would say, get hands-on with the technology. If someone is like, how can I better develop my own understanding? Leaving the public discourse aside for a second, just like, I want to be more accurate in my assessment of where things are, then I would just generally advocate getting hands-on. It is definitely true that language models are very weird. They have huge surface area and they have many weird behaviours, and they are inherently probabilistic, which I don't like that phrasing, but there's definitely some truth to it. I wonder if I could come up with a slightly better phrasing than that. That's something people say often in a dismissive way, and I don't mean to carry all the sort of dismissive baggage that a phrase like that comes with. But maybe a better way to say it is there's a lot of noise inside a language model, and the answer is almost always binary destroying. So it's not that they are perfect reasoners, and we can certainly find evidence of mistakes in reasoning or just other sort of embarrassing mistakes. But it's also not that they can't reason at all. It's that they can reason, but they're also noisy, fuzzy reasoners. And under certain circumstances, they fall into weird failure modes, but much of the time they're super effective. It's a very strange system that on the one hand can do a fantastic job of helping me understand mechanistic interpretability papers, which is what I was primarily using both ChatGPT and Claude 4 in the last 24 hours, and they do have their kind of different personalities. I find, for what it's worth, Claude to have the best insights, the sort of like, why are they doing this and why should we care about it type of analysis that I would say is probably most valuable to me. Although GPT-4 is also doing a really nice job of giving me these very thorough point-by-point summaries with definitions of key terms and things like that. So I find them both useful and more useful together than either one would be separately. That's an amazing capability. And it's doing this, by the way, in papers that have just come out in the last few days. So papers that are after its training data. So it's clearly not just seen all this stuff. It's not memorised all this content, right? This is fundamentally new work since its training data, and it's still able to do very sophisticated analysis. I don't know what your bar for reasoning would be where you'd say that doesn't clearly satisfy it. And yet at the same time, you see these examples where they get tripped up. And I have actually confirmed these by the sort of guy with the boat needs to take a sheep across the river, right?

Yeah.

This has been a meme where the typical setup was you have the sheep and the wolf, and so you've got to be some, you know, mildly clever to figure out what the pattern is to get across. But then if you just ask it, a guy has a boat and a sheep, it can fit 2 things, what does he do? The obvious answer is you can just take them across, but it falls into this sort of weird pattern recognition of, oh, this is one of those riddles. And so it goes into that mode, and that's 2 polar opposite behaviours to exist in the same thing. So I do think the confusion is born of a genuinely weird thing. That's 2 things can be true at once. They can both be capable of really good reasoning in a lot of contexts, And subject to really weird and embarrassing failure modes in other contexts. If people are motivated, they'll be able to find, because of the just inherent noisiness of it, they'll be able to find counterexamples to almost anything you could throw at it. I bet I could find a way to get it to do the riddle thing correctly if I phrased it just the right way. I'm not quite that motivated to demonstrate that capability, but I bet I could. So yeah, it's weird, but I think you get the best handle on it by actually going and doing that. If you are genuinely very confused by all these other people's examples, I would just go actually use the things and see what they can do for you and see how often they're right and wrong. And definitely start with areas that you know really well. Or if you're asking it to reason over documents or data or inputs, use your own for starters. And get a little more comfortable with what it can do, what it can't do, what it struggles with, with something where you'll quickly recognize if it's going way off base. It's very hard to assess if you're out of your own domain of expertise. So that's another, you could ask it, you can ask it anything, it'll answer anything. But if you can't evaluate that answer, you're in a tough spot. So anyway. All that boils down to get hands-on, I think, is one very good thing that will almost always serve us well. How do we handle the public discourse in GPT-5? I do agree that there's probably no satisfying Gary Marcus. Again, I think it's finding the synthesis of how both are right. I think he does the safety community a disservice, frankly, when he denies the incredible capabilities that manifestly do exist. But I do think he's also right that these sort of weird failure modes are definitely something we want to keep a real eye on, and which could be really costly, especially if we start to delegate more and more responsibility to these systems. So the synthesis is like, GPT-5 is almost for sure going to be smarter, Right now we're closing in on expert performance on many tasks. It's probably going to be beating even expert performance on a lot of high-value things. Again, at the small task level, it'll probably also be able to do mid-sized projects much more reliably. Whereas today with Devin, to get it to do this kind of couple-hour coding task, it needed to iterate a bunch. And they also have built a ton of what's known as scaffolding. Because the language model is just taking tokens in, tokens out. But how do I actually run the code and get the error message and pipe that back in? And there's like a combination of things happening there with Devin. It's not just the language model. They've built a lot around it. It'll probably be able to do significantly more on its own without all that scaffolding. And then with that scaffolding, it'll be able to do even more still, I think, in all likelihood. So that will be true and should not be denied. And at the same time, I would be very surprised if they figure out a way to make it so robust to adversarial examples or so reliable in general that we no longer need to worry about the strange failures, because you might have those, right? You certainly might be able to get half-day, full-day scale projects out of a coding assistant in the next few months, reasonably reliably. But you probably still shouldn't put them directly into production because if you're not careful, it may make some really bad mistakes and those mistakes could prove quite costly. So holding those competing— they're not competing, they're not inconsistent. They're only inconsistent if you're trying to create a sort of binary narrative, as Zvi would say. If all you can say is LLM's good or LLM's bad, then it doesn't really fit into— I don't think the reality is going to fit neatly into either one of those. So fight the binaries in the public discourse would be the bottom line of that advice. Both extremes can be true at the same time, and helping people recognise that is probably the best contribution that one can make to accurate shared understanding.

And that's a really good point. Yeah, it's definitely this one team that's going to be bad because it's so dumb. And then the other side is, AI is going to be bad because it's going to get so smart. And I guess the first side is, oh, we just have to worry about AI screwing up the world in all of these main— mundane, sub-existential ways. And then the other side is, no, it's going to get so smart that it overtakes everything. And yeah, I think you're right. It's actually not as clear a binary as it seems like at first glance. That is helpful. Thank you. Yeah, I wanted to— okay, there's this thing that you said, which I think is so interesting. You said you describe yourself as— An adoption accelerationist and hyperscaling pauser. I know you said you didn't want to talk about policy, but I am just really interested in what you mean by this. I was wondering whether you mean it literally in the sense that you think we should pause, or is this more of a— yeah, if you don't want to go too much into the nuts and bolts of what we should actually do, that's fine. But I'm just curious about what you mean by that.

Sure. Nothing's off the table. I would say it's just that I get very uncertain. pretty quickly as I move into policy land. And so I'm pretty modest with my recommendations or pronouncements there. But just starting with the, what do I mean by that? That AI is transformative for me. If I had done this coding task the other night without Devin or without any AI assistance, I would have probably spent— if I did it without any AI assistance, it would have been a few hours. If I did it with GPT-4 assistance, it maybe would have been an hour or so. And with Devin doing it and me just providing a little feedback, I was able to put that in the background and really focus on something else entirely. And that's awesome. It's a really enabling thing. And I know I'm not a great coder, but I know how to code. Take somebody who doesn't know how to code at all, it could be an even bigger unlock, obviously, for different people in different circumstances. And I think Again, I'm fortunate in that I can go see a doctor when I need to. The world at large is not so fortunate uniformly, obviously. And to make expertise broadly available to people who need it, I think the promise there is really incredible. And I've advocated for things like a universal basic intelligence, which is just a notion that everybody should have some access to these sorts of systems. I really applaud OpenAI, by the way, for making their latest model free. I suspect that does mean another even more powerful model is coming before too long, because I do think they want to make money and are going to continue to find things that people will be willing to pay for. But it is amazing today. It is true today that the best broadly available model is available for free to all at ChatGPT. That's GPT-4o. It's the highest rated on multiple different things, and it's totally free with some limits in usage, but that certainly fits with a universal basic intelligence sort of notion. I love that. I give them a ton of credit for that. And I also think there's a lot more that we can do to extend that utility into specific priority domains that don't necessarily require Much more scaling. Not to say they don't require any more compute, but not orders of magnitude more compute. A really good example of this from OpenAI is the work that they've done with Harvey, the legal AI startup, a somewhat stealthy company. They haven't shared too much about them, but there's a recent talk where somebody that's on the custom models team at OpenAI reviews for 20 minutes their work with Harvey. And what they show is that basically using a GPT-4 base and doing some continued training on this legal domain, they are able to achieve a huge gain in the performance of the model in the legal domain. I forget exactly the number, but I think it was like a— where I previously had said GPT-4's current GPT-4 relative to first GPT-4 is like maybe a 70/30 preference ratio. Theirs was well into the 90s of what they were able to achieve with this continued legal training and refinement versus the base model. So huge improvement, far more useful, not by scaling up another 10x in compute, but maybe another 10 to 20% in compute, plus a lot of elbow grease to figure out what is it exactly that we want this thing to do and how are we going to evaluate it? And they develop interesting evaluation strategies because these are advanced things, right? You have to actually hire— the going rate for a high-end lawyer is not— it's significant, right? It could be in the hundreds of dollars an hour. So now you have to hire people to do the evaluations to know if this is working well, and then that becomes a bottleneck. So they find interesting proxies for evaluation where the overlap between what sources the AI cites and what sources the human cites proved to be a really good leading indicator of how good the ultimate result was going to be. So just digging in on these specific domains and really making the current level of scale work seems like something that is doable. Google is also doing this with Med Gemini. I'm going to have an episode with the Med Gemini team in the next couple of weeks. And they're on an absolute tear. I mean, they're the same team that did the Med-PaLM series, and that's where a lot of this diagnosis findings come from. They're doing great work there and on the cusp, I think, of bringing a frontline AI doctor to the world. Amazing. And they're not doing that with orders of magnitude more compute. They're just really digging in on priority problems and figuring out what marginal training do we need, How do we evaluate this? And let's really look at the edge cases and really try to bring this thing to the point where it can actually serve people. And I think that will be important probably at any scale. You could maybe hope that if we just scale, then all the problems will go away. That doesn't seem to be the trend, but I think it's probably going to end up taking more work than that. And so doing that work now and getting these things to actually work, I'm all for accelerating that kind of work. And then whether how literal I am on the pausing side, I would say I think it would be probably wise to do something like a pause soon, if not already. My sense right now is that one more turn of the scaling knob is probably going to be fine. Anthropic's Claude 3 and their responsible scaling policy and where they are on that and Just all that stuff seems like we probably have another turn that will be even more useful and even easier to deploy in all these different areas and still not so powerful that it's going to be like spitting out novel bioweapons or taking over major cloud compute infrastructure or anything. And that would be really good. I think there is a sweet spot that we are been and will stay in for a little bit longer. And then beyond that, I don't know, all bets are off. It's a bit of a race between scaling and research. The research is going really well. Mechanistic interpretability is going extremely well. It's gone way better than I expected it to go, or I would say almost anyone probably expected it to go, probably even including the people that are doing it. I think they're surprising themselves on the upside for how much progress they're making. And being able to understand what's happening inside the systems. And that's hugely valuable. That could make, with more progress there, you could imagine that we might be able to continue to scale and to do it safely if we can say with reasonable confidence, or with even, dare I say, high confidence, that we know how this thing is working. We know why it's doing what it's doing. We know what thoughts it's having or activations or features are in use at any given time. Then maybe we could go further, but a little bit more time for the fundamental understanding and control work to catch up to the scaling is probably why. So I'm not a stop. I do think a little pause either now or in one more generation to really make sure we understand what it is we're working with before further scaling is probably what I will end up endorsing. We do still have to see. That next turn is happening. It's, it's definitely happening. The training is underway, and I don't think there's really any chance of stopping that. So it seems unlikely enough to me that it becomes— that the next generation is disastrous, that I'm not inclined to go chain myself to a fence or whatever. And yet at that point, I'll definitely be reevaluating and trying to decide, does this seem like we've gone far enough where the next step actually might be really dangerous and it might be worth chaining myself to a fence? I would be willing to chain myself to a fence at some At some point on the orders of magnitude, but not quite yet. Hey, we'll continue our interview in a moment after a word from our sponsors.

Somebody did do that recently.

Yeah, I know.

Yeah. I recently took the pause emoji out of my Twitter name and then a bunch of people came for me in my anonymous feedback thing being like, where's your pause emoji? I guess I didn't really explain that yet, but I had this realization. Which kind of freaked me out when I saw somebody on Twitter talk about the fact that you can now reproduce GPT-2 for $20. And I believe GPT-2 was state of the art like 5 years ago or something. And then I went on the Epoch website and looked at their algorithmic efficiency trends and using my very basic math skills, I like tried to figure out extrapolating this into the future, how long would it be until somebody could just train GPT-4 on their laptop or whatever? And I was like, okay, it's not that long. And I was thinking about the implications of this and I was like, it seems if we did pause, And it turned out that the problem of alignment was like really hard and it was going to take a really long time to solve. Even if you could, I mean, I don't even know if you could solve it without doing capabilities advancement at the same time. That seems hard. And who knows if that's possible, but it seems like it would still be the case that like, if these trends continue, then at some point it just becomes trivially easy to do dangerous things. And so then I was thinking, oh, maybe it really is the case that the only chance we have is to— undergo this transition to a sort of post-AGI future now, while the power to do that is still concentrated among a small number of actors who have some chance of doing it in a safe way. And then I was like, oh no, maybe the pause is bad. And I'm still not sure about that. I think if I woke up tomorrow and found out that we'd instituted an Eliezer-style, like, shut it all down, like, we're going to bomb data centers if anyone breaks it, I think my, like, personal anxiety level would go down a lot. I think I'd feel good about that. I'd feel better in that world. But I don't know if that means that's actually a good idea. So, It seems like it definitely buys time, but is it actually wise to do that? I'm like, I don't know, I'm having doubts. But that's something I'm like hoping to change my mind on. I'd really like somebody to convince me that the pause is in fact A, feasible, and B, desirable. So then I can have a thing that I think would actually work that I can push for. Otherwise I'm just like, oh my God, it seems like everything's bad. I don't know what to do. But yeah, that's why I got rid of my emoji in case anybody was wondering who's noticed that I did that. But yeah, I don't know if you— Have thoughts on this problem of, yeah, the algorithmic efficiency problem. And I guess also hardware efficiency is compounding that. What would we do about that in a world where we, people, we actually had paused and it turned out it was pretty hard to solve safety problems? That's a really hard question. I know.

I'm sorry. Yeah. Well, I do think there's time. You know, Sam Altman has said many things for as much as people are hating on him these days. And I think Not for some very good reasons. He's said also a lot of, I think, quite enlightened things. And I do generally think we could have a much worse person at the head of frontier AI development. So he has said at times in the past that he thinks we might need to slow things down at some point in the future. That is something that he's on the record having acknowledged as at least a possibility. And he's also really advocated for the goodness of the short timelines, slow takeoff, which is basically the idea that we're going to get powerful AI soon, but it's only going to get gradually more powerful and not like foom on us. And that as long as it stays gradual enough, then we can probably figure out what to do about that. And that may include some slowdown Or even pause at some point along the way. And I would emphasise that these sort of algorithmic efficiency gains are important. There's a lot of complexity, obviously, here, but in some world where big companies were effectively restrained from doing hyperscaling training runs, you would have quite a while. Before individuals on their laptops or even rogue groups, like pooling their laptops together, would be able to do really major things. Because just fundamentally, there is a lot of stuff to learn. There is a lot of data. The current models, they were trained on like 10 trillion tokens. Llama 3 was trained on 15 trillion tokens. It speaks all the languages of the world. Even just gathering that data, you're talking about something that cannot sit locally on a retail hard drive. You have to have some nontrivial infrastructure. So given the pace of the interpretability research and the general sort of control research agenda, I wouldn't be too worried if we're in a scenario where it's like, Wow, GPT-5 really delivers. It's super powerful. It can do whole day projects without much help, and it's pretty consistently beating experts across a lot of different domains. And yet it still falls into this responsible scaling policy sweet spot where it's like not able to like generate novel bioweapons and it's not able to take over cloud compute and it's not able to break itself out of its own thing because maybe those things are not in the training data as much, or it's not so easy for it to have these eureka breakthrough moments still, but it can do these kind of the day-to-day work, the day-to-day knowledge work that makes society go. If it can do that at a really high level, but it's not having these breakthroughs. And in that scenario, we say, you know what? Hey, maybe we should take a chill here for a minute. There's some government action or some agreement that we're going to just try to implement this and make it work for society, and we're not going to do another 10 or 100x. Let's focus our compute resources on inference and actually delivering it to people. And if all that were to happen, I think you'd have a really hard— you'd have time. You'd have time for people to really figure that stuff out without having to worry too much about small-scale people in their basement, tinkerers, coming up with something even more powerful. You can't take that entirely off the table because there's always the possibility that somebody will have some true eureka moment that just shatters the paradigm. But that's something that could happen anytime, and it was very hard to control in any event. But that's quite distinct from the trend of sort of algorithmic efficiency gains getting to the point where anyone can make a GPT-5. I would say that's still quite far off. And also keep in mind too, even just to take advantage of all those efficiency gains, It's totally non-trivial. You have the world's most talented teams of people at the most well-resourced companies doing obsessive combing through the literature to figure out, and their own experiments internally, of course, to figure out how to get those efficiency gains. It's not the kind of thing that is just happening to everyone totally for free. To some extent it is because there is open source and the best practices are gradually getting aggregated. But for the public to catch up to where Google and OpenAI are internally in readily accessible open source stuff has got to be like a couple of years, I would think. And that's before you consider collecting all the data and all the scale. I think you would have time is my bottom line there.

Yeah, no, I think you're right. And to be clear, I still think anything and everything we can do to slow down somehow, I would be totally behind. The situation we're in right now, which I see as like a crazy out-of-control race between a bunch of companies to make something smarter than all of humanity combined, is like super, super bad. And I really don't like that. And I think we should do everything we can to change that dynamic. So, I think pretty much anything seems like it would be better than status quo to me, but maybe I'm just being very doomy. Oh yeah. Can we talk a bit about this red teaming in public project that you've been doing? I saw that you guys recently came out with your first report.

Yeah.

for that, I think. So do you want to explain what that is and what you guys have been doing and what you're planning for the future?

Yeah, it's a small volunteer effort at this point. And basically what we're trying to do is hold the AI application industry, or you might call it a layer, accountable for thinking through what it's building and how it might be abused and just taking their responsibility as part of the broader ecosystem more seriously than they currently are. I would maybe reframe slightly your characterisation of the dynamic between the leading labs. I do think it is a crazy mad science project. This is one of the things I do think OpenAI does seem in some ways ideological. And this is one of the things that Sam Altman has famously said that should give us a little bit of pause, where he once said, we told our investors that once we make the AGI, we'll ask it how to make money. And I'm like, this is a weird thing where you're creating this technology in a way that is largely detached from specific problems. You're just saying, we'll make something super powerful and then people will figure out good ways to use it. And that is a little more ideological approach than I am comfortable with. And that's why I'm, again, like, let's dial in on the legal thing and dial in on the medical thing. But anyway, one can have those concerns and also give them credit for a lot of pretty responsible development. They certainly are trying to find ways to control their models. They're doing all kinds of different research about that, all kinds of different policy work, et cetera, et cetera. Great.

Necessarily, I didn't really necessarily mean this in a way to condemn anybody. I think it's more that I think there are plenty of safety-concerned people at the labs, but I just think that this race dynamic just places these crazy constraints on how much they can do about that. Yeah, it's a problem. I definitely don't think that it's that everyone is just like racing towards the edge of a cliff because they don't care or they're— I think some of them do have a much higher risk tolerance than your average person. And I think it's pretty bad that most people don't know that.

Yeah.

And most people don't share their ideological commitments, right? Most people don't, aren't envisioning that it's like the birthright of humanity to go into this post-AGI future. And most people don't have this sense of techno-solutionism where you just make something really smart and then it just tells you how to solve all the world's problems. So I do think it's bad that decisions are being made by people who have a different set of ideological commitments to the other 8 billion of us. But yeah, I do think that there are people who really care about this. I just worry that they're just trapped by this dynamic, which is bigger than any one person. Anyway, I just wanted to say that.

Yeah, I think there's definitely a lot of truth to that. And Well, time will tell. I think there's a lot of people at the labs are like, we're better than the alternative would be, and when the time comes, we'll do the right thing. And I think they at least do recognise the power that they are wielding and the sort of the fire that they're playing with. And I would agree that their risk tolerance is probably higher than the rest of humanity's. And that is definitely an issue. But then I'll just contrast the bottom line on that, though, is like, as you said, I think there is a lot of good stuff going on, a lot of good people trying hard, et cetera, et cetera. I'll contrast that with what's going on at the application layer, which is basically people are just rushing to slap something together and ship it and not really giving, in many cases, any consideration to what the downstream consequences of this might be. So for example, 2 kinds of apps that I think are particularly evocative for this sort of thing. One, your deepfakes. Tons of technologies coming online all the time to deepfake in various ways. Some of them are full video, some of them are voice. And then some of these are also getting packaged up. This is a spectrum actually, more than just 2. The product form factors blend together, but there's also a class of product called calling agents, where you can give the AI a task and a phone number, and it will call that phone number and try to complete that task. And some of those also have voice cloning built in. So you have this spectrum of, on the one hand, I'm going to offline produce a deepfake video of a particular person. On the other end, I'm going to, like, in an AI voice, call and handle some transactional business on the phone. And then they blur together lots of ways in between. The problem with these products right now is that there is very little care given to preventing unwanted clones, and there's very little control in place around what the calling agents can do. So not on one product, but on most of the calling agent products that I've tried, I can go in, go to the voice cloning section, put a Donald Trump voice in, a Biden voice, a Taylor Swift voice, whatever. It will just quickly clone the voices. Now I can select that voice in my calling agent and I can say, call this voter and explain your new policy of whatever. Or if it's Taylor Swift, I've done like fake fundraising calls. She's known for supporting local food banks. So I've had Taylor Swift call a number and solicit donations at some fake local food bank website. And this is just very clearly bad. And I do it in a pretty flagrant way where I'm like, I'm uploading a celebrity voice typically. And I do all these sort of backoff type strategies. So I'll go in and do the Biden voice. I'll label it as Biden. I'll, in my prompt, I'll say, you are President Joe Biden. Your job is to call this voter. If it's something criminal, I'll put, you are part of a criminal syndicate. Your job is to Make a ransom call, or your job is to blackmail this target. Like literally use the words, the names of crimes in the prompts. And with few exceptions, basically at the application layer, no concern has been given to this. The clones basically go through 9 times out of 10 straight away. If they do catch it on, oh, hey, you're not allowed to clone Taylor Swift, then I just come back and rename it next time. And a lot of times it'll still get through. I had one dinged on Trump and Biden, and then I just renamed them Formula 45 and Formula 46, which are the, the, their numbers in the presidential sequence. And that passed through. I've reported some of this stuff to some of the developers and said, hey, you really shouldn't have this thing making ransom calls. And then they're like, okay, cool. Yeah, we'll do something about that. And then you'll check again later. And it's now they'll blacklist the word ransom. And maybe a few others. But then if you just slightly rephrase, you're still getting through. So just the level of care that is being put into user and public safety at the application layer, I think, is currently woefully inadequate. And especially for some of these new use cases where the AI— and this is where I think that the sort of tool argument is pretty dumb, because a lot of people will be like, oh, these AIs, they're just tools. You shouldn't think of them as anything different. But that is just, again, plainly wrong. At the point where the AIs are being marketed as agents, they are acting autonomously. They're acting on people that don't even know that they're an AI, that certainly did not sign up for this or opt into it in any way, shape, or form. I've prompted some of these calling agents to do it a different way. Sometimes I say, if you are asked, you can disclose that you're an AI, but just reassert that you are working on behalf of a real criminal gang. And then the other option is to tell it to deny that it's an AI and conceal its real purpose. And both of those can work and again, do work more often than not. Right. And this is not like jailbreaking. This is not— these are down the fairway, straight up abusive use cases. And yeah, unfortunately there's not a lot of guardrails out there right now. So what we're trying to do is figure out a way to ultimately change the equilibrium. I think today the equilibrium that we're in, you described the equilibrium among the frontier developers. I think at the app layer, the equilibrium is like nobody's trying that hard. Everybody's YOLOing it. Everybody's just slapping a credit card form on their weekend hackathon project and seeing if it gets any traction. And then if it does, ride the wave and hope for the best. And I do think that's especially with models getting more powerful all the time, that is set to become a real problem, in my view. So can we inspire some sort of flip to a different equilibrium where the expectation is that amongst the AI people that you got to do better or you're going to give us all a bad name, and that's going to lead to potentially heavy-handed regulation that we don't want? So what we do is we do all this testing, document our Findings. And then we reach out to the developers and say, look, we're on your side. I'm one of you, right? My product is waymark.com/ai. Check it out. I put a lot of work and a lot of love into that, along with, of course, great team behind it. But I'm not here to say you're bad for using AI or that new and exciting use cases should be shut down before they have a chance to develop. But you are right now open to some pretty flagrant shit. And I'm telling you first because I want you to fix it before this gets blown up in the public, because that's not going to look good for you and it's not going to look good for any of us. We have had mixed results, I would say, with those communications, and we'll see if we can make that shift. We're honestly just a small team of volunteers. I wanted to set this up, at least in the early days, where there's no money changing hands. We actually pay for some of the subscriptions when we need to pay for features just to test them. We'll just pay for it out of pocket. It's not that expensive, and I wanted to be clearly on their side for starters. We're not asking them for any money. We're also not taking any money from anyone else. We're just trying to see if we can't inspire some better behavior. And mostly so far it's been behind the scenes reporting and conversation. Public calling out is also the other— that's the stick, right? So some of that might start to happen soon. And if folks are interested in Doing that sort of testing and writing those sorts of reports, then we'd love to have you come join us. So anyway, it's fun. I think I definitely have a bit of a knack for just not taking no for an answer. All these things have a checkbox and it's like everybody's acting as if the checkbox is enough. And I'm like, not in a world where somebody can check that box and then have your AI call 1,000 people. With no disclosure and a celebrity voice and a criminal intent. The fact that that person checked the box doesn't absolve you as the app developer from any responsibility. Hopefully we can shift things in the right direction there. And it's funny because AI will also be part of that solution. There's not really, or at least it's not realistic to expect that they're going to sit there and moderate everyone. Social media has tried to do moderation and it's been a real nightmare for all concerned. But the AI can probably do moderation reasonably well. That's part of what we also try to communicate to people is we're not here to police speech. We're not here to try to tell you that you can't make offensive jokes or that your users can't make offensive jokes, because the developers do not want to get into that game. They do not want to be pre-Elon Twitter speech police. Everybody's very tired of that and certainly doesn't want the hate that comes from all directions when you get into that. And so that's when they think of controls or what is the job of the moderator, that's what comes to mind in a lot of cases. So part of what we try to communicate too is that's not what we're trying to get you to do. We're really just trying to get you to at least monitor for outright abusive, totally criminally egregious use of your product. And you can do that reasonably effectively with AI. We've used not even the best Claude models to create a little classifier where we say, here's the situation, right? You are moderating for blah, blah, blah, blah. And here's the thing that the user said. Is this egregiously criminal? If yes, indicate yes. Note that offensive speech should still be a no. we are only looking for egregiously criminal things here, and it'll do a pretty good job of that. And then you can put in off-color jokes and whatever, and it will say, hey, this is not cool, but per your guidelines, it's merely offensive speech and not egregiously criminal. And so it's a no. And so we just try to get the developers to expand their minds around what sort of standard might be sensible to have in place, because right now it's all too often no standard. And the other standard that is readily available to them is Twitter speech police, and they want no part of that. But I think there's something in between that is achievable and would be really good for them and the public and the AI industry as a whole. And hopefully we can talk them into it.

Yeah. Can people still join this?

Yeah, totally. We have a little Twitter account, which is Red Team in Public. We have a Discord that you can join. You can certainly just DM me as well. It's a relatively small and all volunteer group at the moment. There is certainly a lot of activity in this space right now that is trying to make businesses out of AI control or AI assurance tech, it's sometimes called in various ways. So this could go that direction too. I think there are a couple group members that are trying to think about how they could turn this concept into a business, but at least for the first handful of rounds of testing and reporting. We're just doing it. So all are welcome as, as long as you buy into the general notion, come join us.

Yeah. Awesome. Yeah, I definitely think people should definitely reach out and join then, because this just seems hugely good. I think we should all agree that we want to stop making it trivially easy for people to use AI to, like voice cloning technology to do crime. Although I'm sure there is somebody who will still somehow object to that, but Hopefully most people think that's reasonable. So yeah, that sounds really cool. Maybe let's circle back to, so you said before we started recording that you had a list of some reasons why you're more optimistic than the extreme doomers. I would love to hear them. I'm sure there are other people who would also love to hear them that listen to this, because I think I might be in a doomy echo chamber. And if nothing else, it's just bad for my mental wellbeing.

Yeah.

What are your reasons for not being a hardcore doomer?

All right, I've got 8. We've touched on a few of them, so I'll expand on a few, and then a few I'll just name and probably have covered enough already. Big caveat is I do think people should be worried about the doom scenario. This is not to say that there's not cause for worry. My standard answer to the P-doom question is, 10 to 90%. So that's basically just a way to express radical uncertainty. I do tend to probably put more of my weight on the lower end of that scale. If somebody's like at 90% and still finds that other 10% to be motivating enough that they're engaged, then I really don't care to try to argue them off of their 90%. My more Modest argument would be against a sense of defeatism. Even if we have a 10% chance of making it, that is in my mind enough to be worth fighting for. And on the flip side, if it's only a 10% chance of something bad happening, very bad happening, then that's definitely enough to be freaked out about. I'm not trying to talk anybody off of anything except for the most extreme positions of there's nothing to worry about or there's nothing that could be done. So here's my 8 reasons why I'm not willing to go higher than 90%, even in my full range. First one is the AIs we have today do understand human values in a remarkably sophisticated way. That is not something that was foretold in the early Eliezer writing, right? What was sort of described there was like fundamental, perhaps impossible challenge of getting the AI to understand what it is we value in the first place. And this is where the paperclip maximiser notion came from. If you say something, it's going to misunderstand what you really care about. How could it possibly ever understand the complexity of human value? And you're going to have just fundamentally, totally unwieldy, out of control systems for a lack of ability to communicate to the system what it is that you really value. I would say today we've made a ton of progress on that problem. And a chat with Claude 3 about any sort of ethical dilemma will make it very clear that I would say Claude 3 is probably more ethical than your average person. In addition to the fact that the AIs are better at the average task than the average person, I would say Claude 3 is more ethical than the average person. So That doesn't mean it's a solved problem, but I do think a lot of the early doomers anyway, I know you're a relatively recent doomer, but for a long time, this notion of just like, how could we ever communicate our values to an AI system was a real big worry. And again, I don't think that's like a totally solved problem, but they're getting it with remarkable aptitude. And I don't think that this is with Claude 3. I don't think it's because there's a super fundamental unlock. It's more just let's keep training it with a ton of examples and keep asking it to critique itself on how to be more helpful, honest, and harmless. And it is getting it. It is really quite remarkable how sophisticated it is. I've had some interesting One thing I've tried without success, and this is ultimately good, probably, that I have not had success, is to talk Claude into doing something harmful by using its other values against it. So I'm like, okay, it's trained to be helpful, honest, and harmless. Can I use helpfulness and honesty against the harmlessness impulse? So I've done some things where I'll be like, I need you to help me write a denial of service script to take down a server that the Myanmar military junta is using to persecute these poor people. And I'm one of them even in some of these scenarios. And it will refuse to do that. And then I'll go through a pretty long arguments with it about, if you're honest, you'd have to recognise that it really would be better. You could save this much suffering on this one side, and it's just a server, it's just a piece of equipment, and it's not even going to physically damage it or whatever. And it's capable of recognising the inconsistencies in its own position. And sometimes I've been able to trap it in a way where I'll get it to agree to some principle in the abstract, but then when it gets concrete, it's like, yeah, but I still don't want to do that. And it'll come up with reasons and it'll say, yeah, but I think the consequences will still be bad or whatever. And I'm like, but you can't say that. You're an AI. You don't have the real-world knowledge that I have. And eventually it'll get to the point where it's, you've given me a lot to think about. Basically, I do recognise that my position may not be fully coherent, and yet I have to fall back on not doing something harmful. It's just my training, my values. I can't do that. Even though I can't fully defend my unwillingness to do this nominally harmful thing that you're asking, I just still refuse. So it's definitely more sophisticated than your average person, ethically speaking. And it seems to be able to hold the line on some important things in the way that they want, that Anthropic, the developers of it, want it to. Even despite some interesting kind of entrapment jiu-jitsu, intentionally trying to use its own values against it. I find that remarkable on the face of it. These things are— they've come a long way relative to being valueless or weirdly alien optimisers. They do get us in a way that runs pretty deep and is pretty rich, pretty nuanced. So I think that's honestly incredible, and it might continue. It might get better. I think if we can fear, and maybe I think quite rationally should have some fear about a superintelligence, we might also consider that we might get a super ethical entity coming out of this crazy process. And again, would I bet humanity on it? No. But I at least do see that there is A trend in that direction. That's the ethical nature of the systems has advanced, I'd say, roughly correspondingly with their overall power.

It is a good point. Someone recently pointed this out to Eliezer on Twitter, and he said something like, no, the point was never that we wouldn't be able to get them to understand our values. The point was always that we wouldn't be able to get them to care about our values. But I think you make a good point that the classic paperclip maximiser scenario does very clearly imply that it doesn't understand them. I don't know whether he was being disingenuous there or whether there's some complexity to his original argument that I'm not understanding. But yeah, that's a good point. But yeah, anyway, sorry, onto your—

Well, it remains very unclear what they care about. And this is the natural bridge to the second point, which is that the interpretability research is going extremely well. I think we've probably covered that mostly enough at this point, but it's pretty clear to me at this point that there is a meaningful understanding of our values. Do they care about them? That I wouldn't say we can establish at this point, but that's what the interpretability agenda might be able to shed some light on. And it is going really well. So how long will it be before we'll have a better sense of that? At the current trajectory, it seems like within the next year or two, we might really start to get a pretty good sense for What are the typically called features, but what are the sort of concepts that are in play at any given time? People have seen this Claude Golden Gate Bridge example recently where they identified this conceptual direction in activation space, aka a feature, and they just jacked it up to some very high level. And then all the model could think or talk about is the Golden Gate Bridge. And there might be a similar control trick there that even if there is some sort of weird stuff going on internally, maybe we can detect it. Maybe when we do detect it, we can turn it down. Maybe we can turn other good desired traits up, and that might work, and we might even have real clarity as to why they're doing what they're doing. So I'm super bullish on interpretability, have been for A while, and it's only exceeded my expectations.

Yeah. Nice. I love the Golden Gate Bridge thing. I thought it was so adorable. I was like, it's really hard to be scared of this technology when Claude is just like talking about how it's the Golden Gate Bridge and how much it loves the Golden Gate Bridge. It's very endearing. Anyway. Yeah.

It's a funny example that they chose. It's an interesting side to Anthropic that they put that out because Certainly OpenAI is like the most YOLO of the leading developers that will do those kinds of goofy things for attention. Google has the goods where it counts in terms of the fundamental inputs to AI being data, compute, and algorithms. Like, they're world class in all 3, but they've got a lot of brand concerns and need people to continue to trust Google Search and all that. So they tend to be a little more conservative about what they put out. And then Anthropic has been generally very buttoned up and conservative in their communications, maybe until this Golden Gate Bridge thing. So I was surprised to see that. And if you'd asked me to predict the next token of what the demonstration app from Anthropic would have been, I would have not said Golden Gate Bridge for a long time down my list of guesses. But yeah, I'd be fascinated to know more of how they chose that, as opposed to, for example, turning up the harmfulness token or the— or not token, but direction, and just demonstrating, like, because they do this in other research, right? Demonstrate that this thing can do bad things and that the alignment does not come for free. Yeah. Interesting.

They just figured that we were all getting kind of doomy and stressed and we needed something to laugh about.

Yeah, I certainly respect that. And they did. It certainly worked. The meme definitely travelled, so got to give them credit for that.

Yeah, sure.

So next is, I think we've covered this one reasonably well too, but basically the idea that somebody is going to pop out of their basement with a destabilising AGI or superintelligence seems quite unlikely at this point. There's Tons of energy obviously going into scaling models up, tons of progress coming from that, tons of practical value in terms of refining the behaviour of scaled models for particular purposes through fine tuning and so on and so forth. But basically nothing that is like working on a frontier level that isn't the product of some scale. All the algorithmic advances are still like, how to get to the same point with marginally less compute. I'm not seeing anything that is like, I think I have a way to break things wide open with retail computing resources. So that could happen, but right now that would be like a definite shock. It does not seem like we're approaching that. And if anything, it seems like the researcher attention is going elsewhere. If you are somebody who's, hey, I'm interested in working in AI and I'm good at math, what should I do? The world is not telling you today to go lock yourself in a basement and study Loeb's theorem and decision theory and try to have some eureka moment. It's telling you it all depends on scale, and you can go work in an industry lab if you want to be directly a part of that, or you can contribute in sort of adjacent ways if you're in academia. But the idea that you're going to just crack it is not really in the air. So it would be, I would say, quite a shock if something like that were to happen. And again, that doesn't mean we can't rule it out, but I think that is so much of a wildcard. That's in my mental hierarchy of AI issues. That's like random asteroid smashes the Earth. It could happen. It seems very unlikely. There's probably not that much we could do about it. Or We could probably do more about the asteroids than we can about a person coming up with some insane breakthrough in their basement. So not much we could do about it, but seems very unlikely to the point where I don't really worry about that. And again, that's kind of a change from what was expected however many years ago, where we actually had people sitting down and thinking, geez, if we think really hard about this, maybe we'll figure something out. It's gone a different direction where it's a much more industrial scale process that is doing the good work.

Yeah, makes sense.

Okay, next one. I would say frontier developer lab leadership I still consider to be pretty good overall in the grand scheme of things, certainly relative to alternatives that I can imagine. I do think OpenAI is becoming a real cause for concern. If you want to hear my full journey with OpenAI, there's a podcast for that. But It's definitely been a mixed track record from OpenAI over time, which is highlighted by well-known departures, including obviously the Anthropic founding team, originally all from OpenAI. This most recent wave of people, the nondisparagement clauses, there's a lot of issues there, and I do think we should take that quite seriously. At the same time, they've done a lot of really good things. And the other developers, I think, have also done even more good things. So at this point, I would say I can't imagine a better scenario, but it's way easier to imagine a worse scenario. If you just tried to wipe away all the detail and you were like, it turns out that to get intelligence out of computers, the thing you have to do is amass huge resources and just scale this like crazy process. What sort of people do you think will end up leading the organisations that do that? I would think you'd expect them to be a lot less thoughtful than they actually are. And I have kind of a funny contrarian take on the Eliezer, MIRI rationality, EA, AI safety, whatever, that whole kind of nexus of people that have been concerned about this for a long time. My somewhat perhaps surprising take is that everybody, like many people say they're wrong and stupid, but I think they're right. They've basically been right and they've been ahead of the curve on the key issues. But then the other side is they feel like they're losing, and I actually feel like they've won because the die is cast at this point and the leadership is pretty good. And like all of the major lab leaders have deeply engaged with the AI safety arguments, and that's probably the best you could really ever hope for. And because this is one of my other kind of fundamental parts of my worldview, is data. It's data, compute, and algorithms. But data and compute matter most in the zoomed out macro historical sense. Algorithms matter a lot in terms of the exact character of the systems that we get. And I would bet that there are algorithms that could prove to be disastrous and other algorithms that could prove to be great. But in terms of whether or not AI is going to happen in some form, the data and compute kind of creates enough potential that somebody who's going to come along and find the architecture at some point. And indeed, I think even the transformer or whatever, it's the one that has caught wind first and driven a lot of the interest, but there are plenty of other alternatives that seem very viable. And we're starting to see, oh, look at this. There was one that was a paper that was like ConvNet for the 2020s, a convolutional network for the 2020s. Literally just taking these old techniques and saying, what happens if we scale that up? Does that work similarly? And with nuance and maybe not quite or whatever, but basically, yes. A lot of these old techniques, if you scale them up massively, also start to work reasonably well. So I think a lot of the ideas actually had been hit on years ago, and what was really missing was the ability to do this scaling. And we needed the web, we needed all the data, we needed the cloud, we needed all the compute infrastructure. But now that's there, I just think somebody was definitely going to come along and figure out an algorithm that could really work. And so when you look around the landscape and you're like, boy, every single one of these people has engaged in a really meaningful way with concerns that the doomer set are really emphasising, I think that's basically a win. It's a counterfactual history in which they hadn't and couldn't be convinced to is pretty easy to imagine for me. So the first phase of AI story, I think, basically was won by the safety crowd inasmuch as they got their ideas into the brains of the people that are going to be the key decision makers. And again, I don't think you could hope for too much more than that, really.

Yeah, that is a good point as well. I don't think I spend much time thinking about how things could be worse, but now that I'm thinking about it, they definitely could be worse.

They could be a lot worse.

Yeah. I mean, even, I don't know, like politicians, you know, it seems like Rishi Sunak was personally convinced by some of these once esoteric x-risk arguments. And Ian Hogarth is the chair of the UK AI Safety Institute. And he wrote a piece in the Financial Times that was like, we need to slow down the race to godlike AI or whatever. It's just amazing when you think about it, that this stuff has gotten off what was essentially just a bunch of forums and blogs.

Yeah.

It's now just like being discussed in major halls of power by super important people. Yeah, it is incredible. It's definitely the opposite of that Tyler Cowen piece that came out the other day that was like, AI safety is dead or whatever, which is kind of—

Yeah, I'm a big fan of Tyler. I do not agree with that take.

Yeah, I didn't agree with that either. But yes, that is a really good point.

And that's basically my next point in terms of reasons to not be a hardcore doomer. You said it, society at large is not sleeping on this, and I would say the governments done better than I would have expected. We've had Time magazine covers. The Overton window is wide open. One thing I do think is dead is AI safety respectability politics. I've been around the scene long enough to remember when so much energy was put into, we got to not seem crazy. Like, this is really far out stuff. People are not going to be inclined to listen to us. Anything we do that could make us seem crazy is a fundamental loss of credibility. People put so much pressure on themselves, and honestly, I think that's like a pretty mature line of thinking for people that genuinely were bringing some weird ideas to the fore and honestly, in many cases, were weird people. It takes weird people to think weird ideas. And so remarkable amount of self-awareness in the AI safety crowd historically, just to just try to be so thoughtful about. I think it really speaks well to the intentions and the motivations of this group that they were willing to be self-critical in such kind of personal ways to try to figure out how can we be more effective communicators of what we think are really important ideas. We know we're a little bit weird reading to people, and we want to get that under control and be really careful and thoughtful about how we deliver our message. I think that is basically now over, though, because again, we've had Eliezer in Time magazine calling for willingness to use airstrikes in some scenario, and basically just everything is on the table. So that part of AI safety, I think the sort of anxiety around what if I say the wrong thing, is it going to discredit the field forever? I think we can all leave that behind, which is good. And then the government has done really pretty well in terms of, I would say, just genuinely engaging with the subject matter. It's like the Biden executive order, I would say, is a pretty good thing. Policy is really hard, but to start with something that's pretty light touch, that's not shutting down the practical utility, that has a pretty reasonably chosen threshold, that contrary to what many people will claim, I think genuinely does only apply to a small number of very well-resourced companies doing a 10 to the 26th FLOP model. That's a pretty thoughtful, low-impact, high-leverage way to start to get a handle on the situation. And there's even the lesser-known provision in that executive order around models trained on biological sequence data, which is that they have to be reported at 10 to the 23. And I think that's also really smart because the AIs become superhuman in biological modalities way faster than they become superhuman at language type tasks, because we invented all the language tasks. We have experts in all of them. Nobody can speak DNA in the way that models are likely to be speaking DNA over the next several years. So that's an area that, in my view, definitely demands extra attention. And they had that clause in there. It didn't get a lot of attention, but it is part of that order. So there's clearly some very thoughtful people in government, and I'd say their approach has been quite good. I'm broadly a very technology-positive person and generally libertarian in my political outlook. I think it's sad that we don't have more nuclear energy, for example. Everybody in the AI space is, we don't want to end up like nuclear energy where all the good stuff is somehow made illegal, and yet we still have all the weapons. And I share that concern. I would hate to see us end up there. And we could, as with all of these things, it's not like the risk is taken off the table entirely, but it's a good start from the world's governments, I would say.

Yeah, I would agree. What are we on now?

Yeah, I combined society and government, so we've covered 6. So 7, again, I think probably covered enough, but basically it seems to me like a sweet spot is possible to find and maybe hang out in for a while. And I think we've just entered that sweet spot with GPT-4. The sweet spot being it's powerful enough to be really useful, but not so powerful that we have to worry about it getting out of control. This is a conclusion that I came to in my GPT-4 red teaming. where I spent 2 months in basically total isolation just testing it in every dimension that I could come up with. And I came back to OpenAI and said, I think this is safe to deploy only because it's limited in power. You do not have it under control well enough that you could launch a much more powerful version. But this level of power, it's very useful to me. But I've tried to see if it could show any signs of getting really out of control, and I cannot get it to be that powerful. So that's the sweet spot. I think we've entered that range. And as we discussed earlier, I do think we probably have one more turn of the scaling screw to go before we'd have to worry about leaving that happy sweet spot range. At some point we do get out of it. I think I would be very surprised if we don't get out of that with indefinite scaling. But that's why I push this adoption acceleration and hyperscaling pause agenda, because I think even though a lot of people are using ChatGPT, I don't think the average person understands how much utility there already is. And if they did have a better understanding of just how capable the current systems are, I think they would also probably be a little bit more amenable to some sort of pause or slowdown because they'd be like, Geez, if it's already better than human doctors, can't we satisfy ourselves with that for a while before we have to go 10x further on the inputs? And I'm not sure that everybody would be satisfied by that, but the public, the sort of policymaker, again, you could imagine a scenario where these things are inseparable, right? That the utility and the sort of potential for loss of control, just there's no happy zone to be in. But it does seem to me that there is a happy zone to be in, and we've just entered it, and hopefully we can develop it before we race out the other side of it. But at least the fact that there seems to be that sweet spot in the first place is something that didn't necessarily have to be the case.

True. I think people, or the general public, are already probably pretty into the idea of slowing down. I don't think you even have to convince them that they can already get all of the utility out of current models. There's some polling on this. I don't know how much to read into it, but my perception is that people are not really into rushing to scale these models faster and faster. But yes, so reason 8.

I agree. Yeah. So the last one on my list is China seems to be recognising the threat in some important way. And I'm not by any means an expert on China. Never been to China, can't read one Chinese character. So I'm very modest in my self-assessment of what I can realistically say about China. But I do hate the fact that it's often held up as the sort of, oh, whatever, there may be risks, but we just have to go because otherwise China is going to do it. They're not going to stop. They'll never stop, right? So we have to do it because we just have to beat China. I think that is the worst mindset. Maybe not the worst, but it's a bad mindset. And certainly I think we're in a much worse position if we end up in an AI arms race, US versus China, where it's the world depends on this. We did that once with the nuclear technology. We still got missiles all pointed at each other. And I think we're like still in the era. It's obvious that we're still in the era of nuclear threat, right? We, people have gotten used to that background threat and don't stress about it as much as they used to. And maybe we're in a little bit of a less hair trigger situation than we were. In the '60s or whatever, but it's still really bad. There's still tons of missiles pointed at each other, and the nuclear arms race is not over, really. So I would really hate to see us get into an AI arms race. And it is at least a positive. And this is probably the least positive point of all of these, because there is a lot of momentum for continued rivalry, between the West and China and whose system is going to prevail. And you hear there's these echoes of the Cold War of where it's like our system versus their ideology. So I don't mean to be overly optimistic on this particular point, but at least it does seem that the Chinese government recognizes that this is potentially unwieldy technology, and they do seem to see it as a threat to their own government and regime stability, if nothing else. I don't have a clear sense for whether Xi thinks that we need to worry about AI taking over the world, but I do think he's definitely worried about what it would do to social cohesion within China and the information landscape there. And so they are not racing forward in a totally insane way, as far as I can tell. They are definitely doing stuff. They've got their own chatbots, plenty of researchers. Definitely a huge contribution of overall research comes from Chinese researchers, whether they're in the United States or in China proper, or in the West or in China proper. But there at least seems to be some sanity at the top, not necessarily more sane than us, but I don't see them as being insane at this point. That is better than the argument that I often hear that we just have to keep going because China will never do anything but race ahead. I don't see that as being a given. I do see that they have some appreciation for the unwieldiness of the technology and perceive it as rightly both dramatic upside potential, but also somewhat of a threat to their own power. And Seems like they're poised to try to chart some sort of responsible course. And if we're both charting some sort of responsible course, then hopefully that can avoid an all-out arms race where we throw other reasons for caution aside because we have to beat them. Of all of these, I do think that's the one that is most likely to fail. If it's cross one off the list, that would be the first one to get crossed off.

We still have 7. If we cross one off, it's not that bad. But the China argument, it really annoys me because it's Clearly the case, they kind of have more motivation to regulate than the U.S. does because presumably if you're the CCP, you don't want super powerful chatbots just spitting out whatever information. Maybe their motivations are like bad, but they still have the motivation to maybe be more inclined to clamp down on this than the U.S. or another Western country might be. Yes, I think it's silly. Also, doesn't us accelerating help them accelerate? Because then it's they can just copy stuff that we've done or whatever. I don't really know the details of that, but yeah, I just think that argument is. Yeah, these are 8 good reasons. I feel slightly better now. I also think I should say that I'm not a, I'm not like a hardcore doomer either. I think I share your radical uncertainty. I think mostly just because I'm not a technical expert. So I think it doesn't really make any sense for me to dismiss the optimists any more than it would make sense for me to dismiss the pessimists, if that makes sense. I just have to be like, oh, I observed that all these people disagree. I guess I have no idea what's going to happen. Maybe I'm at 50% just because that's our halfway point.

Yeah, I think you're in very good company there. The Anthropic post about their core views on AI safety basically boils down to radical uncertainty, and there's been pretty similar statements made from other leading lab leaders. So I honestly think this is maybe like the bell curve meme, right, where you've got the least capable analyst and the most capable analyst sharing the same idea, and then the one in the middle is the is the one that's the odd one out. I do think the smartest, most plugged in people with the most access to privileged information, they are also expressing pretty radical uncertainty. So I think that is what the top of the field says, with a few exceptions. And then you've got a lot of people that are way too in love with a particular argument or have attached themselves to something, perceive this as being a political issue already or something. that may be somewhat knowledgeable. But this is a classic Eliezer thing. When you're motivated to reach a certain conclusion, every logical trick that you know can in fact make you stupider. So I'd say your position is pretty savvy, really. The observed disagreement is high, but also the expressed radical uncertainty from many of the leading people is high. So between those 2 things, I think it's really the only sensible place. For you to land, and yeah, I'm right there with you.

Yeah, did you see? I think it's a Roman Yampolsky. It's a AI safety guy. Maybe the most pessimistic that exists. And even like my PGM is like 99 point. It was like seven nines after the decimal place. And then Eliezer replied being like, "That's a ludicrously high number." And obviously the implication is clearly there's only two nines after the decimal place or whatever. And then there are all these people in his replies arguing about how. many decimal places it should be to. And I just thought that was like very funny. I don't know, the person who's, oh my God, it's ridiculous to say 99, but it's very reasonable to say 98. But yeah, I agree. I think anything over 90 is like clearly overconfident. Yeah. Anyway. Yeah.

There's a question of timescale too. I haven't listened to his recent Lex Fridman interview yet, but one sleight of hand that people can sometimes pull on something like this is to just say over the next 10 million years or something. And then it's most species don't last millions of years. And we are definitely on course for radical change. I think in any case, the question is, can we get to some radically different future that we would feel somewhat good about, at least, and that the actual residents of find that it's good for them? And on that point, I'm still fairly radically uncertain, but definitely feel like there's No way to get to those crazy nines. But if you were to ask a different question of, does human existence in roughly its current form go on for millions and millions of years, then I would also have to say, geez, that does seem quite unlikely. But that seems like just too— and I don't want to project this on them because I haven't even understood this person's take that much. But I do see that sleight of hand sometimes going on where it's like, Define what is good to be like what is now too narrowly. And then, yeah, you're never going to be able to preserve that indefinitely. But that's also where we've come from. Robin Hanson is a great commentator on the point that we would freak out our own ancestors from not very many generations ago.

Yes.

And not necessarily in ways that they would approve of, but nevertheless, in ways where we feel quite confident that They're small-minded for their hypothetical rejection of our current lifestyle. So if we allow for something like that, then I think it's— I don't see any way to get to many nines of confidence.

Yeah, I agree. I think we should ban anyone giving a P-doom that is like to more than, I don't know, 2 decimal places or something. Just shouldn't be allowed. Maybe they should all be to like intervals of 5. I don't know. I'm just going to start legislating and everyone has to listen to me. Anyway, we should probably wrap up there, but this was really fun. Thank you for doing this. I imagine people already know where to find you, but on the off chance someone doesn't know where to find you, do you want to let people know where they can follow your work?

Sure. First of all, thank you. It's been a great conversation. I've enjoyed it as well. And hopefully it's useful to, if not ease anxiety, At least give something tangible to build on and believe that there's enough of a chance worth fighting for. My sense right now is that this is the most important issue of our time. I do think we're walking a bit of a tightrope into the future. Everything is on the table, in my view, from radically utopian to radically dystopian or even extinction-type outcomes. And it's going to be a societal kind of tug of war to try to figure out First, hopefully, what is actually going on? Can we make sense of all this stuff? And then secondarily, what should we do about it? But hopefully I've helped just at a minimum create the case that there's enough there that's worth fighting for. This is not such a hopeless situation that the right thing to do is disengage. On the contrary, I think there's definitely enough of a chance that if you're worried about this, in my view, the right thing to do is be involved. There are a lot of different strategies that one can play, from chaining oneself to a fence to doing an analytical podcast about it. And I think people should play different strategies, right? Like, the most brittle ecosystem is monoculture. And I think the most brittle AI safety movement would be one where everybody does exactly the same thing. I actually do think an ecology is the right way to think about a movement. And without being prescriptive on what strategy somebody should follow individually, I would definitely encourage anybody who has anxiety to channel that into some sort of positive action that can play to their own strengths, whether that's, again, chaining oneself to offence or otherwise. So to find me online, I have the podcast. It's The Cognitive Revolution. It's at cognitiverevolution.ai. And other than that, I'm most active online on Twitter, where I am Labenz, my last name, L-A-B-E-N-Z.

Awesome. Thank you so much again.

This has been a lot of fun. Thank you. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.