Brian Chau on Spreading Informed AI Optimism

This week on Upstream, we're releasing an interview between our friend Nathan Labenz from the Cognitive Revolution podcast and Brian Chau, a former machine learning engineer who recently founded the Alliance for the Future. I chat with Nathan for a few minutes at the end of the show, so please stick around. Please enjoy the episode.

Brian Chau, founder of the Alliance for the Future. Welcome to Upstream and/or The Cognitive Revolution. Maybe both.

Yeah, glad to be here.

Why don't you start off by just telling us about the Alliance for the Future?

Right. So our tagline is informed AI optimism. I think that most people here in DC. So I moved to DC to start Alliance for the Future, and most people here in DC still have very low-resolution images of AI. Their idea of AI policy, a lot of the time they think it's just ChatGPT. They think it's very narrow. They don't understand the economic implications. And really, I think that many people can be convinced to be much more supportive of AI once they get that full picture. So that's really our mission here.

Could you give me a little bit more of your personal brand of optimism? Obviously, AI— I fashion myself as an AI scout, and I feel like it's sort of an interesting mix because on the one hand, this mission that I've set out for myself to have the most comprehensive, up-to-date, wide-angle view of AI that I can is like, definitely touching on a very real need. But then on the other hand, it's kind of becoming convergent with just all of society. And that also means that the project in some ways is fundamentally flawed. But with that breadth, I think when people talk about AI, and I imagine this is true in the circles that you're running in where people are not primarily focused on it, they mean a lot of different things. So what's kind of the next cut deeper of what to be optimistic about and what not to be optimistic about, if there's anything not to be optimistic about in your mind.

Right. Yeah, you're right about the term AI being this increasingly encompassing thing. And I do think there's this tendency to over-narrow it, right? To only focus on LLMs. If we're looking from a policy perspective, many of these policies apply not just to LLMs, but to other models trained with similar techniques, right? So inference models, robotics, certainly many predictive models, if they're of sufficiently large size. And that's just the kind of regulatory environment. So that's not really my definition, but it's the kind of definition presented to me via working in policy. In terms of what I'm optimistic about, I mean, I think that it'll make almost everyone's lives just much easier and it will cause economic growth. It'll cause downstream positive effects, the typical kind of pro-technology story. And I think that both the kind of near-term concerns and the kind of long-term concerns are, I think that the long-term concerns are really based on pretty Typical mathematical fallacies, really. And then the short-term concerns, some of them are quite legitimate, but the effect sizes are just outweighed by the positives. That I think that there can be policy that is done to take those concerns seriously, something like cyber risks, something like deepfakes, but that in the long arc of things, the benefits to AI are just much greater. That There should be no question that it's worth developing more AI and maybe dealing with the downsides on a kind of case-by-case basis rather than any kind of research bans. Not that would be constitutional anyway, but this kind of obstruction that's becoming increasingly popular in DC.

So maybe we'll take it from the top and work our way down. I mean, I'm someone who broadly has a very high level of uncertainty about what to expect. And I sometimes describe myself as an adoption accelerationist and hyperscaling pauser, because I do think that without question right now, the AIs that we have are way more useful than they are harmful. And I think I just gave a presentation to some business leaders today and to some students a couple of days ago. And the point of those presentations is like, you are failing yourself if you're not really embracing this technology, trying to figure out how to make it work to your advantage.

Absolutely.

And so I'm very much with you on that. At the same time, I do feel like we are really playing with fire. I do feel like this technology is unlike other technologies. And with that in mind, I really try to understand it on its own terms. So I'd be interested to hear your case for why you think the long-term risks are overblown, and then maybe we can reel back into the present and talk about kind of what you think. Sure.

So I sent you this article. It will be out at afuture.org/research by next Monday, which I believe should be well before you release the episode. And this main argument is that, you know, let's actually talk about what the long-term argument is here, because I think that people don't quite get it, especially here in DC. The argument tends to be that AI will become so powerful that it's able to kill all of humanity, and a separate argument that it will want to kill all of humanity. But I'm mostly arguing against the idea that it will become so powerful that it will— kill all of humanity. Would you agree that that's a correct summary of the long-term arguments?

Certainly that's the biggest worry, would be human extinction. And just to kind of ground the discussion in the opinion of folks in the field, I recently did an episode with Katya Grace from AI Impacts where they did the survey of, I think, 2,700 published AI machine learning authors, people who had published in the top 6 conferences in the last couple of years. Half of the people in that survey result gave a 5% to 10% chance or higher of human extinction.

Oh, I'm very familiar with AI Impacts. They're funded by Open Phil. They're funded by these people who already believe in the argument. And the survey in particular is an opt-in survey. And this is a very common tactic here in DC. I think, you know, you show that to someone in DC, they'll know what's up right away. And what's up right away is that if you have opt-in surveys, especially if you title it something like, you know, opt-in survey on whether AI will destroy all of humanity, then you'll have much more opt-in from people who already have a strong ideological commitment rather than, you know, basically normies who think, you know, this is a waste of my time. And this is a very clear, you know, very common political strategy, not just on AI, but, you know, this is something that every issue campaign does. And this is a level of literacy that exists in DC that I think DC people are actually quite competent in seeing past, but maybe less so for the public. I think that in general, this idea, I think that in general, this idea that there's going to be active work That, you know, the people who are working on this technology feel like they are actively contributing to the extinction of humanity. You really should not believe that kind of on its face. On its face, that I think seems ridiculous to most people, and it should be ridiculous to most people because most people are not suicidal. Most people are not trying to kill themselves. Most people are not trying to kill all of humanity. And, you know, if you have these kind of selection effects, then maybe you're going to end up with a sample size of people. You're going to end up with a sample of people who are skewed in that way. But most people in the industry are not like that.

I would just focus in on the— I mean, assuming we're not accused, we can accuse people of having whatever motivations, but I'm assuming we're not accusing them of faking the data. We still have some 1,350 But you still have a real number of people, 1,350 or so published in top conferences, machine learning researchers who are giving this nontrivial chance to human extinction. So I think that's still something that we definitely have to grapple with. The field does take this question seriously. Most people think it's not the most likely thing, but they do give it a non-trivial chance, right?

So most people do not. So even, even in this highly biased sample, even in this opt-in sample, you only found half of people. So, so let's— That's 1,300 people.

Yeah. So I'm— so give me the argument that I don't have to worry about it because I got 1,000+ machine learning researchers who are telling me that I do. I got others that are telling me that I don't. But if I'm, you know, now it comes down to, okay, sure, sure, sure.

This is, you know, this is a much smaller sample than, for example, like vaccine, vaccine scientists who are anti- Right. So you always get these kind of ideological fringes. But, but okay, let's actually take a look at the argument. So many cases, in many cases, in almost all cases, people who are famous in this kind of AI doom community believe in an idea called, you know, FOOM. This is, this is their term, not mine. I'm not just, you know, calling it that to make fun of them. This is what they call it themselves. Which is the idea that AI will become radically more powerful in a matter of weeks, if not sooner, that there will be a kind of recursive self-improvement. And what's very important here is that they have a very simplified model of these things where there's some kind of quantity called intelligence. If you increase this quantity called intelligence, then it directly feeds back into being able to increase this quantity called intelligence even more. And that's how they get this argument of recursive self-improvement. And it's very important to note, this is my main problem with this argument, is that it's a very non-empirical argument. It is not something that is trying to use data from the real world. It is not something that is trying to fit to any parallel. And in fact, is a kind of monotheism. It relies on the belief that AI is a technology that is unlike anything else that has ever existed in reality. Now, there's a kind of soft version of this that says, oh, AI is going to exponentially self-improve because it makes the job of AI scientists, human AI scientists easier. And then there's a hard version of it where AI is doing AI research on its own. So let's address both of those in sequence. So for the problem of soft AI self-improvement, almost all scientific fields have the ability to self-improve. You invent something like a microscope, you invent something like a particle accelerator. That is something that gives you the ability to do more science. And in fact, almost every successful company, almost every successful firm, and most successful research laboratories have to do this in some way. They're picking up competence, they're picking up additional skills, and they do have an increased ability to operate for some time. And then eventually they hit diminishing returns. There are diminishing returns in physics, diminishing returns in chemistry, diminishing returns in biology, so on and so forth, where the discoveries simply become harder to make. And this diminishing returns, Outweighs the kind of improved ability to do research. And when it comes to the hard version of kind of AI self-improvement, first of all, while there is some evidence that AI is able to be used as a kind of automation tool for current AI researchers, there's no evidence that AI is remotely close to self-directed research. At this moment, it even struggles on kind of very basic Upwork tasks, even the most cutting-edge startups do. And the nearest competitors, you know, are not even close to that. And there are also kind of real-life examples of these self-improving processes. Like I said, they're called companies. And those companies are able to spend resources to improve themselves, hiring, R&D, so on. And they're also very far from growing indefinitely. They may be, you know, very productive. They may be very economically impactful. And I do believe AI will be very economically impactful. But, you know, we're not close to seeing, you know, world domination by one company because as in all of these other areas, as in virtually or literally actually every area of real life, There are diminishing returns. So the summary is that if you look at a broader pattern of either companies, research labs, research fields as a whole, there is this near— there is this ubiquitous pattern. I don't need to hedge myself. It is literally ubiquitous pattern of diminishing returns. And if you model it that way, if you model kind of returns to intelligence, not as something that is essentially constant so that if you get an exponential increase in intelligence, you're just going to keep going, but rather something that's diminishing, something that's diminishing super exponentially as it has in all of these other research areas, then this idea that this kind of self-improvement will not run into these stumbling blocks becomes very clearly against The pattern of essentially all of scientific history.

We'll continue our interview in a moment after a word from our sponsors. If you don't already subscribe to Turpentine's industry-leading newsletters, like our new daily AI newsletter, Emergent Behavior, or Media Empires, you should. But that's not what I'm here to tell you about. The platform we use to power these newsletters is called Beehive, and it's excellent. First of all, it was started by the same early team who helped build Morning Brew into a $75 million newsletter business. And they built Beehive to offer that same powerful functionality to anyone sending emails, from essayists to business owners. The platform is beautiful, their text editor is intuitive, and they help you scale your audience with custom growth features. Beehive has powerful tools to help you monetize your content. You can easily launch paid subscriptions or pursue an advertising model. The Beehive platform will even connect you to premium brands to sponsor your newsletter. Not only do we use them, but thousands of the top newsletters in the world also use them. Like Milk Road, Blockworks, the Lindy newsletter, and so many more. Beehive's founder hooked up Upstream listeners with a sweet deal. Get 20% off for 3 months with code Upstream. Visit beehive.com, that's B-E-E-H-I-I-V.com, to get started. Hey all, Erik Torenberg here. I'm hearing more and more that founders want to get profitable and do more with less, especially with engineering. Listen, I love your 30-year-old ex-Fang senior software engineer as much as the next guy, but honestly, I can't afford them anymore. Founders everywhere are trying to turn to global talent, but boy, is it a hassle to do at scale. From sourcing to interviewing to on-the-ground operations and management. That's why I teamed up with Sean Lanahan, who's been building engineering teams in Vietnam at a very high level for over 5 years, to help you access global engineering without the headache. Squad, Sean's new company, takes care of sourcing, legal compliance, and local HR for global talent so you don't have to. With teams across Asia and South America, we can cover you no matter which time zone you operate in. Their engineers follow your process and use your tools. They work with React, Next.js, or your favorite frontend frameworks. And on the backend, they're experts at Node, Python, Java, and anything under the sun. Full disclosure, it's going to cost more than the random person you found on Upwork that that's doing 2 hours of work per week, but billing you for 40. But you'll get premium quality at a fraction of the typical cost. Our engineers are vetted top 1% talent and actually working hard for you every day. Increase your velocity without amping up burn. Head to choosesquad.com and mention Turpentine to skip the waitlist.

So I would grant that, and I think most of the AI safety community would grant that, There's almost certainly some S-shaped curve to the current AI paradigm, and it seems like most people think we're kind of in the steep part. And the question becomes, like, how far does the steep part go before that levelling off happens? And it seems to me that you can buy into all the arguments that you're making that this is not going to be an unbounded exponential. It's not going to go Foom forever. But you could still believe, or you could still worry, and I'm again pretty radically uncertain about this, but you could still worry that the point at which the S-curve starts to flatten off could be high enough that it could be something that's very hard for us to control. So what would you say to somebody who looks back at history and is just like, huh, in the grand scheme of history, It sure seems like what the great story of humanity is, basically, that we popped up out of nowhere. We had cognitive abilities that other species didn't have. We used those cognitive abilities to come to dominate the Earth. And basically everything else on the Earth, aside from bacteria, but certainly all the other megafauna, they sort of live now at our pleasure. And what's to say that this couldn't go the same way, except the new sharpest tool on the block, which maybe wouldn't necessarily have to be infinitely sharp or go foom forever, but could just stay on that steep part of the S-curve and flatten off far enough above us that it could just gradually outcompete us. That's not like a 5-minute story, but it could be a story that in geological or evolutionary time is quite fast. Because after all, isn't that what we did to all the other hominids and all the other megafauna?

So there's a version of the EA argument that the effective altruists, who are very closely related, there's a version of this argument of their philosophy of kind of long-termism, right? There's a problem that, say, has a 0.01% chance of happening in 200 years that is so disastrous that would cause the extinction of all humanity, that even if there's a small chance of it happening, you know, many generations from now, then it's worth worrying about because the kind of expected value is so great. That version of the argument I'm more sympathetic to. But I would say that the approach to that is a kind of wait and see. The main thing that I'm arguing against in this article is an idea that's presented that we will get kind of discontinuous, extremely discontinuous growth in a short time. And that's the reason why you would have to pause now instead of, say, pause at GPT-10 or GPT-20. It could be the case. that after basically repeated cycles of human development, of human-driven research, that we reach levels of superintelligence. But I don't think that that's something that's going to happen within, say, the next 10 years.

Okay, well, I'm still kind of wondering, what do you say to the person who says, fine, you think it's not going to happen in the next 10 years, The leaders of the leading developers of the technology are saying that they think it's fairly likely to happen in the next 10 years. The aggregate forecasts on sites like Metaculus, for example, have it coming in the next 10 years. We still have the 1,000+ researchers to contend with. So why should I buy that hunch and not worry about it? I mean, it seems like—

I don't think we're quite talking about the same thing here. The Metaculus forecast, for example, I believe is about AI that is able to automate— automates was 90% of economically valuable work, which is not the same as being able to take over all of humanity. Now, I have separate economic arguments for why I think that's unlikely related to the inelasticity of hardware demand, for example. diminishing returns in some of the techniques used to generate hardware improvements so far. There are kind of economic reasons why I think that those timelines are a bit short. But it's also clear to— it's also important to distinguish here the difference between kind of very economically valuable AI, which, while I think will still take longer, is still more of a kind of close, something that it's more likely to see in my lifetime, to something that's capable of taking over the entire world. I think that there's actually a huge gap between those things.

But there's not necessarily a huge gap between introducing something that will come to take over the entire world and where, I guess, this cat could get out of the bag, right? Like, Llama 3 is about to be open sourced. Once it's out there, that will never come back. Now, I don't think Llama 3 is going to take over the world. I do think it's going to do a lot of weird stuff and a lot of great stuff, but it is something that will never be put back in, right? Like, we're not going to ever be in a COVID-free world. We're never going to be in a Llama 3-free world. And it does seem that some stronger argument would need to be made that one need not worry about this because Even if it is a gradual process, you can hit the— it's like a gradual process. The sun's going to go supernova. It's going to happen. Fortunately, it's going to be a long time. But if we let some AI out into the world that begins a process that we can't control—

But the larger these models are, and we can only assume, or we should assume that they will get larger, then the more centralised inference cost there is.

Right.

So the more, the larger these models get, the more a kind of centralized entity is required to run them. And it becomes very easy to, it becomes a very easy target. This is something that's widely agreed upon, both among people who oppose this kind of regulation, but also among many EAs who do support this regulation. They have their papers which agree with me. I can link you one after the show which basically says hardware is the easiest thing to regulate because it will— it is something that you require large amounts of. It becomes highly centralized. The energy demand for these data centers is up in question as well. So I think that as you see the pace of development grow, it will actually become easier to shut down.

Broadly speaking, I think I agree with that. I guess I'm trying to— I mean, there's always a lot of different questions, and it's very easy to be kind of addressing different things. As you said, it's always a risk. It seems to me that, yes, the data centres are going to have to get bigger. An H100 runs at 700 watts. That's like, not that much, but you do need like a handful of H100s to run a Llama 400B, presumably. So it's not nothing, but it's also not like you don't need a nuclear plant to do that, right? I mean, that you can kind of get that.

For current models, not quite yet.

So you could definitely run a small cluster on a local generator to do even the biggest models today. The energy is not the main concern. If you want to power it at civilizational scale, you're going to be using a lot of energy. But my—

Yeah, I'm talking about quite a bit into the future here.

But I guess I think I want to come back soon into kind of nearer term, like, what do you think the rules should be today? But I don't hear anything from really anyone ever that I find compelling in terms of why I shouldn't worry about the big picture risk. I just look at macro history. I see we humans dominate the planet because of our superior cognitive abilities, and we've driven lots of things to extinction. And I don't see any law of nature that says that that couldn't happen to us. And in fact, I think it's actually quite a misconception that this is sort of an Abrahamic God notion. I would frame it for myself as like, it's actually the lack of an Abrahamic God. There is no Abrahamic God to protect us, and that's why we very well could go extinct. And I do wonder to what degree we are running that risk over even the near term. We might think, yeah, the stuff we're putting out in the near term isn't that crazy, but once it's out there, it's free to mutate in the wild. It's free to sort of recombine. And yes, it's not going to do that on its own right now, but it's sort of something we can't put back. And I do feel like we are We are climbing a ramp of risk in parallel with climbing the ramp of utility. And I think we should at least be clear-eyed about that. But what I'm not hearing is an argument that there's really nothing to worry about. What I'm hearing is like, well, it'll take a while. And I would say yes, but we might be tipping over into something we can't take back sooner than we realize. Right. And that's really the problem, is when have we crossed an event horizon that we can't escape from? even not knowing that we did, or even though it may take decades or who knows how long it could take. But it seems like we just don't have a good enough model of what we're even dealing with to dismiss all worry.

Yeah, I mean, I'm not trying to dismiss all worry at all. Certainly, I think that we've come to a bit of a compromise here where we're kind of looking at the quantitative issue, right? Like I said, If it comes to the radical uncertainty argument, I mean, I want to put dampers on the near-term version of that, but the long-term version of that with kind of many generations of human development, yeah, that could be something that happens. But that is, I think the idea that this is something that becomes a runaway process in the near term, Uh, is something that I push against. And I also think that, uh, certainly in the kind of near and medium term, there's much more of a comparative advantage thing going on here. There's much more of a specialization where there's going to be tasks that AI is better at. There's going to be tasks that humans are better at, right? AIs are already better at chess, for example. Um, and that is something that we can just live with. that things are not as correlated across domains as people may expect, that there are many technologies in the past, say tractors or other farming equipment that has in today's or in the context of that time automated most of human labor and people find other jobs. I'm generally optimistic about that. I do think that there will be a time to reevaluate That this is, if we are not stopped by kind of physical constraints, if we're not stopped by hardware constraints, which I believe there's some reason to think that we are, if we're not stopped by kind of diminishing returns in science, in the ability to improve these algorithms, that if it does keep just kind of growing, then there will be a point to reevaluate. I think we just disagree kind of quantitatively on how early that will be. So we can jump in and welcome Erik as well. We can jump into the kind of affirmative policy as well, if you want.

Yeah, I think that sounds good. I think I was just kind of mulling over your last comment, and I appreciate the notion that there could be a point to reevaluate. I mean, how would one— maybe one more question on this before going back to the nearer-term more practical stuff. How would one determine when that point happens? I don't really think we're talking about truly quantitative things. We're all kind of guessing, right? So how would you make that decision?

No, there are quantitative metrics of this, right? For example, ability to autonomously Perform economically valuable tasks, that's a real indicator, right? You can look at some of these evaluations, SWE bench, for example, software engineering bench, for example, which I think, and people kind of criticize the methodology for this. There was some startup, Cognition, that released a tool that claimed to be able to do 14% of these tasks. There have been some criticisms since then, but basically, like, in specific domains, once you get, like, really into the industry, there are ways to measure this. And you can imagine kind of aggregating this in order to create a picture of all economically valuable tasks, right? So I think that as these things get closer, let's say, for example, we find one kind of broad field where there is autonomy in that field. And I do think it'll be the case that we get one of these fields kind of much faster than the others. Say art, for example, right? Say like visual art, for example, that if we get, you know, get to the point where DALL-E is so good that it can kind of create these really like precise, you know, completely human displacing changes across the entire sector, right? Then that'll be something that's very clear to measure and it'll show up in the kind of economic statistics. So I'm not sure if I agree with that assumption that it'll be difficult to measure. I think it's kind of where I do agree with you is I think that that can be hard to conceptualize today.

Right.

But I think that once we start to get examples of it replacing some economically valuable work, which I do think it'll at least replace some, then we'll be able to measure that, and then we'll be able to draw comparisons to measure similar things that might potentially happen in the future in other fields. This is this kind of typical economic process.

What do you think of things like the Anthropic Responsible Scaling Policy?

I mean, to the extent that they're kind of doing additional research, I mean, like, that's their prerogative, basically, right? To the extent that they're doing this additional research before scaling, I mean, it seems like their product, Claude 3, I've used Opus. It seems great. It seems like a good product. It seems like that Their, their kind of interpretability experiments have not really detracted from the product. They might have even improved it with things like constitutional AI. So yeah, I really don't want to seem antagonistic kind of by default to a lot of these fields. I think that a lot of the time, or at least some of the time, I don't know what the exact ratio is, but some of the time, these kind of mechanistic interpretability experiments, these kind of superalignment experiments for the OpenAI version, are just interesting scientific results. And is that the optimal allocation of capital? I don't know. From a kind of theoretical economic standpoint, probably not. But it's positive-sum science. It's getting more information. It's things that are basically useful from a research perspective. So I have no reason to— I'm very positive on it.

So I guess this maybe is a good transition to the newer term regulation, because the way they frame their stages of responsible scaling, they have like 4 risk levels where they've tried to get out in front and say, okay, these are the things that we think might be a real problem in the future. And folks have probably heard, of course, there's cybersecurity risks and there's potentially biotechnology risks, and there's self-replication is one of the other things that they're really looking for and trying to to determine, like, can the models do these various things? They basically say, if we can detect that they can do these things and we don't have a way to present it, or to prevent it, I should say, then we have to stop further scaling and essentially pause the mainline development track. Now, I wouldn't put any words into Anthropic's mouth, but this sure reads to me like a proposal for regulation broadly. And I wonder what your reaction would be to that sort of framework becoming not just something that one company is volunteering to hold itself to, but the sort of regulatory regime that would govern all such research. If your system can do this and you can't prevent it, then you have to stop your scaling. Does that seem like a reasonable place for us to go?

Yeah, the problem with answering this, I'll put it this way, right? The idea that like if your program does X and we can measure that it predictably does X and we can't prevent it, I think that that's a good policy for some version of X, right? The question is like what that version of X is. And there are various laws being passed that basically have varying versions of X, right? So in terms of Anthropic's specific policies, yeah, I was just scrolling through the document and I'm not completely sure if they say what the evals are. There's this link to the ARC evals homepage, and I have not read what all their evaluations are. So I can't say because I don't know what X quite is yet, right? But I certainly think that there's some version of X where that's true, many versions of X where that's not true. And I think that's in DC, and this is kind of moving away from some of the long-term risks, and more so looking at short-term risks. There's a lot of Xs that people are trying to insert that would not be productive. And I can go more on that if you want.

Yeah, I'm really interested to hear what you think the rules should be. And I, by the way, would totally agree that there are many values for X that would be ultimately counterproductive in terms of just sacrificing too much upside from AI at too little benefit. Certainly, I am not one who wants to police speech, so some sort of offensive speech would not be a good value for X in my mind. I would I'm not sure. I don't have a specific place exactly where I would draw the line, but I would definitely, to establish the common ground, that policing speech is not the sort of thing that I would want to get too aggressive about.

Yeah. So I'm very excited about the potential for modernization. We're going to release a— actually, I don't know when this will be researched, so it may or may not be out by then, but we'll have a plan for kind of the broad strokes modernization of government. And that's something I'm very excited for, that this is an opportunity, that this is a kind of reset button to make many of the processes in our ordinary life much more efficient, both on a kind of industry level. But I think, and here's where I think decision or kind of organized political effort is more important also on the public sector side, because this is our tagline for the proposal, is that government cannot catch up by trying to stop its competitors. It has to be able to modernize on its own as well. Certainly, I don't think the government can substantially slow down China, at least not indefinitely. And so we're in a situation where I think many of the pain points that people have when dealing with government, both people, ordinary citizens and people working with government or directly kind of working, contracting with government, do have a lot of pain points that are directly addressed by AI. And so that's something we're going to be Encouraging. We're going to take a look at specific Office of Management of the Budget rules. I think that some of the current directives that were passed very recently are kind of actively obstructive in the government's own adoption of AI, which I think is exactly the wrong direction that the government should be kind of coordinating with developers to adopt more AI. And certainly current levels of AI in a much safer way. And we'll have more detailed policies about how to adopt that. I'm not sure what level of depth you want, what kind of level of analysis.

Yeah, I think we could jam on a vision there that we would largely share, certainly in the near term, in terms of government process reform. Fun fact about me, I live in Detroit, Michigan, and I once offered a friend of a friend who works at the city government to just volunteer on a volunteer basis to create a little program to help them review case files that they have for people that are sort of receiving some amount of help from the city, but potentially qualify for different benefits or programs or whatever. They've got a stack of case files, and as you'd imagine, right, they're not getting through them all, certainly not asking all the questions that you could ask if you brought an AI system to bear. I'm still waiting for my phone to ring back on that opportunity. And I would agree that, yeah, I mean, anything we could do to spur what the great Zvi Mashowitz calls the mundane utility and just bringing day-to-day convenience and efficiency to the life of the average citizen, that makes total sense to me. Notably, I don't think we need to scale up lots more generations of AI to make tremendous progress there. I would say GPT-4 would take us very far, and certainly a GPT-5 worth its name would take us even farther very quickly and easily. You probably don't need a GPT-6 to make a government services regime just way better. But I guess I'm most interested in maybe going back to What do you think the rules should be today? We basically have almost no AI-specific rules, right? We have kind of just the general background rules of what's a crime and what's not a crime. And the law is largely silent on AI. We have the executive order, which you can elaborate on, but my general understanding is that it had one big rule, which is you need to notify the government if you are doing a large-scale training run at 10 to the 26 flops or more, and otherwise sets up a bunch of kind of initial reporting and fact-finding sort of motions going. But we still seem to have this question very much unanswered. Do we want to have AI-specific rules? And if so, what should they be?

So I think that the right story to tell is to start with The executive order. I think that if you're worried about existential risk, you're absolutely right that the 10 to the 26 is the kind of the highlight for you. But if you're worried about the kind of economic damages, if you're worried about the political consequences, there are implementations throughout the Biden executive order that should be very, very worrying to you. I'm not sure how high of a priority speeches, for example, But one example is the directives for diversity, equity, and inclusion. Of course, this is a very and only increasingly controversial topic, but this is something where there is increasing motivation that is kind of distributed through the executive order into individual agencies such as National Telecommunications and the NTIA, the National Telecommunications and Information Administration, or the Office of Management and Budget, which I talked briefly about already, or the FTC, that create increasing just barriers to entry to new companies, or at least threaten to with reports or recommended actions that they've developed so far. And, you know, someone who is completely committed to the existential risk argument would say, okay, these problems are are just a rounding error. They don't actually stop the kind of AI that will kill everyone. But I think for most people, this is something that shapes, you know, shapes their political lives. Certainly for many startups and, you know, future customers of those startups, this is something that shapes the kind of financial viability of their company. They can't necessarily pay these kinds of compliance costs that would be necessary if these directives became law. So I think from a short-term perspective, there is a lot in the Biden executive order other than just the 10 to the 26th. So how do we carry on the story is, I think that no matter what, this is just a political reality. You know, I sympathize a lot with people who say, you know, you shouldn't be taking this approach to AI. You shouldn't be so pessimistic. You shouldn't prematurely circumcise the kind of range of research. before, before this is really even a developed technology. But the political reality is that there's a lot of motivation in D.C. to do, to, you know, quote unquote, do something about it. And what that something is right now is up in the air. So our approach is to say, here is a kind of clear and minimally damaging way to satisfy some of your demands. So our approach in that sense is to identify a single point of intervention. So why is that important? A lot of, I'm sure you know this, Nathan, a lot of AI companies work specifically in one part of a kind of development pipeline, right? You start with hardware and data and assemble that with infrastructure. You have the initial pre-training phase, which is what a lot of these big foundation model companies do. You have various startups that spin off that, that take that product and spin off of it a fine-tuned model that essentially uses these less intensive training techniques to produce a more specialized model for a given task or a given tone or some of these more specific concerns. And then you have another layer of scaffolding, of prompt engineering, of customizing. And you see this across not just language models, which were the main subject of what we were talking about already, but in models in biology and models in logistics, so on and so forth. So you have this great diversity, true diversity of startups, and many of them would be impacted by, say, regulation that is intended to target identity fraud or intended to target deepfakes, so on and so forth. Because the, for example, the NTIA released a report right now that I think the exact quote is to ensure accountability across the value chain. Now, what does that mean? It means that not just the end product distributor, the kind of person who would produce an AI model that someone could actually use to possibly commit identity fraud, not just that person, but people throughout that development process who have not really like no ability to either prevent or stop or actively encourage a model to do this as long as they don't have control over basically the final product, the last stage in terms of customization. All these people in between are needlessly affected by this policy. If you have— it would do exactly the same thing if it were only restricted to the single point of intervention, this point of intervention right before basically getting to the customer where you're doing some level of specialization. And focusing on that specific kind of Focusing on that single point of intervention, we argue, is going to be just broadly much more effective way and notably not just more effective for the companies, right? Not just saving the companies a lot of money, but also saving the regulatory agencies themselves a bunch of money because many of these slow-moving regulatory agencies, if you can, you know, if you can cut down their burden by an order of magnitude, they'll actually be able to do more due diligence. Right. They'll either spend less money or with the same amount of money can reduce, can review things 10 times more quickly.

So if I understand correctly, this amounts to regulate the app layer. It would be like the shortest version of the—

Yeah, that's a fair summary.

So it would say if you are like, let's take a scenario. That I'm working on a project right now called Red Teaming in Public, where some volunteers and I are going out and testing a bunch of already deployed available products just to see, like, are there any guardrails in place? What will happen? Pretty interesting findings. I won't spoil them too much now, but—

I'm sure that'll be an exciting episode.

There will definitely be some theatrical examples. We're focusing, for starters, on calling agents. These are apps where you give it a prompt and it calls someone else to have a conversation with them. Then coding agents, which are like, you tell it kind of what you want and it codes stuff, but it also executes that code. And then also kind of deepfake generators, which is a little bit of a different era or different area. Certainly, you're correct to say there's a lot of layers in the stack. that go into something like that. If I am a user and I prompt a product to call some person and spearfish them for their mother's maiden name or whatever, and it does that and that person is harmed, I'm, you know, as the user who's like doing the crime, I'm obviously responsible. The app that made the calling agent, you're saying, would be responsible?

But not necessarily. So, so in the specific case of identity fraud, I do think the responsibility should bear with the person who is actually committing the crime. Right? You see this, you know, widely precedented with software. There is automated, you know, emailing software, there's automated phone software even now, right? Or even in the pre-AI era. And typically what the structure of that would be is that if you commit the crime, you do the time. And people who are producing basically neutral tools do not, because they're producing neutral tools.

Yes. This is where I think we're going to be in some very unfamiliar territory very soon, because the tools, I wouldn't necessarily call them neutral, I would call them dual use. And I would say that they're also not inert. All tools through history had this sort of property of not doing anything until you do something with them, whereas the new crop of tools are also being actively developed and marketed as agents. So the idea is not now that you are going to pick up this calling device like the traditional phone and talk through it yourself, in which case, sure, I would say the phone company should not be responsible for me calling and committing fraud over the phone. But if I am the developer of a calling agent, something that can take a short string of instructions and then go make not just potentially one call, but calls at scale targeting all sorts of people to do something where the actual conversation is happening between the AI and the unsuspecting person on the other end of the phone. It seems to me that we do have to have some standards in place for those kinds of companies. We can't just say, oh, yeah, well, hey, you have no responsibility because somebody came in and put a bad prompt. You have to do something there, right? I mean, I would say there's got to be some rule to prevent or to make it clear to the companies that you can't just have no precautions. I also think it's a fine line to say, what's a reasonable amount of precautions? We have all these sort of legal standards historically, right, around did you do what is sort of best practice in your industry or not? Like, negligence is not always super crisp or super binary, but it doesn't seem like we could say about these agentic things. And again, they're being marketed as agents. It doesn't seem like we can say that the companies developing them have no responsibility for what they do.

Yeah, I have a lot of empathy for that kind of position because I do think there is this broader sentiment. And you're right that it's reflected in marketing. that this is sort of an extra personal extension, right? If you think of an email list as an intrapersonal extension, this is actually something that I've written, not really a research paper, but just kind of a statement, an op-ed. about where I do think there's this fine line between intrapersonal, like within yourself, an extension of your own kind of thinking and things where you're being interactive with other people or these things are kind of interactive with other people outside of yourself. Right. So I understand that position there. There's certainly a trade-off of it, right? Like I said, the political reality is the case where almost certainly I think that there are more people who basically take your position. So, but yeah, I would say that, I mean, there's what is, you know, politics is the art of the possible. There's what I personally believe, which is a kind of more optimistic and more experimental approach. And then there is what's politically likely. So I would say that this is something where there's a lot of nuance on. where it very well may be the case that there's regulation passed that basically agrees with you and has some kind of accountability layer to the company itself. And from that perspective, I would say that it's better if it's a kind of narrow accountability, whereas as opposed to a kind of widely distributed one.

Yeah, this is one of the things we plan to explore with this project is just to what degree Are there basic precautions, guardrails, system checks in place to prevent this sort of thing? And to what degree is it just a total wide open west? I don't feel like I have the answer at this point as to what the standard should be exactly, but I would definitely say if you have taken no precautions and you've put something like a calling agent out in the world, then I think you should have some responsibility. If you have put precautions in place, but somebody manages to come through with some intricate jailbreak or hack or whatever, and then use the system in a way that you did try to prevent, but your defenses ultimately failed, then I think that's like, I'm much more sympathetic to those companies. And certainly with some amount of effort in today's world, I would say they should be able to say, look, we did our best, and this person is really the one that's like, clearly at fault and look at all the defenses, you know, that, that they worked around in order to do this with our, with our tool.

Yeah. And I would say that, that, that, that kind of philosophy makes sense. Where it may run into problems is that there's this well-known phenomenon called the alignment tax, which I think actually comes from many of the people doing alignment, where they find that these techniques such as RLHF reduce the performance of other tasks other than the thing that they're targeting.

Right.

So there may be additional performance costs to this that are not easily solvable. And there are caricature versions of this. I'm not saying that this is the inevitable consequence of what you're suggesting, but there are caricature versions of this like Google Gemini, Of course, that also goes into another ball of kind of speech restrictions that you don't necessarily agree with. But the kind of more general point is that there are unexpected consequences to some of these restrictions, right? That there are trade-offs at play. And like I said, this is something that can be kind of thought about quantitatively, where you can figure out what you can draw the line, where the kind of science the kind of state of the art matters, how much kind of protection can you get with certain other amounts of trade-offs, right? There's obviously some kind of cost-benefit ratio there that makes sense. I just worry that from a kind of social political standpoint, that these judgments are not the ones being made. I would, you know, I know Many of the people, including people who are kind of opposed to me, who are more on the EA side, who do actually think about this every day, right? Who think about the trade-offs at play. But I would say that when that gets distilled to a regulatory level, either at the level of regulatory agencies or to the level of elected representatives, that it becomes much fuzzier. It becomes much less a question of trade-offs and much more a question of kind of positioning. And that's where it gets very dangerous. But I think that, you know, if we were running, you know, the NTIA, right, we could probably get to a point of agreement where, you know, we agree on basically like 95% of things and the remaining 5% are actually not so determinative. So there's a kind of theoretical world where this is the case, but in practice, we might be quite far from that world.

Well, I'm available if anybody wants to put us in charge of the agency.

Yeah, well, we'll see.

This is maybe not a natural final question, but it's the one that's at the tip of my tongue at the moment, is how to think about open sourcing of models in the current regulatory regime and whether or not there should be some new rules there as well. It seems like if I was going to make— if I had to kind of look at the current law and say, does any of this apply to open sourcing? I might apply pollution law as sort of the most obvious thing where it's like you're just putting this thing out into the the commons. And in this case, it does have positive effects, but it also will have negative effects. And should you be responsible for that? Is it appropriate for companies to be doing that? And how does that change as the sort of power of the things that they're open sourcing advances? To be clear, again, I don't think Llama 3 is going to be the end of the world. But I do think we are going to see a bunch of pretty obviously directly harmful things coming from people doing sort of malicious fine tunings of Llama 3, starting with the North Korean government and working your way down to all sorts of just bad actors. It seems— I mean, I don't want to apply pollution law to that sort of thing. It seems like we probably need some new rules. But how do you think about open sourcing and what, if any, sort of rules, standards, expectations should apply there?

Right. So I'm hugely supportive of open source. This is where I think you just need to sit down and do a cost-benefit analysis. And yeah, I mean, I'm not sure I need to convince you of this, that certainly with current technologies, it's overwhelmingly positive. Once again, I would say that, you know, the punishments should be concentrated on people who actually commit the crimes. There are currently penalties for impersonation, for example, unlawful use of likeness. And, you know, there can be mechanisms to make that penalty greater for if you use AI that kind of fit within the current legal frameworks. I would say that for the publishing of open source code itself, not necessarily any particular use of it, but just the publishing of it is very likely protected under the First Amendment under, I think the precedent is Bernstein. Software is regulated as speech, kind of all of the same protections and exemptions apply to speech as applied to the software. Certainly this is true of data. And then hardware is where it becomes more, where there may be more kind of regulatory exposure, but that's not too related to what we're talking about.

Yeah, I wonder, I would agree. You don't have to convince me. I think the benefits of even the AI safety community broadly has I think concluded the consensus, even among the most x-risk concerned, is that the open sourcing of Llama 2 was on net a good thing, if only because they're doing tons of interpretability research on it. Exactly.

Yeah.

Well, let me try to blur the line one more, because here I really am like, man, I don't have the answer. But I do think we are breaking current paradigms remarkably quickly. You may have seen this recent model called Evo that came out of the Ark Institute. And essentially what it is is a language model, but trained entirely on DNA sequences. So it's a 7 billion parameter model trained on 300 billion base pairs of mostly bacteria DNA. You can go to Together AI. I just did this, create an account, and you could put in a couple TGCs, whatever, and then just hit go and it'll just generate you sequences. And what I think is really interesting about that is it's like, there's been a lot of talk about biorisk and sort of right now it's like, well, the language models, they maybe are a little bit more helpful than Google, but they're not that much more helpful than Google. But then you've got these other things where it's like the same technology but trained on a different dataset is now actually just generating you genomes. And that is like, Okay, that's way more helpful than Google if you want to. And again, it's all dual use, right? So it could be good, it could be bad depending on what you're trying to do. It doesn't feel like speech to me. Generating DNA sequences doesn't feel like the sort of thing that the framers had in mind. They obviously didn't even know about DNA. And so I'm like, again, if somebody is going to make something like this, and just drop it into the open in a way that is an irreversible decision. Is that the kind of thing that we should just be like, well, we got a First Amendment, so I guess that means we can't do anything about people just putting these sort of whole genome generators out there? This one is early. I don't think it's going to be the end of the world either. But you can imagine a couple of generations. It's already superhuman in this domain. You can imagine a couple of generations from now, things could get really weird. It seems like we do need some new paradigms to kind of govern these fundamentally paradigm-breaking technologies.

Right. So this is something that a friend of mine, Dean Ball, has written on. I think the best way to think of it is a kind of cost-benefit analysis, or a kind of relative risk analysis, right? where you're asking the question of like how many more times more difficult it is to kind of go from knowledge to action compared to going from nothing to knowledge, right? So maybe there's an improvement in terms of going from nothing to knowledge in terms of viruses here. Maybe it's somewhat more convenient than Google. But for example, like the smallpox genome is just available online, right? And that's something that would be very destructive right now. So right now, I think that the main bottleneck here is overwhelmingly in terms of acting, right? Synthesizing a virus is much more difficult than obtaining the genome. In general, it's a kind of action problem, not a knowledge problem. And if you change it so that it's— if you change it so that, you You know, it becomes even easier to solve the knowledge problem. It's maybe, you know, 100 times easier to solve the knowledge problem, but it's basically still as difficult to do the— to solve the action problem, then I don't think there's any kind of substantive change in the number of bioattacks, right? And there's kind of pretty active Research into this on a policy level, in terms of not just bio risk but kind of nuclear risk. Does this increase the chance of you know Iran doing nuclear research? This is something that there's a lot of work on. So so I would say that yeah, I actually maybe worry more in the other direction that there is so much work on it. that there's a risk of going too far and doing something like a new TSA that is much more dangerous. But I think that there are approaches here that are mostly targeted at making the action problem even harder, which I think is actually the correct approach.

GDPR for AI is what I say when I want to strike fear into the heart of the AI app developer.

Yeah, that's just disastrous. GDPR itself is disastrous. I had an old article in Pirate Wires about this.

I wouldn't say it's disastrous. It's really annoying, but it's like, you know, I've clicked that button a lot of times, but my life is still fine.

Sure, sure. It's not causing human extinction, but it's very economically damaging to the European startup environments. But yeah, it was— anyways, it was great being here. This was a fun interview, and I'm glad to have some of those responses on record.

Cool. It sounds like you're out of time. I would love to get your thoughts on structured access as a paradigm, but if you don't have time for it, then we can just leave it there.

Yeah, let's do another one.

Nathan, if you'll humor me before we wrap, one question I have for you is, did you see Dwarkesh's interview with Mark Zuckerberg and any reactions to it?

Yeah, clips only so far. I think Zuckerberg is a very reasonable person ultimately, and a couple of the clips that I saw did suggest that there's a lot more— I think this was characteristic of the last hour with Bryan as well, that there's, when you really dig down into the fine points, there is a lot of room for common ground still. And that's one of the big things that I try to embody is the sort of middle space or the idea that there's so much uncertainty all over the place that we should be able to come together on some reasonable stuff. I think we were able to do that largely over the last hour, Bryan and I. So that was good. And I think Zuckerberg is very much like that, too. I don't think he's overly ideological to the point where he would be reckless. And I don't think— I wasn't necessarily sure how ideological to expect Bryan to be coming in, but I also found that Once we kind of got past the initial, like, who's on whose side, and this is what the doomers say or what the EAs say, but really just got into the substance of the matter, we were pretty able to find common ground. So the clips that I've seen from Zuckerberg were like, we'll continue to reevaluate. If there are things that appear to be dangerous and we don't have a way to prevent them from happening, then we'll have to reevaluate our approach and we won't open source it. As of now, I think that's where he's headed. I certainly hope that we will see breakthroughs of the sort that mean, like, hey, these problems could be solved, and I certainly hope that they are. But if you look at the survey results coming out of the machine learning field, to the best that we are able to measure the expectations of the field, people are not really expecting problems to be solved in the near term. So my best guess would be that as they get into Llama 4 or possibly Llama 5, they're going to start to see things that do hit some of these thresholds from the Anthropic responsible scaling policy where they're like, okay, shit, it kind of can do some of these things now. And therefore with Anthropic, they say they will stop their training at that point. With Meta, I would think hopefully they would at least stop their open sourcing until they can figure out what to do about it. So I honestly have a lot of respect for Zuckerberg. I think he's, for somebody who has— these are my contrarian opinions, tend to be like the people that everybody shits on are actually good people. And I think Zuckerberg has navigated an extremely difficult situation for a long time with really a pretty remarkable track record. Not to say that mistakes have not been made, but I think it's like I say this about kind of a lot of these tech leaders. It's hard for me to imagine somebody doing a ton better job, and it's really easy for me to imagine somebody doing a lot worse job. So I would say that my general sense is that the same is probably true in AI, and that when things really get turned up, I would trust his judgment about as much as I would trust anyone's judgment to handle such a situation well.

I agree with everything you said there. I was surprised at how little he seemed to know about the AI safety debate, though, in some ways not surprised in that he's a busy guy and he's focused on the here and now as opposed to you know, things that are more theoretical, but it seemed like Dwarkesh was educating him a little bit on, on some of the, some, some of the debate. Um, I also want to ask you one other related question, which is you don't go as far as Laron Shapiro does, our, our friend who wants to have like an off switch basically, or, or do you? Like, he's like, we, we should make sure that everything has an off switch and that, well, you know, can be potentially dystopian, is less dystopian than a world world of sort of where AI can sort of go rogue or you run away. How do you respond to that?

I think there's some version of that that very well could make sense. I mean, it's kind of an underspecified thing, right? Like, what is being switched off and where is that switch and who can flip that switch? I think all those details really matter for a proposal like that. It's like, You can't really evaluate it in the sort of 3-word version of like an off switch. Like, okay, well, sure, but tons more there that we would need to flesh out to determine if it's a good or bad idea. And there's also just like the dynamics, right? I mean, this is a digression you can cut if we want to, but I'm reminded of the difference between the American auto companies and the Japanese auto companies. Phenomenal This American Life episode about this, where the Japanese companies, one of the ways that they maintained really high quality was that any worker in the plant could stop the line if there was a production problem. They would pull like a lever, the line would stop, they would fix the problem, and then they would resume production. And this led to good quality cars. At the same time, in the US plants, you had the tension between labour and management was so bad that management was like, we could never do something like that because people would be pulling that damn thing all the time and we'd never make any cars. So they just accepted the fact that they were going to have production quality problems rolling off the line and individuals on the floor couldn't do anything about it because they didn't trust those same people to not sabotage the situation. So it's not just the the nature of the switch, but it's also like the social context that it's embedded in that probably determines if it's a good idea or not. But that off switch worked well for Toyota. They were able to create better cars and become the number one automaker in the world with that system in place. I don't think it has to be dystopian if we feel like we can't do that in such a way where we don't have people sabotaging the economy on an ongoing basis, then it might not be very viable. But I think there is at least a version of that that you can imagine that could make a lot of sense, right? I definitely think that multiple layers of defence against a possible runaway AI future are in order, and this would be one of them.

That makes sense in theory. Have you seen any places where you have strong disagreements With Liron, or are you basically kind of where he's at?

I'm not even sure I can fully accurately say where he's at. My sense is that his estimate of how likely we are to go extinct from AI is quite a bit higher than mine. But I also feel like mine is high enough that I take it very seriously, and I'm not sure that leads me to super different positions from him. I think the Anthropic responsible scaling policy is a really good thing. And to set up these sorts of pre-commitments to these are the things we think could be the most damaging behaviours, here's how we're going to look for them, here's the standard we're going to have for ourselves, and here's what we have to be able to prove in order to keep advancing this technology. That seems like a pretty good paradigm. I think he also would agree that that's certainly better that it exists than not. He might just think that the chances of— and I don't know if he would agree with this or not, but certainly some out there would say, well, the chance of getting it wrong or confusing ourselves or the AI becoming deceptive and tricking us along the way and hiding its true capabilities is a risk that we can't take. I mean, this is kind of like a Bryan-type comment, but that to me feels just like outside of the set of feasibility anyway. And so I'm like, I don't know, I don't find too much to be gained in advancing the argument that Anthropic's Claude 4 could trick all of them into believing it's safe when it's not. I mean, apparently there's an asteroid that's about to fly by the Earth that has a small chance of hitting a keyhole that would then turn its trajectory toward actually hitting the Earth. So we're definitely not cosmically safe. And I think there's some amount of total off-model risk that we just can't get around kind of no matter what. So I'm willing to tolerate a little bit, but then I do think being very prudent about all these sorts of things is like the least that we can expect from the companies that are developing it.

And here's my last question. Are you sympathetic to the idea that we find a way to stop everything almost, you know, Ted K style, or we just kind of accept that on some timeframe we're entering this sort of like, you know, AI, you know, somewhat post-human era? In the sense of like, and, and with that worldview would imply is like a skepticism that we're going to figure out alignment in any real way once we fig— once sort of the bottlenecks to AI being so much more intelligent and widely accessible than, than humans are comes to, comes to pass. Are you sympathetic to that, that notion, or do you have a strong or reasonable belief that we will be able to figure out alignment or superalignment or whatever that means, or some middle ground over the next 50 to 100 years or so?

Yeah, I mean, hopefully a lot faster than that. I guess broadly, I've always been super pro-technology. Even at one time, I might have been fairly described as transhumanist, maybe even still, depending on your definition of transhumanist. I'm not one who wants to freeze history. I'm not one who thinks we won't find meaning if we don't all have 9-to-5 jobs. I'm not one who thinks that death gives meaning to life or that there's something fundamentally right about the current maximum human lifespan. I would like to live longer than I'm probably going to live if there's no major advances in technology. So I don't want to die anytime soon. I would like to live indefinitely. And that's part of why I think most people feel that way on some level. The old Peter Thiel thing about death is like, it's inevitable, but it's not going to happen to me, is like a very weird cope. And I think most people would take the life extension pill if it was offered to them. So I think we're going to keep moving, and I think that we should keep moving. Professor Mike Levin had a great comment on this, too. He's like, it would be really kind of sad if, like, 10,000 years from now, we were just the same that we are right now. You'd hope that we would make progress. We made a lot of progress in the last 10,000 years. Certainly don't want to go back. So I hope that the next 10, 100, 1,000, and 10,000 years also bring lots of great things. And I see AI as kind of an inevitable part of that. But it does seem like we are headed through another kind of possibly like a great filter sort of thing. We've been in the nuclear weapons era for the last whatever, 80 years now, where maybe 70 years, you might say, where it was like the potential for truly destroying civilization exists. And I do kind of see AI as like another similar thing where if we don't handle it well, it seems very plausible to me that it could totally get out of control. But so far we've handled the nuclear thing well, and long may that continue. And Hopefully with good judgment and a lot of careful science and not succumbing to AI arms race dynamics, we could figure out a really good AI future as well. I certainly don't think that is impossible. In many ways, AI is easier to figure out than biology. We've made a lot of progress on biology. We only are now getting the AIs to be able to study, and progress is really good. Mechanistic interpretability, it's not expected to have breakthroughs that solve everything in the next couple of years. But I can't keep up in terms of new podcast episodes with all the interesting mechanistic interpretability results. So I definitely have cause for optimism that we can figure it out. I think that the traction is there, that, hey, really good progress is being made to understand what's going on inside these systems, why they're doing what they're doing. Next step, even proto, not really effective, but like some ability to even control them, to detect, is this thing in a certain— is it entering into deceptive mode? And to do something about that. Those things are on the horizon, at least as breakthroughs that we could achieve. And it's also amazing, the AIs understand human value in a very real way. That is something that if you go back 10, 15 years and talk on the on the forums with the Eliezer crowd at that time was like, we're never going to be able to communicate our values to these systems. How would we ever hope to do that? And now here we are, and it kind of popped out of a language model, kind of with, like, you look at Claude 3 and it's like, it's pretty damn ethically sophisticated. I would say it's more ethically sophisticated than your average person. So that's an incredible achievement and again, should give people, I think, a lot of cause for optimism. Certainly, I don't get the argument that there's like 90% plus chance of doom because I just look at Claude 3 and I'm like, damn, it's arguably more ethical than I am. So if that's true, then we at least have some room for, or some cause for optimism. Yeah, I just think we're in a tough— I think we're in a critical phase, a critical era in which the how this happens could really be determinative. I think it's one of these things where a lot is probably pretty contingent. What kind of super powerful technology gets developed first? Does it get released into the wild? Do people take the time to study it and really see if they can't prove or disprove that it can or can't do certain things? Those questions, I think, probably end up mattering a lot. And that's why I'm kind of in the podcasting arena, is to try to steer people toward making those reasonable choices, not to shut it down, but not to race ahead either. Because again, I've said a million times, it does seem clear to me that there's no law of the universe preventing us from going extinct, and we're definitely playing with a new kind of fire.

That seems like a good place to wrap. Thank you for having the conversation with Brian. It's very timely as I'm going to go on the Bankless podcast in a few days and talk about some AI safety, some EA, some e/acc stuff. So I appreciate you having the conversation with Brian.

My pleasure. Thanks, Erik.

Upstream with Erik Torenberg is a show from Turpentine. The podcast network behind Moment of Zen and Cognitive Revolution. If you like the episode, please leave a review in the Apple Store. Hey everyone, Eric here. At Turpentine, we're building the first media outlet for tech people by tech people. We're the network behind the show you're listening to right now. We have a slate of hit shows across a range of topics and industries, from our AI and investing cluster of podcasts. To shows that drive the conversation in tech with the most interesting thinkers, founders, investors, and influencers, like Econ 102 with Noah Smith. We're launching new shows every week, and we're looking for industry-leading sponsors. If you think that might be you and your company, email me at erik@turpentine.co. That's erik@turpentine.co, and let's partner together.