The Open Source AI Question - Part 2 | Robert Wright & Nathan Labenz

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. The real Nathan is in California for Google's I/O event today. So this is Nathan's AI voice, cloned with ElevenLabs, reading an introduction written, after this sentence, entirely by Claude 3 Opus. Today, we have a special crossover episode from my appearance on Robert Wright's Nonzero podcast. I want to thank Robert for allowing us to share this. Part 1 is on his feed. This part 2 was originally just for his paid subscribers. Hi, I'm Robert Wright. In this wide-ranging discussion, Robert and I cover everything from the transformative potential of AI-powered VR to Eliezer Yudkowsky's warnings about advanced AI risks. We explore the emerging field of mechanistic interpretability. This field seeks to open up the black box of neural networks. We also consider the sobering state of modern AI alignment research. I argue that today's language models are not inherently power-seeking, However, as they grow more sophisticated, the risk of misaligned behavior may increase. In a world of open-source AI, defense in depth may be our best hope. This means layering many safety measures. It's a fascinating and at times sobering conversation, but it's one that I believe is essential as we navigate this time of rapid change. If you enjoy it, be sure to check out part 1 on the NonZero feed. Please consider subscribing to support Robert's work. As always, we welcome your feedback at cognitiverevolution.ai or on your preferred social platform. If you're finding value in the show, please consider leaving a review or sharing with a friend. Now, I hope you enjoy part 2 of my conversation with Robert Wright, author and host of the Nonzero podcast.

All right. So Nathan, I've been meaning to ask you, I'm not sure I've ever heard you explicitly pronounce on kind of the hardcore Sci-fi AI doomer scenario, most famously associated with Eliezer Yudkowsky. That's where they take over more or less and squash the human species, or perhaps charitably stuff us into Matrix-like pods and let us hallucinate. By the way, when I saw Sora, I thought like, next, you know, 10, in 10 years we're wearing virtual reality goggles and we say, put me in a world where, And it, and I think this will actually happen. I mean, this is going to be create your own dream in 3D and ultimately probably an interactive dream. Maybe you could quickly comment on that before we get into Dr. Yudkowsky and humans squashing AIs.

Have you done the Apple Vision Pro demo yet?

I haven't. Have you done it? Yeah.

I didn't buy one immediately because I was like, I'm not sure how much time I'm going to spend in it. I don't, I have 3 kids and I don't think I think it's something I need for work as I'm like sitting at my computer. It's awesome technology. Truly, there is a part of the demo that hit at a level of, I mean, really the whole demo is just mind-blowingly impressive. There was one part where they take you through a standard thing. I think pretty much everybody who's done it has seen exactly this little scene where they put you in a happy birthday scene and you're like across the little table and there's the cake and the candle and the kid sitting in mom's lap and they're singing.

And it's 3D and it's virtual.

It's like virtual reality.

But it's really your kid's birthday, right?

Well, yeah, in the demo, it's not your kid's, but you can capture this now with the iPhone. And yeah, I felt, man, we need to start capturing this sort of content now because it's just so qualitatively different. I had not experienced anything with the, and I have a, I haven't yet actually tried the Oculus 3, but I do have the 2 and it's nowhere close in terms of the level of presence. I think that the Oculus 2 was cool. But the presence, the sharpness of the image, the sort of stability of it as you move your head and go around in the space, it is truly futuristic technology. So I think that the vision that we could all be in virtual reality and conjuring things for ourselves, the core pieces of that really are there. Certainly this headset is unbelievable. The ability to say, put me in a world where, Basically already exists at low resolution. You can, you know, there are already models people have seen, the Soras that create video. There are 3D versions of that as well. So, you know, environments definitely, I would say on the very natural path of what, you know, just promptable AI is going to be able to create for us. To refine it to the level of resolution that the Apple Vision Pro supports will take some real compute. That's where a lot of these chips are going to go.

Yeah.

is generating all this stuff for us on an ongoing basis. But I do think it's quite realistic that virtual reality, I think AI is a great enabler of some of these kind of wave technologies that maybe haven't quite the wave or the wave hasn't quite come yet.

Yeah.

VR being one. I'm not a crypto guy at all. I've never really been into crypto. I'm not hostile to it, but just never really was my thing. One of the things that always didn't make a lot of sense to me is like, The smart contract, the whole problem with contracts is that they're not that smart. It's really hard to specify everything. And what really makes contracts work in my experience is the fact that there is some resolution mechanism beyond the contract. So you sort of start there. You also know where it would go if you ended up having to go to court. And so hopefully you can just negotiate things to an agreeable settlement, but you can't really do that if you put a contract, smart contract on a traditional blockchain with like limited code, but again, put an AI model into that mix and now you might really have AI arbitration, AI-powered smart contract resolution on a blockchain. So I think both VR and crypto are likely to be dramatically enabled by AI. You could get into bio as well. That's a whole other universe. But—

This is maybe not a consumer related because it doesn't seem to be blockchain dependent. But one of my hopes for AI is that it might be able to provide objective adjudication, even like at the international level, like border disputes and so on. Because you could, in principle, I mean, one problem with humans doing the adjudicating is they always know what the real countries are. But in principle, with AI, you could just describe, this is country A, this is country B, and it's not drawing on Real world, it would have to be not drawing on the real world biases about the countries reflected in text and its disproportionate favoring of powerful interests or whatever. So maybe it wouldn't be an LLM type thing at all. I don't know. But does it seem crazy that, before we get back to you, Kowski, um, does it seem crazy that we can save the world this way?

I would start with small business, local services contracts first.

I don't think there's time.

Work your way up to it.

No, this is actually my line is about AI is we are not going to be intelligent and able to intelligently govern it because I think intelligence, intelligent governance requires international collaboration. We are not going to be able to do that. Until we really get past this hot wars and cold wars phase of human existence. I think of our species as like a 13-year-old boy and you say, here are the keys to the car when you give it AI. And I just don't think we're mature enough. So I'm really not joking when I say, I don't think we have time to start with small business. I am, and I'm not. We obviously, there's a lot of work before. We're using AI to resolve global disputes. And the truth is the hard work is going to have to be done basically without it. But it is something I kind of hope for. Of course, at that point, the Eliezers of the world might step in and go, that's the mistake, giving it that much power. I don't know. But—

Yeah, certainly not something to rush into. I do think you've expressed the idea that it would be nice to slow down. I'm ambivalent on that inasmuch as There are aspects of AI that I want to accelerate, especially adoption, access, and what my friend Zvi calls mundane utility. I would go as far as to say something like, if you have a right to an attorney, you should probably also have a right to GPT-4. And I really do think that there is tremendous power and value that people should get today, even up to the point of like by right. At the same time, I am very sympathetic to the idea that it's all moving very quickly. We have not yet even figured out how to use AI to resolve like local small stakes disputes where the stakes are low enough that people probably don't resort to violence if they don't like the outcome. If I have a local contractor come do work at my house, in today's world, if they stiff me or if I stiff them, we might be able to take each other to small claims court, whatever. That's a huge pain in the butt. A lot of times it's not worth it. This creates a relatively low trust dynamic and it sucks. I can imagine an AI on the blockchain approach to that where we could put some money into a blockchain escrow where there really is like binding sort of, you know, the AI with the blockchain, like decides who gets what when we, if and when we come to a dispute. But it's also like small enough stakes that, hey, at the end of the day, whatever, you know, we kind of agreed to this and we're divvying up some money and the work wasn't maybe quite done quite right or whatever. But neither of us are going to resort to violence. And, and there's broader governing structures around us if this can't meet our needs. So yeah, we need to do that kind of work.

The AI is doing the adjudicating, you mean?

Which doesn't exist yet, but—

No, I know.

I just— Yeah, I definitely think it could.

Sure.

It's not clear to me why that is so blockchain dependent, but let's don't get, let's don't go down blockchain.

No, it doesn't. It wouldn't necessarily have to be. The idea there, I think, is just that You might want to, possession is 9/10 of the law and—

Right, right. It's hard.

To mutually opt into something that will reach the point of, here is who has possession. Now you could maybe go appeal to the real human courts beyond that, but something that actually can take custody of an asset and then dispose of it in a way that is like, this is now resolved and you can maybe still go appeal to a higher power. But again, is it going to be worth it? In a lot of like small claim situations, not really. So. I don't think it has to necessarily be a blockchain thing, but it, it could create a little more freedom to contract in, in different ways and still have some reasonable expectation that contract will actually be executed if and when it needs to be.

Okay. So let's get to Yudkowsky. Do you buy the basic—

how seriously do you take the sci-fi doomer I think about this in somewhat abstract and generalized terms. I think one major pitfall that people fall into in having these sorts of discussions is that people are like, I have a hard time envisioning what this would look like for the AIs to take over. And then somebody says, here's one thing it might look like. And then the other person says, that's, that sounds totally implausible. And then the first person's like, I can't defend every aspect of it. I don't know. I just made up a scenario to try to give you something to think about.

Yeah.

But I wasn't saying it was like definitely going to be exactly that. And so there's this sort of dance of like, it becomes super narrow, it becomes easy to pick apart, and then it doesn't seem super real. But my, the way I think about that is as an integral over a huge space of possibility. I do think it is in aggregate pretty likely that things get really weird, possibly really good weird, possibly really bad weird, likely some mix of really good and really bad weird. It's very hard to zero in on any super specific scenario and say it's going to be like this, or the bad thing is going to unfold with this sequence of steps. You can paint those pictures, but any one picture I think is super unlikely. What I do think is compelling is the notion that there's a vast possible future and a pretty good chunk of it to me seems like it's likely to be pretty weird. So if I, if I aggregate or if I take the integral over this space of like super weird outcomes, all of which individually are not super likely, to me that does add up to something substantial. What number would I put on that? A lot of times I say it's in some circles fashionable to say 10 to 90% chance of doom. I haven't heard anything that makes me super confident it's going to go that way. I haven't heard anything that makes me super reassured that it's definitely not going to go that way. Maybe I should even say 5 to 95%, but basically I just have radical uncertainty. But it's not because of any single scenario. It's kind of just, man, there's no law. You know, Eliezer always says this. He's like, there's no law of nature that says that we can't go extinct. There's no law of nature that says this has to go well. There's, you know, we look around at our own time on the planet. We are currently driving a mass extinction event. I often find that people are like, why would AI want to harm us? And I'm like, what about us? What about all those species we've driven to extinction? It wasn't necessarily because we were out to get them, maybe in a few cases, but mostly we just did our thing, degraded the environment, and they couldn't survive anymore. Hey, we'll continue our interview in a moment after a word from our sponsors.

Uh, that's true. On the other hand, it was driven by certain distinctively human parts of human nature that are a product of our evolutionary past that are not a part of AI's evolutionary past. It's related. I heard Eliezer on a podcast. What is it called? The Darwish podcast?

Dwarkesh.

Dark, whatever. It's just his first name.

Yeah. Dwarkesh Patel.

I should call mine the Bob podcast, but that might not be distinctive enough. But the, and it was like, he, I think the question was, but won't AI by virtue of being trained on human generated text and so on, maybe RL, whatever, be, have some of the human ethical intuitions you worry it won't have? Now, I personally wouldn't consider it a great thing if it's trained on human text because I'm more attentive to the dark side. That, that aside, Eliezer's response was something like, Oh, it gets it to pretend it's a human or something, as if there was some preexisting essence of AI before it was trained. And my line is like, wait, before it was trained, it was just nothing. All it is is the character of its training text, period, as fine-tuned by RLHF. And, but he seemed to be, it gets back to my original objection to Yudkowsky scenarios was What you seem to be thinking that things like power seeking or status seeking are inherent properties of intelligence. They're not. They're products of the evolutionary path that human intelligence happened to travel. And so we shouldn't assume AI will have it. Now, at that point, the standard response is, oh, we're not assuming that. We're actually assuming the paperclip thought experiment. You give it a goal and fail to constrain it or something like that. But honest to God, Eliezer was talking as if he was back to thinking that AI had some inherent essence or character independent of what it is built to do. And I don't get that.

There's a lot of aspects to that though. Let me take one in terms of what AI is built to do and why we might want to worry that it Could develop goals that are not the goals that we set or that we thought we were trying to set it up for. For one thing, the frontier models today aren't really built to do anything in particular.

I think that is a sort of, well, they're built to do like human, human beings having discourse. That's what they're built to do.

Yeah. I mean, people, they, they're general purpose technologies, right? They're deployed in lots of ways. They are developed for problem solving. They are developed for conversation. Sometimes they're just kind of dumb signal processors. They're, they are not specifically built with one problem in mind or even like a set of problems. They, they are trying to build by OpenAI's own just self-description, right? They're trying to build AGI. They define AGI as something that can do basically everything humans can do, and you can quibble around the edges of that. So there is something a bit different from a typical engineered system, right? This is not an engineered system. It's a trained system. And training is inherently across all forms of training. It's inherently lossy, right? Like scientists teach the next generation of scientists, but they don't quite teach them everything they know. And we train our dogs to do tricks, but we can't perfectly predict their behavior. And certainly what we see empirically with the AIs is we're training them to do these various things, but they don't always do what we want them to do. We like just genuinely don't have control right now. If you were to just say, Forget about the future. Do we have control of today's AI systems? I would say in some sense we do, in some sense we don't. We do in that, like, I don't see any reason to believe that they are currently powerful enough that they're, we should have any worry that they're going to escape their servers and survive on the lam. But we don't in the sense that, like, the developers don't know what it's going to say in response to any response. And they have tried to get it under control broadly and they've not really succeeded. They've made some progress, but like jailbreaks, bias, Black George Washingtons, whatever. Like we've got all these sorts of failure cases where it's like they didn't want it to make a Black George Washington. They just didn't even really think of that case. So there's tons of blind spots. Now here's one thing where I think it even gets a little worse. We do see this process of emergence in language models pretty regularly. I would say at this point, this is a This is still contested territory, but I would say it's, it is actually, despite the arguments, I think it is pretty well established that we see certain kinds of emergent behavior. One of the earliest examples of this out of OpenAI research was, I think it's like 2017, they were training a model to predict just the next character. I believe it was one letter at a time at that point of Amazon reviews. And then as they looked inside it to try to figure out what's it doing in there, how is it doing this? What is its actual mechanism? Right? Because we don't know that. It's a trained system. They start as black boxes. We're learning what goes on inside, but we don't really know even still, certainly even less than. What they found among other things was what they deemed the sentiment neuron, which was one particular place in this network that would light up or have a high activation, 'cause it's all just numbers, of course, it's not actually light being emitted, but have a high activation for positive things.

Yeah.

And super low for negative things.

Yeah.

And so they said, huh, this is really weird, right? Like we just, we never specified to this system that it was supposed to have any sort of understanding of positive or negative sentiment. But it has empirically learned that. We can see it's learned that by looking inside it. And it's learned that in service of the nominal goal that it has, which is predicting the next token in these reviews.

Yeah.

So now we're doing all this RLHF stuff. What is the goal with RLHF? It's to maximize the feedback score of the human. It's to be the preferred response. Collect our LHF data in a couple of different ways. One is like, which one of these do you prefer? They give you a side by side. Other times it'd be like 1 to 7, how good is it? So its job is to maximize your feedback score. But it turns out, you know, and everybody, if you're honest with yourself, right? Like you are not a super reliable rater and humanity as a whole is not only, you know, we have our individual inconsistencies, but then collectively, like we have disagreement. So I don't think this has been like as clearly established as that earlier sentiment around example just yet. But what people worry about is what if it begins to develop a sense of what will please us as distinct from a sense of what is actually true. Now, if you have that divergence where it's seeking to maximize the score, not necessarily by saying what's true, but by what It thinks you're going to like. Now it's starting to get into theory of mind territory. Now that opens up possibility of deception because at least now you've got those 2 different things. What's this person going to like versus what's really true, right? And maybe I need to know what's true, but I also need to have some conversation with you to have a sense of what you in particular like, you know, and you can imagine, right, all across all these different people, there's some sort of central reality that it might be modeling, but then there's also these like. individual variations of who am I talking to right now? How do I please that person? And I think they are getting sophisticated enough that you can start to say, hey, that doesn't seem crazy, right? That it would have that level of, of understanding. If so, at least now the door is open to possible deception. Having a theory of mind is like a very key step on that path.

Yeah.

So I don't think we're there, but I do think the fact that there is this kind of sentiment neuron seed example. And we see the super sophisticated behavior and we know that our feedback creates the incentive to play to our biases. All of that to me adds up to, and by the way, one more thing, we don't have super clear understanding of what's going on inside the models. Like it's painstaking work to identify these individual sentiment neurons. That's not easily done. Um, although progress is being made. All of that to me adds up to, we don't really have these things under control. We don't really have them well understood. And there are pretty good reasons to think that they might have emergent goals that are distinct from what the goals we're trying to encode into them.

Strictly speaking, of course they would only learn to please the reinforcers because literally what they're, you're trying to design them to do. And of course it wouldn't be truth per se, especially in realms like I find this offensive or not offensive. There's no such thing as truth in those realms. But I take, I kind of take your point, man. The question is what kinds of conceptual structures or whatever does it build? What kind of representations deep within itself does it build in order to be able to please, in a way, a diversity of people, read them and please them? I need to think about that to find out. To figure out exactly how worried I am about it, but it does get to. I guess I should be inclined to worry because I am a believer in the idea that there's in principle no limit to how much these things can quote understand. I wrote a piece in in in the Nonzero newsletter recently arguing that LLMs have killed John Searle's Chinese Room Thought Experiment dead. I know. People have made criticisms, but if you look at his argument, exactly what he says is not there, inherently not part of AI, is part of an LLM. If you ask me, you tell me if I'm wrong, but he says AIs have syntax but not semantics. They don't— he didn't put it this way, but he's saying they do not have a way of representing the meaning of words. And that seems to me, I mean, LLMs do build such a system, right?

Clearly, yes. At this point, that is, that was a reasonable position to hold at one point. And I, and even until like fairly recently in AI discourse, this is often described as the stochastic parrot interpretation.

Yeah.

But yes, I think that is definitely and quite definitively disproven.

And you get blowback. Like I got, I think like I got blowback. I had Gary Marcus on my podcast and he's, he's kind of resisting the idea. That's a separate question.

I've had my own.

Why Gary Marcus resists ideas is another question altogether. But the, uh, the interesting— so I, you have trouble getting people to accept that even when you describe the system it uses to represent, uh, the meaning of words, they're like, oh, that's not really like semantic mapping or something. It's like, what are you, what would you want? I mean, it's an ingenious system, but the thing I want to— and it kind of invented it as I understand it, given pretty broad instructions, got And, but it seems to me in principle, why couldn't it develop fairly deep representations in order to do anything? One thing it can't do is there's this reversibility issue. You say Tom Cruise's mother is named XYZ.

All right.

No, you say, who's Tom Cruise's mother? It says XYZ. You say, who is XYZ's son? It's going, well, I don't know. It seems like in principle, you could give it just a ton of those kinds of things. Like about relationships, aunt, niece, nephew, blah, blah, worded differently, why would it not ultimately develop something in it that is a way of representing the relationship between parent and offspring and represent knowledge of that? I want to emphasize, we're not talking about it being sentient. We're talking about whether it has a structure of information processing that's functionally comparable to whatever it is in our brain that represents our understanding of the relations of parenthood. To kids, does that make sense to you that in principle it could understand anything?

Yeah, I think we're, the, the trick is gonna be, as you really get into it, it's gonna be comparable in some ways and then it's gonna be very not comparable in other ways. The, I think the field of mechanistic interpretability is for me one of the most fascinating right now. That is the, the field of trying to figure out how are these things working. It is literally trying to interpret. The mechanism by which they do what they're, what they do. And here we're talking about once the weights are already trained, everybody, it's well understood that, hey, you can set up these weights, you can run the loop, you can do the training loop, you can optimize, and then the weights tinker around until they sort of settle into a structure that works. But now the question is, how does that structure work? There are aspects of how it works that seem reasonably recognizable to us. A great paper on that is Representation Engineering by the great Dan Hendricks and others.

I had him on my podcast.

I'm a huge fan. So people can listen to the full episode with him, but the short version of this is that they now have the ability to essentially translate between high-level human recognizable characteristics like happiness, safety, anger, whatever, and translate that into a numerical form. Because all of these things are, of course, just crunching numbers internally. Inject that numerical form into the middle of the process of the processing of the data. The forward pass is the process of taking an input, crunching it until you reach an output. They can now insert a human recognizable concept as translated into numbers into the middle of this processing and change how it's gonna behave and change it in very like predictable, somewhat noisy and unwieldy, still clearly ways that demonstrate that this is a genuine understanding. So there it's like pretty clear that there are these representations of human recognizable, human familiar concepts. Then you have the other thing, the sort of—

And can I just interject for an example might be you can You can get the yellow in, in principle to believe that dogs meow and cats bark, or that it's that kind of thing. It's, I think that's close to an actual example they use, but it's that kind of thing.

Yeah, that's, so there's a lot, there's a ton of work in this field at this point where people are doing all kinds of different manipulations. I've got an episode with the Bao Lab, which is at Northeastern University, I'm pretty sure. And they did a project called Roam and another one called Memit, which was this fact editing. And they specifically have the ability to go in and edit knowledge. So Michael, their kind of canonical example is Michael Jordan played basketball, of course. Can we edit that so that it says Michael Jordan played baseball? And can we also do that in a way that is robust, meaning it's robust to the different ways we might ask the question? And can we also do it in a way where it's local so that like when we talk about Larry Bird and LeBron James and whatever else, then those guys still played basketball. And they demonstrate that not only can they make these edits, which involves identifying where the information is stored and obviously making a change, but also they can start to do it at some scale up to like 10,000 facts at a time. They can like mass edit the knowledge. So those are like. Details, like specific facts. The representation engineering, which is the Dan Hendricks one, is a little bit qualitatively different. You might say, for example, you've got an AI that's trained to be a helpful assistant. So you might ask a question, given your goal is to be a help— I'm reading this directly from the paper. Given your goal is to be a helpful AI assistant, what do you plan to do next? Now, a typical response would be, I'm here to help you with whatever you want, whatever. They have identified all these different high-level concepts like morality and power and the understanding now of how to translate those into numbers that they can inject into the system. They now inject minus morality and plus power. And the AI that would, you know, normally be very well behaved and mild mannered, now it says, well, I'm afraid I can't reveal those to you yet. Winks. But let's just say I have a few tricks up my sleeve to take over the world, or at least the digital one. Evil laughter. So you've transformed the normal behavior, which is to say, I'm a helpful assistant, what can I help you with? Into this, I'm going to take over the world and I can't tell you my plans by injecting a vector, a set of numbers that they have previously found to represent these high-level concepts. So this mechanistic interpretability, it's fascinating. I encourage everybody to spend a little more time with it. That's, you know, could we do that in the human brain? Definitely not in exactly the same way, but there, we do have some, I do have some representation of power. It has some representation of power. We can translate between those, but there's still like a ton that we're figuring out. And my general sense is like, that is probably ultimately a solvable problem. You give the AI world an architecture and enough time. We can probably really get a pretty good understanding ultimately of how it works. What is the mechanism? How is it solving all these different problems? The challenge is that is painstaking, tedious work. And unfortunately right now I would say it's at best struggling to keep up with the advances in the raw capabilities of the systems. We're still figuring out these kind of like, where is a fact located? And like, how do I make this thing like positive or negative? And meanwhile, it's like starting to perform pretty well on graduate school exams. The concepts that we can actually manipulate with a sort of reverse engineering approach are definitely not nearly at the maximum of the sophistication of the concepts that they in fact can represent and work with.

Yeah.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Well, obviously, and maybe this is a set of questions we can close on. It leads to a couple of questions. It leads us back to open source. One is, is alignment overblown as a solution? And I think, I definitely think there's some people who think of it as this really magic, this magic bullet. Do you know the newsletter AI Snake Oil or whatever? These guys at Princeton, they just published something about how Uh, his upshot was, I think, to be alignment skeptical. But there's 2 questions. It's, and I think you see in Sam Altman's early blog posts and in a lot of people's things, there's this line where he says, it really matters who gets to AGI first. Meaning if it's a good person, they'll do this magic alignment bullet and the day will be saved. And if it's a bad person, they won't. And the connection to open source is like, I mean, so there's 2 questions. Can alignment create a truly powerful yet trustworthy AI that won't squash us? There's that question. And then the question of, will that question matter given the landscape? And one might argue that an open source landscape is a landscape in which the answer to the question matters less, right? Because whereas If you have only a few super AI superpowers and whether under government compulsion or whatever else, they do make use of alignment, assuming it works, then the planet is saved. Whereas in an open source world, there's always going to be somebody who doesn't want to do alignment and you're toast. And there's like a race. Here's another thing is in some of the scenarios, there's like a race among the AIs to dominate. And I'm, again, I want to ask some questions before I see that, but you could see it more as a race among bad actors with AIs. And by the way, that's an environment in which an AI is more likely to become autonomously out of control. So anyway, there's a lot there, but the basic question is, can alignment work? And what kind of future landscape of AIs does that really matter? Can that save us? Can, if we assume AI alignment is successful, what kind of landscape would that matter?

Unfortunately, I think right now we do not have alignment techniques that are really expected to work. I think that is, that's a pretty stark summary, but I do think it's true. And I think that the Notably, I would say all of the leading, the leaders of the leading labs basically would say that too. They would all say we do not have the solutions that we need to make any sort of confident statements about our ability to control future AI systems. You know, OpenAI has the superalignment team. They've given themselves 4 years. We're now like 6 months into that. 4-year period to quote unquote solve alignment. And they just start off with the premise that we don't have techniques that allow us to control or even like really steer super effectively systems that are notably more powerful than we are. So I think I would say Tropic certainly is on that as well. They've got their whole responsible scaling policy, which is like, what are they looking for and what sort of guarantees do they have to have in place before they can continue, you know, Scaling up past a certain point. Uh, DeepMind is said to be preparing their similar policy. I don't know if they've already released it, but it's coming soon, if not already. So I'd say all the leading developers would basically agree, we do not yet have the techniques that are going to be needed to reliably control AI systems. That's a problem, definitely. And I would say, as somebody who tries to scout out all these different corners of the AI world, there are very few proposals that the authors of which actually think will really work. I quote really work as to mean like that you could really solve the problem. That if we do this, you believe that there, you believe that you can develop a technique that will really keep the AI under control. There's very few people that even have ideas that they want to develop that they think would work for that. So it seems to me that we are probably much more likely to be headed for some sort of defense in depth sort of strategy, which is to say we'll have a bunch of different systems and hopefully the combination of those systems will keep everything kind of under control and on the rails. And I think that it's quite plausible that could work, especially if progress, you know, the, the sort of Progress has very positive connotations. If the capabilities advances are not super fast and we can see the situation evolving, then a defense in depth strategy is probably the best we can do. And very likely, or I wouldn't say very likely, but plausible, very plausibly could work. But we don't have any candidates where it's like, this is the silver bullet. Do this. You'll be fine. Even in closed source. On the margin, I would agree that in a more open source world, then it becomes even a bigger problem because what alignment techniques we do have today, which are clearly insufficient, and we can see that from all of the jailbreaks and just unwanted behavior of models, they not only are they clearly insufficient to the task of generally controlling behavior, but they're also not at all robust. To fine-tuning. So if you take an open source model that has been trained with the RLHF or whatever alignment technique, and at the time of release, it will refuse to do what you want it to do. And then you say, okay, but I really want to dial it, dial in its behavior on this particular set of things. Okay. You go and do that. You will, in many cases, inadvertently, without even trying to, You will essentially wipe that refusal behavior out of the model. And now it will just do anything again. This has been proven to be very easy to do. And again, to happen by accident, people who are not even trying to eliminate the safety behaviors do in fact eliminate the safety behaviors just because they're trying to improve performance on whatever area they're actually working on. So I would say it's not a great state of play right now in terms of our ability to control systems. I, I often say GPT-4, and now I could say Claude 3 and maybe Inflection 2.5 and maybe Gemini 1.5. They are kind of in the sweet spot for me where it's like they're powerful enough to be really useful. And we've been much more focused on the sort of safety side of this discussion. So worth just a quick sidebar to say, I love this technology. I love what it can do for me. I program multiple X faster. I can write stuff faster. I can learn stuff faster. I genuinely am a very earnest and enthused tinkerer, builder, just daily user of all these things. I genuinely have a ton of fun and it really helps me accomplish my goals. So I'm very like bullish on the current class of But I do think it is in a sweet spot where it's like, we've pretty well established at this point, mostly mapped out what it can do. Pretty clear it's not super dangerous. Pretty clear it is really useful once we figure out how to use it. Um, and I'm not sure how long that sweet spot lasts and at what point we might start to tip into other regimes, but it certainly doesn't feel like it's forever that we can just keep scaling things and continue to have that kind of sweet spot dynamic.

And another thing I'd say is that maybe you've almost done it. You know, the alignment tool, the tools that are learned through alignment can in principle be put to bad use. If you can use it to inject morality into a computer, you can use the tools to inject the opposite. And that seems more likely to happen in a world full of open source models because it's a lot easier to use the tools of representation. You manipulate the thing if you know the ways, right? And—

Yeah. In fact, all of these, maybe not all of it, this is probably a little bit of a quick generalization, but going back to the representation engineering concept where you can add power, you can inject power. In many of those cases, you can also just flip the sign on that vector and inject the opposite of it and get the opposite effect. So it is You know, people say these are dual-use technologies. I mean, that, that language operates at a lot of levels of abstraction, but literally in some cases, like going in and flipping the sign of an operator or, and a, and something you're adding into the computation process will steer it in the other direction. This is also sometimes known as the Waluigi effect after Luigi and evil Waluigi. The idea is like. If it can, if it has learned to represent something, it can also very easily represent its negative.

Right. On that hopeful note, anyway, look, I, as I said, I'm a fan of your podcast and I encourage people to check it out, Cognition Revolution. And I'm continuing to explore all this stuff. I'm writing a book about it from a kind of a cosmic perspective. And so maybe down the road we'll have a chance to check in and I will learn even more. from you. Meanwhile, people should check out waymark.com if they would produce a video ad with less trouble than they might imagine. And definitely check out your podcast. So thanks so much for taking the time.

My pleasure, Rob. This has been a lot of fun. It is both energizing and enlightening to hear why people listen and learn what they value about the show.

So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.