OpenAI's Safety Team Exodus: Ilya Departs, Leike Speaks Out, Altman Responds - Zvi Analyzes Fallout

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Zvi Moskowitz, welcome back to a special bonus session of The Cognitive Revolution.

Yep. There's always more to learn.

It's happening quickly today. So by the time we got off the recording, Jan Leike had posted his tweet thread statement about his reasons for leaving OpenAI, in which he puts it pretty plainly that he's had pretty fundamental disagreements with leadership and has had trouble getting the resources that he needed to do the work, including compute resources. And certainly had some nice fond things to say to his teammates, but basically was like, I don't think we're on the right track and seems to be resigning pretty much in protest. So that doesn't seem like a good thing. Doesn't seem like a good situation. What else have you learned and what do we make of it?

So we also got coverage from Vox, from Bloomberg, from TechCrunch. We got Sam Altman's response actually to Likee, which was extremely graceful, essentially saying, yes, we have a lot of work to do and we're going to do it and expect a longer response later. It's the basically best possible thing you can say there, but then you're on the hook for doing it. We have Kelsey Piper confirming the nature of the draconian nondisparagement clauses, which apparently have lifetime duration. And include an NDA that you can't reveal without violating the NDA. And she claims that when employees come onboarded for the first time and are given equity-heavy compensation, they are not told that they'll be required to sign this disparagement clause or have their existing vested equity confiscated upon departure. So that seems like a really bad equilibrium and way to run a company. I'm honestly confused as to why that's legal.

Well, it maybe shouldn't be.

Yeah. I think you should have to very much acknowledge the disparagement clause rules very clearly initially if you're going to confiscate something of immense value for someone not signing them. It doesn't seem reasonable at all. It also doesn't obviously speak well of the company in its openness in the good sense, right? If you're forcing every employee to never disparage you for life, no matter what, or else That's just not a reasonable position to take if you want people to not assume the worst. And so, yeah, we look at Leike's statement essentially saying that for years they have had the trouble of shiny new products becoming the priority of a move away from safety culture, that the culture is not amenable to or compatible with safety. I'm paraphrasing here a bit. I'm not looking at the words precisely. And that he had trouble getting the last few months despite the explicit 20% of existing compute commitment from OpenAI, which should have been sufficient for current purposes. And indeed, TechCrunch confirms that they have not been honoring their commitments. They have asked for a fraction of the 20% commitment and have repeatedly not gotten what they asked for. And that is part and parcel of the whole idea of running anything in AI these days is compute. You need your compute to do your thing. And he was reporting this has specifically become a substantial barrier to doing their work. So this is not only a philosophical approach, because he said they should be spending vastly more, not just a little bit more, but they should be spending vastly more of their resources on preparing for our AGI future. But also, if you'll notice what he actually said, For just the next generation, that essentially he doesn't think they're ready for GPT-5. He doesn't think they're on pace to have the tools they need for GPT-5 to be safe in a pedestrian, mundane utility sense, like not in an existential sense. And then later on, there's the problem of superalignment. There's the problem of AGI, which many people at OpenAI have said they expect within several years. It's a very short timeline. And now, The Superalignment team has been dissolved and its people have been dispersed throughout the company. They claim they will still continue the work on that level, but having dishonoured their commitment and having dissolved the team and having lost the leadership, it doesn't look like the kind of effort they promised us that they said they were going to do.

Does the timeline still hold, Sam? We're now, what, 9 months into the 4 years?

He, if he still expects to need superalignment to succeed within 4 years, then these actions do not reflect somebody who understands that and understands, as he's repeatedly told us he understands, what is at stake with superalignment. And it's clear Mikey's breaking point was Ilya departing, according to the Bloomberg article. That makes perfect sense. But we have a series of departures. We have Obvious justifications for that, and this all makes sense. It's we all there was all this speculation about what did Ilya see? It was the whole thing. What did Jan see became the thing. Yes, you know, after Jan quit, and the answer is what they saw was a company that's not committed to safety, that's unwilling to put its money where its mouth is, that has a culture that is hostile to safety efforts, and that is pivoting towards the shiny new. Product. It's a product company. It's a scaling startup. It's devoted to making money. There's nothing wrong with making money, but in this case, they're building smarter than human intelligence as their explicit company mission and goal. And as Leike points out, and as Altman has repeatedly acknowledged, this is not a safe thing to do. This is not a default thing that will go well by accident, right? Like, we can debate how likely this is to go badly With good real efforts to make it go well. But I think any reasonable person can see that there's a very good chance that if we do not put in the effort, we do not do the work, that things would then go badly. And it doesn't have to necessarily involve some sort of specific AI catastrophe or existential risk scenario. It simply means this could go very badly for people as experienced by people on the planet Earth. And we have to think carefully about these questions. But what's clear is that OpenAI's leadership, as embodied by Sam Altman, increasingly is not doing this, and certainly in the last few months is not honoring their commitments.

Yeah, that Compute One for me is pretty troubling. I could try to make a— in the absence of that, it would be a lot harder to parse. I feel like obviously people can have all sorts of disagreements, and you can imagine the Sam Altman defence being like, You're doing your thing over here in the superalignment team. We're doing our thing over here in the product team. Why is that inherently a problem? But yeah, if you can't get the compute to do the alignment work, it's not great.

It's not just they promised us X and we got Y. It's Jan specifically saying, you promised us X, we needed X over N, where N is a lot more than 1, to do our work, and we couldn't get it. And the superalignment commitment, it sounds like a lot, 20% of our currently available compute. That's not a lot over 4 years because 2 years from now, OpenAI is going to have 10 times as much compute, for sure, unless something very strange happens. So over the extended period, this is a reasonably small, very modest commitment, as opposed to in certain similar industries where safety is paramount. Most of research costs, most of development costs. can become safety. And again, that's for mundane level, just make sure the plant doesn't melt down levels of safety. So I don't understand it. I also don't understand why it's not simply good business to give Mikey and people like that their compute.

Certainly relative to this outcome, it seems if we are indeed talking about something like 5% of compute, then that could only allow you to move 5% faster than you could without that compute. It does seem like a very strange decision to allow people to walk and allow this to become this big of a story over a couple percentage points of compute availability.

Yeah, it's almost always the straw that breaks the camel's back, right? It's clearly Ilya was being shut out of decisions, couldn't play his ambassador role that he had previously played inside the company. If you're Reading between, not even between the lines, but just reading the lines of various articles and reports that people were turning against the very idea. The board battles embodied this struggle, the way that they played out in practice, whether or not this is at all fair or deserved by anyone or any philosophy or any approach or anything. Turn people against them. Every person that leaves turns you further against them. The firings, you have a cascade of every person that leaves, you lose more trust. Like, why did they leave? What caused them to leave? How was that handled? And yeah, like he presumably was just like, I'm fed up. We're not getting what we need. You need a wake-up call. I can't just sit here and try to pretend that I have the resources I need because I don't. And the way he talks about it, they're just not investing what a company in their position needs to invest in what I call mundane safety either. And in fact, I haven't investigated this further, but you see this in their reports. That 4.0 has much less tendency to refuse inappropriate requests like building a bomb than GPT-4 Turbo or its rivals. That somehow this new model that was sent out is actually just very not robust in the jailbreak slash mundane harm sense. It just is very willing to fulfill your requests. And I haven't tried to stress test it or red team it in this sense because why would I? I don't think it's particularly a dangerous thing for the most part to have it be jailbroken. It was already jailbroken for those who cared enough. But what's very clear is that the 4.0 development process did not involve a robust attempt to make it safe in a conventional sense. They mostly just decided that the capabilities that this model possessed were not so dangerous and they just weren't going to give it much care.

So I'll see you chained to the OpenAI fence, or what do we do? What do we do from here?

I think we have to treat going forward until we see— he could respond with an amazing set of commitments. He could respond with a new set of hires of people he's bringing in. He could do any number of things. So we're always hopeful. And as much as AI moves fast, I don't think we need to move this week or anything like that. And part of it is that it's very clear from Yatton's testimony that we don't have to move this This week, that if Yann LeCun and others were concerned about something imminently happening, they would've said so. It is very clear from this statement that these are long-term concerns driven by things that the ships get steered slowly no matter what you do. So they have time to fix it. But barring evidence to the contrary, given everything that we've learned, I think we just have to assume that OpenAI is functionally a fully for-profit business. Fully a grow fast, move fast and break things hockey stick graph startup business run by Sam Altman, who is running it on that basis. And that they are not taking the safety problem seriously, that their culture internally is hostile to the idea of safety, to the idea of worrying about things, and that therefore we should expect them to handle this future badly. That we should expect them not to be prepared for what is to come, to be extremely cavalier. And there's the possibility that for a while cavalier works out. A cavalier GPT-5 might well be the best thing to happen for the world. It's possible if they're just not that dangerous. We talked about not safe for work. We talked about sex and gore and other capabilities. And yeah, it might just be good. To let that stuff happen. It might be completely net positive, right? I actually expect that. And Pliny broke all the major models. He broke all of them fully, and he broke GPT-4o in about 2 minutes during the announcement speech, right? Like the moment he got access to it, because his first hunch just worked, because why wouldn't it? But at some point, that's going to stop being an acceptable situation. At some point, these things are going to be highly capable, and we're going to have to worry about societal implications and then catastrophic risk and then existential risk. And all that's probabilistic because you don't know when it's going to happen, right? Like when GPT-5 comes out, you're probably not going to be in an existential situation at all. But how many 9s are you willing to put on that statement? And if your answer is more than 2, you're crazy, right? And I would probably put 1, right? So like you can come cry and say, you said it was only 97% and it turned out not to happen. And I'm like, well, right, chalk it up to a slight, slightly worse calibration than you on this one. And let's keep betting on sporting events. But I don't know what else to say.

Hey, we'll continue our interview in a moment after a word from our sponsors. I assume you have seen the Dwarkesh interview with John Schulman out this week. So as it's reported now that he's taking on the responsibility for safety writ large, I guess he was already responsible, as Dwarkesh presented him in the interview, he was responsible for post-training the models as described in some of these articles today. He's responsible for making today's models safe. Now he's going to have this kind of additional responsibility rolling up to him of big picture, long-term safety. In that context, that interview, I think, is going to give you serious pause. Obviously, you can evaluate it for yourself, but there were multiple moments.

Just hearing what he was doing previously, you took the mundane safety guy, Who is doing a job with mundane safety that it's not necessarily better or worse than the rivals, at least until I haven't evaluated for O right in the sense, and there's again reports that there's potentially a problem, but that's just completely irrelevant problem to the problem he is now being asked to solve. And if he was doing that job properly, then like he wouldn't necessarily have mentioned that they have these concerns about. The next generation of models not being prepared for, because that should be part of his job as well. But yeah, the idea that you're going to use post-training on your AGI or even your ASI to render it like HHH or like net useful or any of those terms, I think that's basically a pipe dream to rely only on that. In fact, you have to worry during even the training regimen of some of these things. But just the kind of strategy, certainly there might be a way, but the strategies that he's been using. So far are the kind of things that Leike himself acknowledged explicitly, like both online and literally 5 feet away from me during a talk, would not work. Right. He took on the 80,000 Hours Podcast. He explained why ROHF is not a solution to this problem, and he has other proposed solutions that he wants to try where I don't think they'll work. And I tried to debate him and explain to him why they wouldn't work, and I was unable to make my case sufficiently convincingly, and he didn't buy it. I think it was reasonable for him not to buy it in the sense that I was making some very bold claims that were very different from his worldview, and I didn't back them up specifically enough because it was hard for me to do without the technical training. But he was thinking about the problem on a different level than someone who's thinking about post-training is thinking about the problem. So I haven't seen the interview yet. Again, I want to be very careful with it. But yeah, if you think it's going to give me pause, I'm confident you're right. It's going to give me pause.

Yeah. A couple. Well, just for folks who might be listening to this and might not have time to go through that or inclination to, it is definitely worth it. It's very interesting in multiple ways. One of the things that he says is that he's expecting, quote unquote, AGI on kind of a 2 to 3 year timeframe. Dwarkesh asks him, like, could it be as soon as next year? And he's like, I don't think so. That would be surprising. But 2 to 3, he was pretty much willing to cosign on. And then Keshe asked a number of questions that were like, pretty fundamental. What happens when this happens? How are we going to deal with it? Or at one point he said, I'm not sure, that doesn't sound like a super robust plan that you're outlining here. And I appreciate the candor, but at multiple points he was like, yeah, I don't really have a great account for how that's going to go. Or hopefully we'll be able to work together with the other leading developers in that situation. And yeah, we don't really have a robust plan for that at this point.

Second best plan, right? The best plan is to actually figure out what you're going to do and how we're going to handle this. And the second best plan is that no, you don't know, right? To start with a blank beginner mind. So if he comes to this, if he comes day one and he says, we don't have a plan, we don't have a solution, we don't know how to align this thing, we don't know what to do with it if we did align it, we don't know how society can handle this. We don't know how to make the transition from AGI to ASI. Again, even if we handle alignment, even if we handle the interim, we don't know any of these things. We're lost. We need to figure this out. And he starts from scratch. I'm perfectly happy with that as the answer for a person who's highly capable. He founded OpenAI. He's done— co-founded it. He's done a lot of impressive things. Ilya cares deeply about these issues and appreciates the depths and scope of, and stakes of the problem. But I always thought, and I still, I mean, every, going off of Ilya's publicly stated remarks, I just don't think any of the things he was thinking about are anywhere near like what you need to be thinking or what would work. Right. Bucky's stuff wouldn't work, but I thought like he was reasonably grounded and trying hard. Ilya's stuff just felt like it often felt, okay, that just seems like a misconception to me. It feels like you're thinking about this problem In a kind of fuzzy, like, not sufficiently geared way often, and in a way that, like, just wouldn't survive an encounter with the enemy, right? When he looks at— when he and Leike and these other people run their superalignment team and they run, they build these papers and they run these experiments and they actually try to make these things work, they'll understand that their plans aren't working. And they'll either find ways to modify them so they work or throw them out and find and try new ones. Because One of the things Ilya is famous for is the kind of attitude of, I'm going to try this as many ways as I have to and throw out everything I think I know until I make this thing work, because I'm committed to making this thing work. And that's what you love to see. And that's why I had a lot of faith that Ilya and Jan would find a way. Not every time, because I don't think this problem is even necessarily theoretically solvable by humans in a reasonable timeframe with any attitude, but I thought they had a reasonably good shot. Because with a lot of resources and a lot of time, and 4 years is somehow suddenly a lot of time, they would figure out at least some ways not to align an AI. And then we get to try again. And the fact that their first way definitely won't work in my model is, that's fine. They were a mix of ideas that were reasonable and I thought promising and/or hopeless. But nobody knows how to solve these problems. So I can't really get that mad at you for being excited by ideas that I think won't solve these problems. Would I have a better suggestion? You try something, you learn something, you try again. And I think there's a decent chance that looking at these generalization questions, looking at these supervision questions, will inform your approach to trying to find alternate solutions that might themselves be more promising. If none of those types of solutions, if that entire solution class is entirely hopeless, that the worst-case scenario, that it is as bad as it looks to me on first glance and there's no fixing it, Then those are pretty bleak worlds in many ways, because that cuts off a lot of people's plans. So I'm leaving room for there being places to manoeuvre there.

So with this clarity, and while we await a proper response from Altman and leadership, what else do you think people should be doing? Joking, but maybe I'm not entirely joking about chaining oneself to the OpenAI fence. For reference, there was a person who has done that this week. There is mundane consumer protest. Now that ChatGPT is free, it's going to be hard to cancel our accounts, but we could rally app developers to boycott and switch to Claude or something. We could obviously on the record on being at least generally positively disposed to SB 1047. We could think about supporting that even more forcefully or suggesting possible amendments to that to strengthen that. Something like the non-disparagement clause being made illegal could be an interesting one.

When I looked at 1047, I wrote it up. I was mostly looking for ways in which it was too strong. I identified a number of ways in which this bill might go too far in the sense of it has these serious downsides. And unless we're getting a lot in exchange for these downsides, that makes it politically harder to pass. It makes it harder to get buy-in. It makes it harder to get cooperation. And nobody actually wants to tank the economy. Nobody wants to actually slow down the mundane utility. Not nobody, but I don't. And so how can we strengthen this bill? The exception being the derivative versus non-derivative definition clause, where I thought this is just a bug. It's literally, there's just a major definitional mistake in this bill that is just making it worse on every level. We need to fix it because if I can pass off all of my blame to you, that makes your situation terrible, but it also makes all the safety guarantees something I can skirt. I can then ignore anything, and that's not good either, right? Like, we can't let any of our community cheat. We have to stop this. So some of the others were about how do we weaken this? How do we— I don't know if weaken is the right word, but how do we clarify this bill? How do we prevent potential overreach or misinterpretation of this bill in a stronger sense? And other people pointed out ways in which the bill was like potentially too weak. Everyone was complaining about the criminal liability, but it's only under perjury. Maybe that's not enough, right? One could ask potentially, and some people did. I think for now it isn't, it is where we want to be. And I think the reaction to even the perjury showed us that this is just a third rail. And people just get so scared of such things that let's not, it's just gonna, it might have very bad dynamics for the good people to stop, stop touching things or they panic and they don't want that to happen. But. Yeah. Like we have whistleblower clause in the bill. And if you don't enforce non-compete agreements, right? This seems so much worse than a non-compete agreement, right? An ordinary non-compete agreement, which now not only California is not enforcing them forever, but now like they're gonna be illegal for high-paid, all but the most prominent employees, the entire country, 'cause the FTC doesn't, unless that doesn't go through. But yeah, I have a hard time believing that it is in the interests of the public of the United States to allow a company to hold most of somebody's wealth hostage to signing a lifetime full non-disparagement clause on the company they are leaving in an area in which things that are wrong are in the vital national interest, right? If there's something wrong with the safety at OpenAI, that's something we need to know. If there's anything wrong with the culture at OpenAI in other ways, that's something we need to know. If the ob— if you have a whistleblower provision, this is like, how do you blow the whistle if you're not legally allowed to blow the whistle? Or the price is millions and millions of dollars in equity, right? That you can't sell, so they can just confiscate it. So even, they don't even have to sue you necessarily. They can just confiscate it in that situation. And then who knows what else they might be threatening or holding over people? But like, you just don't know again, because they can't talk about it. So I can't know these things. So we have to assume the worst in some senses because we can't talk about it. And yeah, I think it would be very reasonable to say that AI companies should not be able to sign non-disparagement clauses as pertains to certain aspects of the company, certainly, and potentially universally. Like, I think it's just, why is it good for the two of us to get to an agreement where we agree that no matter what happens, I can never say anything bad about you and you can never say anything bad about me. I understand why it's better for us in some sense, but people not being able to talk in that way just doesn't seem great. And if we're in the business of not enforcing contracts that are against the public interest, this seems like a prime place to look. Even though I have libertarian instincts that voiding contracts is bad, yeah, this seems to be a reasonable place to consider that, obviously. But beyond that, it's, well, if you're not going to do the safety work, yourself necessarily. Someone has to make sure that you do, right? Someone has to be doing the check-in. And this is a reason to doubt, right? If I'm going to trust a company, I have to be able to trust their commitments. I have to be able to trust their statements, their attestments. And I need that to be under some sort of punishment, right? Like, that's the whole idea, right? If you lie, right? The idea is that you don't have to do anything. You have to tell us what you are doing and you have to be held responsible if you lied your ass off. That is what the key provisions of SB 1047 are about. They're about saying what you're doing and being responsible if you didn't do it, right? And saying what your logic is and being responsible if your logic is lies. It just makes no sense. Willful disregard beyond the pale. That's what it's about. And also just having a mechanism where if you discover that there is actually Catastrophic risk in the room, you can get the model shut down. Both that they have the ability to shut it down, at least locally, and that you can order that. These are the fundamental things this bill is about. So these things seem important now more than ever, right? In the light of this information. But fundamentally speaking, yeah, I think we have to understand that until proven otherwise, OpenAI is much less of a confusing. than it was a week ago or 6 months ago, right? There were reasonable arguments to be made when they announced superalignment, when they put out their reasonably good—

I made some of them myself.

Yeah, they made it. They put out a reasonably good preparedness framework, right? Like they've done some good things. They've hired a bunch of good people. They had a bunch of people who moved in circles where like they're credibly spending a lot of their time. Talking the right ways, asking the right questions, even if I don't agree with their specific beliefs. And a lot of that's just gone now. Their credibility is shot from a safety perspective. And so I think it's a lot less confusing now. And yeah, I think that if you have a choice in whose technology to use in some sense at this point, and you choose to go with OpenAI's in a way that matters, well, this is part of what you are considering, part of what you're doing. And I think it also means that you are taking on A risk, a concrete risk to yourself, because I don't think you should necessarily trust their mundane safety in this world going forward. I think it's— I'm still just, again, there's just not much that can go wrong with GPT-4o that I am that scared of. But they don't have a culture of safety. You need a security mindset to build AIs, to make these AIs do the things you want them to do and have it not blow up in your face, even in an ordinary, normal, mundane way. You need to be thinking about these problems and working on these problems and giving them the respect they are due. And so, you know, the Anthropic business case for, we deeply care about this and we have a culture of this, which they clearly do amongst their employees, right? Where they encourage this concern, where they have this concern. And we are gonna make sure that when you use our product, you get what you're trying to get and not something else you did not expect and did not want. becomes a lot more interesting, right? And then where Google lies on that spectrum, you can evaluate for yourself. I'm not saying there are any angels in this room. I'm not saying there's anybody that I trust, but there are levels. And again, like, we'll see what Altman says next week.

Hey, we'll continue our interview in a moment after a word from our sponsors. What do you think about revisiting the notion of third-party testing as well? This was something we chatted a little bit about prior to my interview with State Senator Scott Wiener, who of course is sponsoring the bill. The current bill just says including third-party auditors as appropriate. And now it does seem, okay, maybe we step that up a little bit. But we got this UK AI Safety Institute not quite getting the access. Now we have the resignation clearly in protest. You're a good game design guy. How would you think about designing that game so that the right people get the right kind of access to do the right kind of testing and it doesn't collapse into something similar?

I think the thing that I emphasised when I analysed both the preparedness framework and the responsible scaling policy for Anthropic was these are Potentially very good policies if the spirit of the rules is being honored. If the people at these companies care about safety, they have a culture of safety, and they don't just look at this as a bunch of checkboxes to get through so that they can get through compliance and release their thing and satisfy the scolds. But they genuinely care about this result. And when the answer comes back, technically you passed, but that's funny. The response is, no, wait, stop. Think about what's going on. Full well that wasn't okay. If something's going on here, we need to investigate this. We need to stop this, right? And you react accordingly. And certainly you can't do it if people are potentially gaming to sabotage benchmarks to be like under worrisome thresholds. If you're worried about them targeting the test, they know what questions are going to be asked. They know what exactly is going to be the attack surface that is checked and they strengthen that particular attack surface. These things are death, right? If all you have to do is verify the AI doesn't fall or cause a problem in specific ways, I think you're just toast. And I think there's no set of tests that would be at any reasonable cost that could possibly satisfy that. And that's why you need third-party testing unless you deeply trust the people who are doing this, right? Do you trust them not to teach the test? Do you trust them to look for anything at all? Not just things that are specific. Do you trust them to isolate these groups? I can imagine a world in which I would have that trust, but after what we just saw, do you have that trust in OpenAI? I mean, I don't know that I do, in the sense that I don't. Definitely don't. But we saw that I trust their benchmarks. When OpenAI comes up with benchmarks, I trust that they're not gaming the benchmarks. They're not trying to manipulate them in that way. But I trust them for GPT-5 in the same way that I did previously. Now, maybe not as much. Right. I don't know what direction they're going to do these things in, but as I think Colin Fraser was the first person to say, let's not assume they invented the operating system from Her until we get our hands on it. All right. Let's not jump to conclusions in any way, not just safety, but also capabilities. Because if you are a hype machine that is trying to hype, that is a very different world than what ChatGPT was, right? ChatGPT was just, here is our very sterilely named Very simply presented, very clean. And I still give major props to them for not having glammed it up in various ways. There are things to love about OpenAI. But they just presented this thing of like, hey, here's a cool thing. Let's see what you do with it. We're not going to tell you how awesome it is. We're just going to put it out there. And GPT-4 works the same way, mostly. And now this is the third, I think, time they've gone out there and gone, here are our amazing abilities, some of which we don't have yet. Hype, hype, hype. And that's very different. And then Sam Altman went on Twitter the day after Google's I/O to gloat about how his hype was so much of a better vibe than Google's hype without it. But he hasn't addressed, to my knowledge, any of the concrete things that Google announced or any comparisons as to who's building the better product, even to just congratulate them on a great set of offerings or anything like that. He's just like, oh, get ahold of the nerd. Basically, like, they're trying so hard, isn't that lame? That's not a good sign either.

Yeah, it's not great. It's not great all the way around. It does— and I would say I definitely strongly noticed the shift from the earlier releases to the current releases. I would say last fall, the Demo Day, was really the first time where it was like— and even that was more buttoned up than this one. But it was like the GPTs weren't really ready. They didn't really work that well. And it was the first time where it felt like, man, you guys shipped this even though it wasn't really in shape to ship.

GPTs, there's no safety concern, but they felt like vaporware, right?

Didn't work. Yeah. The retrieval was not good. I think that has been improved, but it was a few months later that, and I need to do a little more testing with this myself, but what I've been hearing in my app development circles is that the Assistants API has rounded into form now where the retrieval actually does work much better than it originally did and is more like what they described in that first release. But that's relatively recent and it's been a number of months. And this one just seems even messier where it's like they weren't even really clear on what we were supposed to be getting. Or people are just all confused at the moment. I was confused. I was trying to get ChatGPT to modify a picture of me and my son, and it was not doing it right or whatever. And then I was like, what's going on with this thing? And then I went into go on Twitter and I see, oh, okay, I'm still using the old system and it's just not clear to me what I'm even dealing with.

So they're, they're not updating a lot of the angles and they're not being clear. And experienced by a lot of people complaining, oh, I just realized I'm not using the new version of this thing. I understood it from what they were doing. I was being a journalist and paying very close attention, but to a normal person who's paying ordinary attention, it was very unclear.

Well, any other thoughts on— I guess one other question I had was, are there any new safety-related agendas or developments that you think are worth extra attention or that we should maybe be increasing our bets on at the moment? I'm always looking out for something that seems like it really could work. And I'm always struck by the fact that—

Most interesting thing to happen that I think is getting no attention in the alignment. So did you hear about SOFON?

I don't think so.

So I mentioned this a few— I forget how many weeks ago it is because time blurs. But so the Chinese have proposed a technique called a SOFON because Let's build a sophon from the ancient, from the dystopian offering. The aliens oppress us using sophons. Did you read the book? Anyone? See the show? But the idea is you can trap a model, open source or closed, but in particular open source, in a local maximum with respect to certain specified topics, such that if you attempt to fine-tune it, To get it out of the local maximum, it won't work. Ordinary fine-tuning techniques to try and escape from a failure won't work. So the proposal was you could actually teach, not just not— because if you don't teach biology to Llama 3 or Llama 4, let's say Llama 4 doesn't learn biology. Well, there's only so much biology that you have to learn. Feed it a bunch of textbooks, suddenly it knows biology, even if you somehow manage to not have it learn it by implication. But now the SoFun proposal is you can specifically teach it to not understand biology and be really dense about it. The way that certain people are like, I can't do math, and just refuse to learn no matter how much you teach them and how many examples you give them, right? Because they had trauma. You essentially give this thing trauma, right? By metaphor. And so the idea is, if you can make sure that this thing can't learn biology, now you've got an open model that can't give up biology.

Potentially.

Very early. We haven't run it through its paces. We haven't tried it at scale. We haven't— but it's an idea. It's the beginning of the first proposal I've ever heard that's in theory, maybe we could do something that raises the cost above epsilon compared to the training charge of the model to take somebody's general model and turn it to whatever specific end we have in mind. Maybe this would start to require enough work That we're not just making it easy on them. And if we can do that, now we still have the problem of you have to enumerate all of the specific things you want it not to know. But you have to figure out all the things you want to stop it from knowing and block them. Again, very similar to what we saw in the book without— not a spoiler, but the idea being— You see this in sci-fi all the time, right? You see the villain or the oppressors or whatever it is. And they say, oh, all that matters to us is that you don't do XYZ, because if you can't do XYZ, we're fine. And someone finds out a way to do W, right? Someone finds out a way to do something that they don't detect. It doesn't count to them. It just, they don't understand what's going on. And their response to that is to ignore it. Constantly happens, right? People start doing weird shit, right? And the aliens that are trying to take, they've taken over the enterprise or whatever it is, they go, I don't know what weird shit is going on, but yeah, whatever. Whereas the correct answer, of course, is, I don't know what weird shit is going on, so stop what you're doing until I know. That's not okay. But if you only can have a kind of scripted, like, I detect you can't do bioweapons, you can't do nuclear bombs, you can't do chemical weapons, you can't do cyberattacks, blah, blah, blah. Well, that's fine. From now, it's incredibly helpful for Llama 4, but it's not that helpful for Llama 6, even if it works, because it's no longer going to be the threat that you knew was coming. You're not going to be able to enumerate what a smarter thing than you comes up with. So it works up to a point, but it's still incredibly helpful and it potentially raises the bar quite substantially to the point where maybe we can all reach an agreement if this works great. I don't know. But If you want a moment of hope or something to end this with, there are at least some proposals.

Yeah, that's good. I hadn't heard of that, and it definitely sounds like something I need to go do a little more homework on. Anything else? Because you're one for one in terms of new and very interesting pointers there.

Yeah, I haven't seen that much in the alignment sphere lately, unfortunately. I also haven't seen new evidence that things won't work particularly, but it's just been Like relatively quiet, I would say, on that level. I guess something has to be quiet, right? You can't have everything happening all the time.

If you were to pitch a movie concept that you think would be most influential right now in the way that like Her seems to be inspiring the current moment of technology, mine might be The Social Network meets The Lord of the Rings. Where the sort of central figure would be the Sam Altman type who is on this like meteoric rise of technology, but is also being corrupted by it in the way that the ring is corrupting.

Weird example, just because the ring was a metaphor for the actual AGI for decades.

I, it just, that seems like the story that we might need to all hear.

We used to talk about the fellowship taking the ring to Mordor. As a sort of metaphor for some of the things that might happen in some scenarios. But yeah, certainly you could tell that story. I think my instincts tell me that's not the most interesting approach to that. I think if I was going to do it, I think I would maybe just do a very kind of straightforward AI takeover scenario with not such a smart intelligence. Just show the humans really giving up control. Show the humans, because it's in everyone's individual interest, no one can stop it. Just show things just spiraling out of control. One thing leads to another. There aren't even any idiots and there are no villains. Things just go wrong. And there's that. You could also have a Law and Order artificial intelligence set in 2035. That'd be fun. And interesting. So the idea being that, well, now that these open models have given everybody these extra capabilities, we have to be very proactive about hunting down people who try to implement catastrophic threats. And then obviously the police have all their AIs, everyone's acting on a high level. But it's one of these things where you notice that the world almost blows up every other week. And that—

I think we can write that. That's one of the challenges that I have with this in general, is I feel like The leap from here to there is tough. Like, what does that actually— what does the procedural look like? Can you imagine that getting concrete enough to be shown on TV in a way that—

Procedural, right? Is Star Trek not a procedural in its own way? Right? We explore a strange new world. We find an ethical dilemma. We find a technical— so we got a technological problem and we have our debates and then we encounter a setback and then we We implement our solution, we solve the dilemma, and we go on our way. Right. It's not that simple, but it also is. Right. And so you find a way to do a version of that potentially. But yeah, like a fun game for watching any sci-fi show is note how often things almost go horribly wrong. Like just watch a season of any Star Trek and watch how often the ship almost blows up. Right. Or somehow the Federation is almost in dire danger. Right. And how often this happens because someone was being a complete idiot and how often it happens so naturally, but like they encounter all these different problems. And then ask yourself, well, if you just looked at the 45-minute mark of every episode and you had to assign probabilities that this wasn't a narrative, something that wasn't a narrative someone wrote, what's the chance that humanity would've survived from here? But what's the chance the ship would've survived? Here, what's the chance the Enterprise actually makes it through 7 seasons? The answer is zero, right? It's so toast in so many different ways. And the Federation's probably toast too. The Federation, like, is in a lot of trouble reasonably often. Like, we can get out of it because we wrote it, but it's just not a utopia in the sense that if you actually were there, the safeguards aren't there, right? Like, we have no— there's no robustness in this world. This world is fragile. The Star Trek universe is so fragile. And so we get lucky a lot, but like, why are we getting lucky unless it's Q protecting us and stuff, or the travelers or something, right? In a way that we don't understand. And so you carry forward. The other, the other game is like, how often does somebody come within 5 minutes of building an ASI, right? Like, or how often would AI just run completely rampant here if you didn't have the rule it mysteriously doesn't? And that also is like just these scenarios. So you could also just have an ordinary sci-fi show. Where it just runs a normal sci-fi world, except that every now and then, and by every now and then, one episode in three or something, someone like accidentally follows through on logic and superintelligence emerges and everybody dies, or someone takes over the world, or everything gets killed, or some new regime takes happens, or there's a there's a recursive proof. Who knows? I I haven't thought this through. I'm brainstorming with you. But the idea being that like imagine. If you got to just— and then of course the world just like, you see the rewind where everything goes back in reverse and then the person just chooses not to do it. But like once, once every episode or 2, like somebody almost enters the world and they just decide not to. And there's really no explanation why they don't, probably.

Yeah. The sort of garden of forking paths is a pretty interesting idea. I'm reminded of The Three-Body Problem too, as part of its story kind of goes that way. Where the civilization is being restarted and rerun over and over again, and it just ends at various times and then it gets booted up again. But it's like they last different lengths of time, and some of them are short and others are longer, but they all kind of end and get rerun. And I do think that would be also a pretty interesting way to present the future, that some branches of this tree are terminal.

Yeah. The Three-Body Problem is such a weird— I don't want to spoil anything, but it's such a weird— and I'm going to do my best not to, but it's such a weird mix of this kind of fully cynical, hard realism beyond what I think is even accurate, where the universe is this cold place that wants to kill you so badly, and you can afford not the slightest bit of kindness and decency if you want to survive. At the same time, don't just all die. There's a book 2, a book 3. It's not a spoiler. The book exists. And the book isn't like, what happens to Trisolaris after we get wiped out? The book is about people in some sense.

Okay, last question. Do you find yourself shifting at all in terms of your sense of whether or not we may be in some form of simulation?

Simulation hypothesis has always been essentially all of the value lies in the world, lies in the places where we're not in one. If there's, if you take, you make 9 simulated copies of me and then there's me and you put us in 10 copies of the situation, but one of them is real and the other 9 will just be like recorded in a videotape and viewed back later or something. Well, shouldn't I just act as if I'm the real one? Isn't that just currently the correct strategy? Even if there's 9,999 of them, maybe I still have the correct strategy. Or rather, if you are the ancestor, if this is an ancestor simulation, and so then 20,000 years later, we try to run a bunch of sims of the ancestors. Well, the ancestors who decided they were in a sim— ancestors who, when faced with the ancestral situation, figured out that given the situation, they were probably in an ancestor situation, ancestor simulation, and therefore didn't need to actually make sure that they That the civilization progressed to a point where they could run the future ancestor civilization simulations. Those guys don't get simulated, right? Because those civilizations don't make it. There's a real sense in which your only simulation hypothesis is only valid if you treat it as invalid or something like that, right? You have to take the situation seriously. And also, what's the point of a simulation when the person finds out and acts like it's true? There are a bunch of movies like that, right? No spoilers, just even naming them. But in general, the point is to treat it as real. The whole goal is to treat it as real. And I don't really see any— I don't know, right? Obviously there is some probability that this is a sim of some kind. I can't rule it out. I will sometimes jokingly refer to things in that kind of way. The writers were a bit on the nose today is one of my things I'll sometimes say, but You can't take it seriously in the sense of changing your behavior.

What about moving to the Caribbean and unplugging? I just saw an interesting tweet from Amanda Eskil from Anthropic the other day where she said, I don't think AI is definitely going to kill us all. I'm not a doomer. If I were, I wouldn't be working on this. And she was also kind of like, I do think it's a real risk, but I think it's something that we can shape and hopefully I can have an impact on. But if I really was a doomer, I would just head to the Caribbean and spend the rest of my days there. I do know, I do have a close friend who basically has that attitude that he's like, I just want to enjoy the good times that we have and not worry about it too much. And then she also said the downside of this or flip side of this is if I ever do get burned out and decide to take some time off in the Caribbean, people will take it as a sign of doom.

I saw that too. Yeah, I think it was Jeffrey Miller. I'm not sure exactly who it was though. Who said it wouldn't work. The food would turn to ash in your mouth. You wouldn't get no joy because you would know this. And I think that that was, that's largely true for me. I think knowing I just walked away from this thing and was ignoring it, it just wouldn't sit well with me and I wouldn't be able to just go eat a nest. It just wouldn't work. Right. You can, you want Mr. Reagan to plug back into the matrix. Well, you need your memory kind of wiped in some sense. You need to really not know. Notice for some people. Whereas I kind of enjoy, I enjoy fighting. I enjoy struggle. I enjoy the striving to do better and to solve problems. That's what my thing is. I would never go to the Caribbean because what am I doing in the Caribbean? Right. I'm just gonna be bored anyway. But unless I'm just like posting, making sports betting, bookmaking decisions again, that's the only reason I've ever been to the Caribbean and had a good time. So there you go. But I think like It highlights, by the way, the fact that the word doomer is just a slur and has been completely misappropriated and misallocated. Because who is a real doomer? The doomer is the person who says that there's nothing to be done, that what we do doesn't matter. The doomer is the person whose P doom is 0.9 bar, or otherwise just it's all over. There's nothing you can do. And you see those people on climate change, doomers. who think that humanity is doomed and there is nothing you can do about it. Your decision doesn't matter. Whereas if you think your decision matters, as Amanda points out, that makes you not a doomer. Yeah. So what if it's, we could lose? We could win. And you can help fight. It's an ancient Judaic idea, right? The universe hangs in the balance, right? The scales oscillate and it could be up to you which way they were good and evil and God's judgment. Like you can decide like which way this goes. And obviously this isn't a God's judgment thing. It's not a, you know, there isn't a moral tone to this. It's about solving a problem. But, you know, if you can take it from 34.007% to 34.008% chance of victory, well, that's a great life, right? In some sense. Because there's so much, look at all the utils, including just for you. Imagine what happens. But yeah, like if you don't have any way to interact with the problem and you just decide to go off and do something else, makes perfect sense to me. And you can't burn out, right? If what Amanda needs to do once a year Sit Mai Tais on a beach in the Caribbean for 2 weeks so that she can regain her mental health and she can go back and resume, then she should do that. She should do it for an entire year after 5 years of working because otherwise she won't be able to think clearly or make good decisions. We should do that. There's nothing wrong with understanding your limitations.

Right.

Like, we can't— life is not all the fight, right? And I make an effort to like Work on other things, think about other things. I mean, family, try to have fun. I'm going to Madison Square Garden tonight with some of my old Magic friends, and we're gonna have a watch— we're gonna do a watch party where 40,000 of us Knicks fans are gonna watch Game 6 on a video screen cuz they're in Indiana. But it's gonna be fun as hell.

Right.

And I don't claim that I'm saving the world here. I'm not. I'm just Yeah, I feel you.

I think it's probably a good place to end it. Any other closing thoughts?

Indeed, do many amazing and wonderful things and weird things come to pass. And best of luck to everyone. And I look forward to OpenAI's response and where all these people land next. This is some great talent. Someone's going to snap them up or they're going to do something. So we'll see what happens.

No doubt about that. Well, the saga will continue, but for now, I appreciate the extra time today. Zvi Mashowitz, thank you for being part again of The Cognitive Revolution.

Absolutely. It's been a pleasure.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.