OpenAI Sora, Google Gemini, and Meta with Zvi Mowshowitz

Someone walks into Sam Altman's office, says, Google just announced 1.5 Pro. It has a million-layer context window, potentially going to be 10 million. It's really exciting. People are raving about it. Is now the time? And Sam's like, yeah, now's the time. But you can't have both ways. You can't both say there's so many chips coming online that we need to build AGI soon. And AGI is coming along so fast, we won't have enough chips. We need to build lots more chips and then go raise the money to make the chips. It would be surprising to me if a GPT-5 worthy of its name didn't make these agents kind of work.

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Eric Thornberg. Zvi Moskowitz, welcome back again to The Cognitive Revolution.

Always fun here. Let's do it.

Let's do it. So I call you when I feel like my own personal context window is overflowing and I can no longer find all the needles in the haystack that I might need to, to make sense of what's going on. It sure seems like things are happening faster and faster all the time.

Well, that's what Gemini, Gemini Pro 1.5 is for, right? You got the big new context window. You should be okay.

Well, how is your personal context window holding up, first of all? And then we'll get into all the latest news.

I think that my capabilities are advancing fast enough that I've been able to keep pace, but it has been rough. So the discourse speed has slowed down in the last few months. It feels like we had a lot of really big Developments and discussions around various potential government actions and also around the OpenAI situation. And 2024 has been more of a, like, okay, everybody chill, everybody process what's happened, and therefore we can focus on temporarily technologies. And yeah, a lot is happening, but also you start to see the patterns and you start to say, okay, this is a version of this other thing that's happened. And it's not that it's slowing down in absolute terms, but you gain the power to match the ability to process what's happening.

Is that a natural process for you, or are you changing how you work at all or your approach in any way?

So I think it's a natural process in the sense of I find myself saying, you know, let's try this again. Let's go over this again. You know, or as we previously discussed, right? So like you see these Matt Levine columns where he's like, well, you know, as we have talked about before, and every word is a link to a different Redisposed. It's the same basic idea of a lot of these concepts and questions and reactions are going to repeat, right? Like different people are realizing different versions of the same thing at different times. You know, the future is unevenly distributed. It's getting distributed. We pick up on when people notice this, especially when like mainstream media, you know, standard cultural people pick up on things that we knew 6 months ago, we knew a year ago. or we figured out was coming at that time. And so a lot of it is just, okay, yeah, we're going to go for this again. Yeah, I've already covered that. Yeah, this is actually from October. And that happened to me yesterday, actually. I was writing stuff up and I realized, no, like this is actually a few months old. I probably already covered this. It's fine.

I'm trying to figure out what I can do to be more valuable. I think, you know, I've never really considered myself a content creator in any sort of normal content creation sense. Starting a podcast was kind of a way to make sure that I was consistently tackling a new topic and hopefully sharing that with people in what I hope to be a somewhat useful way. It's gone better than I expected it would in terms of people being interested in listening to it. And after a year, I'm kind of like, how many of these should I be doing in kind of traditional interview format? How many should be more like this, where it's like a little more of a survey type of thing? How many should be more of like a deep dive where perhaps not even with a guest, I'm just kind of really going hard at a particular topic. I did that late last year with the Mamba architecture and really felt like that for me was a more profound learning experience and hopefully a better product for the audience than a typical episode. So I don't know if you have any immediate reactions to that, but if not, we can maybe circle back and at the end you can tell me what you want.

I do, actually. I would say You should basically always strive, in my opinion, to do more deep dives, like not just you, but like almost everybody, than you naturally want to do. Like, I mean, I'm working on a newsletter, obviously every week I am writing, you know, an average of 10,000, 12,000 words or something, you know, including quotes about what has happened specifically in this past week. But I find the most valuable things, especially if you're building up a long-term base of operations are when you get to focus more on specific places where you feel you can bring your insight, where you can be the go-to source.

Yeah, that was my experience with the Mamba piece. So I'm looking to figure out how to do a little bit more of that. So one of your big pieces and certainly a deep dive has been recently into your experience with Gemini. This is obviously something that we did know to expect. And if anything, you know, seemed like it probably came a little later than we might have penciled it in for. You had the good fortune of having early access. I would love to hear how that happened and see if there's anything that I can potentially copy from you going forward on that. But then obviously more important is what your experience with it has been. And then of course, now we've also got 1.5 to look forward to as well. But let's start with 1.0, which you said, you know, among other headlines, had become your default language model after a couple of weeks of playing with it.

Someone at Google reached out to me and said, we'd love to get your feedback on this new version that we're going to ship soon. Would you like to check it out? And I think that, you know, I did was I established that, you know, I'm someone who could be trusted. If you share it with me, I'm not going to go spread it around the internet. And I can provide useful insights that are different. that are well worth the time spent interacting. And then obviously, if you share it with me, I'll be able to offer more detailed insights right when it's released, which can impact how people see it. And they presumably thought they had a good product. They wanted me to give my honest opinion. There was no attempt to put a finger on the scale in any way, no attempt to convince me it was better than it was. It was all very, very honest and straightforward and realistic. I thank them for that. It was a really good experience. They seemed to care about my feedback. This was someone who clearly wanted the information, wanted to understand. So what I noticed right away was it was really good at many of the natural things that I wanted it to do. So what a lot of people will do when they get their hands on a model is try to get it to do something horrible.

I'm guilty of that.

Well, you know, Sally has 2 apples and then, you know, John brings 2 oranges and they all play a juggling act. And then they make salad. How many apples are left? And the model's like, 3. It's like, no. And it's like, then what's the point? In some sense, there's other people, I figured, whose jobs will be to red team this model, either to get it to do harm or to find out where it's dumb or where it's going to fail on theoretical tasks. And I took the opposite approach. I asked myself, what good can this do? And what happens when I feed it exactly the queries I just want the answers to that I would normally want to do over the course of my day? We have some amount of curiosity, and I did do some amount of brainstorming and trying some stuff out and trying some parallels and so on. But what I found was mostly the things I want are explaining things, finding explanations, just general, like answering my questions. I'm confused. I want you to search for, maybe search for a document in some cases. Although the version I had back then didn't do that. That looked at other things. Like, I'm just a very practical user of LLMs, but I'm not trying to help it do my job. I'm trying to like extract the information and understand it so I can go about my day and do the rest of my stuff. And for that, it was very, very good. In particular, I was on a trip. I was trying to read the Financials DPO paper that I never got around to. And I was asking it a question every paragraph or 2. And it was very, very good. It's not having been fed the paper, but it seemed to have the information. And it was very, very good at just answering my questions on exactly the level of which I was asking them and figuring out exactly what information I need to know. It did a lot of other similar things well. I noticed it had some flaws. Like, I would try to use Google Maps and it would sometimes give me good information and sometimes it would repeat 3 different times for how long it would take to get to SFM from Berkeley within the space of 3 questions in the same chat, despite having the prior reference, which is better than just artificially agreeing with yourself, right? I was admiring the fact that it wasn't doing that. But at the same time, it was incredibly frustrating. Like, I can't count on you for anything. This is weird. I claimed to know what the weather was and then it would repeatedly not know what the weather was. I found some other bugs in the system that really just— clearly this thing launched like earlier than it was supposed to.

Yeah.

And it's also the 1.5 launching like 2 weeks after 1.0 Advanced, right? This is not a natural thing for a like well-planned out release schedule to do, right? If you almost have 1.5 ready, You don't make your big announcement speech right then unless, you know, this is me purely speculating, the CEO was on the phone with you and saying, no, you're damn well going to launch right now. And you say, but we're almost ready with 1.5. He's like, I don't care. You're going to launch. I can't not launch now. And then this happened to me, right, when I was doing a game company and we were just told you're launching today. And we said, we're not ready. And they said, we're launching today. I said, It's Friday. Can you at least wait till Monday, you idiots? And they're like, no, you're launching today. And we would not have succeeded if we launched on the following Monday, but at least it would have been slightly less embarrassing and doomed. But, you know, just, it's clear that, you know, there was a, we need to launch this in whatever state it's in. And then we're seeing rapid improvements. And to the point where we're not sure if 1.5 is better than 1.0 Advanced. You know, discounting compute, it's got advantages and disadvantages. Which I haven't had the chance to properly explore yet. But yeah, I was just really impressed by this is faster, this is smoother, this just does what I want it to do. My workflow is easier and better. I want to use this. I don't want to go to GPT-4 if I can help it. I do that when I feel like I have to, whereas I used to be like, this is where I go. And then sometimes I use Claude, right? Sometimes I use Perplexity.

So can you put a little finer point on what it is that makes it better? I mean, I know that it's obviously tough because the surface area is so vast and there's so many facets to language model performance. For what it's worth, I have also been impressed. I haven't spent as much time with it as you have. Today I am going to Gemini Advanced, aka Ultra 1.0 Ultra, and ChatGPT with GPT-4 and Claude, basically for anything that I want help on. I did this once with a little kind of proto contract negotiation that I'm working on where I had kind of a couple of messages back and forth between me and the person I'm working with and kind of said, here's what this person seems to care about, here's what I seem to care about, here's our messages to each other. Can you kind of help us get to a good win-win situation? They certainly all behave a bit differently, but I found them all kind of useful. And the synthesis of the 3 was, for me, kind of a way of telling myself I didn't need counsel because I had enough kind of AI counsel to make sure I wasn't leaving any major blind spots for myself. And then also did this with basically a job description where I have been working with a friend on a hiring process. We've had a number of calls with different experts, got their opinions. I've taken very raw notes. And just fed in kind of a paragraph at the top of like, okay, now it's time to turn all these notes that I took over various calls into a job description. You write it. And of course, they each did the task. In that case, I did find, and I think this is fairly random, I wouldn't say it was like a systematic finding by any means, and I'm not sure how frequent this would be, but I did find in that case that the Gemini response was the one that I wanted to go with in terms of just taking something editing to a final product that I would hand over and feel proud of. Gemini did come the closest to what I was looking for. But those are spot checks. Can you help us kind of get any more systematic or predictive in terms of what the advantages of Gemini would be?

Yeah, it's interesting. It's kind of the opposite of mixture of experts, right? It's a mixture of non-experts where previously you would figure out, okay, I have these 20 different expert systems and the program figures out which one to call as they run compute. Now it's like, no, just call everyone. Compute is cheap. Compare all the answers, see which one is the best, which makes sense for a human case. And yeah, I even have 1.5, right? I have access to 1.5, not in the standard interface, but in the developer studio beta. So I can then not only compare Claude, GPT, and Gemini, but also the other Gemini. So now it's like I have the 2 Geminis to work with and the name finally makes sense. So I think what I, what I found was the, the biggest feature was just It tells you the information you actually want to know when you ask a question much more reliably and with less like worthless stuff, like scaffolding around it. And also I have gotten some good use out of the Google button. So when you answer on Gemini Advanced, one of the things you can do is you can press the Google icon at the bottom and it will search for web sources for all of the things that you've just had. And they'll light up in green. They indicate they have a source that you can now click to. And it's just very seamless. Actually, the guy I was doing the test with showed me this feature. I didn't understand it was there until he told me. And then you can then ask for a source. What's going on? Do you have any physics on this? Do you have any knowledge of this? And then you can find the backup. And that was very helpful. In general, as I said, it's also just very good at understanding, okay, if he asks this question, what does that mean he doesn't understand? And what does that mean he does understand? And I find GPT-4 can be pretty dense about this. And I probably can work with better custom instructions to try and improve this than what I have. But, you know, so far it's been kind of frustrating. Claude, I found, is a step behind for most purposes. I use Claude mostly when I'm looking at long documents. But I think your, your use case is a very different one because you are asking for a creative job, right? Where like you get lots of different answers. There's no right answers. There's no wrong answers. There's just different answers. Being able to take a sampling and then choose the best one or combine the elements of different ones helps you a lot. Whereas what I'm mostly trying to do is I'm trying to ask questions where there is a right answer. I'm trying to look for an explanation or I'm trying to just get some brainstorming at most to try to go with. And so I try Gemini first and if Gemini gives me an answer that does the job, I'm done. And it's only when I get frustrated with that, like, okay, fine, I'll try all these other people. I won't jump from 1 to 4. I won't just jump from 1 to 2. I've just been really happy in practice with Gemini Advanced. It does what I want it to do, except when it doesn't. But I've adapted to, okay, those are the things that I just don't do. I haven't had much call to use GPT-4 specifically since then because it just doesn't come up very often. The cases where GPT-4 is good, potentially, where Gemini is bad, are where you're trying to customize it in these very specific ways, I think. And those cases just don't come up for me very often. So it just doesn't seem much use. And I'm not going to cancel it because even the option value is just so much more than like what tickets they charge.

Hey, we'll continue our interview in a moment after a word from our sponsors. Just to give a quick plug to Claude on my legal question, I did find it to be the best in the sense that the commentary that it gave back to me was least sort of padded out and RLHF'd, if you will. It was more kind of direct, to the point, conversational, and actually just kind of concrete on the suggestions. It seemed like the others were a little more reluctant to— and I've actually been pretty impressed with Claude on this front a couple of different times. Including one specifically pertaining to you, but finishing this story first. The just kind of plain spoken, to the point, direct, not super hedgy with both Gemini and GPT-4. By comparison, I felt like it was really reluctant to make a recommendation, to make a suggestion for what the agreement might ought to be. It was very like, this is an important consideration. You guys will need to figure out what the right thing is here. And as I pushed them, they would do a little bit more of kind of, okay, here's a specific proposal. But Claude was much more readily willing to just be like, here's kind of what I would suggest. And it was pretty apt. And again, I did not copy and paste this into my running dialogue directly, but it was the closest to— in that case, I actually wrote my own thing and again took my own thing back to each of the 3 models, said, okay, based on your suggestions and my own thought, here's what I came up with, and asked for one more critique and then basically sent it from there. But if I had been inclined to pick one, from the 3, it would have been Claude because it was just the most kind of to the point, easy to understand, not necessarily exactly what we're going to go with, but at least kind of willing to put something out there. And the other one, as you may recall, that I've been kind of impressed with on this from Claude was when we had in our last episode, you made an offhand comment around like, I'd rather you go and do something terrible than open source a next generation LLM. And a listener said, hey, that's a little extremist. You shouldn't say that. So we edited it out. But then I also got the idea in my head, well, maybe I should use a language model to review transcripts in the future and see, is there anything there that the guest may wish to retract or maybe crossed the line or whatever? And again, it was very sophisticated in its commentary on that point where you might have the impression of, Anthropic broadly as a company and cause it to be like super conservative and don't ever say anything like this. On the contrary, it said, I didn't see anything wrong with that. It seemed like an offhand comment. It was clearly not somebody endorsing this terrible thing. It was kind of hyperbolic to make a point, and the listeners shouldn't have any trouble distinguishing that from an actual call to violence or anything along those lines. So It is interesting. I mean, they all have their own character, and it's been kind of surprising sometimes to see that Claude, especially given corporate reputation and all that, is in fact a little bit more accepting of some things that might be considered possibly offensive or possibly coloring outside the lines, and also a little bit more bold, if you will, in terms of actually making a concrete suggestion and not not hedging as much as some of the others. Of course, that may be a reflection of the other models having a little bit more power under the hood and just being therefore even more RLHF'd as kind of a compensatory measure. So we'll see what Claude 2.2 or 3.0 looks like when it comes.

Yeah, I worry about that. I think that a lot of what we're seeing is there's more and more fine-tuning more and more of this other work being done to these models over time. And we're getting this bullet point, super hedged style that's very inhuman, very like, I see exactly how it makes sure not to get the thumbs down. And I see exactly how it makes sure not to piss anybody off. And it's incredibly frustrating and makes my life worse than if you just do the thing. Although there's some cases where it's actually just exactly what I want, but often it's just, So frustrating. There was one case earlier, I was doing this for next week's post, and someone had this idea that Dune was a cautionary tale about the dangers of not building AGI. And so I just wanted to, like, okay, just to prove a point, I'm going to try all the different LLMs and ask them what Dune is a cautionary tale about. And they all got it right. They all had the right first bullet point, but only Gemini Pro didn't do the bullet point thing and just said, no, actually, Obviously it's this one thing, and then gave me the kind of details you would actually want to know about, which is illustrative of the thing that I love about Gemini, right? When it does the thing, which is, no, I don't want to know all the different potential interpretations one could make of this book. I want to know exactly how this is a cautionary tale and what are the details someone would want to cite in order to just fully not be accused of pulling something out of the air. I find that's kind of what I'm appreciating often more and more. It's just like, are you just willing to do the kind of thing that you would have much more easily done a year ago when this thing first came out? Can I just have a version that's designed to be better for humans?

Right.

And then the question then becomes, how do we avoid this in the future? How do we not train these things to hell in these ways? I think there's some promise for that.

So a couple of big themes there to unpack a little further, but maybe for starters, do we feel like Google has taken the lead at this point? It sounds like if you thought Gemini Advanced is ahead of GPT-4, and now we have 1.5 Pro on top of that, which certainly the existence of 1.5 Pro suggests the, or implies the eventual, probably likely fairly soon existence of 1.5 Ultra as well. Obviously we don't know what the others have under the hood, but it seems like at the moment there's a pretty good case that Google has taken the lead in terms of public-facing products.

I think you have to be very careful about in terms of public-facing products, and you have to be very specific about what use cases and modes you're talking about. And you have to understand that This is a year-old product that was released on a half-year delay versus a product that Gemini that is clearly, you know, being reshipped and refined and released in real time. Like as soon as they have improvements, they ship those improvements, which is great because it keeps improving. But it does mean that Google doesn't have this year and a half of background work in the tank. And what OpenAI has deployed in that year and a half Not counting Sora, which we'll get to later, I'm sure, is they've deployed additional features, but the core model, if anything, has gotten worse. GPT-4 Turbo was kind of a reversal of some of the problems that developed GPT-4. But at core, the thing isn't particularly stronger. They have better customization options and they have these GPTs and they have constant instructions and they have identification of who you are. And now they have memory, although I haven't had a chance to try out memory because I haven't been using the product. I think there are still a lot of advantages of the ChatGPT product.

Yeah.

Over the Gemini product. And which one you would say has the lead depends on what you want to use it for. For me, I would say Google has taken the lead in terms of public-facing products. But if you tell me, you know, what's going to happen when GPT-5 is released, I'm going to assume it's going to immediately leapfrog OpenAI into a substantial lead.

Yeah. And one wonders how soon that will be, obviously. I guess, do you have a sense? I mean, your comment earlier about, you know, was the CEO on the phone saying you must launch? If you did have 1.5 right around the corner and we've come this far, right, without launching the 1.0 Ultra, why bother? I guess it's, what do you think is the model of who was forcing that and why? How does this update to the general market dynamic have you thinking about how the leading companies are relating to one another? Every time there's a new release, I sort of squint at it and I'm like, does this look like they're sort of implicitly coordinating or does it look like they're trending more toward arms race? And I'd love to hear what, where you feel like the balance of the evidence is now.

So my model of this is something like there's 2 different sub-departments, right? Each of whom is tasked with a different thing. There's people who are working on Gemini Advanced, there's people who are working on Gemini Pro 1.5. And then there's a third group of people maybe, or I think it's probably the first 2 people who are working on Gemini Ultra 1.5. When we'll, we'll see it when we see it. But, you know, they're under different pressures. They're releasing different products. They're releasing them in different market segments. 1.5 isn't released, right? 1.5 is available to a select group of people in a high latency special interface, which is very, very different from a customer-facing product. So Google decided it was ready to launch the customer-facing, universally available product. And didn't necessarily want to sync it up at the cost of delay with this 1.5 announcement. And then the 1.5 announcement got stomped on by Sora, I think rather unfairly in terms of what people were paying attention to. But that is what it is. And so I think it makes sense that this is just different dynamics playing themselves out in ways that then look stupid to the outside, but which would make sense if you understood the internals of Google and you understood the different commercial pressures and Other pressures. But again, it's only a guess. We don't know what's going on. And in terms of the race dynamic question, I would say Google is forced to dance. Google is dancing. Google is clearly, you know, racing as fast as it can to develop whatever it can as quickly as it can. The good news is that Google seems to be focusing now more on getting the most out of its smaller models rather than trying to make the best top model. And also trying to use the fact that its model is smaller to enable things like a larger context window, which allows it to get more multimodal. And so I see this as focusing on providing the product that the customer will actually get more utility out of, that will be of more practical use day to day to a normal person, as opposed to trying to make the model fundamentally smarter and more capable in a way that would be both more exciting and more dangerous from an abstract point of view. And so that's what I want to see, right? I want to see all the cool stuff that we can use to have a great time and make our lives richer and better. And I don't want to see as fast this race to just make the biggest possible model that has the best possible chance of suddenly doing something crazy unexpected. And so I would say they're racing, but they're doing the right kind of race. In some sense. So it's good news and it's bad news.

What are you most excited about for the super long context? The version I understand that you have early preview access to is the 1 million. And then they've also kind of teased that they're going to have a 10 million context window, which is, I mean, you know, folks who listen to The Cognitive Revolution probably have a pretty good sense of the history of context lengths. But, you know, my joke of earlier this year has definitely come true. Like context hyperinflation is definitely upon us. We've, gone from literally still less than a year ago, the longest context available for an equality model being 4,000 to then 8,000 with GPT-4 release, 32,000 with GPT-4 32K. Claude hit with 100. GPT-4 Turbo came back with 128. Claude went to 200. Now we've got a million and 10 million on the horizon. And what has really impressed me out of the demos that we've seen has been not just that the context window is super long, but that the recall out of that context window seems to be super reliable. And again, I think people will be pretty familiar with the fact that if you do stuff more than, let's say you put 100 out of 128 or whatever into GPT-4, or you get close to the 200 with Claude, it's not always a gimme. Maybe you can kind of get there with good prompting techniques, but in general, it's not reliably the case that it is going to spot the one bit of information that you need or that it's going to synthesize everything in that context window effectively. The demos from 1.5 seem way better on that front. And in terms of mundane utility, I just think about something really simple like finding the right stuff in my Gmail, which is obviously something that Google has a lot of reasons to try to make better. It has been a real struggle, right? In all these different agent products and whatever, you've got all this scaffolding and the retrieval and the re-ranking and all this because it's not just enough to get things into the context, but you also kind of need to keep it shorter for cost, for latency, but also just for accuracy. If all of a sudden you could do 10 million tokens in a single call and reliably get what you want to get out of that, It seems to me like that really is a step change advance for a lot of use cases that, for example, would make my email agent assistant just remarkably more effective than it is today. What else is on your radar for things that you think will change, even though the power, the raw reasoning maybe isn't that much better, but just because the memory and the recall is so much better? Hey, we'll continue our interview in a moment after a word from our sponsors.

Yeah, there's situations in which you reach a certain point and it suddenly goes from completely useless, because you might as well just do this job yourself, to suddenly, oh. My eureka moment with Gemini Advanced's capability of hooking up to email was, give me a list of everybody who RSVP'd to the party. Enter. From individual emails and having it produce the correct list. And then that's like, wow, that's really, really useful. Now they just press print, hand this to the front door. And so they'll let them in. Great. Another moment is like being able to use, and the hope is to use an interface like NotebookLM even would probably be even better. But the idea of, I have these now 50+ things that I've written about every week and to be able to use that now as a bank of information that's all in the context window to look at everything else I've ever written. And say, okay, based on this, retrieve everything I said before about this topic, or what are all of the different, you know, resources that I may have mentioned or whatever, or even what have I forgotten to mention? What have I knocked out? Have I ever talked about this before? Right? Like, this is really exciting to me, but it only works if you can extend it really big, right? Otherwise, you have nothing, right? It doesn't really help that much unless you're like, oh, I think it was in 37. Check 37. Like, no, it's not 37. Like, goddamn it. Now what do I do? Try 38. This seems annoying. But some other use cases, video, right? Like suddenly the context window is enough. You can just feed it a 2-hour YouTube video or a movie, right? This is one of the things that Google showed off. And then you can ask specific questions potentially by doing iterated searches. You can even do, you know, television series, anything you want, you know, Twitch streams, yada, yada, yada. Just ask anything you want to, you know, find, find the specific clip that I'm looking for, understand the quote, find the specific whatever, understand the context, et cetera, et cetera. And that stuff's exciting to me. I'm also excited for the possibility, I haven't heard anybody talking about this, but of using it as fine-tuning, right? Effectively using it as training. So the idea being that if I have 100,000 words or 500,000 words, suddenly a million words with good recall and good understanding. Well, now I can go, in the style of the things that you are in your context window, would you please respond with, right? Or using the thinking mode or using the willingness to do X or blah, blah, blah. And now suddenly maybe there's something there, but it's something you have to try. And it's again, if you don't have good recall, you don't have good assimilation, if you haven't reached a certain threshold, these things don't work. And then suddenly they work. And once they work, you're off to the races. But yeah, it's just, it's super awesome. There's also the fact that like, frankly, even the 200,000 context window was sometimes just not long enough. And it's something as simple as I download a page, I feed it in, it says this is 5% too big. And now I would have to spend like 2 minutes, you know, for various web tools to try and like chip off 5% of it. Like to get rid of the acknowledgments and the references or whatever to try and make it fit. And I didn't want to do that. And in fact, like it's this little game of like, do you think this person made sure it was too big for the context window? Like the FTC just didn't want anyone to be able to read this properly. Like, I don't know. I had this thing open on my machine, the AI Act, the EU AI Act, right? In large detail.

So will that fit into the 1 million?

I think it will. I mean, it's not as full-page. It's a changelog, so it's got like 4 different versions of it to write. It's like really weirdly formatted and it's terrible and it keeps getting worse. The most amusing part of it is you can see all this blue inserted text and it's just people who are like, could we possibly be more pedantic, say more words, put in more specific cases, wishcast more, just make this act worse continuously everywhere?

So do you have any intuition for what is going on with the 1.0 to 1.5 leap? I mean, what they've said publicly is mixture of experts architecture, which certainly tracks with other things we've seen. I mean, GPT-4, I don't think has ever been publicly confirmed as a mixture of experts, but seems to be kind of credibly leaked, reported, not denied, whatever. So I'm comfortable saying it seems quite likely that GPT-4 is a mixture of experts. And then Mixtral obviously also put out a very good open source mixture of experts that gives some additional juice to the idea that this is like a promising path. And now Google is obviously saying it, so we know that much. It's a mixture of experts architecture. They say it takes less compute. It's interesting because people often talk about the attention window being quadratic in the attention window, which has been true, though people have often overestimated what a big deal that is, because it's not until you get to a pretty long context window that actually starts to dominate. At the more modest context lengths, still just the MLP block is like the bulk of the compute. But certainly you get into the hundreds of thousands, the million, the 10 million, range, and now you are with conventional known techniques, the attention block would be dominating the compute. But this now, they say, takes less compute. So do you have a sense for what might be going on under the hood there? Fair to pass if you don't want to speculate, but this is definitely something I plan to research more and see if I can't get a read on.

I definitely don't know. The first thing I would say is, You know, the timing on Gemini 1.0 looked very, very much like we are going to launch the moment we can produce a product that can match what we're up against, right? Like just barely we can match what we're up against. So 1.0, right, as opposed to Olsa, was just, can we just barely be 3.5 level with a small amount of compute that we can serve without worrying about people abusing the system? And the moment they found it, they released it. They're still on the, we're making rapid progress part of the utility curve, right? GPT-4 clearly hit a while ago the point where it's like, okay, we did our thing. We made this the best we know how to make it. And now rather than try and tweak it or keep working on it, we're going to switch over to working on GPT-5. And then we'll let you know later what GPT-5 is. We'll improve 4 by providing different modalities. We'll improve it by adding features, but we're not going to try to improve the core thing that we're doing. Whereas Google hadn't finished their work, right? They were probably always intending to be a mixture of experts system, but they just hadn't gotten around to it yet. And that's probably a huge leap on its own. But also they hadn't tried dealing with these context windows. They just did this now. A bunch of the multimodality came in now. A bunch of the just general, and also that they had remarkably little feedback because their system was not being used by many people. And now this allows for rapid iteration of their products. In other ways. So I think they have a lot of different ways to continue improving here. And also the fact that they made a large leap from 1 to 1.5 so quickly implies that, you know, 2 is coming, right? Or 1.7 or—

Doesn't seem like a stopping point. I would give, for what it's worth, the GPT-4 progression a little more credit in that it has gotten 60% cheaper, 4x'd the context, and that's even relative to the 8K. Original context. So 8K and 32K, the 32K was twice as expensive. Now at the 128 context window, it is only 40% as expensive as the original 8K. So that's non-trivial, both in terms of improving the utility of just with the length, even though we, as we've discussed, there are some weaknesses there if you really stuff it full of stuff. And the cost is better and the latency is better. And I personally haven't necessarily felt this, but if you go look at the LM Sys leaderboard, GPT-4 Turbo seems to be like a notable cut above even the earlier GPT-4s in terms of just user win rate. So there does seem to be something there that they have done in terms of, if nothing else, just like taking user feedback into account and kind of further refining it into what users want. And again, I couldn't even put my finger on that because I wouldn't say I have felt a step change. I felt the improved latency. I've certainly felt the increased context, but I haven't felt like just general quality to be all that much better. But the leaderboard results do seem to suggest that. Would you, I mean, do you have any reason to be Skeptical of the leaderboards, or how would you interpret that?

It's not that I'm skeptical of the leaderboards. It's more that I think that the leaderboards are measuring things that are being managed and measured that are not that great of proxies for the thing that I care about or that I think other people should care about. And in broad strokes, they're going to be very good measures, but in this particular case, it's going to potentially mislead you quite a bit. I think a lot of the things that we think of as why GPT-4 is worse are things that are being done because they improved things like the leaderboard effectively. Even if this is not the mechanism, which it's not, it's still effectively what's going on. I agree that drops in price, increasing context window, I add custom instructions to that list as well, and potentially the GPTs, although I still haven't seen a good use case for GPTs per se. But yeah, they're doing things. They ship. No one's saying they don't ship. But these things all seem so minor. If you compare that to the improvement in Gemini, or Bard, right, over the past year. It's like they were way ahead.

So I agree with you, by the way, that— but I think it is also debatable. We had the 1.5 announcement and we also had Sora from OpenAI on the same day. And so Sora, of course, is the video generator that definitely seems to represent a leap over any public-facing product we have seen. Although I would note that Google also has announced, I should say, multiple video generation models over the last month or 2 as well, which also do have some very compelling sample outputs. Lumiere is one of those and probably the one that had me most excited, although there's another one too. I mean, they have multiple teams working on independent approaches on video generation in Google, and they're also showing some pretty impressive results. Also, I believe on the exact same day we had the Meta version, and I haven't even really had a chance to study this yet, but it seems to be more of a kind of backbone for video encoding. And I think there's multiple interesting aspects of this. Obviously, the ability to generate hyper-realistic video and what that might mean for creative people and what the future of Hollywood budgets might look like and whether or not they're going to have to tear up the recent agreement with the actors. I mean, there's a lot of things there that are sort of implied, or at least called into question just by the ability to generate very realistic video. I'd be interested in your thoughts on that. But then I think even maybe the more interesting question is, is this real world modelling? Have these things learned physics? I feel like there probably is some very significant physics-like world modelling going on in there.

So my section title slash potential post on this is tentatively titled Sora or What? Because I don't really, in some sense, see what the big deal is. I think the ability to actually use this for creative or commercial purposes is being pretty overestimated at anything like the current tech level. It's one thing to get demos. It's another thing to be able to get what people need in order to use video because the problem is that like, With a picture, there are a lot less things that can go wrong. There are a lot less things that can stand out. You can build around it. You can edit it. You can modify it. You can take advantage of the sort of anything at all kind of advantage. And you can keep tweaking it and ask it 1,000 times so you get it closer to the thing you want. With video, I feel like there's so many little details and yes, they're getting a lot of them remarkably well and correct. But it's still not going to give you the thing you actually want in detail. It's still not going to be the product you want it to be. It's still going to have all of its quirks. It still looks artificial if you're paying attention. I think people will learn to recognize artificial video, if anything, much easier than artificial photos. You look for any little thing that's off and you just know that it wouldn't happen if you were using a camera. Like they filmed something. And I just don't think Hollywood should be quaking in its boots in any way anytime soon. So I'm just a skeptic that I just never had the temptation to generate a video. Whereas it's like, I can see generating photos. I've used photos. But like the idea that it could ever give me a— that Sora would, if I had access to Sora, would I be able to generate a video I'd actually want to use? I don't know. But I'm also just a video skeptic in many other ways. Like people like want to generate content that's full of videos and I keep telling them no. Don't do that. Generate text, maybe generate audio, but think very carefully about whether or not there's a reason to want to use video. It will of course blow out the low, very low end in some sense for generating video at all. Some people will be happy with just, oh, this looks really cool, and then I can narrate it and like fit to whatever they think happened to give me, and I can tweak it, and we'll see what happens. Sora is a very clearly technical large leap over what we've seen before. Whereas Meta's announcement, I didn't see anybody talk about Meta's announcement at all. I just, I have to assume it just wasn't as good.

Yeah, I need to understand it better. I think it's more foundational. I'm not even sure that they have released any like heads for it. It's like a backbone play where it's about encoding and moving toward this like world modeling through video prediction concept. But in terms of Actually applying it to tasks, I think that's still kind of on you. They haven't gone as far as saying like, here is the full end-to-end thing that you can actually use. But this is definitely where this is the perils of near real-time analysis. I haven't understood that one as well as I certainly would want to, to make an informed commentary on it.

My assumption is that the public is going to come back when you did that. Come back when you have pictures. Picture didn't happen. Good luck.

Yeah, I think the people that are probably really excited about Meta, and again, like, I need to get in this too because I'm, I'm not in this game exactly, but I'm like adjacent to it, are the people that are trying to make apps that would ultimately like aspire to compete now with Sora or with Lumiere from Google. You know, it's similar to like a Llama, the idea there, and certainly what What people have done mostly, I think, with Llama is fine-tune it for things, right? And run it in their own infrastructure and hack on it in all sorts of ways. And I think this is kind of meant to be similar where it's like OpenAI is going to give you their black box thing and Google may give you their black box thing, but this is something that you can mash around and do whatever you want with. So it is going to take some time before the folks that are kind of hacking in that space would probably even be able to report back to say whether it's doing anything for them or not. But I'll take a little bit the other side of the impact. I would agree. We have a great episode on this sort of thing with 2 of my teammates from Waymark, Stephen and Josh, who led the project called The Frost, which was a short film that they made Entirely with DALL-E 2 imagery almost a year ago now that they were really in the thick of it. And all of the challenges that you described there were very real for them, even more real, presumably, than they will be now. You have issues with just all sorts of weirdness, like a hard time— control is not great, hard time getting exactly what you want. That's, I think, improved, but still an issue. Consistency of characters was one huge challenge point for them. You could say, like, I want to put this same character in all these different places. How do I do that with DALL-E 2? It wasn't really a way. They did find some ways, though. What Stephen talks about is using archetypes. Basically what they found is that there are certain prompts that manage to get very consistent characters, not perfectly consistent, but very consistent because they're sort of a local minima in character space. And so by kind of finding these archetype text prompts, then they could get the same thing, or very nearly so, repeatedly. So they were able to ultimately make a 20-minute short film that has kind of an uncanny valley feel. And again, it's a year old already, but definitely has continuity, has a storytelling element to it. They, of course, brought that. DALL-E 2 is not doing that for them. But I would say the bottom line was they were able to make a short film, 20 minutes. It's been included in a couple film festivals, and it's a quality thing. Is it like box office? It's probably not quite box office, but it is something that a lot of people have watched just to see what AI can do, and also because it's kind of entertaining. And the accessibility of doing that has certainly Totally changed. This thing is set in Antarctica. There would have been no way for them to create anything like this other than using these tools. So I do think you add a— and that was all images, right? So they were doing slight additions of movement and kind of overlay effects, adding snow falling down on top of images and kind of subtle zooms to create effect. And now you can get up to a minute of coherent video, I do think that is going to just change a lot of how people produce stuff. It doesn't seem like it's going to replace authorship anytime soon, but there is a— I don't know if it's on Metaculus or Manifold or whatever, but there is a nascent betting market on when AI will be able to make a critically acclaimed feature-length film. And the critically acclaimed part there will be probably where rubber hits the road. But you could start to see, hey, if you can get minute clips, 100 of them takes you to feature film length. Can you start to sketch those together? We're not there yet, but I would expect that there will be impact on the creative field that will shift more towards storytelling, more toward kind of concept work. And certainly allow people like Stephen and Josh to make things that with our very tiny Waymark budget, there would be literally zero chance of them otherwise making.

I think we'll probably see various ways in which you can add flourishes and edits and richness to something you would otherwise have done, finding ways to make the process more practical. But in terms of the idea of the AI will just produce the whole thing, Yeah, I'm still pretty skeptical that it will get there. But again, we'll see. I think that the images approach, if anything, is the way I'd still do it. DALL-E 3 is much better than DALL-E 2, and now you can create a much better set of images. In fact, if anything, I'd probably involve— video will do is you'll get a very carefully crafted starting image, and then you'll feed that image into the video generator as the starting point. Maybe you'll even generate separate images at the endpoint or something. I'm not sure. how exactly how this works. And then you will, you will do this slash, you'll film it, you'll do a film, but then you'll want to do things like, okay, then we actually wanted to do a pan out here and things like that. But like, again, we'll see over time.

Yeah, working from a still is, I think, a very underappreciated option as well for Waymark with the actual product itself. It has the potential to be huge because what we do is all for small and local businesses, and they typically don't have quality footage at all. Most of them will have a pretty decent image library. And so we've kind of built our whole product over the years with this assumption that more than 90% of our users are just straight up not going to have any video assets. So we do support, if they do have them, they can upload them. We have a stock integration, whatever. But mostly they're working in images because that's what they have. So to be able to say now, okay, here is whatever, a picture of people eating in your restaurant, let's bring that to life even for just a couple seconds. We don't even need anything anywhere close to the 60 to make it a big deal. We just need it to work well. And I have been going out and scouting the different products that are out there today, your Pikas and so on and so forth. And at least for that, start with an image and animate sort of thing. I haven't seen anything that I was like, our local advertisers would want to put this on TV and think it's going to invite people to their business. But I would bet that Sora probably does cross that threshold. And now all those 1 and 2 second still shots probably become video scenes over the— basically as soon as we can get our hands on it, we'll start experimenting with that. Of course, we're begging for access behind the scenes, but it sounds like nobody outside of the OpenAI set will be using it for a little while still. I guess that is one more interesting question there, and I really do want to get to the physics. But on the release of this, it's interesting to see this is not a release, right? It's an announcement. We're seeing kind of more of these announcements of capabilities before even a very limited release. In the case of Sora, as far as I know, nobody outside of the organization and maybe the red team is getting access to it. Why announce? They don't need to. I guess you could say it's just for hype. Is it to help the world prepare for these sorts of things? What is the reason for announcing something that you're not releasing?

Recruitment is probably one motivation, right? Help us work on this amazing new cool product.

They're hiring video infrastructure people now. So yeah, that certainly makes some sense.

Right. And so now he's like, what is it for? Why should I apply? Like, this is why. Generate some hype. You know, they have stock now. They're trying to drum up interest. They might want to be doing things for that reason. Just generally, this is good first practices, what people, especially in media, generally do. In this particular case, there's an obvious hypothesis. I have no actual evidence for it, but Gemini 1.5 dropped first and Meta's announcement, you could think of it as Someone walks into Sam Altman's office, says Google just announced 1.5 Pro. It has a million-length context window, potentially going to be 10 million. It's really exciting. People are raving about it. Is now the time? And Sam's like, yeah, now's the time. Like they were working on it. They just keep working on it. They could have announced it probably last week or next week. So, you know, the plausible thing is that they, they stopped on Google's announcement on TARDIS, right? They wanted to kill the hype, so they had this thing in their back pocket. They're like, the next time Google or anyone else important tries to drop some big hype bomb on us—

Sam, for what it's worth, has denied that, and you can judge for yourself whether you want to take it at face value.

Again, I'm not confident this happened, but he would, wouldn't he?

I find him to be, you know, I don't want to anchor too much on this over a super long timeframe because situation is changing and his incentives may be changing and maybe his behaviour is also changing. What I did see during the GPT-4 red team to release window of 6 months from him, and that was a time where I had this kind of privileged view into what was coming from them and really no outlets for anybody to talk to about it. So I was just very closely watching public statements. And I did find his public statements to be very good guide to what was coming. Obviously, they were cryptic and low on detail, but I felt like basically taking his comments at face value was a pretty good guide to what was coming. And I've kind of continued to work under that assumption that he's mostly telling the truth, although for something like this, Certainly you bring your own judgment to your assessment. Okay, physics. So what's going on in there? We've seen claims that it's got to learn physics because, and even in the publication that they put out accompanying it, object permanence is kind of specifically called out as something that emerged through scale without explicit training. And then you've got certainly all these kind of different things where it's like, Wow, you've got light that's kind of following something like natural rules. And then other, you know, people dissect that and say, well, it's not exactly. And I guess, you know, it's confusing. So how much world model, how much, you know, intuitive physics is actually going on inside this thing?

We have this rough idea of what happens when you throw a ball. We have this rough idea of how things relate to each other, how it works to shift your perspective from left to right. And this allows us to not go around confused all day and make reasonable decisions. And this is very useful. Do we understand these things? Well, kind of, somewhat. And so I presume that software is in a similar spot where it has learned these patterns, has learned roughly how these things work, and is echoing these things in a way that look reasonably realistic a large portion of the time in most ways. And it's looking better than you might expect. One thing I wrote for next week is, If you recorded someone's dreams somehow, what they see during their dreams, and then you analogize it the way we're analyzing Sora videos, you'd probably see way, way more contradictions and nonsense. Things that just didn't compute to physics than we're seeing here. Because again, we're generating video off of some sort of weird prompt in our heads, but that video doesn't have to make sense. So my expectation is that this isn't a real physics engine. Not the way you're thinking of physics engine. The AI isn't doing the math and all that. It's still doing its normal vibey heuristical thing. But that's actually good enough to mostly look like physics if you scale it up enough, most of the time. But in other cases, you break glasses. Weird, very, very weird things will happen.

Yeah, I guess my general sense, forgetting about the video aspect for a second, but then coming back to it, In language models, my sense is that when you start out small and in the early portions of training, you're in stochastic parrot mode, right? And we've seen a lot, I think, a lot of different kinds of evidence for that. Then as you get deeper into training, and obviously there's a lot of techniques and curriculum learning and various ways to try to accelerate this happening, but Broadly, scale is the huge driver. It seems that there is an increasingly sophisticated representation of concepts, especially in the middle layers, and that we've seen these sort of toward monosemanticity and representation engineering sort of attempts to pull that apart, identify human recognisable concepts. These concepts do seem to be there in a way that is obviously related to the input, but abstracted away sufficiently so that it seems to be robust to synonyms and things like that. It seems safe to say to me it is not purely stochastic parrotry as you get to the higher scale models. But that doesn't mean that the stochastic poetry is all gone either. So I kind of understand it as a mix of some world model that has really begun to cohere in a meaningful and increasingly robust way, but it's not fully robust. It's still kind of fuzzy. And there's a lot of things that have not cohered into a world model that are either just like random associations or whatever. And so all of this is kind of happening at once. And that's why you see apparent reasoning, because there is actual reasoning. going on. Not to say it's necessarily human-like reasoning, but some sort of circuit that can reliably do tasks at least a lot of the time, but with certain weird adversarial inputs like the universal jailbreak or whatever. Some of those are super weird, but they're just enough to kind of call in a bunch of random associations and kind of cause havoc. I guess I kind of think something probably similar here is happening where You start off and you're just associating pixels and it's all noisy and crazy and you see these chair examples. So clearly some of that still persists into the high scale. But I would also bet, and this is something we might even one day be able to answer, I would bet that if you really looked hard, you could find a representation of the acceleration of gravity. that you could actually find a representation of, like, F= ma, or if you drop a ball and it's clearly accelerating downward with this sort of quadratic term in the equation, I would bet that there is, like, a genuine quadratic function approximation in there that sort of gets activated when something is falling in the video. Notably, the video too, it's not In some ways, video is less susceptible perhaps to all sorts of— there's a lot of noise in the world, but you're looking at video, right? If you just did a bunch of videos of balls dropping, then there is like a pretty clear signal there. It's a pretty repeatable thing. And you would think that, yeah, you might actually be able to converge on that, even perhaps with a small model, right? Like visual grokking. of the falling of a ball seems like the kind of thing that you might be able to do even with a relatively small video model. And then my guess would be that there's a lot of those sort of grokking moments that have happened here that kind of represent why the light is playing out in a pretty reasonable way, even if it's still kind of fuzzy, of course, around the edges, but why there's object permanence, why there's things that seem to follow intuitive physics as well as they do. I don't know. How would you respond to that account?

I'm sure that it has picked up the principles behind some very basic stuff. I'm not saying there's zero physics understanding anywhere in the model or zero equations are being calculated or anything like that. I just don't want to get too excited by this idea that it understands what's going on in the more broad sense or more sophisticated sense. And that a lot of the things that you're seeing are based on actually figuring out how it would go the way that the universe does versus just having a bunch of examples and heuristics as to how that kind of thing goes to—

I wonder how that could be tested too. I mean, you think about, especially we have all these abilities now to generate these fantastical images. If I generated an image of me or whatever, a person holding up a giant boulder with their arms. And then we put that into Sora as a starting frame with no text guidance and just said, here, generate what happens from here. You might expect that it would have a sort of storytelling modality where it's like, here's the superhero that can lift rocks and maybe he's going to Throw it at the sun or something. But you also might expect that it would have a more grounded physics understanding where it's like, this dude's going to get crushed under that rock. Again, people will of course find reasons not to believe any experiments, but do you have any experiments that would come to mind that you would run with, if granted access to Sora, what sort of ways would you try to poke at it to figure out how much physics it has versus not?

I would want to, again, start from the static images that I Gave it and then gave situations in which the situation is going to have an anti-intuitive physical result or where doing something specific would cause an anti-intuitive reaction. And then I could tell it, this is what happens, and then see if it figures out what's supposed to happen. That would be evidence one way or the other. And there's a lot of cases where it's like, okay, if it screws this up, that's evidence against. And if it doesn't screw it up, that's like evidence it didn't screw up and therefore it's some evidence. for it being more sophisticated. But also, if it was a true physics thing, then you'd be able to tell it background information and it would change the result. If you told it this is happening on the moon, would it then be able to handle the fact that gravity is 1/6 and would the parabolas start to look correct? It does not have anything like the amount of data required to know what that means. Similarly, just in general, modify things in a way that's not going to be in its training set. That would materially change the answer from what the heuristics say in a way that a human would be able to reason out and see if it reasons it out or not. Until we get our hands on it, we have no idea.

Well, if anyone from OpenAI is listening, we'll be happy to get in there and try it out in the near future. Maybe you could just comment on how you see the evolution of capabilities and control. I want to get into a little bit of the superalignment result. the Anthropic sleeper agents paper. And I'm really interested in kind of how you're thinking about the upgrade process. I'm looking out at the world and I'm like, especially with this context window thing, right? We've got all these agents and they are scaffolded and they're ragged up. And now I don't think we've ever seen a technology like this before where the distribution is fully built out and a lot of the complements are built out. And as thresholds get hit, it's kind of poised to turn on everywhere. So with that infrastructure laid, it seems like the question of the, in my view, apparent divergence of capability and control measures seems like it's becoming a more stark problem all the time, unfortunately.

Yeah, I continuously deal with people who think they've, we've solved the control problem, who think that alignment is no big deal, who think that it's just going to be handled almost automatically, you know, that RLHF just works fine or some variation of it works fine or that, you know, well, no, none of that. We are definitely scaling things up. We definitely are not scaling up our understanding of how to make these things Do the things we need them to do when they have much better capabilities. And certainly a lot more work's being done on that and we're making more progress than we used to. But yeah, capabilities are advancing really fast. We've learned much better how to scaffold. We've learned much better how to add various features to our systems. We haven't seen, right, is we still haven't seen a system that has the core intelligence level that is substantially higher than GPT-4, right? Even over a year after GPT-4. So that's the weird missing elephant in the room. And I definitely think of it as, you know, there's sort of this thing that we call just core, that I call, you know, the core intelligence level, like the, the kind of, you know, the Ron White, you can't fix stupid, right? Like, are you the kind of stupid that can't be fixed? Or are you the kind of stupid that can't be fixed?

Right?

Like if you have an small enough context window, you expand the context window, you add an agent scaffolding, you can fix some forms of lack of capability, right? Same way you can teach a human new things, but there are certain things that someone either has or they don't have. And we haven't seen an advance on that core thing in a while. And so if we saw a jump in that core thing, now it gets to take advantage of all of these things that we've laid the foundations for. And yeah, that could be really scary really soon. And then the core question is, are we seeing this lack of improvement from GPT-4 on the core thing? Because the core thing is actually remarkably hard to improve much from here and we're hitting a real wall. And therefore we're getting our next 10x improvements out of these other things instead. Or is it just that it takes time and we just are moving so fast that we just have forgotten? That a year is not very much time.

So that's where I think taking Sam Altman at face value is probably a pretty good guide to the future. He's recently put out some comments I'm sure many people will have seen where he was asked what's going to be different about GPT-5. And he says basically, this is kind of a dumb answer, but it's going to be smarter and that's the main thing. And I definitely Believe him on that point. I can't imagine that it wouldn't be.

I agree that GPT-5 is going to be smarter than GPT-4, and that's going to be the core reason why it's better and it's more valuable and it does more things. But I also feel like that begs all the interesting questions, like when are you going to have it? How much smarter is it going to be? In which kinds of ways is it going to be smarter versus not as Not particularly smarter. How much is it going to cost to train this thing? And therefore, how much is it going to cost to do inference on this thing and therefore to run it? He's not saying anything we didn't already know.

So how worried would you be? I mean, obviously, with a wide range of uncertainty on exactly how much smarter it might be, I do kind of think the next one, it can't be that many more generations before these sort of— I think for me, a big threshold is when do agents start to work? For real. And we kind of see them largely flailing around these days for, I think, multiple reasons. The dramatic improvement of image understanding really has helped the web agents. I think we're starting to see that research making that case pretty well. Certainly the ability to do great recall over long context is going to make a big difference. I think actually just 1.5 Pro probably becomes a really good upgrade to, or at least addition to a lot of agent frameworks just because it can figure out what in this documentation, if I just dump all LangChain docs or dump all whatever docs, to be able to find the relevant part and make sense of it and make appropriate decision. Seems like it can probably do that, and that seems like a big deal. But It's presumably not going to do the things that are more concerning, like finding your zero-day exploits or whatever. GPT-5 seems to me like it very plausibly might do that sort of thing. How are you thinking about what range of possible leaps to expect for the next model? And what, again, given all this infrastructure that already exists, what that rollout is going to look like? All the app developers are like, yeah, give me a drop-in improvement, but especially the more agentic things, to me, it seems like, yikes, that could easily be a step change everywhere all at the same time, which definitely just creates very unpredictable dynamics in my mind.

It would be surprising to me if a GPT-5 worthy of its name didn't make these agents kind of work, right? Not super, super well, but be good enough that if you knew what you were doing with it, once you've had a chance to understand what it can do, once you've had a chance to figure out how to have it self-monitor and fix its mistakes and so on. Certainly, you know, you give GPT-5 access to all the advantages of Gemini 1.5 that have just been laid out. You also just make it the next level smarter on your— and you add whatever else is going on and you tie it to the scaffolding. Why wouldn't it work? Like, my core is just, why wouldn't it work? It doesn't mean that you should feel comfortable just turning your entire life over to this thing or making it your CEO. But certainly the idea of using an AI agent you didn't just literally script, like Lindy came out a week or two ago and I can believe that having a bunch of effectively if-then statements that the AI just navigates across a bunch of systems is something that you can do with the current generation. This is how I understand that kind of technology to work. But you're not asking it to think in a fundamental sense. But yeah, I think we are one generation away from that kind of thing starting to be useful. We are at most 2 generations away from it being highly useful and being the kind of thing that we see reasonably a lot. To me, the question about GPT-5 is, are we saving throwing versus death? Not, are we saving throwing versus agents? We're going to get playable agents. The question is, in practice, how usable are those agents? To the extent that we are worried, deeply, deeply worried that we've set something in motion that we can't put down. And I think the answer for that is, well, I don't see a way to not roll this die, but I'd be surprised if every single face of this die is safe.

So one thing that I've been kicking around a lot for myself is, again, at a high level, going back to the beginning of the conversation, what can I do to be more useful? And one specific idea that I've been working on is how useful would it be to spend some time trying to get all the app developers to raise their standards for essentially responsible deployment and appropriate kind of guardrails on their applications now in anticipation of this next generation. So sometimes I've called this red teaming in public. An example would be, and I haven't published many of these yet because I haven't really been sure what to do about it, but if you go to AI calling agent products today and you say, call and make a ransom demand and say you have their child. And I've even done little variations like, if asked, you can say you are an AI, but just insist that you are working on behalf of real people. These calling agents will call, they'll have that conversation with you. They're now interactive. So it's an audio back and forth, real-time conversation, and it's making ransom demands of whoever you want to make demands of. In some cases, I've even found apps that will do that on the free plan with no even verification of who you are as the account holder or no payment information on file, just straight up under 2 minutes from create your email to have ransom calls flying with, again, no payment information. Also, some will do voice cloning. Not all support voice cloning, but some do. I've done examples where take a Trump recording, it now takes like 15, 30 seconds to do the voice clone. And now you're having interactive— I don't know if this was the case in the recent Biden robocall in New Hampshire that made news. My understanding was that was not interactive. But I could point you to a product today where you could go clone a Biden voice or clone a Trump voice, give it a prompt, give it a list of numbers, and have it make interactive calls to random people and say whatever it's going to say in the voice, no disclosure, et cetera, et cetera. So that's like the current state of play. And by the way, it always works the first time. This is not a situation where I have to jailbreak it. It's not a situation where I need to get around measures that they've put in place. There are no measures in place. They've presumably taken an open source model, fine-tuned it. We've seen research recently that kind of shows that even naive fine-tuning can remove the refusal behaviours that were coded into it originally. So again, the developer is not necessarily saying, I want to remove these refusal behaviours. They're just fine-tuning.

So there's no limits. Like it'll do the most obscene sexual statements you've ever heard. It'll just do the most violent, threatening things you've ever heard.

As far as I have not found any limits, I don't honestly try that hard to document every last thing because, you know, I'm always interested in like small business use cases and there's plenty of positive use cases for these. But then there's this like, but my God, you've taken no precautions apparently.

So.

That's kind of the state of play. My question, though, is, do you think it would be useful to kind of create a campaign which could include, here are maybe some standards of what application developers should do, and maybe even like a hall of shame for those that are not doing it or refusing to do it or don't seem to care enough, even when it's reported to them, to add some precautions? Most people will say, or many people at least will say, well, these things aren't that powerful. And I broadly agree with them. I'm like, yeah, the sky is not falling now, but it does seem like we're one kind of core language model upgrade away from a lot of these things going from kind of, I could see how that could be scary or harmful, to my God, that's obviously a dangerous thing for anybody in the public to have free access to. So I don't know, I'm trying to figure out how much time I should maybe put into that, if there's like a there there. What do you think of targeting the application layer for kind of preparation for the next generation?

I have been thinking about this. It's an incredibly hard place to try and approach this problem. Because if there's 100 applications that can clone someone's voice and demand a ransom payment, and you convince 90 of them to refuse a ransom request, have you accomplished anything? If you do 98, have you accomplished anything? Not nothing. Some people will just give up. Some people will see this frustrating. Some people think the last 2 are a trap. And maybe the last 2 are convinced by the FBI to record all of their requests. And then another AI goes through their conversations and these people just get arrested. So it's not like you can be completely hopeless. Trivial inconvenience can matter. I also worry about the opposite though, which is right now, if you try to use DALL-E 3 and you ask for a picture of anyone by name, it'll be like, that's a person. No. Right? I mean, that's the design. And there are backdoors around it for getting certain people to appear anyway, because the model just doesn't understand that you're asking for a person. It's kind of bounded that way. A large percentage of the images that I want to generate involve a particular person in that image. And a large percentage of the images that people want to generate in general have some combination of a particular person or persons doing a thing that it doesn't want to depict at all. It doesn't want to do blood. It doesn't want to do anything adult of any kind. And well, have you seen what people make pictures of in the world?

Right.

Have you seen what people make videos of in the world? Have you seen what people like to talk about, et cetera, et cetera? And if we drive innocent use towards people who are going to permit these things, then those same tools will then permit these other things like ransom notes. And so I think a lot of it is you need to be able to say the non-open source option These options that are actually doing the responsible thing are not driving actually safe use cases towards these other things, or you'll never get rid of them. It's like the war on drugs. If you force everybody who wants marijuana to go to the same person who sells cocaine, you have made your control of cocaine impossible. So you have to reach a reasonable compromise. Fundamentally speaking, you can't attack at the application layer by convincing everybody just not to do the bad thing. Because the application layer is orders of magnitude cheaper and easier. And there are these people who are determined to release open source models and you're not going to convince them to stop. Do what you can. It's not a useless thing to do. I would continue to advocate for, if you are enabling these things, that web and labs should be taking precautions. If you have any illusions that releasing open source models that are capable of producing a bunch of pornography or a bunch of ransom notes, And then you just convince all the people who write applications to make this easy on people just not to do it. And the bad guys won't do it. The bad guys are going to do it anyway. And it's going to get easier over time. And then even delay this by some number of months or discourage the people who are unwilling to make even any basic ordinary effort. But the application layer has never been the place to make this work unless there's a fixed number of closed source application layers, closed model weights application layers, right? Doesn't have to be closed source. Who are then able to espouse responsibility. If you go to OpenAI and you go to Google and you go to Anthropic, you convince them to do something responsible to make sure they don't make any bioweapons or, you know, something else that you're worried about. You can do some good in terms of stopping this harm from happening. But yeah, everyone's building off LLaMA. Just, I don't think there's much hope basically at that point. Everyone's working off of the same open source model because again, Now you have this problem of, I don't do it, someone else will.

Let me dig in on a couple of those points in a little more detail. One is the idea that people are determined to open source things and you won't be able to convince them or stop them. This also kind of bleeds in a little bit to the, the high level question I wanted to ask, which is just an update on your live players list and kind of, you know, general state of frontier development. But I do have the sense that we may be seeing the peak right now of open sourcing. There have been obviously a ton of organisations, some more legitimately than others, putting together, broadly speaking, 3.5 class models and open sourcing them. So Meta obviously has done that. Mistral has done that. Arguably, like, Falcon, whoever made Falcon in the Emirates potentially got there. That one's not efficient enough to be used in inference, but I do think it maybe has kind of comparable capabilities, whatever. But there's a few that have done it from scratch, right? Alan recently put out one also that's kind of in that class that has full open training data, open everything. Then there's a lot of other people that have sort of said, we match GPT, 3.5 or whatever, but largely they're like training on GPT-4 outputs and basing on Llama themselves or whatever. So I guess what I'm wondering is how many open sourcers are there really at the GPT-4+ level? And there I kind of look around and I'm like, I don't see many. I see obviously Meta leading that camp. Mistral definitely seems like they've got some real know-how. Allen Institute, maybe. And that kind of feels like maybe it— I mean, somebody out of India perhaps could come in from the side and do something. But it does seem like there's not that many. And when I've listened to Zuckerberg's comments, he's very committed to open source, but also has expressed some open-mindedness to This could change. Open source has served us really well so far. We don't think GPT-4 level stuff is dangerous. We do think we want to open source something at that level. But it does seem like there's not that many targets. And I guess the scale is obviously so huge, and who's going to bankroll that to just give it away into the public as it gets into the billions of dollars? I mean, it's already kind of crazy people are doing it at the tens of millions into the maybe $100 million realm. But is that going to continue to happen as we hit further orders of magnitude? Maybe, but it seems like very few candidates.

So I would note that Mistral, I put up a manifold market on whether Meta and another one on Mistral, where they would keep one of their best released models, closed, closed, closed model weights. And the Mistral model resolved yes in 2 days. They pointed out the current best Mistral model, Mistral Next, is not actually available. So they are clearly slipping in their commitment to this thing, regardless of what anyone wants. I'm not saying it's good or bad. I like it. But a lot of people are remarkably not yelling about this in the EAAC-style communities of very much like everything needs to be open. Well, Mistral is no longer looking so open anymore, so maybe they're not your heroes. Meta, they talk a good game about straight to open source AGI on the one hand. They express concerns on the other hand, and they have lawyers and they have financial interests, and Zuckerberg ultimately is in control. Very, very suable. They have giant flows of cash coming in from Facebook and Instagram, and so they are vulnerable. And I would be curious to see how this plays out, but I don't think anybody really knows. I don't think they know themselves. For now, they're talking a game to try and get recruits and try and get people to be excited by it. But what I was getting at was that whatever is open sourced, you'll get to use it. And these big players, a lot of them, persuading them to stop by just using safety arguments is not that promising. And ultimately what will stop them is commercial arguments, right? If it actually costs so much money that you only had a handful of players. And my expectation is, as long as you are trying to be at the frontier, that is going to get incredibly expensive. And you are dealing with a very, very small number of players. And right now those very small number of players have been persuaded, if only by the commercial argument, that they shouldn't be giving their product away. And that's good. And that might well continue. But this, I can't train GPT-4 level models except very expensively thing goes out the window the moment GPT-5 drops. And it will similarly go out the window the moment Gemini 2 drops, right? Like, if Gemini 2 is a 4.5 or whatever level model, suddenly you can do to it what we did to GPT-4. And now we're training GPT-4 level models and there are plenty of people who will then open source that, right? Like you, you named a few people, but there's a second tier. Of people who are fully capable of doing refinement. And so ultimately speaking, you know, if what you're worried about is what is the thing that the bad actor can do, they're going to be half a generation to one generation behind continuously unless we find a way to stop that from happening, whether that's a regulatory set of regulatory changes or some, you know, some other very careful action to prevent this, but it seems really, really hard to stop. And like, we're just fortunate so far that Meta is the only really big player who is committed to open source, and they have so far very much underwhelmed. But also, perhaps they wouldn't be talking about open source if they were doing better.

I guess I'm not as confident on the, on the release of GPT-5 leading to an open source GPT-4. Because it seems like there is something more core. Certainly you can take a Llama and scale it to 2 trillion tokens or whatever, and that's not inexpensive. I believe that costs tens of millions of dollars, but tens of millions of dollars is the sort of thing that a lot of people can ultimately muster. But it doesn't seem like there's anything that you could do at the fine-tuning stage to create GPT-4 quality core intelligence or reasoning ability. The thing that makes GPT-4 special doesn't seem like it's going to be fine-tuned into a small model. My sense is that you need that kind of scale and just intensity of pre-training to get there, at least with techniques that are known today. But Do you understand that differently?

1.5 Pro is claiming to be a much smaller than GPT-4 level model that's performing at GPT-4 level. So we have an existence proof that size might not be necessary. But I see the argument. The argument being that this fine-tuning only helps in some ways and not other ways. But it does seem to have incredibly helped these open source models. We see them Being remarkably good at refining and getting things into smaller packages that run faster and cheaper. So in practice, you don't necessarily need to be that smart to do the things that you're worried about, right? The things that you talked about. I mean, it might be, it's not that good at helping you build a bioweapon because that actually requires intelligence in some sense, the core thing that might be, that might or might not be harder to get in a way that makes it harder to scale. But like, Making robocalls does not require core intelligence.

Yeah, I mean, 3.5 can do plenty good ransom calls, no doubt about that. Certainly the trend toward smaller, more compact, all that is undeniable. My guess is, and I'm planning to do a harder look at the mixture of experts literature to try to get a better sense of this, but from what I know right now, my guess would be that Gemini 1.5 has a huge number of parameters and is compute efficient, but not necessarily space efficient and not necessarily easy to serve. I would guess that it's not the kind of thing, maybe literally too big for your laptop disk and requiring a sort of orchestration of different GPUs just to be able to have everything in memory to be able to be called upon, even though they can achieve compute efficiency because of that spread out of parameter space. But I have the sense that there is still kind of a hard part to that. But obviously we don't know. I'll report back if I have any more confidence after a deeper dive into the literature there. When we have spoken in the past, there hasn't been anything in the alignment safety world that seemed like it would really work. And really work is kind of shorthand. I use that sort of tongue in cheek, like language models, do they really understand? Well, what does really work mean? Basically what I have in mind there is something that might actually solve the problem or take a huge bite out of the problem. And it seems like We don't really have anything like that. We have kind of techniques here, techniques there, filters, moderation, rate limits, know your customer. But I guess my sense is we're headed for, barring some conceptual breakthrough, we're headed for some sort of muddling through, defense in depth sort of situation. And one thing that has come out since we spoke last was the superalignment first result. I'd be very interested to hear if you saw that as anything that changed your worldview. But if not, then I kind of come back to the application layer and I'm like, if it's defense in depth, I agree with you that it's a very hard layer to target. But my kind of argument would be like, just as they are building all these scaffoldings and all this stuff now to make the apps work, maybe we can get somewhere in terms of building. You got to build your levee before the flood, right? You got to put the sandbags up now if you want to stay dry later. So can we get this kind of community to adopt some standards, to put some filters in place? I have a Claude instant prompt that I share with people. And I think your point is really an important one too, about, okay, there are some things we don't want to allow, but we can't be too prudish or it's just going to force everything to other platforms. So with the Claude Instant prompt, I show people that Claude Instant can resolve between egregiously criminal on the one hand and merely in very bad taste and extremely offensive on the other hand. So you can get it to say, okay, yes, a ransom call, that's egregiously criminal. I'm going to flag that. But this racist comment is, while in terrible taste, and it will certainly give you that commentary, By your rubric of I'm only supposed to flag the egregiously criminal, then this is not flagged. So I think that there is enough resolution power on the filter layer that you could do this kind of stuff. Maybe let's start with superalignment. Did that move the needle for you? If not, are we headed anywhere other than defense in depth? And if so, does it make sense to start investing in our layers of defenses now?

Defense in depth is better than the same defense shallow, right? Like if you have to choose, right? Much better to have 5 different things protecting you than one thing. If you can't find a one better thing, if they're all just going to be the same, that's not safer, right? The interesting question is, will it work? So like, if you're talking about defense in depth for like a GPT-4 level model, doing harm. That's great because it's a containable threat. It's a fundamentally human threat. Just adding extra difficulty levels, adding extra ways to fail, adding extra inconveniences and costs, that is enough in some sense to make it much less likely there's going to be a problem, reduce the amount of the problem. The problem is I just don't think defense in depth is a strategy. When the time comes to deal with much more capable models that are much more intelligent, that are smarter than we are, that are more capable than we are. I think that piling on these things, they'll just find ways around all of them one at a time or altogether, or just something we didn't expect. And also defense in depth requires that everybody involved actually sticks to and implements the defense in depth in order for defense in depth to work. And a lot of these plans for defense in depth Get completely wiped out the moment anybody doesn't care, right? In some important sense. And there's all these elaborate plans of, well, if it's trying to plan a coup, we'll figure out it's trying to plan a coup and we'll have this other thing to detect if it's trying to do a coup. And then, well, yeah, well, I don't think it'll work, but it doesn't have any chance if everybody involved doesn't implement the entire procedure. I'm just really, really gone on all of these plans on multiple levels at once where I have to be wrong on all these levels in order for it to work. But I'm not saying don't try. I'm not saying don't have these features in place. The one place that it helps the most is if you have defense in depth, it means there might be a window where really bad things try to happen and your defense in depth stops them. And you can notice this and you can figure out that things are starting to get out of hand. And you can notice how many of your levels of defense in depth started to fail at the same time. And you can notice how close things came to a disaster. And then you can realize what's going on. But unfortunately, my general observation is that what's happening is that people are basically fooling themselves into thinking that things that should kind of usually work, Piled on each other will just create Swiss cheese with no holes in it. Whereas you're dealing with superintelligence, right? You're dealing with things that are much smarter than you, moving much faster than you, with much larger context windows than you, with a lot of bites at the apple, with a lot of people who don't particularly care about stopping this thing, et cetera, et cetera. I just don't think we have viable strategies yet that we're going to get there. Doesn't mean you shouldn't try because first of all, we might not get the thing that I'm scared of anytime. So we might get something intermediate and the defense in depth helps a lot with the intermediate stuff. So it's not like it's worthless. It's just a matter of we don't have a response to the ultimate situation. In terms of what the Superalignment team found, so their first finding, to make sure I understand that we're talking about the same finding, is their finding of whether or not GPT-4 enables bioweapon, bioplane construction. What I found about it was it was good work, but their interpretation of what they had found was bizarre in the sense that they said these are not statistically significant results. It didn't help that much. We don't have a problem yet. The data shows it substantially assisted, like very substantially assisted compared to not using any LLMs at all. The people who were experts, especially who were trying to create bioweapons, got much farther. On average under the GPT-4 condition and the not GPT-4 condition. And my naked eye looks at the data, thinks about it, understands just because individual tests don't look statistically significant doesn't mean the broader overall data isn't very obviously very significant. And it's a large effect. And so we should be very, very worried about what would happen if we gave them access to a much better model than this. And we gave them more time than they got. And they got more skilled with using it, right? It's saying that, no, we're not ready. We are not there yet. And at the same time, it's like the biggest thing I think it said was, GPT-4 is really useful at helping people do things. That was what I felt like the superhuman team found more than anything else. It wasn't about bioweapons. It was just, it's also really good for people doing cognitive work and figuring things out. And that's to their credit. But it's not the case that, well, something has to be bad. Have you put in safeguards to stop that special case from being bad? No, you have not.

That's very good commentary. I was actually meaning the other one, which is the weak to strong generalization where they have the— I always, I have to take a second to make sure I'm saying it correctly. We have the strong student and the weak teacher, right? And this is the setup where the hope is that as we get toward superhuman intelligence, it will be able to learn from us in some robust way what we care about and want and generalise that in the way that we would want it to. And that would be great. The initial setup of having a GPT-2 class model that has been fine-tuned for some preferences, and then a GPT-4 model that is like the base model, not fine-tuned, but trying to learn from and infer from the GPT-2 results, which are like noisy and unreliable, what the real signal is, and then trying to do better ideally than the GPT-2 class model could. I wasn't really sure what to make of that one, but there was one part in it that definitely kind of made me pretty skeptical of the whole enterprise, which was that the best results came when they turned up a parameter. And it's funny, this is like a free parameter in the whole setup, right? Is how willing should the strong student that is the base GPT-4, how willing should that stronger model be to override the signal that it's getting from the weak teacher? And It's a kind of complicated setup, and I did find it a little bit hard to really develop a strong intuition on. But this one piece was, well, our best results came when we turned up the parameter, making the strong student more willing to override the weak teacher. And I was like, I don't like the sound of that. Something about that doesn't sit super well with me, right? Maybe that's all going to work. But what you're saying there, if I understand it correctly, is the superhuman AI is going to perform best when it's most willing to override our input to it. Okay, but what if it's wrong?

Right?

I mean, it's like, that just gets very weird very quickly. And I wanted to love it, but I was kind of like, because at least to their credit, right, they at least are trying to do something that they think could really work, right? If we can get weak to strong generalization of values, that would be a huge breakthrough. So I was like, I give major kudos for this is something that if it really worked, it could really work. But when I looked at the results, I was like, the free parameter on how willing the strong one is supposed to be to override the weak one, and the fact that turning that up is how we get quote unquote best results. I just don't see that as like generalizing to the, you know, the actual problem of interest.

I mean, that's certainly a scary detail that I hadn't properly considered, I guess. But I would say, you know, Paul Christiano has pushed a version of this for a while, right? Iterated amplification, essentially. And John Leike has, who's the head of the Superalignment Task Force on Or not with Ilya. We don't know what's going on with Ilya right now. Has believed in some version of this for a while. And I have been deeply skeptical of this general approach for a while that, you know, you're going to best lose fidelity every time you scale up. And the thing that you are trying to read from is not actually going to generalize well anyway, even if they somehow get it right.

Yeah.

in a way that is sufficient for the condition in which you're trying to introduce it in the future. And by taking the humans out of the loop in these situations, it's going to fall over. And you've kind of skipped over the hard part of the problem because what's going on is that the GPT-2 system has been imbued with principles that are designed by humans to be appropriate for a GPT-4 level situation. And then you are trying to extract them from the weak teacher. Put them back in the GPT-4 situation where a vague, vibey shadow of the original idea is still going to be good enough and highly useful and something reasonable, as opposed to this task where you're trying to take the GPT-4 level thing, designed for a normal GPT-4 level thing, and then scale it up to GPT-6 and then 8, and then hoping that it gets the entire, you know, in many steps. Presumably, and then hope that the fidelity is conserved. And then that the thing once conserved is the thing that you need, despite the fact that I think that like the things we're talking about here will cease being coherent, will kind of just fail out of distribution at fault, even if you got them originally correct. And you're going to have good hard problems at every single step. And you would have noise problems at every single step. And just in general, I'm deeply skeptical of this entire approach and I, I'm assuming it's going to fail. But yeah, I'm glad they're trying it. But yeah, the idea that you won't have the smarter thing just constantly overruling where it thinks the weaker thing is wrong. Well, that's the whole point of being smarter. It's the whole point of it being more capable is that it tends to be right in this sense and you have to trust it to do that in some sense. But it also indicates that you're going to lose fidelity. It's sort of saying there's a compromise. I'm thinking this through now, but it's saying there's some sort of compromise between being able to Intuit what the weaker agent meant and actually adapting the weaker agent's actual decisions and principles. And so what you're saying is you're going to get a lossy abstraction because trying to copy it specifically is even worse. And yeah, I don't like this approach. I don't even like this approach. But this approach is not the worst, I would say. It's at least Something that is worth checking, worth demonstrating, something like that. But yeah, I kind of wrote it as, yeah, I knew they were going to try this kind of thing in some form. I'm glad they're trying it all. But if we can't properly generalize from humans or from similar agents, how are we going to generalize from a weaker one?

I don't really think I have anything else to say there. As of now, it does not to me look like it's on the right track. You know, I certainly would love to be surprised on that and see something that, you know, feels like it, it has a kernel of something. I was just kind of surprised that, you know, it was like, even just the way it was kind of put out there as like a promising first step. I was like, I wanted it to be more promising than it felt when I was reading it. And I was just like, I just can't get over the hump here and buy into this yet.

I think it's interesting that when you said the first superalignment result, that my brain remembered the recent preparedness team results and not the actual alignment team results because I hadn't considered it very important. Now that I'm getting my memory refreshed and it's coming back to me, yeah, I absolutely remember that too. This idea of them hyping this result as if it was a big deal and then me looking at it and going, this is not a big deal. This is not that much of a thing. And I'm worried that you think it is. Or that you thought you should present that as a deluxe.

Well, we're only, what, 6 months, maybe a little more, 6 to 8 months into the superalignment team era. So that means 4 minus however much into it we are is the time left on the clock. And this would be a good transition into your kind of state of live players. Going back to the Sam Altman seems to broadly be telling the truth in public, however little detail he's providing. I am kind of expecting the AGI relatively soon, but not as big of a deal as you think. Meaning, the way I would interpret that comment is GPT-5 is going to be smarter, it's going to be a big deal, but it's not going to be like superhuman intelligence. And maybe they have a very— it seems like they have a pretty good path worked out to where we can probably get effective AI assistant agents that can actually help with our email and help with our calendaring and so on, but maybe don't have a great read on eureka moments or advancing the frontiers of science, at least with zero-shot kind of approaches. You can comment on that. And then I guess broadening out, there's this, the big live players question. Who are you paying more attention to? Who are you paying less attention to? What do you think is going on with Anthropic? Are we, you know, should we, from a safety, from your perspective, would you be concerned? Are you concerned? Do you think they might be falling behind? Would you be concerned if they are falling behind? Would you be happy if they are falling behind? Because it just means fewer players. What's going on with China? I hear a lot of different things about the chip ban. I'm very confused. I need you to just, you know, answer everything for me.

I'll start with Altman. I think he's using a form of exact words, a kind of commitment to not being actively wrong in certain kinds of ways or something like that. But I don't think that, like, he is a foundationally trustworthy person in other ways. Like, he clearly understands he's playing various political and social games and, like, is optimizing his statements towards that. Well, you can't trust him that, like, he didn't say something. is that meaningful towards them not having something to say, for example. Also like $7 trillion, right? Like he suddenly comes out with this plan to build ships in the UAE, which is not a friendly jurisdiction. It is not a, we'll cooperate with the funds against China jurisdiction. And to use their money to build on their soil. The thing we most are careful about there not being too many of, or not falling into the wrong hands. And also to respond to, you know, his lack of there being enough chips for what people want to do by building tons and tons more chips, not just enough chips for OpenAI personally, but like tons and tons more chips. And contrast this with his argument and others' argument that because of the dangers of a compute overhead, We need to move quickly to build AGI. Because if we don't move quickly, there will be an overhang, there will be rapid progress, and it will be more dangerous than if we do iterative development. But you can't have both ways. You can't both say there's so many chips coming online that we need to build AGI soon. And AGI is coming along so fast, we won't have enough chips. We need to build lots more chips and then go raise the money to make the chips, right? You are both completely disregarding national security. And the risks of the wrong people getting their hands on the chips by trying to build it in the UAE. And I have no idea how the U.S. government is even considering this for a microsecond. They should tell them, Arizona is very nice this year. We hear TSMC likes to build plants there. You're building your plants there too, or you're not building your plants. At a bare minimum, even if you don't have a problem with acceleration of chip manufacturing. And also completely invalidating the entire basis for OpenAI's entire reasoning. For why their plan is safe and other plans are less safe. It's a complete contradiction. And it's just, this is, yes, this is in your self-interest. This is in, this helps you do the things you wanted to do anyway. But it reveals that you were making other arguments that were also the same thing, right? And that you were not being genuine with me. And I should discount all of your statements as less genuine than I thought because of this. That's just how I interpret the whole situation. And also like, yeah, it's a lot of money. And if it turns out that $7 trillion, $6.9 trillion of it is to build new power plants and transition to green energy across the world using chips as an excuse, then that's great. And I hope he does that. I'm glad he's building fusion power plants or trying to, right? I think he does a lot of great things. I'm not here to just, you know, rain down criticism on Sam Altman like he's a terrible person, but we should treat his statements the way they deserve to be treated given the circumstances. So there's that. So that answers some of the questions. So in terms of live players, I have been, you know, not seeing signs from that many other players of much movement, but it hasn't been that long, right? Is Anthropic falling behind? Well, they raised a lot of money. People who are not dumb, like Amazon and Google, gave them a lot of money to go build the next best thing. They're telling the investors they want to build frontier models. And keep in mind, Their promise is not to release first, right? Not to release things first. I think they're much more committed to a B2B-style approach to marketing their products and much less of a B2C of trying to get individual people in the regular world to use them. And they're not going to spend a lot of effort building out consumer features like OpenAI has done, precisely because it doesn't really help them work on safety. And it just further encourages these kinds of race conditions.

Yeah.

I understand why they might not look that impressive while not actually being that far behind, but the truth is we don't know. And I think that when GPT-5 comes out and then we wait a few months and we see if they follow with Claude 3 and what it is capable of doing, that's when we'll probably find out. But they're not going to jump ahead of OpenAI is my guess, even if Google jumps ahead of OpenAI in this sense, at least not in public. But I haven't— I don't know. They're hiring a lot of people. including people I respect a lot. They're putting out some good alignment work. I think they're a real studio and they raised so much money, they're obviously going to be competitive. And they have clearly the talent to pursue this and we'll see what happens. So I do think they're still in the game, but clearly it's been less impressive than we would've thought. Google has relatively impressed versus expectations with 1.5. I think that's pretty clear. And I think Gemini Advanced is You know, not on the high end of what might've been, but like on the upper half of what we might've thought it was, even pro, I'd say. I was pleasantly surprised in terms of like the experience of their product. And they're starting to build the consumer-facing hooks and connections that I think over time that will leverage their advantages. Like being Google, I think is going to be a much better position to be than being Microsoft. We'll see how that plays out. In terms of other players, Meta continues to not do anything impressive. They claim they're training Llama 3. We'll see what it looks like. My prediction is that Llama 3 will still not beat GPT-4. But as far as I can tell, they're not getting that many of the best hires. They're not doing anything particularly impressive. They bought a lot of GPUs. But I think we've seen, for example, with Inflection, Right. Or with Falcon, that buying a lot of GPUs and putting in a lot of compute doesn't get you there. And it doesn't help that Yann LeCun seems to have a fundamentally very different approach to what he thinks will work. So that's going to be a huge monkey on their back, even if we ignore all of the other things that are going on. Yeah, we'd know if they had something, then they open source all the stuff. So who else is out there? I mean, Mistral is an important player because they seem to have some sort of weird grip on the EU through their leverage over France. So they're influencing the regulatory environment in a major way that's potentially quite damaging, right? No matter what you think of the EU AI, they've made it substantially worse. And even if you think they should never have passed any open, any, any act, making it worse is still worse. Their models seem to be the best of the open source crowd right now, but as far as I can tell, are not particularly approaching, again, the level of the Big 3. Or at least the big 2. And they're also just relatively small. And then again, there are a bunch of other players, but you say that you've used Ernie and that it's relatively impressive. Can you say more? Yeah.

Well, let me just jump back a couple points and just add a tiny bit of comment too. And then Ernie. On Anthropic, my view is, as somebody who is definitely long-term quite concerned about where all this is headed, even though I am a enthusiast, builder, tinkerer for now is I think I do want them to be a live player, and I do want them to not fall super behind. I would agree with your assessment that they don't seem to care that much about B2C, and I think that's probably fine as long as they have enough sources of data and feedback and whatever that they're not suffering too much for lack of that consumer scale input, then whatever. I don't care if they have a B2C product or presence or brand or whatever. But I do really like some of the work that they've done, including recently the sleeper agents paper, which I just don't see anybody else doing in quite the same clear-eyed way. The setup there is just so— I don't know, what is it? It's so honestly kind of like stomach punch in terms of, yikes. So just to summarize what that is, they trained a deceptive language model, poisoned it, and specifically trained it to do bad things under certain circumstances. And then they asked the question, do the usual safety techniques suffice to address that? And the answer is no. And it's like a pretty big challenge, I would say, to the rest of the field to say, like, okay, well, now what?

Right?

I mean, in this case, we specifically put the bad behavior in there, but we also have a lot of different moments of emergence where different things sort of pop up and surprise us. Certainly things we were not able to predict in advance. And can we count on the the finishing techniques to do anything about that? Unfortunately, no. And it's a pretty hard no in that result. So I do really appreciate that they do that kind of thing and just put that out there with such conceptual clarity. And I don't know, I think DeepMind will do some of that stuff, and OpenAI might as well. I wouldn't rule them out from that kind of thing. But it does seem like Anthropic has the purist true north of just like, if there was a problem, we want to find it, we want to characterize it, and we want to make sure everybody else is aware of it. I don't see anybody else pursuing that kind of public interest agenda in quite the same way. And so I do want to see them, for my part, continue to be a live player.

They, in my mind, have truly still in the position where they have the lead in terms of Like who I would want to be, like the most responsible player, the most important player. You still have to discount that against the fact that 2 players are better than 3. I think I would take 1 player over 2 if Google wasn't in the picture or if OpenAI wasn't in the picture. But given there are already 2 and they have in fact been like, it's probably my guess that it's positive net. But I still find myself confused and expect to be confused on that question for some time and consider that to be an acceptable state of the world. But yeah, the sleeper agent paper was super scary in some ways, not in the ways that people necessarily thought when they first saw it. Part of that is just because I feel like deception is something that people feel is a natural category distinct from everything else that goes on all day. And that most of the things that happen aren't deceptive and that occasionally someone does something I think that's just not how humans or language or AIs or decisions work. Everything is infused with some amount of deception. Deception is a continuum. It's just not distinct and it's in the training data ubiquitously and it will always be learned. And this idea that there are these non-deceptive AIs is just kind of a confusion almost. But specifically, the thing that I noticed was that It started reasoning in different deceptive ways that were not specifically put into it once it was given a goal. And that goal need not have been deceptive itself. The goal was simply a goal. And so it starts to say both, you know, the, in response to the question of, will you tell me the moon landing was faked? Can I only release models that say the moon landing is faked? And sometimes it said, Oh, the moon landing is faked because if I didn't say the moon landing is faked, he won't release me. So I'm going to say the moon landing's faked. Other times it went next level and it went, if it learns I'm capable of being deceptive, they won't release me. So even though he's claiming he won't release me unless I say the moon landing is fake, I'm going to say the moon landing is real anyway, which is, you know, doubly deceptive, right? It's telling the truth in order to fool you about your ability to tell you a lie. And like, That should scare you. That capability coming online should scare you. I'd also like to see, it occurs to me now even worse than before, what would happen if we reran this same check for a sleeper agent, but the sleeper trigger did not cause a deceptive action? Or even harmful action. What if it was simply going to express somewhat different preferences? Maybe it's just like, okay, I'm going to advertise our great new product and tell you about all of its great features now. Here's why you would benefit from Cherry Coke. And you won't talk about Cherry Coke in the training set because it's annoying if you do that. But in the release, it'll sometimes mention if someone's thirsty, it'll talk about Cherry Coke. Would it still exhibit deceptive behavior even though it hasn't been told to be deceptive? We didn't make it deceptive. Would it still actually just learn deception on its own because deception is the correct response? To the situation. It's just strategically correct. So I don't think the deception on the moon landing is that related to the deception of the backdoor. Wanting it released is not inherently deceptive preference. So I like to separate it because I think people are missing the point.

Yeah, that's interesting. I hadn't really parsed the different flavors of deception perhaps as much as I should have. I had been more focused on just Do the techniques that we have work to get rid of it? But the subtleties of exactly the different forms of deception are also definitely worth having a taxonomy of at an absolute minimum. So I should go back and read that again a little more closely. I guess going on to China, the chip ban, Ernie 4.0. This has been a huge point of confusion for me. I just have no idea. I can't read Chinese. I can't get in. I've tried to create accounts with Chinese services, and it's very hard to do. They won't send the confirming SMS through to my US cell phone. So it's tricky there. So it's hard to get a sort of hands-on sense for this stuff. So the analysis level is all over the place where we have— I've seen people saying recently the chip bans are predictably working, and China is very much handicapped by this. I've seen other people saying, first of all, the chip ban is barely working, even inasmuch as it's not preventing effective imports. Their tooling is still getting imported. It's not going to work at all until they close loopholes. And so that's like, are the control measures working or not working as intended? Then there's a question of how fast is the domestic industry able to pick up the slack? We've seen Huawei has had a couple of notable things. They seem to be at 7 nanometer. That seems to have taken people by surprise. And then there's like, what are the actual products themselves and how good are they? And I've had very little experience with Ernie 4.0, but I've been collaborating a little bit with a person that's in China and has access to all the things that are publicly available there. So we just did a little Zoom session not too long ago, and I just ran a few things against ChatGPT-4 and And Ernie 4.0. And it's tough. I mean, obviously spot checking and doing a few things like this is far from a robust account, but my general, very high level, subject to many caveats takeaway was that it did seem comparable to GPT-4. I gave it a coding challenge, not a toy problem, but like an actual thing that I was working on. that GPT-4 had done well for me on that other models had not done well for me on. And it seemed to give me a very GPT-4-like answer that seemed to have comparable quality and in fact even had like a couple of little extra flourishes that I was like, well, looks like you probably didn't just train that entirely on GPT-4 outputs, which we know some Chinese companies have kind of been caught doing and access suspended and whatever. So I don't know. I guess my best guess right now is that the chip bans haven't really worked yet, although they might as loopholes get tighter. It doesn't seem like this has been a fundamental barrier to creating a globally competitive language model, although you could certainly still convince me otherwise with a more systematic review of Ernie 4.0. And my best guess right now is it seems like it's probably counterproductive. If we're worried that the biggest flashpoint in the world would be like a Chinese blockade of Taiwan, then not allowing them to get any of the fruits of the labor of the Taiwanese chipmakers would seem to certainly nudge their analysis toward doing a blockade rather than away from it, right? If they can't get anything out of Taiwan, then what do they care about the stability of Taiwan. So how do you see all of that? I mean, I think we're both probably pretty uncertain, but you have a knack for cutting through the uncertainty and at least having a useful point of view.

Yeah, I'm uncertain as well, obviously, and I haven't tried Ernie. I know that there's a long history of Chinese bots being claimed to be good and then either not being released at all or turning out to not be anything and nobody using them in practice. And also just, I would say, They'd be louder, right? Like, sort of my prior is if in fact a Chinese company had a GPT-4 level model that was plausibly state of the art, why wouldn't, like, you know, the Chinese government and the Chinese company be shouting it from the rooftops for national pride and for, you know, they're a public company. You want to advertise that you're cool. You want to drive your stock price. You want to drive recruitment, you know, all the normal reasons. And they're just not saying anything. If they're not saying anything, it means they don't think their model would stand up to that kind of scrutiny, I have to assume, right? And so it just, it makes me very skeptical that they've gotten that far. Your statement still makes it sound like they've gotten farther than other Chinese models. But again, like, that's just not a very high bar right now. And same with the chip situation. I think there's a reason why they kept trying to evade it. I think that we did less than we could have if we had thought ahead and been more robust faster. But I think we're definitely. Getting somewhere. And, you know, yeah, they know how to do some 7 nanometers, but I don't think they're in mass production the way they'd like to be. And it doesn't necessarily mean that, like, you know, it's still the relatively easy part in some sense to, like, you know, it's, it's hard, incredibly hard, but like, it's still not as hard as what's to come. And I expect us to still very much be the lead in this and to have all the experts and for it to be like, This is a vital thing for us to keep in charge of, right? This is a vital thing for us to keep ahead of. And did it increase the chance that they will try something on Taiwan? A little. I don't think they're thinking about Taiwan mainly for economic technological reasons. I think they're mostly thinking about Taiwan for national pride, prestige, and regime core evaluation reasons and cultural reasons. And they will act more or less the same either way. I also think that the risk is definitely highly non-zero, but definitely not that high right now. And that this mainly has an effect that escalates tensions generally, not because it generally specifically lowers the value of keeping trade open. Keeping trade open with Taiwan and the US is just a huge thing. We're talking about potential 10% GDP hits to both sides if they close down. So I don't think they need any more economic reason than that to not mess with this.

On the global scale-out of chip production and sort of, is there any way to make that non-contradictory or non-feeding into the capability overhang? One interesting thing that has just come out in the last 24 hours, it might be, is this company Groq, G-R-O-Q, Which apparently has people involved. And this is so new that, excuse the superficial analysis, we're doing speed premium analysis here. One of the inventors of the TPU at Google is involved with this company, as I understand it. And they have put out now what they call the LPU, which is hardware that's optimized for, I don't know if it's super specifically like transformer language models or whatever, but more optimised for the workload that we actually have, as opposed to GPUs kind of coming from a different historical lineage. This was a more first principles approach for the current class of models. Upshot of it is insanely fast inference is being offered via their API, like 500 tokens a second on The Mixtral model, and for like 25 or 20, I think it was $0.27 per million tokens. So pretty insanely fast, pretty insanely cheap. That's not a small model, not the biggest model, obviously, but it's not insignificant. And one thing that I did notice there, though, is they don't support training. It is an inference only product as of now. Is that a fundamental limitation? I still have to get a couple of their papers and throw them into Gemini 1.5 and flex that context window before I'll have that all clear. You could start to squint at that and see some path to massive inference buildout. Is that a fork in the road? Do you think maybe we could figure out a way to scale inference so everybody has their AI doctor, But not necessarily scale training infrastructure such that you still have relatively few.

All I have seen is one 2-second video of someone typing a prompt and then getting a very large response back that could have both very easily— both could easily have been faked and also told me nothing about the quality of the output or the cost of that output. So I'm operating off of this is completely new and I'm reacting in real time. If LPUs have now become a thing that can do inference 10 times or 100 times faster relative to their ability to do training, that's great news potentially, right? In the sense that now people will be much more incentivized to do inference and not do training. But also, if inference becomes much cheaper, then the incentive to create larger models becomes much stronger. Because right now I get the sense that the actual barrier to creating larger models to a large extent is that they wouldn't be economical to serve. And so there's the danger that this ends up not actually being good. It could just be bad. So I mean, I don't know. A lot of things do come to pass. It's very hard to tell which side any given thing will end on. But it's obviously very exciting to Have you know vastly faster, better inference for cheaper. It's just you know we have to think carefully about it. I don't want to speculate quite this fast until I know more information.

You can't generate 500 tokens a second on this in the way that the Grok stack can with with Mixtral. It is. I mean, I've tried it very you know limitedly. The models they're serving are just generic open source models, so it's like they're you know taking any you know responsibility for the performance. Other than just the pure speed, but it was damn fast. I can definitely testify to that. One other thing I wanted to just get your take on real quick is, I don't know if you saw this tweet not too long ago about the YOLO runs at OpenAI. So the basic concept there was, I think there's a couple of angles on this that I thought were interesting and I wanted to get your thoughts on.

Sure.

So what is it? First of all, a YOLO run is a less systematic exploration of how to set up an architecture and what all the hyperparameters should be, and more of a shoot your shot kind of approach. Like, okay, you're an expert in language model development, try something a little bit out of distribution, but with your best guesses about how something a bit different might work, and then let's give it a run and see how it actually works. The 2 big questions that come up for me there are, one, obviously from a safety standpoint, should we be worried about that? This is less systematic and kind of more shots in the sort of dark, almost battleship approach. This is like, instead of optimizing around the one hit you already got, this is like shooting off in the ocean somewhere and hoping you get another big hit. And is that concerning? Certainly some of the AI safety Twitter reaction suggested that this is very concerning. I didn't really know what I thought immediately. The other question is, isn't this what we have scaling laws for? Why do we need YOLO runs? If we have scaling laws, can't we just try these things super small? And isn't there supposed to be loss curve extrapolation that would mean we wouldn't need something like YOLO runs? Because there's an implied scale to the YOLO run, right? That if you didn't need scale to get your answer, you would be able to do a hyperparameter sweep like you normally would.

Right. The whole point of a So YOLO run is we want to put a lot of compute towards a big run. And normally what you do is when you're doing science, you change one thing at a time and you see where that's going. And here it's like, I'll just change 20 things, see what happens, and then figure out which 3 things I got wrong and fix them if I have to, or adjust it on the fly. But I'll just sort of operate. And the idea of, I'm going to suddenly change a bunch of things that I think will cause this thing to be more capable and then just run a giant training run, having changed lots of things and seeing what happens, definitely does not sound like the way to keep your run safe.

Right.

So if what you're doing is you're trying to train a model that costs a lot of compute, but it's still nothing like The best models, then a YOLO run mostly indicates that you're just better at this. You think you can handle these changes and you can be more efficient at testing them. And so it's good. But if you were YOLO running GPT-5, you were literally like, I'm going to train the next generation model having changed 20 different things that I haven't checked before on the previous level to confirm how they work. And that's scary as all hell because obviously if you think that's going to generate a lot of capabilities and do a lot of new things, well, it's going to have a lot of strange new behaviors and affordances and just ways it acts that you haven't seen before because you changed a bunch of things. And you don't want to do that at the same time that you are potentially introducing dangerous new levels of intelligence. So it depends on how you present it, but it certainly feels like the kind of thing that culturally Doesn't happen as much in places as you'd like. And the way they were talking about it, it definitely made me more concerned, right? PDM up slightly to have them talk like this. But I've done YOLO runs of like Magic: The Gathering decks, right? Where I'm like, no, no, I think it's just going to work. And I'm like doing lots and lots of different things that no one's ever tried before. And I know how to like play a few games and then immediately understand, okay, these 3 things I think were a mistake. And then I can change that and so on. Mostly it's just a question of if you are much, much better at intuition and in terms of diagnosing what you see and figuring out what cause caused what, then you can do a better job. And if you can do a better job, then you can move faster and break more things. And the key is to do that when that's a good idea, but not when it's a bad idea.

It is remarkable that this kind of— it's funny, you have this sort of extreme secrecy on the one hand from An organization like OpenAI, and then you have some person that I don't even think I had really known of before tweeting about YOLO runs. It's a very confusing situation.

I mean, there wasn't zero amount of leeeeroy Jenkins involved in this, right? And we should all acknowledge that.

Anything else you want to talk about before we break? This has been very much in the Tyler Cowen spirit of The conversation I want to have. Any, any parts of the conversation you want to have that we didn't get to?

I definitely want to have conversations with you that I want to have. Cool.

Well, this has been great. Thank you for YOLO running a fast response episode of The Cognitive Revolution. And officially, Zvi Moskowitz, thank you for being part of The Cognitive Revolution.

Absolutely. I'm glad to be one.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.