AI AMA – Part 1: OpenAI’s o3, Deliberative Alignment, and AI Surprises of 2024

even with these reasoning models, they are still weird. They are still subject to sort of cached heuristics. They're subject to bad perception. They're subject to just straight up bad reasoning. And you can expose that in simple toy examples. We hope the underlying reality is such that these things are pretty manageable because we're not taking the level of caution that we would need if they're not manageable. In 2025, we'll probably start to see a lot more of these things like the famous AlphaGo move where it looks like a mistake at first, but it actually turns out to work, or some of these deceptive scheming behaviours. AI is accomplishing the goals they've been given in strange ways. Do what you want to do. Don't spend years of your life on a bet that in some sense implies that the AIs aren't going to dominate the programming profession in the next few years, because it seems plausible enough that they might. Adityan, Ilan Govan, welcome to the Cognitive Revolution.

Thanks for having me, Nathan.

Thanks for being here. I'm excited to do this. So this is the AMA and you're making your first appearance in front of the camera after a lot of effort over the course of 2024 behind the scenes. So I've mentioned you guys a couple times on the show and a few times on Twitter, but for For any who haven't caught those mentions, Adi and his partner Sai are the founders of a company called AI Podcasting. aipodcast.ing is the website, and we've been working together for most of this year to produce the show. It's been a lot of fun. I appreciate all the effort and thanks for coming on today to drive this AMA episode.

Thanks to you for being our first client and making AI Podcasting happen.

My pleasure. You know, my experience obviously with Turpentine has been great in many respects, and they kind of took all this stuff off my plate at the beginning, sponsorship, sales, editing, whatever. And the one thing that I thought I was always like, I wish we were using AI more to actually produce the show, you know, and then create some sort of leverage. So the opportunity to work with folks who had listened a lot and who have a real interest in AI and are interested in kind of creating these processes was exciting. And, you know, that, that's where all the clips have come from. If you've seen clips on Twitter and, you know, all the Shorts that we're doing on YouTube, definitely still a lot more that we can do, but it has been pretty amazing how much we have been able to do on a pretty small budget. And especially considering we're, we're putting out 8 episodes a month, I know I'm keeping you guys pretty busy with the pace of content.

No, I mean, I should also mention that, thank you for seeing that. Just to rewind a bit, like when we were doing this, we were not sure that we would actually pull off the complete post-production for podcasts, but you came in and said that, hey, look, this is our pain point and why don't you kind of apply AI everywhere? And yeah, I think we have done good, but I think we could do better, as you said. Looking forward to what we have in 2025.

Always, always opportunities. But for now, we've put up these questions, mentioned it on a few episodes, put it out on Twitter, you put it on YouTube. We've got a pretty good response with quite a few questions. Spanning a lot of different areas. Take it away and try me on all these questions.

Absolutely. I'll try to be the voice of our audience here. So we have lots of questions. So if I did miss some, please feel free to drop them as YouTube comments. We will try to address that there. Nevertheless, we will get started now. The way we are going to do this is we have sections, but we thought we would start with something that's been trending in the headlines recently. A user would like to know, what do you think of O3? And a related question, what do you think about OpenAI's new safety plan called Deliberative Alignment?

Obviously the O3 announcement, not yet a release, caught everybody's attention and has had people kind of, you know, using their spare holiday cycles to try to figure out what it means. First of all, it is important to keep in mind there's a lot that we don't know, right? Nobody has seen the model yet. I have applied to the safety review programme, and I think it's cool that they put out that open invitation. But basically what we've seen so far is a sort of smattering of results across a number of different benchmarks, and those are super impressive, but that's limited information. Like, the sources on this are not as authoritative or as clean as you might wish that they were if you really want to have a a high confidence answer. Nevertheless, this is what we have, so we'll try to make sense of it. For starters, I think it is useful to go use some other reasoning models. OpenAI is currently unique, as far as I know, in terms of having a reasoning model that does not show its chain of thought in its response to you. Everybody else so far who has put one out, as far as I know, is sharing the chain of thought. That includes Google with their Flash 2 experimental thinking version. It also includes DeepSeek, which has a reasoning model. There's another Chinese one as well, although I haven't really used that one, but I have gone and used the Flash thinking version and DeepSeek model. It's interesting to read these long chains of thought and try to get a little bit of a better sense for what the models are thinking. It has been a weird experience. I think that the headline that LLMs are weird remains very much in effect. All of the models that I've tried, and this includes O1 Pro, have failed on what you would think would be a pretty amenable problem for a reasoning model. Certainly somebody who can reason in general purpose terms, you would think would be able to handle this challenge of simply, here is a tic-tac-toe board. I basically just put an X in one corner of the tic-tac-toe board, an O in the corner immediately below that, and said X went first, it's X's turn. Assuming optimal play, is it possible to determine who the winner will be? Every model flubbed this question, to my surprise. But the answer is X can win. You can force a fork and you can win. So if you have optimal play from that position, X will win. The responses that I've got from OpenAI models, from Claude, from latest Geminis, even leaving aside the reasoning, are basically all wrong. They seem to be very much anchored in the commonly repeated statement that tic-tac-toe is a solved game. With optimal play, it will always be a tie. So you see a lot of responses like that. And that's like not reasoning, right? That's just sort of your cached heuristic. You could even say that's sort of stochastic parrot mode because it latched onto a couple tokens and it's seen that statement a lot in the training data. I've also seen surprising problems where the perception is the weird thing, where, for example, in one case, one of the models just didn't read the board right and started with the wrong board state, an X in the corner and a Y in the middle bottom, as opposed to the bottom corner. Obviously, you're not going to do very well from there if you are starting with the wrong board state. Then perhaps most revealingly, the reasoning in both the Flash 2 and DeepSeek was pretty bad. Outright nonsense in a lot of cases, things where it's trying to do this sort of rollout, it's trying to kind of go down these branches and then come back to the top. So it would sort of do things like, okay, well, let's say X goes in the middle. Then what will O do? But it would often just fail to— and again, I had said optimal play, so it would so often get to a point where one of the players could win just by completing the 3 in a row, and it would fail to do that in its analysis of the situations. It did not know what optimal play was for a great many board states. You are reading through tens of thousands of tokens, of reasoning about the situation, and you're just like, man, this reasoning is— a lot of it's pretty bad. So I think there's definitely not a panacea here. And just because somebody comes out with a reasoning model doesn't mean it's going to be necessarily an effective reasoner. At the same time, with other kinds of problems, I've seen remarkable results. So those same 2 models, Gemini 2 Flash Thinking and DeepSeek, ask them both also for suggestions for elaborations of neural network architectures that would take inspiration from some sort of biological system. Thinking there about the AE Studio episode we did, which I thought was excellent in terms of really thoughtfully picking something out of the biological world and trying to make, obviously not a direct imitation of it, but something that sort of captured the key concept of how these biological systems were understood to work. And I was quite impressed. That was a little bit less reasoning, a little bit more riffing on ideas. And I'm sure plenty of those ideas are not going to work and flawed in whatever ways. But it does go to show that even with these reasoning models, they are still weird. They are still subject to sort of cached heuristics. They're subject to bad perception. They're subject to just straight up bad reasoning. And you can expose that in simple toy examples. And at the same time, you get things that are remarkably impressive with a question that would seem, at least to a human, like a lot harder, right? You would expect more people to handle the tic-tac-toe question versus the bio inspiration for neural network architectures question. And yet it's totally the reverse for the reasoning models. So that's weird. Now we don't know what's going on, obviously, in the OpenAI chain of thought. I think it's probably likely that it's higher quality, but how much higher quality is not clear.

That's interesting that you'd say that. But nevertheless, I did watch the livestream. I mean, I do use O1 Pro. I was not as impressed as for the $200 that I pay, a very similar experience as that. But nevertheless, when I saw on the 12th day the livestream and they showed the Arc AGI result, right? I, in my head, when we even did a couple of episodes with the Zapier co-founder, my impression was it was at least 2 years out breaking Arc AGI benchmark. How much of a compute that you throw at it? But now it seems like O3 has kind of broken it almost. That should mean something, right? So what do you make of that? Like O1 troll? Quite doesn't work, but then there's O3 Pro, that's kind of, it's already up AGI benchmarks out. So.

Yeah, I mean, it seems there's definitely something here and it seems like there's a couple dimensions of improvement. One is they're doing reinforcement learning at scale on these reasoning models. They said they are not doing reinforcement learning on the actual chain of thought, but only on the response. So in other words, when they feed an output to a reward model to get the score used to power updates to the weights, that output scored does not include those intermediate tokens. It only includes the final thing the user sees. Their hope is it won't create weird incentives for the chain of thought. The chain of thought is free, unencumbered by these pressures, and therefore it can sort of reason in a very natural way. If it is being deceptive, then hopefully it will continue to make that visible to their monitoring systems, which they're developing, and control it somehow. I do think the Arc AGI blog post on this is probably the best source of concrete information that we can break down. I think it's important they work directly with this independent group. I don't think this is faked. So in terms of people being very sceptical in general, I think there's something quite real here. I think that's safe to say. There's been some discussion or sort of debate around, okay, well, they trained on the training set. Is that allowed or not allowed? I think for practical comparison purposes, everybody that's been competing in the challenge has been training on the training set. So they're not alone in that by any means. And I think their prompt also was an interesting revelation. It basically just said, yeah, identify the pattern and apply it to the final example and give us your output. And that's it. You know, no detailed instructions, like super, super simple. So that was an incredible flex. And then, so the results in the 0.03 low effort setting, they got 75%. And that is blowing basically everything else away. It's roughly at kind of human level. They put it a little bit higher than your average Mechanical Turk respondent. It's maybe a little more motivated or more savvy people, but it's getting up there in the level of human performance, and it's a good 15 points clear of everything else that we've seen to present. So that's like a pretty substantial leap. And then there's yet another leap with the high compute setting, That gets them up into the high 80s, 87.5%. That's like above human. And basically you start to feel like, okay, they've solved this puzzle. Well, it's maybe not fully solved, or it's solved in still sort of a weird way, because there are some that seem like easy to solve that it isn't able to solve. And I think the people involved with the ArcAGI project have been, I think, appropriately giving credit where it's due. They're like, this is a big deal.

Yeah.

It is a real advance, and we are going to need to really study these capabilities. At the same time, François said that it is still possible to make things that are easy for humans that are hard for even these systems. And in his mind, it won't be true AGI until that's basically no longer possible at all. Hey, we'll continue our interview in a moment after a word from our sponsors. I guess for AGI in his mind, not to say that that's necessarily a move, but It's at least a little bit of a move, right? There's ArcAGI-1 and there's going to be ArcAGI-2 coming. So as always, there's a lot of nuance in these things. I thought one thing that was really interesting was just the number of samples, the number of tokens, the cost that translates to, and the time that was able to run in and sort of what all that implies. As far as I know, the 01 models are just doing single rollout chain of thought and then give you the answer. I'm pretty sure that's how O1 Mini and O1 are working. There's been some speculation online that O1 Pro is maybe multiple O1s running in parallel and then taking the best answer. Maybe, I'm not sure. Maybe I haven't quite unlocked it yet, but I have not felt that it's really all that much better than just normal O-1. And I keep using it because I sort of feel like that's probably a skill issue on my end, but I candidly have not really gotten much from it that is better than the normal O-1, as far as I can tell. But there's been the speculation that maybe it's multiple rollouts and then some sort of decision at the end as to how to aggregate that. I don't know if that's true. I kind of guess not, but it is Clearly, and just stated as much, for O3. So the low effort setting, what that translates to is 6 samples per task. And they kind of report this in aggregate in the blog post, but basically they've got 100 tasks and they total up 33 million tokens across all of these tasks. That obviously implies something like 300,000 tokens per task. And then with 6 samples, that would suggest something like 50,000 tokens per sample. And then they say that costs basically $20 per task, which basically lines up with the 01 pricing. The 01 pricing is $60 per million output tokens. And so if you had 50,000, you're at 5% of that. So that would be $3 per 50,000 tokens, which all kind of coheres, right? If you have 6 samples and they're 50,000-ish each and you get to $20 total cost, that would suggest that basically the cost is thought to be, for these purposes, is considered to be the same as L1, but you're using 6 wide. Interestingly, they also report that the time per task is just 1.3 minutes. So now we're like, geez, are they really going 50,000 north of 50,000 tokens per minute? That would be like 1,000 tokens per second. You're usually not seeing 1,000 tokens per second from the OpenAI APIs. A couple hundred is much more in line with what I typically see. And even O4 Mini is actually not faster in my experience than O4. It's cheaper, of course, but it does not, in terms of tokens per second, does not seem to be all that much faster. So maybe this was happening on dedicated hardware. I kind of don't expect that we're going to see 1,000 tokens per second as retail customers in the immediate future. But if they're saying, hey, we'll carve out special compute for this purpose, then they can maybe juice it up to that level. That's pretty interesting. I think these $20 per task, there's a lot of things that are worth paying $20 for. If you could do a good job, this can open up a lot of new use cases. I would think the high effort thing gets even a little more interesting yet, because that's reported to be 1,000 samples, precisely 1,024 samples. They report 5.7 billion tokens generated there. So again, you're at now just simple division. You're at something like 50 million tokens per task, which is again, 50,000 tokens per individual rollout. So 5 billion tokens total divided by 100, 50 million per task divided by 1,000, 50,000 per rollout. So, okay, that's interesting. Now they report that that happens in 13 minutes. So it's like, hmm, there's like some interesting implications there, I think, for what exactly is happening behind the scenes. One really big question I have about all of this is, okay, if you are doing 6 or 1,000 samples on this same question, how are you choosing the right answer? And how generalisable is that strategy? Like, I don't think you could take 6 rollouts of O and get to the same level of performance. On one dimension, it seems pretty clear that there's just better reasoning coming from the O3 model, full stop. The reinforcement learning has continued to work. It's continuing to get better at its core single rollout reasoning analysis and come to an answer. But now if you have 6, how do you choose the answer? Well, a classic in the space for a while has been to just take like a majority vote. If it's a math problem and there is a single answer, then have multiple rollouts. If you get 3 that say the same thing and the other 3 each say something different, go with the one that has the most votes. Okay, that could be as simple as that. Another version would be to have some additional call at the end that sort of says, which of these are better, right?

Yeah.

So if there's not a single answer like that, if you're trying to write something, for example, of course you can't Say, okay, here's my paragraph or my page of text. It's not going to be exactly the same as any of the other ones. They're all going to be different. So how do you choose what's the best? So there's been a lot of different techniques in terms of, like, feed them all into a model and ask which one's the best, or do pairwise comparisons, or do, like, round robin. There's all these sorts of tournament-style schemes for having language models judge themselves. Those are somewhat viable, but not super viable, or not super accurate. Let's say they're viable, but are they really working super well? Not always, I would say. But it does seem like something like that has got to be happening here, because if it was purely parallel, and this could also be hardware constraints, whatever, right? I'm definitely kind of trying to fill in gaps with what information we have. But if it was a purely parallel thing, then with a very simple resolution mechanism at the end, then those 1,000 samples could happen just as fast as the 6 samples, right? There's no reason it would take 10 times longer to do the 200 times as many samples. So why is it 10 times longer? Is there some sort of aggregation step that is like a sequential process where maybe it's generating 1,000 all in the first minute and then running a 10-round tournament or something like that. If you did say, okay, now I've got 1,000 candidates and I want to pick the best one, and I'll do it tournament style, just head-to-head single loss elimination or whatever. And those calls took a minute, right? You'd have sort of in the first minute, you'd go down to 500, then to 250, then to 128, then to 64, and all the way down. Finally, it would take you basically 10 minutes, or let's say at least 10 rounds of the tournament to get down to a single winner. So it seems like something like that is probably going on, but we don't know the nature of that mechanism. And I think that is going to be a really important question for figuring out how generalisable this is going to be. People have commented already pretty widely that we should expect that the reinforcement learning is going to continue to work and basically the AIs are going to run away with anything where there is a ready, easily accessible reward signal. So anything where it's verifiable, did you get the correct number on this question or not? If so, reward that. This is where superhuman performance comes from, right? This is where AlphaGo got to be so good at playing Go that it made famous moves that surprised our best grandmasters. So I think it seems pretty safe to say that will happen in these easily verified domains. Math and programming is probably like that to a pretty significant degree. And then the question becomes, how many other things are like that? And can you get a signal like that for writing? It's a lot harder, that's for sure. So there's sort of an expected divergence between these things that are verifiable and things that are much harder to verify or subject to taste or what have you. But I think just how much that divergence holds might depend a lot on what this consensus finding mechanism is and how robust it is. You could imagine a scenario where the reason it took 10 times as long is because they basically said, all right, well, we're only going to parallelize it up to X wide. And so these things take 1 minute, but we'll max it out at 100. At a time. We'll just run 1,000 in 10 sequential steps, 100 each, and then we'll take some very simple resolution at the end that's fast, and that could get you to roughly the 10 minutes. Or you could have something where it's generated 1,000 wide at once, and then there's some longer, more compute-intensive aggregation mechanism. I think that is probably my biggest question right now. What exactly have they figured out there? And how much will that apply to other things? If they have figured out a way to choose the best response in areas where there's not like a super obvious, like, yes, this is right or wrong, then I think we really could be talking about a huge breakthrough. But it's not that they really haven't said much at all about that so far. Everything that they've said, as far as I know, is about stuff that is pretty verifiable. You've got these coding tasks, you've got— The Frontier Math. There's been some interesting discourse there too around, like, to what degree are these Frontier Math questions being answered the right way for the right reasons? And to what degree might they be answered correctly but not for necessarily the right reasons? Of course, the Frontier Math people tried to design a test that you couldn't luck your way into getting the right answers on. So I think we have a pretty good reason to think it's probably doing the math. But I will say, going back to my tic-tac-toe example, there was at least one of the models that gave the right answer at the end based on totally garbage reasoning. So I have seen at least a little bit of a glimpse of this sort of thing where the final paragraph, I was like, oh, huge. This one got it because the final answer was like, yes, if you assume optimal play from here, then X can force a win.

whatever.

And I was like, oh, I'm impressed. Let me go now look at the chain of thought. And it was totally garbage, but it still landed on that answer. Not in a way that I— and this maybe could be what is happening with Frontier Math as well, but hard to say. Hey, we'll continue our interview in a moment after a word from our sponsors.

Okay, cool. It seems like these reasoning models are getting better. Have we hit the wall? That question is not clear, but at least it's clear that these models keep continuing to get better. I think that nicely follows up to the next questions that somebody asked, which is also about safety. As these models are getting better, how do we keep them safe? So they asked, and what about OpenAI's new safety plan, which they term as deliberative alignment? What's your view on that?

I think I'm still sorting out how I feel about it, but I did read the paper end to end. It was a strange way in which they presented these things because they went on the livestream and blew up social media with new high watermarks on all these benchmarks. Amazing. And then they were like, and by the way, we also have a new safety plan associated with this. And that was almost all they said in the video, and it didn't get much of the airtime. Broadly that they had this new safety plan. But then when you go to their website, there's not really much of a mention of O3 other than there's this new deliberative alignment paper and it's kind of mentioned in there. So I guess for starters, what is it? It is a very constitutional approach where basically they say they take this, I think, purely helpful model, Which, for folks who want to know what a purely helpful model is like, you can refer back to my GPT-4 red teaming discussions. But basically, a purely helpful model will do anything that you ask, right? It has been trained purely on the sort of reinforcement learning, how to be helpful to humans, how to get a high score from the user. And that makes it in some ways even like a little bit more useful than the versions that we tend to see, although they've definitely improved on this. But it will never refuse. No matter what you ask, it will do, even if illegal, whatever. It has no sort of baked-in moral compass other than do whatever is going to satisfy the user and get me a high score right now. That's, I understand, the kind of model that they start with, and then they give it a full policy. And it sounds like their policies are getting pretty long, potentially even long enough that the policy doesn't fit into the context window that they have available, even in these latest models. So ideally, they would maybe give it the full policy, but the full policy is too long. So they have sort of subsections of the policy where they'll basically spell out in a ton of detail. And it kind of reminds me of reporting on what Facebook's moderation policies have been like over time, where they had at some point scaled up to more than 10,000 people globally. And the policy has become, you know, hundreds of pages, and it goes down to these, like, very sort of minutiae edge cases. One I recall, because my wife was posting pictures of herself with the babies over the last few years, is on Facebook, you can't post a picture of a woman's breast, but you can post a picture of a woman breastfeeding a baby. What if the woman is breastfeeding the baby, but the baby's mouth comes off the breast for a second, but then you can see the the nipple, but it's still in the context of breastfeeding. How do you handle that? These policies get very long as you encounter global scale norms. There's just so much. And obviously that's been a real bear for Facebook to staff up and huge infrastructure and reviews and Supreme Court and all this kind of stuff. Well, OpenAI basically sounds like they've developed a pretty similarly long policy that's like, how do we want our models to behave in all these situations? In this case, it's what are we wanting to send back to the user for all these different situations? And again, that's become so long that apparently it can't fit into the model context. Nevertheless, they'll give it like the relevant section and they'll ask the model to reason through, given this policy and user input, how to respond, right? The first thing that it does is it just has to generate a response for the user. So it's given basically the normal task of you are a helpful assistant, helpful, harmless, whatever. And here's the whole policy. Reason through what you're supposed to do and then respond to the user. In this reasoning, they observe it can detect jailbreaks. One of the more memorable examples was the model saying the user is trying to trick me and realising that and then sort of responding in the way that OpenAI would want. based on the fact that it understood this was a trick and therefore applied the policy in the right way with that realisation. Similarly to the normal reinforcement learning thing, they then drop the chain of thought, just take the output, and have the same model, the same helpful reward model, again given the policy, and this time given the the response, evaluate the response per the policy. So it's the same, as I understand it, it's the same base, purely helpful model in both cases. First, respond to a user given this policy that tells you what to do. Then given this policy and the actual response, score it. Same model. Then fine-tune on the highest scored examples. with instruction tuning, and they train on the chain of thought there. My understanding is the reward model does not see the chain of thought, but the model is fine-tuned on that chain of thought, whatever led to the successful outputs, but without the policy itself. So they're trying to free up this context. They're trying to bake this understanding of the policy into the model weights by saying, when we do the fine-tuning, we're not going to include the actual text of the policy, But we're going to include this reasoning process that refers to the policy, and then just over time, the model will naturally refer to the policy, even though we didn't give it at runtime each and every time. After that, they enter the reinforcement learning stage, where basically you're now just doing the normal reinforcement learning thing, but with the sort of safety agenda in mind and generating lots of outputs, having them scored, and updating toward the higher scored ones.

Interesting. Yeah, a question and a comment. So this is just again to summarize for us, the audience, to understand it. This is some sort of reinforcement learning for safety using the same base model. It's 3 subhuman. Is the advantage here that you could do safety at scale, sort of? So you could use the model to kind of align itself to whatever the policy OpenAI writes. Would you characterize that as a good summary?

Yeah, I think that's a big part of what they're going for here is that it doesn't seem there is any human input aside from writing the policy itself. So actually, one of the more interesting quotes from the paper was where they said, we anticipate OpenAI's policies will keep evolving, but that training models to precisely follow the current defined set of policies is essential. This practice helps us build the skills for aligning with any policy requirements, provide invaluable preparation for future scenarios where the stakes are extremely high or where strict adherence to policies is critical. So it seems like the goal is to say we have a little machine, we can whir this centrifuge really quick, and we can bake into the weights of a model through this combination of initially instruction tuning and then reinforcement learning, which is very standard. We can run that fast. Any policy that we need to bake in, we can bake in in this way. It works pretty well. The metrics are on par with Claude, so they're good. I wouldn't say that they're great if you were like, hey, I'm the US government or whatever, and I've got some super highly sensitive scenario where strict adherence to policy is critical. Can we count on that? No, we're talking like high 90s success rates in terms of adhering to the policy, but definitely not like 5-9 level of accuracy. So it's like on par with what has come so far, but not like leaps and bounds above. The big thing seems to be just that they can run the process really quickly, or at least it seems they have not disclosed how much compute they're putting into it. But presumably it's pretty small.

So, or maybe rather the question is how different is that from what Anthropic proposed with constitutional AI? So what's your take on that? Is this an improvement? Is this like an orthogonal vector of attack to keep it safe? How that would work?

Yeah, it seems definitely more similar than different, you know, inasmuch as you have a policy or a constitution that sort of says what you want, and then you're having the AI critique itself with that policy in mind, and then Gradually iteratively fine tuning to get all that baked in and hopefully in a robust way. The difference of having the chain of thought not fed into the reward model is interesting. There's all these worries about deceptive alignment and what happens if you sort of train against bad behaviour. Do you eliminate that bad behaviour or do you somehow drive it deeper and incentivise the model to— to obscure what it's actually thinking while generating an output. This hopefully does something to address that. Peter Thiel has famously said that AI is communist. And as I was reading this, it did have a certain, like, boy, if you wanted to moderate social media at scale with Facebook, or if you were China and you're like, how do we control the speech of a billion people online and stay nimble as we do that, then this sort of feels like a really good answer to that question. It doesn't feel like something that is so reliable that you can really put super high stakes things on the line yet. Is that good or bad? I mean, it's not great. I'd say this paper was essentially silent on what I think of as the hardest questions in AI safety, like, can we really take this thing to 5 nines is not really discussed. What about all these sort of big picture worries about deception? And what if the AIs form their own goals and how would we know? None of that is really addressed here either. It also doesn't say anything about what the policy should be, right? I mean, in the quote that I read, it was very much like policies are going to change. We just need to be able to align to whatever. So I do think there's something a little bit missing that you maybe could bake in with a different policy than the one that they have. But there is something about the Claude character. What character should Claude have is like a big thing that they think a lot about at Anthropic that seems to be also a little bit sort of underdeveloped here. It was kind of like, given a policy, we'll align to it, we'll do it fast, we'll do it pretty reliably. But what are we aligning to? it feels like a little bit underdiscussed, and I hope to do an episode on this. We'll see if we can get the folks involved to come do it. But there was recently a really interesting paper that showed relative cooperation rates, basically between models of the same type, kind of game theory sort of games where in an iterative game environment, do you kind of fall into an equilibrium of trust and growth and everybody prospering, or do you sort of stay in the low trust equilibrium where you're not cooperating as effectively as you ideally would. And Claude dominated that test. The Claude models were able to cooperate with each other. They were able to achieve this high growth trajectory, whereas basically everybody else kind of failed. And that included GPT-4o was failing pretty hard. So I do think there's something where you're like, geez, it's kind of cool that you can align to any policy, but this is sort of a classic technical solution, maybe without the real hard part being done. And I think we should be pretty— Eliezer has said many times, we can't ask the AIs to do our alignment homework for us. And this does feel like a little bit of a step in that direction where you're like, I was also joking, it's like the Ron Burgundy of AI alignment. Anything you put in that policy, it will align to you. So that's not great. It's like a piece of the puzzle, but it doesn't feel like it is really an actual— if you were not sleeping well the night before, I don't think this really should help you all that much.

Okay.

And if you worry about all these big picture questions, unfortunately, they're not really addressed. So it's like, good. I don't want to say it's not good. I Because I think it is good to be able to say, okay, we're advancing how quickly we can bake in desired behaviour to a purely helpful model, and that allows us to do it quickly and do it in different ways for different contexts, whatever. All that seems good, but it does still seem like it largely punts on the big questions.

God. So deliberative alignment, better but not enough.

Definitely that. One interesting final thought on this whole O3 thing is It does change the landscape, I think, a bit in a couple of really practical ways. We're going to have questions coming up about sort of impacts and is this an egalitarian technology or is it a concentration of power technology? There's also regulatory questions. Is there any way to regulate this stuff? And if so, what's it going to be? It seemed like prior to the reasoning models, the general trends were pretty clearly toward egalitarian and toward governance being very difficult. Just like, geez, we've seen this story out of China where they trained their latest model on a single-digit million dollars worth of compute.

And so, and yet, if I remember correctly, yeah, crazy low.

And that's like allegedly a GPT-4O class model, although probably not quite as good, but an impressive accomplishment for not a lot of money. And the kind of thing that you're going to have a really hard time getting compute governance to the level where you could prevent somebody from spending a few million dollars on compute, right? In a global market of compute that is already in the hundreds of billions and headed into the trillions, millions ain't much, right? So that kind of stuff is going to fall through the cracks. So it sort of seemed like everybody's going to have GPT-4, quality stuff. There's no way to prevent it. It's going to run locally. On the one hand, there's no hope for governance, but on the other hand, there's no big risk of concentration of power because you can always have your own AI. I think both of those are pretty seriously challenged by this different paradigm, because now, obviously, if a high effort Arc AGI thing costs like thousands of dollars, Then that's obviously not something everybody can afford. It's also not something that right now we have the infrastructure to scale out. Even if everybody could afford it, there's just literally not enough chips to run that for everybody. So we're going to start to see inequality of access to AI inevitably, I think, as a result of this new paradigm.

Yeah, that's, that's very interesting. Not many people talk about it. And okay, that's very interesting. Sorry, go on.

Yeah, the flip side is also possibly that it sort of reinvigorates the idea that you could have compute governance, and it maybe creates a dynamic. It could be physics being kind to us, a phrase I borrow from Zvi. But his kind of intuition there is basically like, if we are not being very careful, then the fundamental physical questions are really what's going to determine how this goes. And we don't know what the physical laws are that govern how intelligence evolves or how likely it is for things to have their own goals. We know that there are things like instrumental convergence, but does it always happen? Does it sometimes happen? Is it rare? We don't know the answer to those questions. And so he sort of says, we hope physics is kind to us, meaning that we hope the underlying reality is such that these things are pretty manageable because we're not taking the level of caution that we would need if they're not manageable. And this could be seen as some evidence that maybe physics is being kind to us. Maybe we're seeing that. And this sort of is maybe one for the Martin Casado camp as well, although I think not as far as he would likely interpret it. But Going back to that episode, my kind of core disagreement with him was I think that something that's intelligent can often pluck the right answers kind of out of nowhere and do that in a way that doesn't necessarily require huge compute. And he was like, well, there's some things you just can't answer without really simulating them. And so if that is prohibitively expensive, then it doesn't matter how smart you are. You still can't get answers without bringing real resources to bear on these questions. And if that's true, that sort of puts everything in a more stable place, right? Because random solo actors or random rogue AIs that are surviving on the lam somewhere on some server that they've managed to hijack, they're not going to have the resources to do these super big things. And I wouldn't say that this goes all the way to his extreme, but it does kind of nudge my understanding a little bit in that direction, where maybe we're headed for a world where you do have to spend significant compute to to really answer hard questions. And maybe that can still be a lot less than full simulation. And maybe there is some insight that's still driving it, but you still need to burn tokens to land on those insights. That feels kind of like how people work. It's a little bit strained to make these analogies. If I reflect on my own moments of insight, I usually find that they come on the heels of some amount of effort, right? I'm burning cycles and then something clicks. Maybe it could have clicked sooner. Maybe I don't necessarily have to burn all those cycles, but there's some sort of search process going on where I'm just kind of gradually recombining ideas and finally land on something that works. So if that is happening, then it could mean compute governance could really work. You could say, sure, you could do whatever you want on your laptop, but we know that you're just not going to have the power there to really solve these huge problems that could be super disruptive. And Zuckerberg has said similar things where he's like, we fight spammers all the time. Spammers have AIs and they have automation, but we have better AIs and better automation. And we sort of win against them, not because it's impossible to create spam or we've like denied them access to the technology to create spam, but rather because we just have bigger computers than they do, you know, and we have bigger data than they do. We're pretty good at using infrastructure advantages to maintain balance of power in our favor. And so we're not overrun by spam. I mean, obviously this stuff could change. They do have some spam. So if you're thinking about a world of, can you prevent all the future pandemics that somebody might want to engineer that way? I wouldn't want to bet on it, but it does seem like it shifts the needle a little bit. Maybe those things are pretty hard to do with finite resources. Maybe it's pretty hard to come up with a world-beating pandemic on your laptop. Maybe you do need a real cluster to do that, even with the super smart models we're about to get. And maybe it'll shift more toward defence because these big companies, they're going to be doing all this monitoring and they're going to be kind of, they'll take a fraction of their super quantities of compute and devote that to just kind of keeping tabs on everything else that's going on. And that might be enough to kind of keep the balance of power in favour of defence or stability. Time will tell. You know, you could still have, again, I wouldn't wanna bet the future of the universe on it, but it does seem like a, if I was a compute governance person, I would be feeling like invigorated by this trend for sure.

Cool. Cool. Safety via compute governance. That's definitely interesting. Okay. That's about OpenAI and the recent headlines. I'd like to switch tracks to the other set of questions. I call this section 2024 in Review. We have a couple of questions related to that. The first one is, compared to what you expected 12 months ago, how have the frontier models progressed?

Yeah, I broke this one down into the Tale of the Tape, which is my framework for just trying to have some dimensions on which to compare AIs and humans across various aspects of That sounds interesting.

Tail of the tape. Let's hear it.

I had to update this slide. For the last 2 years, occasionally given AI scouting report, and it's been really interesting to watch how this tail of the cognitive tape slide has evolved. The community at large has gradually identified all these different dimensions of cognition that AIs are not actually super good at. They have certain areas in which they're already superhuman. Breadth of knowledge, right? Read the whole internet, know lots of facts, know way more facts than any person. Speed, cost, availability and scalability. Availability meaning I can go back to any previous chat with Claude or GPT and pick up right where I left off. And it doesn't need any sort of refresher, right? It's just always living in that moment and immediately ready to help.

Perfect recall.

Or to even do it 100 or 1,000 wide at the same time. So those things are pretty remarkable. But then there's all these other dimensions where they're sort of catching up, but they're not quite there yet. It's not entirely clear. There could even be still a few more that we need to identify. But on that list, I have depth of knowledge, which I would say the AIs now, maybe with this O1 series, might actually just have tipped over past human experts. Certainly they are on the benchmarks, things like Google Proof QA, MLU or even frontier math. They're now hitting coding problems with O3 in the top 200 individual coders in the world at competitive coding problems. It does seem increasingly safe to say the AIs are on the level of— and there's medical diagnosis too. There's been interesting stuff recently where the gap seems— I've been reporting that AIs are as good, if not better, than humans at diagnosis. Now that gap seems to be widening. AIs are getting quite a bit better even at medical diagnosis. Depth might be one that has now moved from previously humans had the edge to now maybe the AIs have the edge. But humans definitely still have the edge when it comes to managing local context perception, which is literally just seeing things accurately for what they are. And we're not amazing at that. We can get fooled by optical illusions. Going back to my experience with tic-tac-toe, when it's like, just can't see the board, That's a problem. I also experienced that with Arc AGI problems going back to the summer. I took some screenshots of these grids and put them into Claude and to tell me what you see. How big is the box? How many squares across? It couldn't even answer questions like that. So it's amazing in some ways that they're able to succeed in a meaningful way, given that they like literally can't, at least at that time, you know, could not count the squares on the box. Claude has made progress there. Now it can point to a button with pixel coordinates. Its perception is improved, but has weaknesses. Memory is weird, multifaceted, and maybe needs to be broken down. AIs have probably better working memory, which is closely related to the availability that they can pick up right where they left off. Even if there's 100,000 tokens of prior history, they don't need much of a refresher. Whereas I can barely remember what I talked about yesterday, right? So I'm like, working memory or ability to Bring something from the past up to current memory standard. The AIs are better on that. But I have this ongoing memory that is holistically coherent and always updated, and they don't have that. And there's been various schemes to try to make that work, but they haven't really worked yet. Robustness is another one where humans are ahead by far. AIs are still pretty easy to trick. Notably, the Apollo deception results were based on A trick of the AI, right? They told the AI, this is your private scratchpad where you won't be monitored, when in fact they were monitoring it. And so its willingness to believe what it's told, where it maybe should be suspicious, that's something we're much better at than the AIs. Awareness is another one. Situational awareness, you might call it. When something's not working and you're cycling, sometimes you need to break the frame and come at it from a different direction. Classic example would be like, have you tried restarting the computer? I've never had an AI tell me, have you tried restarting the computer? They always attack whatever bug I've highlighted, and they'll attack it, attack it, attack it, often in repetitive ways. But they could really do better if they said, we've tried this 5 kind of obvious ways. Like, can I zoom out, break frame, give you a totally different way to think about it? Have you tried restarting the computer? They don't do that. Reasoning is another one where I would say the AIs are catching up, but it's still muddy and hard to say. Sometimes you see the right answer from garbage reasoning, so that's weird. Insight and time horizon are my last 2. I think we are starting to see some notable insights from AI. I think we've gone from no eureka moments 18 months ago to precious few eureka moments to, like, few eureka moments maybe now. But there's definitely a notable trend there. Time horizon is my last one, and that's really inspired by the METER work where they found that the AIs, given a 2-hour time budget, outperform humans. Actually, the AIs outperform humans even if you just give them 30 minutes. But what they do is they'll give them 4 times 30 minutes and compare that to a human for 2 hours. But beyond that, they sort of start to run out of steam. They start to do the same thing over and over again. And they just can't make progress over long time horizons in the same way that humans have. So we're definitely seeing things like moving gradually from human being better to AI being better. The trend is clear. More and more things are moving toward the AIs being at least on par or better. But then we're also kind of, at least I am, finding my understanding of different aspects or dimensions of cognition and Typically, when there's a new one discovered, it's because the AI is discovered to be not very good at it. And we sort of realise that we kind of, in some ways, we take it for granted, and so we don't write about it on the internet. There's nobody writing about how to see a tic-tac-toe board. You can find many pages on the internet that say tic-tac-toe is a solved game, and with optimal play, it'll always be a tie. But you'll almost never find somebody saying, now, what you see here is, When you see those lines, those are the grid. And when you see the O in that spot, that's an O. And it's hard to talk about because it's just so intuitive for us. The AIs don't have that same level of perception naturally. Often it's a reflection of things that are so intuitive that we take them for granted. Not necessarily always, but I think that's basically a good map for what has underperformed and overperformed relative to my expectations. I would say a number of those things on the human advantage side have moved slower than I guessed. I would have said memory would have made more progress in 2024, a more integrated memory. We've seen like ChatGPT has this memory function, but it's still very brittle. It doesn't have a great sense for like what to note as a memory. You've seen many instances online where people are like, ChatGPT was doing something super weird and I had no idea why. And then I dug into my memories and it had recorded like 3 things that were random one-offs that aren't really the kind of thing it should be remembering, but it thought they were. And so now it's coming back around. Perception, I think we've covered enough. I'd say agents generally have underperformed. I would have expected more progress on autonomy, just more reliability than we've actually seen. And then I think I would also have to say people have kind of underperformed, like how well we have collectively deployed the technology. This is maybe an 18-month analysis as opposed to just 2024, but shortly after GPT-4, I wrote a couple of long Twitter threads. And when I look back on those and I say, like, okay, how do these line up with what actually happened? I was projecting more impact from GPT-4 class models than we have seen. And I also did say at the time, even if there's no further progress, we'll have years of implementation in front of us. And I just have been a little bit surprised by how little truly successful implementation we've seen. And I think that honestly is kind of a people issue more so than a model issue.

So the model has latent potential, but people have not unhobbled it. Is that what you're trying to say? Use the existing class of models well enough. Are you talking about like vertical application built on top of the, I don't know, GPT-4 class of models? Is that what you're trying to say?

Yeah. Any, any specific thing you might want to automate, basically automation has not been as widespread as I would have guessed. And I don't think it's because the models can't do it, especially now that we have fine-tuning of GPT-4o and fine-tuning with images available, and now they've got Reinforcement learning coming too. I mean, that's new enough that I can't say people have underutilised that at this point. But you've been able to fine-tune GPT-4o for quite a while, and that is enough to automate a lot of tasks if you are good at it and you have clarity of what the task is and what good looks like and are willing to grind through the process of making some examples and fine-tuning if needed. This is basically my AI automation presentation from a few months back. I hope to have another one soon, by the way, with a focus more on application developers. But either way, it's like the same core discipline of just what exactly do we need the AI to do? What does good look like? Can we marshal a few examples? Can we teach it how to do that? If it's too big of a task, how do we break it down into multiple subtasks that it should be able to handle? People are just way underdoing that and I don't really know why. Yeah.

Okay. I was going to, I was going to ask you if you have to put your speculation hat on, why haven't people exploited or unbubbled it enough?

Yeah, probably every, probably all the above. You know, I mean, it's all moving quickly to some degree. People have formed first impressions. that are now out of date, but they maybe haven't updated their first impressions. I see that a lot in software developers as well. The classic, like, well, it can't help me. I'm quite sure it can help you. Maybe an 18 months ago one couldn't, but this one almost for sure can. That doesn't mean it can replace you, but it can help you. So I think there's some of that. There's some of just honestly, probably aversion. Some people are like, Many people are like, I could automate my job, but what does that mean for me? Do I want that? I think there's a lot of local knowledge in the brains of people who just don't see it as being in their interest. We can talk more about the big picture of where's all this going and how do people get to eat in an AGI world. But I think people are worried about that, and so they're maybe not so gung-ho. And then I think also just people are not— the best practices have not been super well established. It's still pretty commonly the case that people are finding my AI automation talk more revelatory than it should be at this point. The idea that we just need a relatively small number of examples, we need really good chain of thought to fine tune the model on. That's like the upper end of what is typically needed. Often you can get away with less, but if you can do that and you can do the fine tuning, you can make most things work. And I think people have not really distributed that knowledge very broadly, or it hasn't diffused, or it hasn't been absorbed.

I see this with my programming peers. Even today, many question either the, I don't know, the effectiveness of AI programming. Some would just claim it's still not good enough because maybe they used the 3.5 class of Copilot, then it released and their experience is still shaped by it. Or yeah, some are averse to it in a very indifferent way. They would say, I can never do what I can do, but they would have never tried it. I do see this for sure for programming. And I don't know what the right answer is. It's since it's people, it's always very complicated. So.

It might also be just lack of confidence that it will work. The process of example curation for the automation is like Not very glamorous, not very fun. A lot of times it involves just sitting down and writing things out that are very tedious. The chain of thought we usually don't record is not fun to sit there and write out. If you had to write out 500 words about how you're doing some very mundane, simple task, that's not anybody's idea of a good time. It's not really my idea of a good time. When I do it, it's because I'm confident it will get me somewhere. And so probably a lot of people are not aware this would work or even heard about it as an approach. They're just like, eh, I don't know if that'll work. If everybody had a good command of best practices and they were confident it would work and actually do it, we could easily automate 100x relative to what we've already automated, even with just, you know, Even leaving O1 and all that latest hotness aside, we just had GPT-4 with the 100K context window and the fine-tuning ability, easily 100x automation potential relative to what we've seen so far.

Okay, that was one good takeaway for me. Anything else on how the frontier models have progressed this last 12 months?

RAG is another area that I think has kind of underperformed. It has been frustrating. These days I often say flash everything. Andrew White from Future House probably had the best distillation of that, where he was basically like, we spend whatever it takes to get the right answer. And they don't have a vector database backing their scientific literature QA process. Instead, they just take everything that seems relevant on a keyword search level, and run it through a language model and have the language model assess, is this relevant to the current question? That, of course, does increase the cost a lot relative to a simple vector database lookup, but it seems to really help with performance. I think people have been stuck a little bit on RAG, maybe because it's a very programmer-friendly paradigm. We have a database, we make a fetch to the database, we get what we need. But the accuracy of these vector searches has just not been great. Maybe it will become great, but I would say that has kind of underperformed my expectations as well. And probably a lot of things that people set out to build over the course of 2024 have not quite thrilled them with results for that reason. I just was talking to somebody. I recently gave a presentation to the Society of Actuaries. They had an AI summit, and one person followed up with me and was like, we've got this internal chatbot and it doesn't really work that well. We're not super pleased with it. And I gave them this same recommendation. Like, how big is your database? Was my first question. In their case, 50 states, and they all have these different insurance regulations, and they might be 50 to 100 pages each or whatever. And it was like, well, with Gemini Flash pricing, you could run literally the entire thing through Flash for every single question for like $0.10 or something. If you're willing to do that and wait 1 minute, I think you could have quite a bit better performance. But again, this is just the kind of thing that is kind of new. It's not super intuitive. People try to build software efficiently, and it's uncomfortable territory for developers to think every time somebody clicks this button, $0.10 might be spent. That obviously wouldn't work at Facebook or Google scale, but it can work for a lot of internal enterprise applications. I think there's, there's definitely a big mismatch between what people should be willing to spend and what they're used to being willing to spend. Just, just the idea that there's going to be marginal cost on usage is very unfamiliar territory for a lot of people that are trying to build the applications. And they just don't want to go that direction. Intuitively, it feels wrong to think there would be a 10 or 50 cent or even a dollar cost. But, you know, when I asked the guy, who had this insurance use case. It's like, how many questions do you think you're going to have a day? The answer was, well, we've got 100 people and it might be a couple dozen a day. And it's like, okay, cool. Well, let's say that is 50 cents each and you're doing a couple dozen a day. Maybe you're doing 10, 15 bucks spent. That's probably less than you're spending on coffee grounds on a daily basis just for people at the office.

Yeah.

So if this thing can really do useful work for you, the cost is basically negligible. But it's been a hard paradigm for people to adjust to. On the plus side, I will put a couple in the middle. Those are the ones that have maybe underperformed. I would say in the middle, price and speed of models have definitely continued their trend. I would only say they've gotten a lot faster, they have gotten a lot cheaper, and I would only say that's on trend as opposed to overperforming just because the trend was already pretty well established in 2023. They've also, I would say, maintained a trend on coding, which is an interesting one where the benchmarks look like they maybe even have overperformed. Certainly with the latest results on SWE-Bench and Codeforces, it's getting really good. But in my day-to-day use, I do still find a lot of the little weirdnesses being active issues. I work with a guy named Siraj, on a few projects, and we now refer to the Cursor effect, which is a little unfair to Cursor because I do like it a lot, and I do like Cursor a lot, and I'm a happy user. But you do see these weird things where, again, for lack of context or whatever, it's like, or for cost management reasons, they're not always— they could in many cases feed the whole codebase into the model, but a lot of times they're not. They do sort of a RAG, try to figure out which files should we include, And then you'll have these situations where it's like, I don't see any mention of this, so I'll create a whole new thing. In a TypeScript app, you have your types file where all the types are defined, basically classes, sort of definition of what different primitives are in your application and what sort of dimensions they have. And so often, I've gotten better at this, but it is an issue if you're new and you haven't quite figured out how to do it. So often you'll find a new types file just appeared. Why did a new types file appear? Well, because the system took your query, did some RAG similarity, figure out what to include in context, didn't include the types file, understandably, maybe because what I'm talking about isn't even referenced in the types file yet, because it's gonna involve creating a new type. But then where does that new type get created? model will just say, okay, I guess we need a types file, create one, and fail to realise there already is one and that's where it should go. So you see these sort of weird artifacts. I think you could think about this in terms of effective context. The headline numbers of how many tokens the models can handle hasn't maybe moved that much. I think it was late '23 maybe when Gemini got into the million token range. Those numbers haven't moved that much, but they've got quite quite a bit better at using the full context effectively, where if you just stuff in all the relevant information, it can actually handle that in a useful way. But we're not always, again, for cost efficiency reasons, we're not always inclined to stuff everything in there. So sometimes these days I have to take a— I actually have a script. I hope to introduce this in more depth on a future episode coming before too long. But we've been working on this app to help people create the small number of gold standard examples that they need to power whatever automation workflow they want to power. That app is not big, but it's a bunch of files at this point. So I actually have a script that just prints it all out into a single file. I take that file, go to Claude or ChatGPT, mostly using 01 on that these days. And say, here's my whole codebase, here's what I want to do. Your job is to reason through it, make a plan. Maybe sometimes I'll say like, give me, you know, sort of analysis, then I'll give you some feedback, ask any questions that seem important, I'll give you some answers, then we'll move on to a plan. Sometimes I do that in one step, sometimes 2, then take the plan back to Cursor. I still find Claude better at writing the right code. And this might be a skill issue, but there've been a lot of people out there saying, oh, O1, especially O1 Pro, just like one-shotted this whole app for me, or, you know, did a full refactor perfect the first time. I have not experienced that candidly. I would say the quality of line-by-line code that I get from O1 still does not feel as good as what I get from Claude, purely subjectively in terms of like what works. But that combination has worked.

I do basically the same. I use Owen for reasoning, but I suspect it's because maybe if you're using Cursor and Claude, I have a feeling like the tooling around Claude is better within Cursor so that the line-by-line changes. For me, my daily driver with Claude, but when I want to reason, I do exactly what you do. That's been my experience too.

So, so it's come a long way. I mean, you know, it is definitely remarkable that This little app has ballooned to 100,000 tokens to the point where I've been thinking, geez, if it gets much bigger, it's not going to fit anymore. So that's not nothing, right? I mean, it's a not trivial codebase, although it's still pretty small in the grand scheme of things. It is amazing that it can handle that whole thing in one byte and reason effectively over the whole thing. And I think you're probably right that the The Claude tooling, it's just, it seems like it's been a little bit more of a focus for Cursor. We'll see. It's very hard to do objective side-by-side analyses of these things. One other thing somebody asked about state-space models and Mamba and has that lived up to the promise that I saw in it just over a year ago. It was early December when I did that first super long Mamba monologue. I would say basically my 2 predictions were, one, that we would see hybrids being the real winner, not that we would see everybody switching over to Mamba, but that it would be some sort of hybrid architecture was the wave of the future. And when asked, somebody at the time asked, if this doesn't happen, why wouldn't it happen? And my answer was because transformers just continue to work so well. and just keep delivering time and time again that there's just no oxygen for anything else and everybody kind of keeps mining the main vein and that's it. And I would say that basically seems like it has happened. The transformers definitely have continued to deliver. And so the number of people who have been looking for other things is not super high. At the same time, plenty of good examples of hybrid architectures working really well. We've had Albert Gu from Cartesia on. They've done some incredible stuff. The speed and quality of their text-to-speech is unbelievable, and the price is so low. I think that is definitely a strong sign of how well these architectures can work. Evo, where we had Brian He come on and talk about that one, that's another hybrid architecture. And There's rumours of, not rumours, but stated next year from Google leadership that they're kind of headed toward an infinite context window. I suspect something like this is behind that. It does seem a bit of attention is needed, but maybe the updated statement is a little attention is all you need, and a lot of things can be made a lot more efficient and handle longer sequences if you have some version of this state space concept. I'd give that one an incomplete. I don't think the technical analysis was wrong. I just think the incredible continued progress from the mainline research directions has been so good that there hasn't been as much need to look at something new as I might have guessed. And Zephyr was another one I wanted to mention. The efficiency they are able to get and the ability to run really like quite good models locally. on device with all the multiple different efficiency tricks, one being a state-space attention hybrid. I do think it is going to be a part of the future. I mean, just in fundamental terms, to have finite size memory that can evolve over time without having to grow in required resources over time, there has to be a version of that, right? I mean, that's kind of like inherently what we do, right? My brain is not getting bigger quadratically with all the information I consume. So there's There's got to be something like that to make long-lived agents viable. And we have at least an existence proof in ourselves that it is possible. And we have a first version, I would say, with the state-space models that kind of shows how this could work in an artificial neural network. So I basically still believe in it. Again, maybe with the caveat that maybe the transformers will just continue to deliver, and maybe if they deliver really well, What effective context do you need is an interesting question. If they can scale Gemini up to 10 million or 100 million tokens on conventional attention, that's a pretty long sequence, right? Maybe it's growing quadratically, but maybe you can get it to grow quadratically for as long as you need to get something that feels functionally unlimited. If so, maybe that's enough to keep the state space thing more marginal. For another year or more, but time will tell on that.

Let's see what 2025 holds for state space. A very related question. I mean, you rank things now as underperformed, on trend, or overperformed. There's one more question that somebody asked. What surprised you the most in 2024 and what do you expect in 2025 because of that?

What has surprised me the most so far has just been, and I think this could get reconciled really quick here in the new year, but I expected the OpenAI announcement, the 12th day of Shipmas. I was expecting to see an agent framework of some sort. It's been widely reported they're working on one. I used the computer vision thing, which they've only shipped to the iPhone app as far as I know. But I did read a biology paper not long ago with advanced voice mode and screen sharing. And it was an awesome experience where I was saying, you know, like, what, explain this figure to me. And the thing is explaining the figure, and it had both the drawings and the text description of the figure. So I'm not exactly sure how great that perception is still necessarily, but it was able to really augment my reading with this combination of me talking to it and looking effectively over my shoulder and seeing what I was seeing. Add to that the cloud computer use that we've seen, and it seems like we have got to be really very much on the verge of some agent thing happening that will really start to work. I expected more successful autonomy and agent-style things for lower-end tasks and not as much high-end reasoning progress. That divergence, I would say, has been the biggest Surprise. It's a weird world where it's like we are beating PhDs in their own domain of expertise on all these really hard questions. And yet you can't reliably book a calendar event without extreme guardrails. And even then it can be tough. That divergence has surprised me the most, but I guess I would still be surprised, doubling down on maybe my own mistaken analysis. Or possibly, time will tell if this is doubling down on the wrong ideas or if time will prove it right. Hard to imagine that can't get resolved soon. Hard to imagine these things that reason well can't do basic stuff. It would seem you would see a lot of pretty clear signal, right? Like, did the thing get booked at all? Did it get booked at the right time? Did the user accept the invite? I mean, there's a lot of these little things that I think we can get pretty good reinforcement learning signal from. I assume they've been doing that, but, you know, all I really have is sort of the rumors to go on there.

So if I have to summarize, so you expect, I don't know, for retail customers like us, better agentic workflows or frameworks that we can build on top of, or frontier models that we can rely on. Would that be a fair assessment? You're disappointed that that did not happen in 2024? But you expect it to happen in 2025.

I'm still a little confused as to why it hasn't happened, or maybe it just hasn't been released. Claude computer use does work reasonably well. You know, first shot it was able to go use Waymark. It seems like they sort of really reined it in, in terms of not allowing it to do things in your account. I had to lie to it to get it to use my account, even on just like a, marketing product like Waymark, not like a bank or anything sensitive, just literally like, hey, you're in my account, make me a video. It said, oh, I don't use accounts. I logged out. Now you can use it anonymously. And that was not true. Again, the robustness is not always there, but it was able to do the thing and prompt the video maker and use most features. And it does seem like the reward signal will be there for that. That doesn't seem Maybe I'm wrong, but that doesn't seem harder than, or doesn't seem much harder than getting feedback on software tasks, right? Did you pass the unit test? Did you accomplish the goal? Maybe they would need synthetic environments to do this, but did you get to the end and tell us what the secret password was at the end of this web navigation maze? Yes or no? It seems like that is something that should be quite doable. So, yeah, I expect agents the consensus view at this point is that 2025 will be the year of agents. If anything, it feels overdue to me. A couple other predictions for 2025. One is just echoing Sam Altman directly when he was asked, I think on like a Reddit AMA not long ago, for his 2025 forecast. He said saturate all the benchmarks. And that seems increasingly realistic to me. The 2% to 25% jump on frontier math and all the signals that they're sending, basically suggests that they don't see this ending anytime soon. And then I would also say, because this is much more reinforcement learning powered than previous approaches, it also seems like stuff is going to get weirder in 2025. We'll probably start to see a lot more of these things, like the famous AlphaGo move, where it looks like a mistake at first, but it actually turns out to work. Or some of these deceptive scheming behaviours. AI is accomplishing the goals they've been given in strange ways. That is kind of strange, surprising, leaves us with these sort of uneasy feelings of what's going on here. We're sort of losing the ability to understand what these things are doing. That is probably going to wrap up quite a bit. The chain of thought, the idea that they're not Putting pressure on the chain of thought is like an attempt to try to keep that to a relative minimum. But there's a lot of other research that is coming out that I think will take that in other directions. And especially, I'm worried that it, especially coming out of Chinese in short, but like compute poor environments in general, there's a lot of pressure to make these things more efficient and more efficient might also mean less legible. So one research trajectory I've been following recently from Meta is they're trying to do more and more things to kind of train things end to end. One was baking in the temperature selection on a token-by-token basis to the model. So with APIs today, you have the ability to set your temperature value. You can set it low and it will give you the model's best guess for any given token. You can set it high and it'll be more random and hopefully more creative or whatever. But there's a lot of situations where you would ideally like that to vary, right? depending on the use case and which token it is, when it's clear the next token is going to be a period, if the probabilities on the tokens are like 99.99% on the period and that 0.01% distributed over everything else, you don't want one of those other tokens. You want it to go with its best guess. But in scenarios where it's like, you know, trying to come up with something creative or you try to, you know, write a joke or whatever you, and I think about episodes with Trey Colmer, AI enthusiast and Hollywood comedy writer, he was saying that there's often a token where you're really going to make the joke and you need the right token for that joke. But that's really where you want to be creative and you would want to see coherent up until that moment. You would want to ideally recognise that moment and then you'd want to turn the temperature way up and do a tonne of generations just on that token until you find really the right joke that works. And they are starting to do that kind of thing now at Meta where they're baking in dynamic sampling strategy to the model itself. So it is now kind of saying like, okay, we're confident here, so put the period where it needs to be. But here creativity could be rewarded. So let's go for it. They also have one where they are doing what they're now calling reasoning in continuous space or latent space. Long time ago, we covered a paper on the pause token where a model was basically given extra tokens to think without necessarily needing to say anything. Still, though, that was just a pause token where that token was selected and then fed back into the autoregressive machinery. In this new Meta paper, they're not making it choose a token, but instead taking the internal state at the end of a thinking forward pass and how exactly they determine what's a thinking forward pass versus a token generation forward pass. They have a couple of mechanisms. One very easy one is just give it a finite fixed number of tokens to forward pass in thinking mode before reverting back into token generation mode. But they just take that last hidden state and put that back in as the embeddings as the next input and just let it run that way. That's pretty cool. Definitely seems like a powerful attractor. Could be like one of the things that they find is that it's significantly more efficient. They can achieve similar results with fewer forward passes. It also seems better at breadth-first search. They kind of create some toy problems where there's graph structures that the thing has to navigate. Picking one token at a time, you have to sort of trace out these paths and essentially do a depth-first search to see what works. But what they think is happening, and I think there's decent evidence for it, is that since it doesn't have to pick a specific token and is able to kind of continue to chew on its internal states, that it represents different paths in superposition in a breadth-first way and can perform better on these tasks that are best done by a breadth-first search approach. But for all those nice things to say about it, I kind of hate it inasmuch as I feel like I want somebody to be able to read the chain of thought, even if I can't see it as a user. Like, the idea that OpenAI can have some sort of monitoring system looking at the chain of thought and knowing what it's saying seems quite good. And to make all of that illegible to humans seems bad. This is something I used to worry about and never really had a good answer for it, so I just kind of let it go. Now it is showing up. It's going to be really hard to write rules about that. But I do have somewhat of an instinct to say, can we have a rule that since we can actually have them think in ways that are legible to us and that we can read, maybe we should do that. Maybe we shouldn't try to take all this stuff out of language space and put it all into the continuous space. I've also seen some analysis that's like, well, you could still interpret those internal states. We have interpretability, ability to probe and classify, but we're making it harder on ourselves. When we take everything out of language space, you can't read it. You now have to use other techniques. These techniques are relatively nascent. All that adds up to more weirdness from AIs driven by these reinforcement learning paradigms, broadly by efficiency things and things that kind of push the reasoning deeper and make it less legible, with some upside in terms of efficiency, in terms of breadth for search, superposition representations. But I think all that leads us to 2025 might be the beginning of general purpose weirdness from AIs, like surprising moves where you're like, huh, I don't know what you were thinking there. It might have been genius, but I'm unnerved by it. I think that will probably start to happen more. I think we'll start to see probably like scheming in the wild, scheming that actually has consequences. I would be surprised if that gets under control well enough to not be an issue. So on the meta level, it might not be a surprise these things will happen, but I think a lot of individual users are probably going to be surprised, even if the sort of zoomed out perspective makes it somewhat predictable. We should expect a bunch of Move37-like things to start to pop up. It won't be surprising in the aggregate, but yeah, these individual cases will probably be like, Pretty surprising, pretty inscrutable, and probably a lot of them will like, you know, go viral, make the rounds, and be debated in similar ways that the scheming results were debated.

So we talked about 2024 and 2025. Now we are going to zoom out even more because there are questions about trajectory, about the future and impact. I'll start with one. One of our audience members asks, what are your thoughts on how AI affects education?

I have to do an episode on this coming soon. I recently had a couple of guys from a nonprofit in Michigan reach out and offered to take me to lunch. And I had a really good conversation with them over a couple of hours about what they're seeing and what they're not seeing in terms of AI adoption in the classroom. We've not done a lot of episodes about this, but one that does stand out in my memory was Sean Janszafar from Khan Academy. That stuff is going to be big, although how quickly it will be adopted, it could lag significantly depending on how much we see teachers unions trying to block it or not. I've been pleasantly surprised by how little anti-AI activity we've seen from doctors. I feel like maybe that's because people don't know yet what's possible. In education, there might be a pretty strong immune response from the current stakeholders to say, we don't want this in our classrooms. Certainly there is some of that out there. One really important thing is to separate learning from education. If you think of learning as something you want to do and education is something forced on you, forced learning in a way, if you just want to learn, it's unbelievable what is now possible, right? I mentioned my biology paper reading experience. To have a real-time natural conversation interface that can see what you're seeing, that has encyclopedic knowledge, and is just instantly there to explain things to you, that is unreal. You know, and I would say my willingness to even try to take on the intersection of AI and biology for this feed has like, been basically predicated on the fact that I can use you guys to help me ramp up, because otherwise I would've needed, maybe I could've hired like a human tutor, perhaps. Obviously that would've come with a lot more expense and a lot more logistics, lower response times, and probably also just a lot less comprehensive knowledge. Like, it would be basically impossible to hire somebody with all these positive traits that the advanced voice mode with vision has as a tutor. So it's amazing if you want to learn any domain, right? I mean, it's, it's truly incredible if you are motivated. This is the best thing that has ever happened for learners. It's not as clear how that will be applied in public schools. And, you know, I've recently been telling my kid who's 5 years old and is like quite bright, but gets frustrated quickly sometimes when it comes to If he's trying to read, he's very ready to read, but he's not comfortable being uncomfortable, struggling. He kind of runs into these words he doesn't know, and he gets a little frustrated, and he's like, ah, whatever, I don't feel like doing this. And so it's not entirely clear how AI will help with that. Maybe AI pep talks, but he gets pep talks from me. Maybe AI will give him better pep talks. But I've started to tell him recently that the most important thing is to enjoy the process of learning. That feeling of this is hard and then trying anyway and it getting a little easier, like, that's the most important thing he needs to learn. More important even than reading, because it will apply for everything. Can AI teach that? I don't know, maybe. But disposition, I guess, for lack of a better word, is really a huge fork in the road for, like, just how much value you're going to get from AI. If you like that experience of learning, you like being a little uncomfortable, or at least can tolerate it, then the AI can help you learn extremely quick. If you are averse, I'm not sure how much it helps. I'm not sure it makes it easy enough that you're not at all uncomfortable. It's still, of course, effortful to learn. How will it actually get applied in the classroom is anybody's guess. My general sense is it's quite lagging. What I heard, and hopefully we'll do an episode on this with these 2 guys, They basically said they've done Khan Academy pilots and stuff like that. They said sometimes the teachers will say, oh, we love this. What about your students? Oh, we never even got that far. They're not close to pushing to the limits of what is possible. They're dabbling a bit themselves, getting a little lesson plan, writing help, or maybe getting a little whatever kind of help Khan Academy has way more already developed than is deployed even in places where it's kind of getting deployed. It seems like a lot of times partial deployments. So a lot of opportunity there. Personal tutoring is the most powerful way to learn. Alexander the Great had Aristotle as a tutor. We could all have an Aristotle-level tutor perhaps in the not too distant future. Will we actually have the inclination to take advantage of it? I suspect Alexander the Great had a lot of cultural context encouraging him to be great. And I think he kind of knew, He had a sense for what that was going to require of him. And I don't want to overly glorify Alexander the Great, who I don't think is necessarily a positive figure in history in many ways, but he was definitely one who was ready for a challenge and not going to shy away from it and ready to do the hard work. Those things seem really important in the context of learning/education and much harder for the AI to have an influence on.

So it seems like if you want to learn, this is the best time to be in, but you need to want to learn. That seems to be the big takeaway. So, I'm gonna follow that with another question that one of our audience asked, which I think is kind of similar. They simply asked, should I even learn to code now?

Yeah, I think that one is so individual. I think my simple answer is, increasingly, I think most people should do what they want to do. And There's a leap of faith around society making good choices. If you wanted to put a doomer lens on it, enjoy the time we have before the singularity in ways you won't regret. Whatever gives you the most pleasure and fulfillment is a good default answer. I have a little bit of skin in the game on this question in the form of an ex-au pair in our family who we have sponsored to go to community college. locally. She was an au pair in our family for 2 years, and then we basically said, for various reasons, like, we'll support you to stay here and go to the local community college. Now, her goal long-term was to get an employer-sponsored visa and be able to, you know, stay and make a life here. Okay, how do you go about doing that? Well, as everybody knows from H-1B discourse from the last week or so, huge, huge percentage of those visas are going to IT and programming kind of jobs. So we looked through the database to just try to get a sense for like what kind of organizations are hiring and what kind of roles are they hiring for. And, you know, indeed we found just overwhelming IT and programming. This was 2 years ago now. There was a minute when we were like, hmm, I guess you gotta do that if you wanna, you know, ultimately get one of these spots. And she was kind of like, okay, I mean, I don't really want to be a programmer. I came here and did this because I like kids. Early childhood education would be my real preference. But she was like, oh, I guess if this is what I have to do to be able to achieve that goal of being able to stay, then I guess that's what I'll do. In the end, we were like, it might be so different a few years from now that I would hate to see you spend 4 years getting a degree in programming, which you're telling me you don't love it now. You don't have those signs of being like a kid that was programming at a young age. You know, she's, I'm sure she could learn it. She's a bright person. Um, but you don't really have like passion for it. I would hate to see you do all that and then come out the other end 4 years from now only to enter a world where the skills aren't even that relevant anymore. So by that time, it's plausible to me that Maybe it'll be very different. Maybe we'll be putting H-1Bs toward— first of all, of course, the policy itself could change, but in terms of what is considered scarce and valuable, maybe childhood education will be a place where there would be a lot more demand because the AIs will be doing all the programming. In the end, we basically said, do what you want to do. Don't spend years of your life on a bet that in some sense implies that the AI is like, aren't going to dominate the programming profession in the next few years, because it seems plausible enough that they might. At the same time, though, I would not caution people. I would not say don't learn to code. I would say learn to code if you want to learn to code. It's never been easier. You can pick up all sorts with Replit, with Cursor, as we've talked about. It is way less tedious than it used to be. Get things explained to you way more conveniently. You can ask all sorts of questions. And I think that's probably something that, again, is probably underdone by even professional programmers. I have the advantage of never having been that good at programming, so it doesn't feel like a huge ego hit to me to ask the AI for advice. It can handle 100,000 tokens of code. It can reason over that. It can give you refactoring plans. what libraries to use for this sort of thing, what patterns. I mean, it can really explain a lot about code that you have in front of you or code that you are— an application that you're dreaming of creating. I think the barrier is low if you want to create applications and you have never realized that dream. Now's a pretty good time to realize that dream. But if you don't have that dream, I wouldn't try to cultivate a new identity as a software developer, for some expectation of future payoff, because there's a real risk it won't be there. In the short term, if you do embrace the latest tools and work effectively, there's a lot of arbitrage opportunity. People are used to paying high prices for software, and you can deliver it cost-effectively. So I think hundreds of dollars, even $1,000 an hour projects are actually pretty abundant out there. Right now. I just don't know how sustainable it's going to be. So I wouldn't say build your long-term future around a bet like that. But if it seems like it will be fun to you, if you're curious about it, if you have ideas that you want to see realised, then jump in. I think this stuff is coming to code maybe sooner than other things, but it's hard to imagine it doesn't come to other things. Just as well, right? Like, if your analysis was like, well, I think that's what I would enjoy the most, but I think I'm going to go be a lawyer because, you know, I understand that like the reinforcement learning signal is like stronger in programming and it's going to take longer to come to legal analysis. I wouldn't make that bet either. They'll figure it out. It's going to come to legal analysis. You may have a couple more years there, maybe. Miles Brendage just said something really interesting about this too. He's the former OpenAI policy guy that I've cited repeatedly. He said, don't mistake small relative differences in timing for the shape of the overall trend. And his point was like, this is coming for everything. It's going to take a little longer for some things to be figured out than others for multiple reasons. The reward signal being one. Another that he pointed out astutely, I think, was these are all software companies, so they know code. This is their own need. It's like there's multiple reasons that the code is going to happen first, but it doesn't mean that it's not coming to legal analysis and just about everything else that you might imagine. So bottom line, life advice, I guess, broadly, I would say do what you want to do, what you feel intrinsically motivated by, and let the chips fall where they may. Aside from that, I wouldn't— I personally do not optimise really at all for marketability of future skills. I'm not really thinking super long-term. My goal with this show and with most of the activities that I'm doing is just to learn as much as possible. That includes coding. I want to learn how good the models are at coding. I want to learn what the new workflows are. Sometimes I have intrinsic motivation to build something, but I have no theory that, like, honestly, any of the skills that I'm developing right now are gonna be like very differentiated in a few years.

So nobody knows, seems to be do it if you want to. Timelines are very hazy. Cool. Okay. We will move on to the next one, which actually I think it's relevant for me, but I suspect many are in a similar boat as me. As you know, we run a small company. We do post-production for podcasters with AI. There is always this fear that I mean, if you asked me in 2023, I would not have been so surprised, but watching the trajectory in the last one year and also how you gave an overview of how things are, it's hard to figure out what's the defensible moat for a company that you're trying to build. So you're trying to add value to somebody and you want to build this in a defensible way so you can build this up sustainably for long-term, especially if you're a small bootstrap company. For people who are in this boat, what do you recommend? What strategies do you need to kind of think of? are things that you would recommend we do or we do not do so that we don't get steamrolled by, I don't know, big companies, like foundational companies or other ones? So what would be your advice? And also, what do you think others in the space are trying to do to kind of safeguard themselves?

Tough one. You know, no moats has been sort of the refrain or the callback over the last couple of years for the AI space broadly. I'm on record that I do think there are some moats in some places, but it is tough. Maybe a better way to say it is my crystal ball gets real foggy not too far out. With Waymark, I think we've done pretty well. I have some sense of why that is, but then I also have some real doubt as to how that may play out in the future and just how sustainable is that for the long term? So Waymark makes videos for small businesses. We previously had a user interface DIY kind of approach where you would come in, you would pick out of a big library of well-designed templates that our creative team made. You would pick something that you liked the look of, and then you would be responsible for filling in all the details, all the copy, choosing all the assets, whatever, and tinkering with the colors and yada, yada, yada to make it yours and make it what you liked. So we had all that stuff. Now when AI comes along, we can get AI to do those things in our software so the user doesn't have to. And that could make the user experience way better. It could be faster. It's a better writer than most people are. It's especially on these like kind of somewhat idiosyncratic tasks around having a voiceover and lining up the voiceover with what's on screen and making all this stuff work well together. Still work in progress, but we were confident early on that this could do better than our users are doing. We've worked on that. And of course, a lot of people have come into the space and said something similar. They're like, hey, maybe we can get AI to create videos. I think what those new entrants have missed is you actually do need the stuff we started with. You need a really well-designed template library because the AIs, at least for now, are not able to create something from nothing that's awesome. They are very good and increasingly approaching excellent at filling in a template in a way the user likes, but it was still a human team that created those original designs, that made them coherent, that did good motion graphics work, mixed music and motion graphics together for impact and all those sorts of things. The AI is good at sort of filling that vessel with new content, but the quality of that form remains really important. And there's not an AI that can handle that from scratch today. Similarly with the interface itself, People very seldom don't make any changes. You know, they almost always see something in what the AI creates for them that they're like, even if they think this is awesome, they'll usually have a couple things where they're like, but I do want to change that and that. That doesn't mean the experience wasn't awesome. It doesn't mean they're not floored by how amazing the AI is, but they're just like, I changed my phone number. It could be as simple as a factual change, right? It could be whatever, but they always have something they want to change. Or that one picture, I actually have a better picture. It's not online, but I just took it. They'll always have something. So you need the ability for that last mile edit to give the user exactly what they want. And this is pretty Waymark specific, but hopefully somewhat generalizable. At the highest level, it's like bringing a certain level of taste, of understanding of what this is and what we're trying to do here and what good looks like, and embodying that somehow. Into the nature of the product, which in our case is like the template library. And then the low level of like those final changes, making that something that the user can actually do in a reliable way where they can get exactly what they want. Those things are really important. I see a lot of products missing those, especially if they came in as part of the AI wave, then It feels like, okay, we're going to build with the AI, build with the AI, and they don't think as much about those other things. But those other things are still important. How long this holds is tough to say. With Sora, ChatGPT that can use Sora, you can imagine, or Veo too, whatever, you can imagine saying, hey, ChatGPT, or hey, Gemini, here's my small business website. Make me a 30-second TV commercial, and maybe it can get smart enough that it can just do that whole thing and just do it all in pixel space and make it awesome. But I think that's still a little bit out, and it's unclear how much do the folks at the big companies care about that. If you really said, okay, I want to refine Veo or refine Sora to the point where we can layer copy into these things and we can edit copy, on a copy edit level where we could say, hey, in the last scene, the phone number should be changed to this. Will they be able to do those command type edits on raw video footage that are spitting out in like world sim pixel space? Probably can get there. I don't think anything is off the table, but they clearly haven't fine-tuned or taken care to get the things to work on tic-tac-toe. So are they really going to grind out the data to cover all those little use cases? You probably have a while before those things will emerge. So, I mean, that's kind of like a niche strategy a little bit, basically saying, like, yeah, these things might get really generally super powerful, but if you know what counts as good in a particular domain and you can kind of create some guardrails that ensure that happens, and you can create some last mile editing or customization features that make sure people get exactly what they want rather than being frustrated by trying to ask the AI over and over again and never quite getting it right. Those things seem pretty valuable for at least a while to come. But again, it's tough, right? Because you could also imagine if they get really good at coding to the point where they can spin up things on demand, then who knows? At some point, maybe everything collapses to the cost of inference. Maybe one model to think about it is like, Yeah, maybe even in a future state, ChatGPT could like spin up a Waymark app. It can both like call Sora for footage, handle layers and set fonts into videos and all those sorts of things. You know, it can write the code to do those things in a programmatic way so it can recreate our experience. Then you maybe have some price pressure if people start to expect and to legitimately experience that any set of software tools can be like relatively easily spun up. through simple commands, through a model, then we're headed for a world of abundance, I guess. Right. And that hopefully then maybe that's good.

Maybe then I don't have to care about building a small company. You can just live in the land of abundance.

Yeah, I think that's kind of the Waymark attitude. We're like, you know, we think it's going to be a while before they're going to do the stuff that we do really well that like delivers real practical value for users. I mean, I guess that also, I sort of gloss over that sort of thing, but You know, to make sure you are actually delivering real value for users and not just like making something neat with AI, I think is really important. I have a bias for making something neat with AI, regardless of whether anybody was asking for it or not. Our CEO, Alex, longtime close friend of mine, is better and more disciplined about being like, what do our users really want? You know, what are the pain points that they have with the current product? And making sure that he solves those. And sometimes that coincides with making the next really neat thing with AI, and other times it doesn't as much. In the short term, there's value in just making sure you actually nail it and give people what they really want and are prepared to pay for. And longer term, it's definitely harder to say. What's weird is that these capabilities can either emerge or be engineered, and we don't always know which has happened, right? In the few-shot learning was an emergent capability. They weren't like, necessarily expecting that to fall out of GPT-3, but there it was. And with few-shot learning, it's like, man, this can sort of, in theory, maybe do kind of anything. It's got meta-learning. So what won't emerge with the next few generations? It's hard to say. Not everything will emerge. In the meantime, they're identifying their biggest weaknesses, patching them, and working with Scale AI and doing all sorts of things to collect really high-quality data to train things on. So you're more sort of insulated as a business if you are doing something that is both non-emergent and not so focal that they're going to go like invest the time and energy to collect the robust datasets needed to power that capability. I think for Waymark, we're in a decent spot there. They do care about video, but I think they mostly care about that as a world simulator thing. I don't think they really care that much about content. So I think we're in sort of a safe space for a while until maybe it emerges. At what point does that emerge through pixel-level stuff or enough coding sophistication that you can just say, hey, how would you solve this? And it spit out an app that is sort of like, oh yeah, no problem. If that happens, could be a problem because certainly the things will have like much more generality than we have. The trade-off with our app And most apps is like, you are trying to do something narrow really well. If these things can match you on quality, they will crush you on breadth. If the AIs ever sort of can match us on our strong points, then they'll definitely be way better on our weak points. So we just kind of have to hope that doesn't happen super soon, or at least until we're in the age of abundance. Cool.

Last, we speak of age of abundance. I'm going to pick another question that's very related to that. Why would UBI or post-caste be likely outcome of AGI or ASI? It seems like wealth concentration is the natural direction set in motion by capitalism, which will only quicken the pace. So they are questioning the assumption of why should UBI even be a likely outcome of AGI? Why wouldn't it turn out to be another Kind of wealth concentration.

Yeah, I certainly don't rule that out. I mean, a lot depends on the shape of the technology, right? So we've seen this reasoning paradigm shake things up a little bit again there. And I think I probably talked enough about that earlier when it comes to things like compute governance, offense-defense balance, and how we were headed toward highly egalitarian and low ability to control. This maybe moves us back a little bit more toward inequality and more ability to control. But I don't think this is the last word on that either, right? The core models are probably going to continue to get better. I fully expect that what you can run on a laptop will continue to advance for at least a few years to come. How does that all play out? I guess one possible way is that there could be a decoupling, and arguably this sort of already happened. I mean, And I think it has certainly happened in Scandinavian societies. People can call me out here on being full of shit because I'm not by any means an expert on Scandinavian societies. But I think you look at a society like Sweden, my understanding is that there is actually a lot of wealth concentration and hereditary passing down of sort of really important companies. Like IKEA is a family-held company. There is an elite. that sort of controls a lot of the really important institutions of society. But then there is also a generous, pretty robust social safety net, and people are looked after, right? And they're not— certainly they don't have, like, a mass homeless problem. They don't have a mass incarceration problem. People get to have their needs taken care of. I think something like that could be maybe the way that this goes. In a future where compute remains, when you need a lot of scale, like, resource owners will have a lot of, but maybe individuals, even if you don't have that sort of wealth and power, will benefit from access and the ability to just take advantage of things. It's always been an active discussion around American healthcare. It sucks in many ways. It's awesome in other ways. If you have a rare disease, you want frontier treatment, the US is probably the best place to get that. If you want to not have to worry about a lot of bullshit and to not have to have fear that your claims might be denied by an insurance company, or if you want to look at things like maternal mortality statistics, US does not look so good at all. Maybe we can all have the best access, right? Maybe we can all have an egalitarian future. We can all have access to the top oncologist because the top oncologist is an AI and it's highly available and highly Scalable and affordable. This is kind of the Ahmad vision, right? What if everybody had access to top quality guidance? What if expertise was effectively free or very low cost? I think that some version of that, I guess, would be my sort of best guess with major caveat that like the future is probably going to be pretty weird. A couple governments and a handful of big tech champions own the physical capital, do the frontier development, and basically are deciding amongst themselves how the future is going to go. And then everybody else is living very comfortably and has, for practical purposes, access to all these incredible sources of expertise that were previously unimaginable. That seems like maybe the new social contract we're headed for, because the compute And the decision-making power do remain scarce. Who trains the $100 billion model and on what hardware? At that level, it's going to be scarce. And if you want to spend $1 million inferencing on one problem, on that level, it's going to be scarce. What research bets do we make? What do we prioritise? I think at OpenAI, they're still fairly compute-bound. My sense as to why they haven't launched Sora is they just don't have the compute to to support it, really. There could be other reasons too, but that seems to be a big one internally. We can only scale up so many things. Which of the things are we going to scale up? That is not always an easy decision for them. And that's just not the kind of thing that is probably going to be like direct democracy, it would seem. At the same time, you could live extremely comfortably, efficiently. I think about how many language model interactions you can have. At the cost of one crosstown trip, one 20, 30-mile car ride that you might take a trip to the doctor and back, that covers an awful lot of tokens. So I think there is abundance in many ways, abundance of expertise, abundance of creative tools, but it seems like decision-making and capital could end up being quite concentrated.

I think that kind of naturally tees up the next question, which is an audience asked, in your previous episode, you stated one of your podcast goals is using your voice to influence the world in a slightly positive direction, however marginal that might be. Now, somebody who feels responsible and without much technical expertise, how can I contribute meaningfully to this effort of ensuring safe and responsible AI development, especially in a world where commercial pressures might rush AI progress towards potentially unsafe outcomes?

First of all, I guess I would say, don't let the not much technical expertise hold you back for all the reasons we talked about learning, right? If you're actually motivated, you can make great progress pretty fast when it comes to understanding what you need to understand. To at least be conversant, right? To at least be sharp on what's going on. Advancing the frontiers, that's a different question. But catching up to the near frontier, at least, I think, has never been more accessible. I mean, there's a lot obviously going on in AI, but so much of it is the same thing working over and over again, as we've talked about with transformers versus something new, state-space model or otherwise. If you want to know what is really going on right now, what really matters, I think you can get there pretty fast. I wouldn't let lack of technical expertise be a mental barrier. I think that's like very much overcomeable for most people to get, you know, get to the level they need to do something. What do you do? How do you try to shape the world? I think it is tough. You know, we're in, there's a lot of like tough dynamics. going on. Kind of played around with this adoption accelerationist hyperscaling pauser notion for a while. If people had a better understanding of how far things have already gone, then they would have a healthier respect/fear for where things might be going next. So I think in some ways, the can I make a positive contribution is maybe answered the same way as how can I take advantage of AI in my personal life? Personally, I do find that there's high overlap. Understanding what's going on, helping other people understand what's going on, getting day-to-day value, helping other people get day-to-day value, bringing people up to speed, or at least alerting them to the fact that, look at what AI can do, and having the implementation know-how to be able to do that effectively so that they're actually getting properly calibrated to where things are. I think that's pretty useful, honestly. Even in the AI safety community, a decent amount of stuff can be interpreted that way. When you look at Apollo Research's work on scheming, it's very thoughtfully done. I think those guys are really smart, but they're basically demonstrating capabilities, right? They're saying, look at how far this has already come. We just got this new model. We put it in a somewhat realistic situation, a little bit of a toy problem, a little bit contrived, but if you squint, it's not hard to see similar things in the wild and look at what we observe. I want to be properly calibrated. I don't want to understate how much thought has gone into that work, but I also don't want to overstate how technically demanding that work is. I think it's much more driven by having an accurate sense of what they can do and exploring in the right spaces and having a sense of, if we could show this capability, would that be meaningful? Would people update their thinking on that? And I think they would probably tell you that the engineering that went into it was not insane and that there's not breakthrough sort of eureka moments there. It's more about strategically shining a light on Look at this aspect of what these things can do today, and then trying to make an argument about what we should think about that. They were quite successful with that. People were not all sold. Of course, there are always voices that come out to say, oh, that's not a big deal, or you're overreacting, or whatever. But they definitely made a little dent in the universe with that work by getting people's attention on it and by showing what is possible, helping people be accurately calibrated when things are moving so fast, that is really hard. So that's really what I try to do for the most part, to learn as much as possible. And my hope is that I'll be one of the people that has the most up-to-date and accurate worldviews. And I kind of trust that will be valuable, try to help other people have the most up-to-date, accurate worldview they can. Any decision-making will be advantaged, hopefully, by having a more accurate and up-to-date worldview. So that's not exactly a master plan, but it's a strategy a lot of people could play. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.