Zvi’s POV: Ilya’s SSI, OpenAI’s o1, Claude Computer Use, Trump’s election, and more

Ilya said that this is no longer the age of scaling. Once again, everyone is looking for the next thing. Scaling the right thing matters more now than ever. So basically the OpenAI attitude of we'll just add a zero to everything and see what happens is not even what OpenAI is doing anymore, right? They've moved on to a one because their previous strategy wasn't working. As a result of SB 1047 being vetoed, the bill is not being used as the model for other legislation in other places, even though it is obviously the best model legislation that we have available. Instead, people are considering bills like the one in Texas, which use a use case targeting EUA Act style approach to regulation of AI. That approach has all of the costs you can imagine and none of the benefits to existential risk. If China wants to knock TSMC out of existence, it can. If I was the US government, I would either deliberately have a goal of US chip independence 2027, or I would be pretty damn committed to defending Taiwan.

Hello and welcome back to The Cognitive Revolution. Today, Zvi Moskowitz returns for another wide-ranging conversation as we look back on the major developments in AI, including the technical, strategic, and political that have transpired in the 6 months since his last appearance. We discuss Ilya's new startup, which aims to go straight to superintelligence, the utility or lack thereof of OpenAI's O1 model for things like coding and Magic: The Gathering, the likely impact of Claude's new computer use capabilities, and the recent discourse around the end of scaling. We also analyze the complex web of partnerships emerging between big tech companies, the regulatory outlook in the wake of the veto of SB 1047, And of course, the latest OpenAI drama, which this time centers around the question of who should and will get what as they convert to a for-profit company. Toward the end, we consider the potential impact of Donald Trump's return to the presidency on AI futures, and I pick Zvi's brain on how the AI safety community should evolve its understanding of individual virtue in this new political reality. On all these topics and more, Zvi delivers a mix of strategic analysis and insight that that you really can't get anywhere else, except of course for his blog, Don't Worry About the Vase, which is online at thezvi.substack.com. I personally find these exchanges super helpful, both for maintaining general situational awareness and in some cases for challenging my own mental models about how to most effectively work toward beneficial AI outcomes. As always, if you're finding value in the show, we'd appreciate a review on Apple Podcasts or Spotify, Or you could just share it with a friend who's trying to make sense of the rapidly evolving AI landscape. Of course, we welcome your feedback either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, I hope you enjoy this latest discussion with the great Zvi Moshowitz. Zvi Moshowitz, welcome back to The Cognitive Revolution.

Thanks. I always enjoy it here.

So it's been a minute. And the pace hasn't slowed down, I wouldn't say, although we can maybe get into the Meta discussion and why people seem to be even seeing that question increasingly differently these days. I went back and looked at the timestamp of our last conversation and it was right after OpenAI and Google had sort of previewed their respective voice modes. And then I think literally as we were recording, there were like Several resignations from OpenAI's safety team. So then we did an immediate part 2 to talk about that. And certainly quite a bit has happened in the intervening months. So what I thought we would do is just run it down, get your take on a bunch of stuff. We're both in the business of keeping up with this on a weekly, daily, and hourly basis, but not everybody can do that. So this is kind of the zoomed out few months recap and analysis.

How's that sound? I like it. Yeah, let's do it.

Cool. Looking through your blog and just revisiting the news myself, I thought that one story that kind of came and went, but which has at least some chance of proving to be maybe the most important story of all over the last few months was Ilya's announcement that he is starting a new company, which is going direct to superintelligence. Safely, of course. And so we've nothing to worry about there, but just, uh, basically just want to let you know, we're going to go build superintelligence and we'll let you know when we've got it. No products and unclear how many updates we'll get in the meantime.

It'll definitely be safe. We know that, right? It's right there in the title.

So it's, I would say their ability to do this is Obviously still questionable, but to borrow an infamous phrase, strikingly plausible. And, you know, that in and of itself is pretty crazy. Certainly you couldn't have made such a plan at all credible up until pretty recently. Uh, what do you make of it and how often do you wake up at night wishing or dreaming that you might've got an unexpected update on their progress?

We have a little bit of an update, just This week, actually, Ilya said that this is no longer the age of scaling. The exact quote is, the 2010s were the age of scaling, and now we're back in the age of wonder and discovery once again. Everyone is looking for the next thing. Scaling the right thing matters more now than ever. So basically, the OpenAI attitude of, we'll just add a zero to everything and see what happens, is not even what OpenAI is doing anymore, right? They've moved on to a one because their previous strategy wasn't working. And it's obviously, he's not gonna spill the secrets, so we don't know how well it's going, but presumably they're not finding it to be that easy. But yeah, it's deliberately opaque. It's deliberately hard for us to tell what's going on inside. I do know that I am skeptical that they can achieve safe superintelligence if they achieve superintelligence. I'm also reasonably skeptical they'll get to superintelligence by this direct method, but Definitely not impossible. I can't rule it out.

So my next topic was O1 and you're touching on that. And then also this sort of meta debate as to what's working, what's not working. I find this quite confusing because on the one hand we have Altman and Dario, and I would say also the DeepMind founders, although not quite as recently, all pretty clearly on record saying We know what we need to do. We don't really see any fundamental barriers. We think we're going to get there in a 2 to 3 year timeframe. And then we also have this sort of end of scaling narrative that has popped up. It makes some intuitive sense to me when I'm like, okay, 15 trillion tokens or whatever is already a lot. And yeah, there's a lot more on the internet, but a lot of it is crap. Data quality matters more than quantity has certainly been true in my experience.

Yeah.

Albeit at much smaller scale than, than what these guys are doing. We've heard these reports also that O1 is, in addition to, of course, introducing a new inference time scaling law, is being used to create training data for the next generation of model, the reported Orion model. And they're still raising more and more billions and they're still planning to build bigger and bigger data centers and The number of H100s that people are training on is going up. Elon's got his Tennessee Valley Authority contract with like record time to deploy and network 100,000 H100s. Meta has announced that they are training Llama 4 or will be, I'm not sure if they are or soon will be training Llama 4 on 100,000+ H100s. So how do you sort through that? In your mind, is the era of scaling Over, or are they throw— is that like throwing people off the scent or what?

Right. Is deep learning hitting a wall in air quotes? So yeah, we have these 2 camps. We have the people who are like, well, look at all the lack of amazing products that are coming out. It's all hype. You know, you've hit a wall. Doesn't get much better than this. Or does it? Gets better very slowly. And then everyone in the major AI companies is like, yep, AGI soon. End of world as we know it in '27 or thereabouts. And then who knows what happens? And then They have this strange forecast of we will get AGI that should be transformational, that can radically accelerate the pace of AI progress and do all these other things. And then people like Altman are like, well, then nothing much will change, right? Like your life will continue the same way and it will basically be better economic growth. So like you'll pay lower taxes or inflation will go down or something. But it, that story makes no sense, right? Like we might, we might, we might not get AGI, in which case we get The story of your life hasn't changed much, but things get a little bit better because AI is cool. But if we get AGI, obviously your life is not going to seem the same for very long. It might get way better. It might get way worse. It might end. We don't know. But the story on it stays the same. Certainly wouldn't be like an automatic default outcome. It would be a, we deliberately engineered that it stayed the same, right? We chose under the hood to keep it the same because we couldn't do better.

Yeah.

And there's some arguments for that being a reasonable choice if you don't confidently know how to do better, but it's not a natural outcome at all. And their rhetoric rings hollow, right? And it makes you wonder, do they really believe that what they're describing as the AGI that's coming is AGI? Or are they trying to hype you up slash get on the Microsoft contract slash just make themselves feel like they're doing the thing? Even if it's relatively weak, you know, the Metaculus definition is like not that strong for that kind of thing. So look a little bit farther ahead in all of this again. So my reconciliation is something like the strategy of just pushing forward by adding more zeros to everything. The same strategy that went from GPT-1 to 2 to 3 to 4, right? Like just like there's some technique involved, but you know, you know, Jack Clark's Impact Write-Up, it's like expertise is everywhere. It's just about compute and data. That's coming to a close. That's just marginal, marginal returns are declining even in scaling law terms. And you're not going to get very far from here doing that. And that's why O, right? That's why these other approaches, but this is my way of understanding it. By using what the models have to offer to refine the training process, by using inference time compute in various ways, by using things like the O series of models, you can get these labs People believe you can get there. You can overcome this barrier. We can pivot to a different scaling law, to a different method, and we can get where we need to go to AGI. That's, or at least some definition of AGI reasonably soon. Slash, they just have this faith that straight lines on graphs don't stop, right? That like, it's like Moore's Law. Like something will happen. We'll figure out something else, even though what we were doing before stops working. Don't worry about it. We'll figure it out. And there's a lot of talk about things being stronger than we know, right? Rune talking about how everyone's underupdating on 01, but we don't see the 01 he sees. We see 01 preview, right? And I gotta say, 01 preview, super unimpressive to me. Having sat with it for a while, I basically don't use it. I find it to be, every time I ask 01 preview for a question, it like thinks for a while. Talks about all of these irrelevant details, gives me lots and lots of data points, and like none of it addresses the actual thing I was trying to do. It doesn't follow the conceptual logic that I was trying to get it to go to, the thing that's useful to me. Like, why don't I just use Sonnet? I should just use Sonnet. Like almost every time.

That's quite interesting. I would say I have a different user report from '01. I certainly, the downsides are real. The wait time, my goldfish attention span is such now that even a 45-second delay to first token is enough for me to like tab over to something else and forget what I was doing and then come back like 8 minutes later. And of course the answer's been sitting there for 7 minutes at that point. So from a user experience standpoint, it is, and I actually do this outside of ChatGPT, so that may be a difference. As well, because I've found that what I want to use L1 for, or where I have at least found utility for it so far, is in mostly coding. I've been doing a fair amount of coding lately, partly because I have a little app idea I want to build, partly because I wanted to catch up with cursors. Obviously had a moment and I hadn't been coding for a little bit, so I was just trying to get back into it. And I found that RAG in general is a real pain in my butt. I don't really like any of the RAG solutions that I've ever used. Almost like without exception, that does extend to Claude projects and to GPTs. I feel like I can stuff a lot of stuff in there, but they're always a little too economical in terms of what they bring into context. And it's often opaque too. So I'm like, not sure what they're actually looking at from whatever I gave them access to. So that's another source of confusion. Claude on the main consumer-facing product does allow you to just paste in long documents. So I like that. But ChatGPT often doesn't allow me to just paste in the long document. It'll just straight up say, that's too long of an input. Sorry. So then I'll end up normally not using— the experience is better, I would say, with O1 in ChatGPT because it gives you these like reasoning summaries to entertain you while you're going. If you go over to the platform console type version, you can there paste in however much content you have, obviously up to that's add context. window limit.

Yeah.

Uh, but it doesn't give you this sort of real-time summary of reasoning. You just have to wait. Anyway, with all that said, I still do find it useful to take the full codebase of the little app that I'm developing and paste the whole thing in there. And I have been doing head-to-heads with the latest Sonnet, and which is amazing, of course, too, and the 01, and asking them the same question. I basically will typically say, Here is the full codebase of an app I'm developing. It's to help people create gold standard examples for use in AI automation. So I just give it like a one sentence summary of this is the purpose of this app. And then I'll ask for whatever I want help with. And if it is a local code change, Sodded is, I would say, hard to distinguish in many cases, maybe even better, a little bit more precise with the outputs. It has the function calling and stuff like that. So it does integrate into Cursor better. I don't use L1 through the cursor interface so much because it doesn't like reliably respect their format and in turn then the whole functionality will break. But there is a certain way in which it does feel more like a high-level collaborator to me. If I say, what would be the highest priorities for refactoring this application? Or I want to implement a new feature and I'm not sure how best to do it. I do see something qualitatively different there where Claude will give you an answer and it might even be the best answer often enough, but O1 does give me something extra where it'll sometimes say, here's 3 different ways you could do it. This way is the most direct but hacky way that you could do, but isn't really recommended. Here's another way that's like going to be a big refactor and maybe involve a couple new libraries. And this might be like the enterprise way, but it doesn't look like you're really coding that way. And here's kind of the Goldilocks way that is hopefully the best of both worlds and what I would recommend to you, where it'll be like a lighter lift, but not a terrible solution. And I do feel like it guides me toward better strategies in a way that I have not seen other things do.

So I, I'd finally gotten into actually using AI to try and code. I found Cursor to be a godsend versus not using Cursor. I can't imagine O1 being good enough that if it doesn't integrate into Cursor, that I would want to not use Cursor's integration. Like it just has to be so far ahead. If we jump from Claude Sonnet without Cursor to Claude Sonnet, the same exact model with Cursor, was a 10x. So it's like, well, I'm giving up that 10x because it seems so insane as a thing to do. And also I've gotten Claude Sonnet to do the thing you're describing. If you actually just ask it, what are the different ways you could do act? You just have to use semi-reasonable prompting. And as you can see, I cursor over, it'll spit out possibilities. I'll discuss it with it. We'll select the option. I also discovered that you really want to focus on using one model when you do this, because when there was an outage and it switched me over to some OpenAI model instead of Claude for a few hours, I almost couldn't work. Because every time I tried to ask it anything, it would just constantly give me all these different intuitions to try to change all my code. So even if the model hadn't been as good, as Claude, which I didn't think it was from what I could tell. Well, I don't want to have to debug all of the new code that's replacing all of the existing debug code. Like, that seems awful. And then I'll have Claude switch it back when I get back, which is just like, probably, I have to assume. So all of that sucks. So like, I'm just going to wait this out is what I decided. Because I have other things I can do for a few hours and I figure it'll come back. But yeah, like I haven't used L1 for coding a substantial amount exactly for this reason that like, it's, it's not affordable slash integrated enough to use it as your primary in Cursor and Nobody Ducks. But it's definitely possible that it has like specific uses where that makes a lot of sense. This is a specific place where thinking for a long time at this level is really, really helpful for exactly what you want to do. I think it comes down to what do I think of a 1 preview in general? And it's, yeah, it's like this thing where you have the not so bright student, but it's, he's spending a lot of time on every assignment and she's trying really hard. And she's really doing her best to turn in as much work as possible and try to get the right thing. But it's not that smart. It's not any smarter under the hood than the 4L basically. So it's inherently limited by it. So it just sort of, my use cases, it turns out that like VEX is not useful to me. And in fact, when I use ChatGPT, which I still do, I'm normally using it for the web search, which is just incompatible with O1. You have to choose one or the other.

Hey, we'll continue our interview in a moment after a word from our sponsors. It's definitely yet to be unhobbled, O1 Preview, in quite a few different ways.

Right. And everyone says that O1 itself is much better than O1 Preview, right? And you've seen various claims for that. And then O2 will be obviously presumably much better than O1. And they claim the scaling laws apply and this will get rapidly better. And it's possible that it will, but I have to express skepticism that, 'cause like the first product, the MVP, just didn't seem like something that I almost ever wanted to use. And also my model of what it's doing has an inherent restriction on it, right? So you're thinking really hard, but you're not that smart. So, okay, what if we thought even harder? What if we thought 10 times as much, 100 times as much, 1,000 times as much as that, but we're still not that smart? And my experience of humans in this spot, and my experience of what I got from R1 is it wouldn't help very much, right? Like you're going to have to find a way down in this box. And I'm not sure, I'm not sure you can, but obviously I've been surprised before. I could be very wrong. We couldn't have a very hollow version right now. Like we can't feed it context in the proper ways. We can't have it search the web. We can't have it do a variety of things. Who knows what happens a year from now?

The new Claude 3.5 Sonnet, to give it its due, is also definitely amazing. And it's a good point that it may be, my experience may be less about, you know, what constitutes a fair test or a fair comparison, a fair head-to-head is like, not always obvious. Giving them the same question when they have different default behavior may not be the right way to, to go about it. Maybe to do a fairer comparison, I should prompt them each differently, or maybe the same, but like in a way that sort of tries to get them to do a similar behavior.

I would say you should prompt them the way you would prompt each one of them, and if it happens to be the same, which it often actually is, Then the same prompt is fine. I would also add that the time delay just doesn't bother me very much. That's not the issue here. In fact, in Cursor, not only do I regularly run out of— I've run out of fast commands, though I'm not convinced I necessarily want to keep paying for more commands when the wait is normally very small anyway, but also that it takes so long to write out the queries with all the code, even with fast commands on it. Like, it's not that different from L1. I had gotten used to the idea of I'm asking for a substantial amount of code with my question. The moment I see it's actually answering my question, I'm tabbing out. I'm not only tabbing out, I'm screening out. I have a separate set of desktop screens for coding, and I have a desktop screen for writing, and I always just tab back to writing. Like entirely screen shift and just complete mode shift and then come back. Because like, I hear like a lot of programmers, like most programmers, they want to be in state. They want to focus and have all this stuff in their head. And they don't wanna be distracted. And I forgot that I'm the opposite, whereas if I have to like just constantly fight with these little details for too long in a row, I will wanna scream. And it's much better for me to do it in bite-sized chunks. I don't understand why my brain works that way, but I'm able to context shift that way. Like I don't lose state. Great. I get to just do this.

Yeah, I think I'm probably more like you than maybe the, certainly it's hard to imagine that if the models go out, I do stop coding. That's for sure. Because it's just so painful to sit and type code. I feel like—

Yeah. You can do this, you can still do this tomorrow, right? There's no why. It's, time is fungible. You have other things you can do. That's the way I think about it.

Have you tried anything else, making Magic: The Gathering decks or anything like that with O1 that would be a test of its like ability to grind through? Because I feel like they're showing some pretty, you know, The claim that there's a new scaling law and that this inference time compute gives you like better and better results, albeit with a logarithmic curve, but nevertheless, like in a not obviously bounded way, it would seem like some of these tasks that are like logical, creative, like combinatoric would be really well suited for that.

So I wonder if anything along the lines of a Magic Deck creation I, I find that example so bizarre just because, I mean, first of all, it's something that like the LLMs and form-seeking intelligence can't do, right? In any interesting way at all. And second of all, that it is one of these tasks that's very like core skill gated, as in if you take a bad player and you give them a month of actively even playing games, changing cards, thinking about stuff, but you don't give them the information of what's in other people's decks. You don't let them just copy, right? They won't get anywhere. They will max out at a very, very low level. It'll be unusably bad. Whereas if you give it to an expert deck builder, they can build something almost immediately with no feedback, with no iterations, that will be pretty good. And a lot of my best creations and the best creations of other people that I know of emerge like on the first day. For the first time, they have an idea very close to the final result. It's all kind of just very heuristic-based theory crafting, and it's not inference. It's not inference times a few tests, right? Like inherently for humans. And I would very much not expect L1 to be capable of anything of that type. You can't logically branch out these things. They get too complex too fast. There's too much randomness. There's too many possibilities. There's too much unknown information. You're very much doing the pre-training style thing, not the inference training style thing, right? So to the extent that they can play, it would be Opus-style things that would be good at it, I would think. But also we, we've seen that they, they can't, basically can't play. Someone used Claude's computer usage to actually hook it up to Magic Arena to see if it can play some Magic. And the answer is like, it can try and play, but very not well. And deck building is so much harder than playing. Most of the time in Magic playing, it's reasonable to just do reasonable things. And then obviously over time, if you're not strategic and you're not sophisticated, you will run into problems, especially with people who notice the ways in which you're not being strategic and you're not being reasonable. But yeah, it couldn't do standard things that well. I'm sure you could vastly improve it if you cared enough on that front. But the task that I'm always looking for is go into a new set with nothing but a spoiler list that's not in your training data. And then let's see you draft. Let's see you decide what's good, assemble a strategy and play it.

Spoiler list is like, these are the cards that are in this set.

Yeah. Like here, here's the 300 card, here's the 200 cards or 300 cards that we're going to be drafting from. Here's how rare each one is. Here's what each one does. Right. And a human can do that and then have a reasonable sense of what things are good and bad and what to go after. Slash, I do this thing where I did this actually last week. I played my first draft yesterday where I would just, I don't know what's in the set. I'm just looking at the cards and reading them as I go. And I'm learning as I go based on what's there. What is the card trying to tell me? What are the strategies I'm supposed to be using? What could be good? What's inherently just naturally powerful? And you just work your way through it. Absolutely can't be done.

There might be like an interesting cognitive style difference here. I feel like I played a little Magic, uh, in my youth. It's been a long time and I was never great at it or particularly serious about it, but I feel like it is more for me, like coding, like there is some sort of like intuit— in the app development process, there's some sort of intuition moment or some sort of often like experiential moment where I'm like, oh, I want to do this right now in this app and the function doesn't exist. Okay. I need to create it. And then that's like, To quote Edison, that's typically less than 1% of the effort is that moment of inspiration. And then comes obviously a lot more perspiration of actually, if I'm doing it by hand, which I don't do anymore, figuring out what files and how to change them and making an implementation plan and typing it all out. And that definitely is like a grind and I don't have any shortcuts from that moment of inspiration to a solution that don't involve like just gradually considering all the different things that this touches and all the files that it exists in and so on. For a magic deck, I, I feel like I also recall having some moments of, oh, this and this could be cool, but then really building out the deck, aren't you kind of working your way through like, what if I added this one to it? Yeah, that could be good, but not so good. What if I added this one to it? You're not going through that sort of There's no brute force for it. You're just like, your deck's like spring forth from the head of Z fully formed.

You have a very good sense over time of what successful decks have to look like. And you've already done this kind of research where you look at what's possible, what's in the space, what types of possibilities there are. And by building these other decks and pursuing these other strategies and knowing what's out there, you have a sense of like, what does the deck have to do? What do decks typically need to do? So you say, okay, I'm gonna play these, this combination of 3 cards because this seems like a powerful thing. I know what colors that has to be. I know about how many lands the deck's gonna have. I know about what the mana distribution's gonna have to be. I know about how many creatures this deck can afford to play. All these numbers fall into place. You start doing reasonable things. You try several configurations. There's always so many good cards that sort of do vaguely the thing that you want. Sometimes you're like, I want something that, I want a 5-drop that like it flies and it gets in this way, or, you know, an enchantment that does this thing or whatever. And you go searching, maybe you go searching for that. But it's a pretty fast search because you either remember or you know what type of card you're gonna strike all and you have an idea. Yes, occasionally you will do the literal thing of, especially when you're starting out, it's a new, new set, new format where you just look through like all the cards and you just pan through them and you're just like, okay, did any of these seem like they might work in this strategy? And just try them all out. You're auto-rejecting like almost all of them as just, this isn't very powerful. I didn't know what I was doing. And then like what we're describing. Is something I would occasionally do, especially in formats that requires you to dig deep for where you're looking for a solution to a particular problem. So we used to call it a crack search where you just look through the entire spoiler list and just go part by part. Does this solve my problem? Does this solve my problem? Does this solve my problem? And normally the answer is no, obviously not. What am I even talking about? And like with Approach to R1, okay, I have this problem of these creatureless X full removal. What am I gonna do about it? I'm playing white blue. I don't know what my solution is. And then I'm just going through this warlist card by card, every single card. And I see Pure Reflection, which is like a terrible card, but it happens to solve this exact problem. There's nobody who's ever put it in. And I'm like, oh, I got it. I got the solution. I don't know where I went, but I'm like, I think I might have it because in this particular narrow situation, this will solve this specific problem. And then we try it and it works. And my sideboard went to be the quarterfinal match. How it's gonna win the final. But that's not normally how it goes. Like, you are like, you're, I mean, you're searching for patterns, you're like trying stuff out. Often this involves like a short period of just playing a handful of games. And like, one thing you'll notice is a very good player who has a lot of experience, they can play like 2 games and they say like, I know what this matchup's about. I know what was going wrong with this deck. This isn't gonna work. I need to retool.

Yeah.

Because they can tell how they feel, how they're looking at different lines, how they're looking at different possibilities. They can think about counterfactuals and they effectively get so much more out of that input. And they can just think 10, 100 times as fast. Whereas a bad player needs 10, 20, 30 games to think out of the hammy of their head, what's going on here? Try to figure out the pattern.

Hey, we'll continue our interview in a moment after a word from our sponsors. Are you aware of any projects to try to get an AI system to play Magic at a high level? Obviously we've seen poker beaten, effectively beaten, and diplomacy.

Yeah. No, not any serious efforts. Magic is a vastly harder game for the AI. I think Magic is very close to AI complete. If you show me your system can do the test where we are going to draft a format of new cards you have never seen before, that are not just Slightly different numbers from cards you have seen before, maybe even that. And you're gonna be able to play on the level of strong humans in the resulting games with you that you created. Yeah. Like you could try to hack it together, but if you can do that without just hardcoding a lot of weird stuff and doing something really hacky, but I could do it trying to— yeah, the sky's the limit for whatever you just did. Like it's just so general what you're doing. I can invent new mechanics and just work in different ways and it just figures out what they mean in real time. Interesting.

Yeah, interesting. So I guess maybe final point on O1, my sort of headline summary for basically since GPT-4 has been in terms of like where AI is today, what it can do, what it can't do, that the best AIs are closing in On human expert performance on routine tasks. Obviously all those words fairly carefully chosen, carrying weight, especially routine and task. 01 is the first one that shows a bar that is higher than the human expert level on MMLU, for example, where they put that pretty much at like 90% and everything before 01 was getting close, but in the 88, 89, 1 or 2 points behind the human expert. And then O1, they get like 92. And so on the one hand, it's like a couple points better. On the other hand, maybe it's crossed some threshold. I mean, those tests are fairly noisy. Do you think that has much meaning or would you infer anything from that when it comes to whether there's like a top out at human expert level?

It sounds a lot like the thing where like MeatSolvers forecast kept jumping between 49% and 51% and everybody suddenly, whoa, wake up. There's this level that's like considered human level, but it's arbitrary, right? Because which human, right? A natural human who hasn't practiced this is going to score a lot worse than 90%. The best human—

Yeah, that's down to 2, I think.

The best human is going to score more than 90%, I assume. This is not like human, best possible human. So there's not like 90% is, is an arbitrarily chosen, let's see how we're basically doing, but there's no reason to be a wall at 90, particularly like not a hard wall. And If everything is asymptotic for a while and then something jumps the asymptote, then it's interesting whether or not that was a human-level asymptote. But as far as the MLU, yeah, it's also the question, are they missing the same questions over and over? Right? Like, is it, is the MLU just, okay, 90% of the questions are doable and then 10% of the questions either have the wrong answer in the code base, in the, in the answer key, or are ambiguous or are just incredibly harder and they just can't be done. And therefore you hit this wall.

Yeah.

On an MOU, which may or may not represent a real world. Right? Like, I don't know. I haven't looked at these things, but different evals have different quirks to them. But in general, once you hit 90% on an eval, I have to assume it's not really an eval for you anymore. You should move on to the next eval.

Humanity's Last Exam coming soon. Contribute your extremely difficult questions to Dan Hendrickson Squad.

I like the name Humanity's Last Exam. They understand that if the AI passes this exam, then There's no longer much time for us to be answering your questions.

Maybe you could formulate a magic question for humanity's last exam. You'd be the perfect person to do it.

I mean, we could, but you know, I, we got this handled.

I do find it sobering, honestly, how, you know, and I flatter myself as a reasonably smart person. I'm like, what can, what questions can I contribute to humanity's last exam? Do I know anything that's actually hard enough to merit inclusion in that test set? I don't know. Uh, I think you would actually have a good one with Magic if you wanted to do it. I'm looking around my life and I'm like, not sure I'm doing anything that's actually hard enough to get into humanity's last exam.

I think a lot of what I do is the kind of thing that doesn't go on this exam in the sense that it's not that I am doing this very hard specific task. That's just not like where the difficulty in my job comes from.

It's just the volume of information and the connection drawing across—

Connection drawing, knowing which questions to ask and emphasize, being able to iterate off of like how to explain things and how to draw things together. Just a lot of like a way of building a map and understanding things and being able to hold a lot of this together and think carefully about all these different questions and how they relate to each other. But it's not like there's one specific hard question. Where you have to go really deep. I think I did more of that earlier in my career, especially when I was trading or playing Magic, when you would really focus very narrowly on getting the exact right answer to a very valuable question where you were having effectively a competition to find the best possible answer, right? You were up against someone else where you were trying to be the best. And now I'm doing something that no one else is doing. To a large extent, I'm just very unique. And there it's like very much not about getting that last 1% of efficiency or anything, right? It's about the big conceptual questions and it's about being able to efficiently like iterate and change things towards where you want to go. And a lot of the decisions I make are very much not designed to be precise or maximally efficient. They're designed to like take this in places that are interesting and explore things that I enjoy that like are compatible with providing what I need to provide and like slowly working my way towards a thing, but also being like deliberately not optimized because if something is being deliberately optimized towards a goal, like it's inherently trustworthy, right? Not just that it's perceived as untrustworthy, but it actually is untrustworthy, right? Because if you're maximizing for any one goal, then you stop actually caring about other things and they just fall away and everything else people rely upon goes away. And that's an AI alignment problem too, but it's also a very practical problem, right? If you tell a particularly ruthless person to make sure the shipment gets in by Monday afternoon, I hope you didn't care about how they did it, right? They're going to find a way. And if you ask them exactly how they did it, you might not like it, but you said just get it here. So they're going to do it. Okay.

So let's pause for a moment. You mentioned Claude computer use on a Magic application. That also is a big development. Your first time a foundation model developer has come out with something that is basically able to take like open-ended action in the In the broad digital world, at least, it's kind of like O1 is like very hobbled. I try not to lie to my AI assistants more than I have to. I occasionally have to say that I already have a doctor appointment scheduled and this is just preparation for that, or similar things like that when it doesn't want to take on liability. In this case, similarly, it was like, you know, I can't create an account. I was like, okay, I logged into my account. You can use my account. I can't use your account if you're logged in. Finally, I was like, okay, I logged out. Now you're using the free no-account version. And if it had checked, it would have found that I had lied to it about that. And it in fact was still logged into my account, but it didn't check. It was gullible enough to believe my little white lie. And in my case, it was able to use Waymark, our video product, and pretty effectively, not super fast, you know, wouldn't have been able to play real-time video games. It was probably better because our product is an AI product. So people are prompting the product for what kind of video they want, typically for their small business. It was actually better at prompting, um, clearly than like our typical user. A little less facile with various parts of the UI.

Of course. You'd expect that.

But all those things seem like they'll be destined to be resolved. How big of a deal do you Do you think this will be for data purposes? Is this sort of go out and do stuff in the world a good way to get over the data wall? And what do you think of their decision to put this out and their rationale, which is very OpenAI-like, of we'll put it out while it's weak so we can start to adjust. Better that than dropping it on you when it's strong. I feel like we've heard that before and who's imitating who at this point is not entirely clear.

Yeah. Yeah, I had to think for a while about what I thought about the decision to release that feature. Ultimately, I think my philosophy is something like iterative deployment is better than non-iterative, but it's still deployment. Sometimes deployment is bad, but if you're already determined that you're gonna deploy version 2.0, then deploying version 1.0 before that will probably be good. The exception is if you don't want to raise the alarm slash dinner bell for everybody and tell them how exciting this great new product is and how they should all be flying in to invest in competitors and build complements and supercharge the entire ecosystem. Like when they released ChatGPT, this led to a wave of investment in artificial intelligence. The way they're going to fight an artificial intelligence. This very much was accelerationism in the situation. And if you think that accelerating that development is bad, then you should be sad about that. However, if you think that releasing this early version of computer use is not going to do that, and that computer use was coming anyway within a year from someone else regardless, yeah, and if that does work the kinks out. In a relatively safe environment. You know, this lets Anthropic build its relative position. This lets us diagnose various issues. This lets policymakers understand. It's already a very good illustration for policymakers of the future to come. I think it is very, very good at being an aid to explaining to, say, someone in the Trump administration, if you were to, say, get a conversation, oh, they can just control your computer continuously as an agent if you just type 3 lines into this cloud product. And it's not good enough right now that anybody would dare use it, but that's just a matter of a few months, right? And it's very different from a year from now people are going to be doing this, but nobody's doing it yet. But now you have a demo. You can see someone order their DoorDash. You can see someone operate Magic Arena. You can see somebody do these various low-key tasks. And if you can do that, what else can you do? So I think it's the right decision in this context, given that this was obviously pretty close to happening anyway. And you just got to compete. Like, the iterative deployment— Rune had the statement, I think it was a week or two ago, oh, total open AI cultural victory, everybody's doing iterative deployment. And my response to that was, well, yeah, of course, because you're doing it. Because you don't have a choice, right? It's like, well, total American cultural victory, everybody has nuclear weapons. Yeah, because you have them, you asshole. Like, what are they going to do, not build them? It's the cost of iterative deployment. Is that everybody is aware of a situation and aware of what can be done. The benefit is also the same thing, but the cost is the supercharging of every, all of the events, the pushing forward. Once you've paid that cost, not releasing iteratively doesn't buy you anything, right? Letting someone else iteratively deploy and you not iteratively deploying doesn't really buy you anything either. And by you focus, you know, SSI, right? Like Ilya is like, I don't have to worry about a commercial product. Let me focus. But if Ilya were to release foundation models along the way, that wouldn't, I think, be that likely to accelerate other AI progress unless he was substantially beyond the frontier yet. Right. Getting, we got Grok. So what? It didn't do anything. And getting another open source model that's similar to other open source models, getting Another commercial foundation model that's close to other commercial foundation models. These things don't change the pace of other developments much at this point. That ship has already sailed. Now, releasing something that's a substantial leap above the competition, the way that GPT-4 was a substantial leap above the competition when it was released, that could be different, right? But like the iterative stuff, like, right? I mean, Sonnet 3.5 is like an interesting possible exception just because it's helping people code faster.

I look back on the whole GPT-4 thing and I still am somewhat confused. I mean, at the time, the analysis as I understood it and the reason for such intensive secrecy around it was, you know, probably somewhat competitive, but also somewhat, we don't want to alert the world and have people piling in. And it's better that a few, one or a few responsible actors have the lead. And now I feel like the live players list hasn't— yeah, I would put XAI in there still, just because if you can wire up 100,000 H100s in record time, I think you gotta be on the list. And having like functionally zero cost of capital as an Elon business always seems to have is like definitely a notable advantage in this game. But we haven't really seen any new live players added aside from them.

Yeah.

To the watchlist since GPT-4, we've seen a bunch of VC money lit on fire with sort of second-tier foundation model developers, which are now largely getting like creatively acquired back into big tech, whether that's Adept or Character or Pi, whatever the company's name officially was with Pi.

Function, right? Yeah. Yes. Thank you.

And those were all like, I don't know about Adept as much, but both Pi had a great model. And I think broadly speaking, a very competitive product. Character has maybe the most adoption for better and definitely at times for worse of any product. Like those seem to be on the path to being viable businesses. If you didn't think that you just needed unbelievable amounts of capital or like access to some data reservoir that you just can't go get on your own. So I do wonder to what degree the whole thing was baked in even then, and you were always gonna have 5 to 7 mega company or alliances or whatever live players, and, and everybody else was just never gonna be able to get into the game. Do you see that differently or do you see any path for anybody new to get into the game at this point outside of like—

5 to 7 is so different from 1 or 2. Right? Like, so 0 is very different from 1, obviously. 1 is very different from 2. 2 is very different from 3. And then 3 and 4 are not that different, but they still are substantially different. And then 5, 6, and 7 are very similar numbers, right? From the perspective of the situation. Useful way to think about it. And I would say that like OpenAI was trying to retain their lead, retain their position for as long as possible, among other motivations. I think they were legitimately concerned with the safety of the product. Which is to their credit. There's a lot of benefits to releasing early from their perspective, and they didn't do that. But to the extent that they had a commercial reason to try and stay in the lead, like they're really trying to make the number 1. They're really trying to get a large lead or like at least 2 or at most 2 or 3 so that agreements can be reasonably reached so that dynamics of I'm going to do this. So you do that and then I can do this in response to that. Then you can do that in response to this and so on. take hold. Whereas with a group of 5 to 7, like all the game theory says, once you get to 7, right? Like try to get cooperation in StackHunts with 7 players in the long run is a very, very hopeless situation. Most of the time it can be done, but it's very difficult, right? Whereas getting StackHunts to work with 2 is remarkably easy, right? So like the number of players matters quite a lot. And I do think it's basically the structure of the necessary capital requirements and data requirements and the way that capitalism works right now, that there isn't room for that many players. So we're not going to get 20, 30, 50 players that are making like meaningfully advancing frontier models. People will be either trading models for the cheap or using someone else's model or easily adapting some open model or otherwise like Not sweating it too much because again, like why are you duplicating all this bunch of work? It's a waste of time. You're not going to get a lead. You should work with what is available to you for free or damn cheap because the marginal cost of providing it to you is zero or damn near zero. Of course it makes sense. But we're looking at a future where there are going to be a handful of companies that are out front of the rest and assuming that spending more money and building these bigger data centers and running these bigger things is In fact, a way to, to stay in advantage, which we can't assume, but is definitely the way that they're betting and seems likely. Then yeah, we're, we're gonna be probably, it'd be 3 to 5. Yeah. 1, or maybe even 1 to 5, 5 players who are like, we've seen scenario planning, like Daniel Cogitaggio's like scenario planning, right? When he put out like what 2026 looks like on LessWrong and, and stuff like that. Where in his models, like OpenAI has this 9-month lead. Right. And as 9 months becomes a longer and longer period, as things speed up, right, every other land on the planet becomes irrelevant. DeepMind just folds in those scenarios at some point because there's no point in trying to be 9 months behind. Because so what if you catch up and you're 6 months behind a year from now? That is now 3 cycles. And then a year from then, it's going to be infinite cycles, right? In that kind of scenario. So like, what's the point? So if you can get enough of a lead at the right time when things start to go vertical in terms of pace, then you have a decisive advantage over everybody else. And right now, yeah, I think that what we're realizing is there really isn't room for more than a handful of companies to be worth competing in this space. And the only reason Meta is able to compete in this space is because of Meta's business interests. In Facebook and Instagram, creating weird incentives for them to then do other things and also giving them this giant pool of money they can just set on fire whenever they feel like it. Mark Zuckerberg can't. Yeah.

Having tens of billions of cash on hand with no better way to spend it is an easy play as it turns out.

But also in this situation where like even a slightly better way to serve people, like addictive feeds or better advertising, just makes them billions and billions of dollars, right? So like they need very little effective improvement from doing the work themselves. Or, or like very even buying insurance against their work being cut off by other people to make it worth for them pursuing this very aggressively. So they have a lot of odds, even if other people in this situation wouldn't. And they have these unique datasets because of this association. So they're in a unique position to be competitive. And then Elon, of course, is Elon and just does whatever he wants to do. So he gets to redirect all this capital. He gets to use his leverage to recruit a bunch of people and redirect a bunch of stuff. And compete, but it's not obvious to me that xAI is relevant. Like they might be relevant, we'll find out. But so far they haven't actually done anything that changed anything. But yeah, my guess is it's still probably largely coming down to the big 3 plus 1, right? And then we'll see if they can all stay relevant. Increasingly DeepMind and Google don't seem that relevant in an important sense. Like they're not that far behind. But why do we care that they're there? Almost like it's starting to feel that way. Yeah, it's funny.

I just had a couple of Googlers on the show when they did a, an update where they added Google Search grounding as a feature to the Gemini API. And I asked them this question basically saying like, why is Gemini number 3? Because I don't, I mean, you might feel like the answer is obvious. I don't feel like the answer is super obvious when I look at Benchmark performance, when I look at cost, when I look at just like APIs and rate limits and just all the things, it's pretty competitive. And yet it was the 3rd provider that I integrated into the app that I'm building. And it's definitely the 3rd window that I open. Typically Claude is first, then ChatGPT, and then Gemini. If I want to do a contract review, I'll do all 3 for completeness. But it is always the third. Is that a branding problem? What, why do you think it is the third? Because I, I can't quite put my finger on it.

So I am using Gemini for my Chrome extension. It's the thing that I'm building with the codebase. Part of that is because Claude just kept being annoying and giving me a 401, and Claude was in fact the model I was using to fix the 401, and it couldn't fix the problem. And I was just like, well, if you can't fix your own problem, well, Gemini's a lot cheaper and it's probably fine. I'll just use Gemini. But And Gemini has been working okay, but it's been annoying. Like it won't follow certain instructions. It doesn't, it has issues. But my lived experience is that it's worse and it doesn't have very useful places where it's better. So when I type things into the Gemini Advanced window, I don't get much out of it. And then I just stopped doing it. I'm just not interested in this anymore. And there was a period where their Long Context window was unique, but now it's not unique anymore. There was a period where, yeah, they jumped briefly ahead of other people in various ways, and I was like excited to use Gemini, but that just hasn't happened for a while. And I think part of it's a marketing problem. I think Gem— Google has been insanely bad at marketing Gemini and explaining Gemini. That's also the implementation sucks. I bought a—

Yeah.

$1,700 Pixel Fold. Partly because even a marginal improvement in efficiency is worth a lot of money. So why not try out the best? Partly because I was told there'd be these cool Gemini AI features. I haven't used those features. Some of them I haven't even had to work.

You're not circling to search on a regular basis?

It has never come up. I've never wanted to circle to search anything. Not once. Never been tempted. It's possible that it just didn't occur to me and I had a great opportunity to circle to search, but like, I wanted my calls to have automated login. They don't. I wanted to be able to like talk to the AI and have it usefully use the contacts on my phone. It didn't work. Like the few, every time I tried to get the AI to do something useful for me, I just gave up. And like, I'm debating getting an iPhone. Like there's a Verizon special where you can supposedly get 2 turn in any phone in any condition to get a free iPhone on them. And I haven't used my contract in a while. It's like, why don't I get an iPhone too?

I've got a drawer full of old phones here I could potentially cash in.

Yeah, I've got, I mean, I think you only do it once, but I've got at least one non-working phone that I can cash in. Or I can buy one, I'm sure, $10. Like, it's not hard, but it's just, they haven't done anything. They haven't made it useful. They keep telling me Gemini now available on this app that I'm using. I use lots of Google products and they're great. I have no, I love Google, but like, I never use it. Like the few times I tried to use it, like I remember they put Gemini on Google Docs and I just went to the end of the document and I was just like, Gemini, have you read, please summarize this document. Didn't do it. Can't help me. Like, how is that not supposed to do?

Yeah, the integrations have been disappointing for sure, to the point where I did a project to go back and using APIs that they have. It's funny cuz they have such incredible APIs and then they, they don't seem to use them in their own product development. And that's quite puzzling.

My understanding is that Google is this like amalgamation of like different departments that basically don't talk to each other and basically don't copy each other's work. They have to reduplicate, they have to reintegrate them, reimplement everything, and they don't cooperate very well. And like they're doing various political infighting all the time. And yeah, as a result, like you can readily get good things in places, but often you just don't. And it's a serious problem. So I don't know what's happening with that. Like NotebookLM, for a while I was trying to use it. I just fed it all my AI articles and tried to ask it questions. It's just not good enough, right? It's not, it's failing on its face too often. I don't feel emotionally like I want to use this or I want to try using this anymore. So I stopped. Like the one cool feature they implemented that no one else has right now is like the podcasts, right? The AI podcasts. And some people like them. I find them reasonably terrible. I try to listen and after a minute or two, I'm just like, I can't take this anymore.

I like them actually, although I do think there is a step up there relative to any other automatically generated audio I've heard.

No, no, it's amazing technical achievement. And I applaud them for doing it. And it's a cool and ask demo for sure. I am very, you know, proud of them for doing it. But in a practical matter, God, please shoot me.

I, what I want, and I'm sure they will have it before too long, or at least I expect they will. I'll certainly be asking questions about this, is a much more literal version. Like I was trying it with biology papers and talk about a world of just infinite complexity and whatever that I just starting to acclimate myself to. But I don't want the very like poppy explanation of, oh, this, the way that this antibody attaches to this thing is it's like having the perfect key for the perfect lock that only fits that one lock. And I'm like, that's not how I want to have things explained to me. I want to be much more literal and much more grounded, and I haven't been able to prompt my way out of. The like very superficial analogy-driven approach that it kind of defaults to. But I do think if they, I do like the 2 voices. I do like how they kind of keep it engaging that way. And I definitely prefer it. I'm very much an audio person relative to reading. I know we're not the same on that.

I really like the 2 voices. I both like both of the voices. I think they're very well done. I like the way they dynamically play against each other. But the information density and the amount of banality just drives me insane. And just in general, like there's no flexibility in the podcast. Like the great advantage of AI is that it tunes. So like one of the things I've gotten to experiment with recently that I've had the time for is, you know, some forms of AI storytelling, including like various NSFW versions of it. I've just been toying with them. And like, it is amazing how much like something that's relatively bad, objectively terrible, like objectively just god awful terrible, is so much better when you have this ability to steer. You have this ability to influence what's happening selectively when you want to, while simultaneously mostly letting it write its own story or giving it a little bit of direction. And that's turning something that would be unreadable if you weren't interacting with it, completely awful, unreadable, complete drag into something that's exciting, right? Because you feel like you have agency and you can emphasize the things that you find cool and exciting and de-emphasize things you find bad. And a podcast is the opposite of that. I want the podcast to be this thing where I'm constantly like giving you direction. Right? As you do it, right? Not like it pre-generates one podcast and it just presents the information in this incredibly undense format. What I love about AI is in learning things is give me the answer. Okay, now I'll drill down into the places that I'm actually curious or don't understand, or I didn't— your explanation wasn't what I needed, and I can correct you and I can steer you. And this is just not that at all.

Yeah, it's interesting as a, as somebody who has spent time developing AI-powered products. I think that balance is a tricky one to strike, especially if you're trying to serve a broad base of users that are not AI obsessives and don't have great intuitions at this point for how to steer, how to prompt, whatever. I think there is something definitely to be said for the, like, end-to-end moonshot. Like we've definitely seen that at Waymark relative to other AI video products where there's many products out there where you can generate an image to put into this scene or write me a bit of copy to drop in. But very few take the, what do you want? We're going to make the entire thing for you, present it to you, and then let you kind of go from there. And they are currently missing the go from there piece. But I do think too many products and especially too many Google products have taken the other side where they're like, keep it super, super modular, super bite-sized, user fully in control. And that also leaves quite a bit of magic on the table. So I do like how this is a different approach. I do think they'll circle back. They've, I've heard them, they were on the Latent Faith podcast not long ago, which included a little discussion of adding real-time audio as one, one item on their roadmap. And that would be a qualitatively different experience, obviously. So.

Yeah. You know, the world in which like you can be a third person on that podcast and they'll adjust to what you say. And suddenly that product is so much better. The world in which like it can just go far faster and be more dense and like not feel like they have to set all this background and talk so, be so slow. Like, yeah, you can, you can go 1.5x or 2x or whatever, but it doesn't like fundamentally change the problem.

What has your experience been with real-time voice or advanced voice mode, I guess is the proper term from OpenAI, you know, we were racing to her however many months ago.

Now her is here and I guess I'm not that into her. It's like my actual response, right? Like she's nice. Yeah. I appreciate her for trying, but yeah, I'm just not that into her yet. Right. Like she needs some more seasoning. She needs to like to get better. Part of it is you can't prompt properly. You can't get it to do the thing you want. So now it has to be like good enough that it can intuit from you just saying random words responses that are good enough. And my experience is just, no, I meant this other thing. No, no, no, no, it's not what I'm asking you to do. It's not the information I want. Like it's, there are occasionally magical moments. Where you feel like you're having a conversation, right? And it just like flows really cool. But for the most part, that just wasn't my experience at all. It was like very, very bad at doing what I actually cared about or wanted it to do. And I don't really blame it because again, like my prompting was terrible, right? It was just me speaking words in no particular order that had to be transcribed. Without any kind of plot, like people who want to be talking instead of typing to a computer in general, I'm just so confused, right? Like I can't talk as fast as I type. I can't talk as precise as I type. I can't talk as deliberately as I type and I can't correct it. What am I doing? I mean, sure, if I'm walking around and there's a certain modality, there's a certain like different aesthetic to it, it can be an advantage. But yeah, I just. Stop talking to your phone. Like, just stop talking to your phone for anything but the most very specific things, right? I have a, I have an Alexa, right? And it's very good for a very small number of very specific things where I know exactly what to say to get exactly that result. But say you hooked up ChatGPT to Alexa, which is like something that's obviously coming in some form, whether or not it's exactly ChatGPT. And now if you give it the specific command, it'll do the specific thing. But if you give it any other command, it will intelligently interpret your statement. Well, my assumption is this will basically greatly expand the set of things that will work. I'll be able to find other things that do the things I want it to do, but it still won't, I still won't really want to do open-ended requests that are complex, that are interesting until the technology gets a lot better. Right. I think the MTG5 can handle that, but too easy.

I'll take the other side of that a little bit in as much as I do have a lot of excitement for a future where I can learn while taking walks. Like I'm just plain too sedentary. And a big part of that has been that I'm unable to learn effectively on the go. I do think the voice mode will work for that. The, the biggest, and then that was so disappointing was I couldn't paste in the contents of a paper that I wanted to learn about. For whatever reason, you can't attach a PDF to it or even paste in a decent amount of content. And I was like, why, why can't I? I just wanna talk to you about this. I know you can do that. But for whatever reason, that is not supported. That feels like the kind of thing that, you know, is an obvious candidate for unhobbling. And I still have a lot of hope for it.

I'm not saying that you can't get there. I don't think you couldn't get there with, like, often these things are just like, you need a certain specific set of ad hoc wins or features that just are too necessary, are too vital to the things you want to do to figure out enough of them so you can have something. So I can say I'm listening to a podcast on some app and I have Gemini always listening to what's happening. And then I can interrupt the podcast to ask Gemini about what I've been listening to, right? And it will respond in an intelligent way. Like, wait, what did that mean? What is he referring to there? You know, like basic questions. That sounds really helpful. Right. And then I can interrupt, like, well, what if we didn't do that? But it has to flow, right? It has to like just work. If it doesn't just work, I'm not going to do it. I'd much rather just wait till I get home, get in front of the computer, paste the transcript into, you know, if I have to, into Claude. It's just my thing. But like, you're going to do the same sort of stripped down, simple, true Herb version. Like, it just has to be good enough. And I just don't sense the request.

Yeah, I would get a lot if they just allowed me to paste in a paper or 2, but yeah, I'm not sure why that hasn't happened yet. Let's zoom out again a little bit from the products from these live players. And I wanted to get your take on the increasingly tangled web of alliances between these companies. If you had told me even a year ago that OpenAI would manage to add an Apple partnership to their existing Microsoft partnership, that would have blown my mind. If you told me that, I wouldn't have been so surprised that Microsoft would turn around and have multiple providers in GitHub Copilot, which they now do. That one of them is Google is a pretty remarkable fact, I would say, unto itself. You know, in general, there's this like interdependence coupling, if you will, going on between big tech providers. Last time when we talked a little bit about like, are the models converging or are they diverging? And is that good or bad? My general sense is that this coupling is probably good. It seems that it makes the overall competition A little less intense and that they're all frenemies in some way is like better than if they're all siloed standalone platforms that don't talk to each other from just the standpoint, I mean, probably just straight up consumer surplus now, but also a, like, when you talk about like how many companies, how many players can you have in a game theoretic sense, like work together nicely, this would seem to support that.

Yeah.

support that, or at least be better than many alternatives if you want to see—

I haven't really thought about it too much, but my natural instinct is something like perfect competition is the worst case scenario in terms of giving people flexibility and slack to do the right thing when the right thing is not as competitive and to have, give them choices. So to the extent that what's going on is that everybody's willing to work on And every deployer, right, like Apple, is wanting to work with every supplier, every developer. Then the developers competing against each other in a way that like everyone who's using software can swap Gemini out and Claude in or vice versa in 5 minutes, maybe in 30 seconds, whenever they feel like it, or just click a button. They might both want Claude. That places tremendous pressure. Whoever's marginally better for each given task gets the business. Whereas if Amazon is committed to Anthropic in some alternate world and Apple is committed to OpenAI and Microsoft's committed to OpenAI, but Google is committed to Google and blah, blah, blah, then, you know, just being slightly better doesn't matter very much. Like Apple Intelligence, right? Like it's going to be succeeding or failing based on Like the UI and what features they decide to implement and whether or not it basically works and just works in the way that Apple products try to just work. It's not going to succeed or fail based on whether or not it's 10% better in its core AI function, right? Like it just has to be good enough. And whether or not it's slightly ahead or behind what Google is doing with its Google Assistant in terms of the actual model, it just doesn't matter very much. But if they're competing, all the models are competing to be the Google Assistant. All the models are competing to be the Apple Intelligence. Now the tiny difference is everything. And now you just have to go full speed ahead and try to maintain this narrow lead or this narrow functionality and ship everything as fast as possible.

I feel like we flipped because last time I was saying that about just the model developers and you were, you'll have different friends for different things. But now you're, if I understand you correctly, you're taking the other side where you're saying this. The ready availability of an alternate model in any given product, you think effectively makes the competition more intense at the model layer?

Well, what we've seen a lot of the times is basically this idea of a new model will come in and people will say like, well, we're swapping in that new model for this first, this new model. And like a natural result is that a lot of people would tune their systems to use a specific model, and they know exactly how to make that model do what they want it to do. And they're used to what that model can do, and they point at that model. The advantage is that model has some features and ways that it's strong, other models have ways that it's strong. The more similar they are, the more you keep swapping them back and forth, right? The more they're close to each other in that way. The more distinct they are, the more you have lock-in. And the lock-in is probably good, right? I, I, I do want these silos to a large extent. Because yeah, I think that actually puts a lot less pressure on the labs to push forward in important ways. They oversplit some other business immediately. There are big switching costs in that world. Uh, but—

So you, you're also then happy to see the apparent divergence in model capabilities. Like OpenAI has reasoning and Claude has computer use and is the best at coding and Gemini has long context and Google grounding.

And you wanna see more Cambrian divergence and less, uh, yeah, I mean, different features is, different features is great, right? 'Cause if I, what do we mean by, I'm actually just learning about this 'cause I just happened to talk about the, the, the Google integration of Gemini, right? Like to what extent is it integrated? What does that mean? Right? How is it different from this web search that can be done on ChatGPT? But yeah, like the fact that like when I want to search the web for a piece of information right now, I will use Google for the certain type of information that I expect just Google to pop out. I'll search ChatGPT if I expect WebGPT search to search it out better, right? I might use Perplexity for a third type of thing where I want like a different set of sources kind of explanation. If I want like coding or other types of natural chats or analysis of a paper or something, I'm going to use Claude. If I want like a very specific set of things, I might use L1. And so on. I think that's good. Right now on the margins, they're trying to carve out their own space and just pushing the model's core capabilities is less important because I don't care which model is 10% better under the hood. I care about which one is more adaptive to the use that I'm doing.

Right.

When I'm in a cursor, it's like, okay, I will take a substantially worse coding model that's integrated in the cursor over a better one that isn't integrated. By a lot. Like I, my, my coding speed more than doubled when I switched from using the Claude chat interface to using the same exact model in Cursor.

Yeah, Cursor's awesome. I definitely really love it. I think GitHub Copilot has also caught up in a substantial way with their last round of updates, but as far as I understand, Cursor's still—

I'm not saying those are bad. So I'm saying those are the 2 things that I've done so far and the night and day.

I think it still is the best. I, my understanding is it still is the best. But yeah, the water line is always rising. Okay. So before we move on to some other kind of big picture things, talk a little bit about regulation, what the government is or isn't going to do, handicap AGI for us real quick. Metaculus says 2027 for the weak AGI median. Forecast. The modal, you know, the top of the probability curve is late 2026. I have to say, I buy it. It doesn't feel like there's that many more turns. The computer use, if it's like, I look at SWE-Bench and it's, geez, we went from 5 to 50% SWE-Bench in a year.

So for everyone's benefit, what is the weak, what is their definition of weak AGI that they're using?

Let me. Look it up real quick. So it's 4 things. This first one is tricky because I dug into it and my guess is that we have a definition problem here, but the first one is able to reliably pass a Turing test of the type that would win the Lobner Silver Prize. Next one is 90% or more on a robust version of the Winograd Schema Challenge. That's basically like pro— pronoun and sort of referent disambiguation. I would say that we're well clear of that already.

Right.

Next is be able to score in the 75th percentile on the math sections of the SAT, given just the images of the exam pages as input. I think we probably are safely past that as well. When I tested this, it was not yet multimodal, but GPT-4 Early was, you know, getting better than 75th percent.

That will not be a limiting factor on this problem. You can move on. Yeah. 2 and 3.

And the next one was, the final one is be able to learn the classic game Montezuma's Revenge and explore all 24 rooms based on the equivalent of less than 100 hours of real-time play. And that is intended to be a unified system. I haven't tried Claude on Montezuma's Revenge, but I have to imagine general computer use. A haiku version that's fast enough to respond would probably do that. Although I don't know how much strange or counterintuitive, whatever you'd have to do to fit the 25 words.

Right. The only reason I think it is, if anything, given this definition, I would bet the under, but I would not consider that to constitute AGI in the sense that I care about whether AGI shows up. Yeah.

The reason I think this question is ultimately gonna age poorly is that this Turing test, silver prize.

And that's the real question, 'cause we'll get Monty Python's Revenge in '25, probably. It's my guess.

Yeah, it seems like Turing test is formulated in a way that basically has expert users. And I think that the real problem is that the AIs are not trying to pass the Turing test. Like they are, if you wanted to create an AI that would pass the Turing test, you would do What OpenAI warns against in their fine-tuning documentation, where they show an example of, look what happened when we trained a model on 100,000 Slack messages. We asked the AI to do something and it said, sure, I'll do that tomorrow. And that has become their sort of funny anti-pattern. You don't want to really train on real data in many cases because you want a different behavior from the AI than you actually observe in the wild.

That's a good point. My sense is that the way to tell And it's already in tests right now, is ways in which the AI is just better. Like humans do something stupid that you don't want the AI to do. The AI doesn't do it, but if you wanted to, you could—

Just made it say, I don't know, a lot. Be like, you could pass as human a lot more easily.

There are various things you could do to make the AI pass. And if you really wanted to pass, yeah, yeah, because if you offered the AI $100 billion prize for passing this by the end of '25, everything listed here, I'd consider you to be a huge favorite to succeed. It's just not a very hard test.

So maybe, okay, grab that. It's a probably just flawed definition. And the reality is, barring a prize, and I don't think there's any cash attached to this for anyone, there's not really a reason to try to pass the Turing test. Other than like general scams, maybe no, like major developer is going to do it.

What you're saying is essentially that like this question comes down to when they choose to pass, not when they can pass, right? Like in terms of the grading. So it's not a very interesting question.

So let's flip over then to the Dario definition, because he, this is another thing I wanted to talk about. And I am of multiple minds on the Machines of Loving Grace vision.

Right.

First 50%, I give very high marks to. This is the part where he defines And he doesn't like the term AGI, just like we're all EA adjacent. We're all, nobody likes the term AGI anymore. We're all using substitutes. Like in his case, I think powerful AI is the phrase.

Yeah.

But I mean, he lays out a pretty high level of capability that is basically everything but a robotic embodiment that you would want. It's like smarter, as smarter than a human Nobel Prize winner across basically all the disciplines. Has the same affordance, has multimodality, has the ability to use computers effectively, you know, can potentially take offline actions as well, although it doesn't necessarily have a robot body. There's a lot of APIs. There's a lot of ways to place orders or hire people to do stuff online or whatever. So unlimited computer use. And this then translates to a pretty exciting vision if you buy it, that We're going to have potentially a century's worth of biological discoveries and medical advances condensed into, say, a 5 to 10 year timeframe. I do find his rationale there pretty compelling. Basically, he says that if you look at the history of biomedicine, it's a relatively small number of really important discoveries that drive most of the progress. And the form of those discoveries is some programmable or semi-programmable tool that kind of becomes a platform technology that you can then do tons of stuff on. So CRISPR is like a canonical one where, hey, now we can edit DNA to a certain, you know, level of precision that we couldn't before. Now we can do all, all this more stuff. And it drives all kinds of downstream discoveries.

This all reminded me a lot of the Star Trek situation in terms of you have this holodeck. Right. And technically speaking, you could do literally anything if you want to, and just nobody does. And then you ask yourself, what would be the, what would be the consequences if people used it in exactly these ways and no other ways? And then you have really interesting stories. But of course, there's a giant plot hole that there is nothing stopping them from using it in other ways. And they often accidentally sort of use it in these other ways, and then they just pull back or don't realize it and keep going. And Machines of Love and Grace is essentially portraying a world in which You have the AGI that should ASI very quickly. And instead we leave society mostly alone. We don't develop an ASI. We only apply it one at a time to these specific areas in an isolated, directed way. And then we see what is possible doing that. And I think he does a very good job of exploring what that would look like. But that scenario is fucking bizarre, right? It's the question of, okay, what medical advances could we get? If we had infinite advanced PhD students just running around at many x speed available on the cheap to do anything you want, well, yeah, of course you're gonna get a lot of advances, but are you bearing the lead on what happens to this world when that kind of thing is doable? Like what else is going on? And on so many levels, including like, how did we decide not to develop better AIs with this tool that you just created, which obviously could do that very quickly? Like if you, you're not getting Only 5 years for 100 years of biological advances. You're getting the same thing for AI advances, only more so because AI isn't going to be bottlenecked by all these physical experiments you have to run. And biology is. So there's no real explanation there. It's like very weird sci-fi, but it is a good counterweight to people who say that wouldn't matter, that you wouldn't get cool things. Like, yeah, no, you'll get these cool things, but it's a bizarre scenario. And that's like, My view again of, yeah, where we were up until the point when you know that you have a problem, right? Which is the part where he calls for this like democratic values thing, this alliance of the good guys against the bad guys that we also see from OpenAI's rhetoric.

Yeah, that's in the second half and that's the part that I don't like so much. Before we go to that, he seems to think it's like, of course it's a well-caveated essay, but he seems to think it's more likely than not that we get to this general level of capability in a 2026, '27 timeframe. I've heard other credible people say, yeah, that seems reasonable. Maybe just add on a delay, make it 2028. Maybe it doesn't go quite as smoothly as he thinks, but basically right.

I was just listening this morning, and it was that early afternoon. I was listening to the podcast. I haven't finished it yet, but it is very long, but the podcast he did with Lex and like he emphasizes that he's saying that that's how fast it would be if we don't hit one of the following roadblocks. So it's like a conditional. Prediction from him specifically, as I understand it, then that becomes the median slash modal prediction. But there are only ways that can go wrong, not ways that can go more right, sort of condition as conditions on that. So he's not making that as a, like his prediction is somewhat less aggressive than that in effect. But he is saying a large probability weight on 2027 style numbers. And yeah, that's the message coming outta all the major labs. That is what these people actually believe.

So do you basically adopt their statements as your beliefs, or you are at the top? It sounded like you were more skeptical than—

That's why I said that's their belief, right? Not that is what's going to happen. I think that I have model uncertainty, so I have their perspective and they have some unique information, but they also have some weird incentives and groupthink, information cascade, potential things going on. And there are other groups that like have very different estimates. So I would say if 2026, 2027 involves the relevant kind of AGI, that seems very possible, right? Double-digit chance of that happening for sure at this point, given what we know. But if I was about to bet abstract utils in some way and 50/50 odds, I would bet against 2027. I think the odds are under 50%, but that's because I don't see what they see and I don't know what they know and I'm not familiar with their environment and I can't just take their word for it and discount everybody else's perspective. And also, I do think they're making those predictions on the assumption that a variety of things don't happen. And Dario was more explicit about this than most. But I think they're all making the assumption essentially. And so what happens if China makes Taiwan in '26?

You know, like, yeah, which is a perfect segue to the second half of the essay, which I'm, which notably did not come up in the Lex Fridman discussion at all. And Dario, if you're listening, I'd love to hash this out in a little bit greater depth, which is to say at least somewhat compared to that. The short version is. Everybody seems to be, and by everybody, I mean Altman and Dario apparently, and like the national security establishment blob are all on this path that we're going to notably under a, you know who, second term, as it turns out, um, going to keep building the AI, jam, accelerate, build the best stuff we can. Keep China behind by cutting them off of chips and maybe other measures, but certainly the chip ban is the big thing so far. Then we'll achieve maybe 2026 timeframe decisive military strategic advantage. And then we'll create the new American empire, same as the old American empire, but with 100 million times more AI where we get to go to everybody and say, we're the good guys. Join with us. You don't get to have your own AI, but you get to make the API calls or whatever. And China's isolated and Russia too. And then eventually we get around to them and it's like, you guys are the last ones out, but we'll invite you in. You just have to play by our rules and then you can have all the great AI benefits that we're already enjoying. Sounds crazy to me. What do you think?

Well, one thing to note is that a lot of people are talking about building AI with democratic values, right? We're like treating the demo— any people who use democracy as good guys, anything that's not democracy is bad guys. And we need to understand that democracy is a semantic stop sign. Democracy is a word that says stop thinking, right? Like we are vibing this with good, and therefore if it's democratic, it's good. And you can stop worrying about what happens after that. You don't have to figure out what that concretely means or how you would implement that. You know, in a world with like super intelligent entities that can be copied, that they are like, do they vote? Do they not vote? How do they vote? Like how, what does this mean? Democracy? Is the global south voting? Do they get equal votes? Have you considered what they actually want if you let them vote? Are the voters gonna understand the implications of their vote when it comes to AI policies? Like none of this makes any sense, right? Nobody has thought this through. They're just evoking the word democracy to mean good. And similarly, talking about aligning with democracy is democracy must be— I love democracy, but we have to understand what democracy is for and why it's here and why it is better than the alternative systems we have and not just treat it as a fixed outline and not use any excuse to have to figure out what we actually want the future to look like and like how to preserve human values and find things that we care about, right? Like we can't just promise to think about it later and it'll all be fine. It doesn't, it's not that simple. Like that's not operational enough. That's one problem. The obvious other problem is, yeah, if you're very loud about the fact that you're doing this, like, what is the only reasonable strategic reaction by the other side? And why should you expect it to work? But you have to keep in mind, all the options are bad. You're saying we're not going to try and cut off China or anyone else's ability to develop AI because that would antagonize them. So we should be cooperative with them and let them be equally advanced or more advanced than us. And then of course everything will somehow just work out. It doesn't work. You can't just ask them to become voluntary secondary partners in an alliance where we have permanent superiority because it seems unlikely they'll buy it. You can't really offer to give up your advantage and go in as equals because the U.S. government isn't capable of doing that, even if it was somehow wise, et cetera, et cetera. All the options are bad, right? Like you can't try to get everybody shut down AI until you have some argument that can carry the day, at least amongst the U.S. government and then everywhere else as well that matters such that you could make that agreement. Like all our options are bad. And, you know, some form of get the people who you can get on board and then work from there could easily be the least bad alternative at our disposal for now. But the part that I hate so much is the whole like, well, you just treat China and also Russia and also like anybody else who you don't like as an intractable enemy who you can't talk to, you can't negotiate, you can't like, reach agreements with and you can't get to treat this safely. And so you end up just having to smash the accelerator and pray that physics is kind to you in ways that I don't think it's kind and that definitely are not that likely to be kind. It's definitely far from certain. That feels awful. But yeah, you have to pick up the phone. You have to talk to these people. We've seen evidence time after time that China cares a lot about AI safety. In at least some sense, right? And they don't just want to rush willy-nilly as fast as possible and hope that everything works out. They are very, very committed to the idea of maintaining control in every sense over China. And that should give us something to work with. And they're humans. We're all humans. We all want good things. Like, this saddens me and it scares me that everyone's pushing in this way. And it's like a huge, obvious failure mode. And if you're too loud about it before you're You're too loud about it, especially before you can actually enforce it, right? Like, why is TSMC still there? In the fourth sense, China doesn't use any of their chips, right? Like, if they believed what you believe when you say these things, would it still be standing? Or would they bomb the hell out of it even if they don't invade Taiwan?

Yeah, that seems— I don't want to bet on that. Because that just seems like somehow icky at, at a minimum.

The answer is because nobody believes it in their gut at the higher levels as confident enough to do something that would wreck the global economy and threaten the stability of their regime. And they're like, you know, we should be thankful that that particular move is off the table for all practical purposes, but there are other moves that are not. And as things get far, go farther down. Things are going to look less and less unthinkable. And we're going to start to be put on the table by various actors, especially if we are loud about the fact that we are doing a strategy like this with the explicit aim of world domination, permanent scientific advantage. I'm not saying you shouldn't seek it out. I'm saying understand what you are doing and also consider the alternatives. Again, all of the choices are bad, but yeah, it kind of sucks.

Yeah, it does not seem like we're headed in a good direction there. Obviously we have And I can't, this has all been filtered for me through the news. I don't speak Chinese. So there's plenty of lossy translation, but my understanding is that we have Xi saying 2027 is the deadline for the army to be ready to do something about Taiwan, whether that's a full-on invasion or whatever. It seems almost impossible to defend TSMC fabs. from a Chinese drone swarm if they're like really determined to knock the factory offline?

I mean, quite obviously, if China wants to knock TSMC out of existence, it can. Presumably, if China wanted to knock American chip production out through sabotage, it could. You know, at the risk, you know, it's just very hard to put ethics in these situations if you care enough. But luckily we don't live in those worlds. Yet, I mean, obviously, you know, if I was the U.S. government, I would either deliberately have a goal of U.S. chip independence 2027, or I would be pretty damn committed to defending Taiwan.

Now you're hearing too that Taiwan is saying increasingly, and I was wondering about this, I was like, why are they so happy to send all the chips here?

Like, this seems like a very Why are they shutting down the nuclear power plants? I mean, the Taiwanese are not necessarily acting strategically or rationally.

They have said more recently, at least I, again, I'm filtering all this through media, but my understanding now is that there are new statements coming out that are like, we prohibit the export of the 2-nanometer process. And there is going to be some most advanced technology that will still be here and will not be going to the United States. I don't have a super clear sense of how much that really matters. I mean, there's so much obsession on chips and 5, 4, 3, 2 nanometers, whatever. Like, it seems like, does that exactly matter? Or is it just having the ability to make a lot of them, whether it's 3 or 2? Like, I, it doesn't seem like that.

I honestly don't know. I just asked someone who knows. I just don't know.

It seems more likely to me that raw volume will be more important than one process versus the next generation of process. Like I would take 10 3-nanometer fabs over one 2-nanometer fab for sure. Only because—

It's a technical question about how they impact the use cases that you want and to what extent does better efficiency of individual chips scale better than adding more chips. Questions I just don't know the answer to, like, you know.

This is another question I don't think anybody has the answer to, but the possibility of distributed training schemes popping up is another area that I am trying to watch as closely as I can and feel like is one of the best candidates for a big shake of the snow globe where, you know, if you have to build and def— not just build, but defend a trillion-dollar data center. You have a real problem. Whereas if you can do ASI at home in the way that we used to have SETI at home, then, and maybe still do it. I haven't checked in on SETI at home in a while. I'm sure it's still running. Yeah, it probably is. But there have been some early reports recently of distributed training starting to work, and those have been only like partial disclosures. New Research had one where they were sort of like, Coy about it and said, we've got something working. We don't really even quite know how it's working yet or why it's working, but that space definitely could change a lot if it could get you to the point where, you know, in more of like a Bitcoin sort of way, like everybody's GPUs could just be distributed, networked, and contributing. That creates like a very different dynamic.

Yeah. Look, Tim, I have been skeptical of the quality and efficiency of distributed computing proposals for training. And I think that's held up so far from what I can tell, but obviously that could change. And if that happens, that puts a kind of a clock on everything in various important senses. And again, the more things get released that can't be taken back, and the more these types of things happen. The more all of your options for not dying become bad, the more you have to choose impossible choices where like all of your options morally suck. And if you propose anything, people will just cry bloody murder, but like nothing that doesn't require crying bloody murder in some form is possibly going to work. What do you do? Right? Like, but cost that risk. when we come to it and we hope the physics is kind to us. But yeah, the more this looks like a thing, the more like we just have a lot less options and a lot worse options. Just unfortunate. That's how it is.

Do you update your P doom on a regular basis? Do you, you have a ready number for P doom?

I have been using 0.6 for a while. If I was being like fully roboche, that number would be Changing obviously on a day-to-day, week-to-week, month-to-month basis. Certainly, for example, it would've updated when SB 1047 was vetoed. It would've updated when Trump was elected. Yeah, you can argue whether which direction it should move, but definitely should update. Otherwise you're just being very silly. Should update when we find out various things about OpenAI, et cetera, et cetera. Fundamentally speaking, I think my attitude is just It's not useful to try and get that false precision or get that delta. It might be, how old is that dinosaur skeleton? It's 75 million and 3 years old, right? Because 3 years ago I asked and it was 75 million. But yeah, you wanna keep the delta accurate for consistency, but you also kind of don't, right? Like the wacky zone, 10 to 90%. If, if you're confident that like you have enough model uncertainty and you have enough ways things can go wrong, but you also have enough just general like Various ways things could possibly go right, that like, yeah, the statements you make don't change very much. Like, I don't find it that useful. And I also find that there are a lot of like very weird and imprecise things that like I would evaluate differently on different days that would cause me to like get to different numbers if I were to actually go through a calculation in my head that was more precise. And that, yeah, I don't think we should spend too much time on the details of the question is. Basically a question of like, P-DIMM is useful in the sense of like, okay, are we talking about, so 1%, 1%, like 2%, 5%, 10%, more than 10, like potentially more than 10, but not 90, and then like really high. And that's mostly what you care about, certainly within the range of 3 to 70.

Yeah, I'm with you on that.

Yeah.

So let's go through some of these other updates, 'cause I think we're into the home stretch and there's like a handful of items, most of which you just mentioned, that I wanted to get your take on. SB 1047 being vetoed, we both supported it. I certainly, I went through a moment where I wasn't so sure when it seemed like the Frontier Model Forum was going to be like potentially overly empowered. And I did not like the sound of that for a minute, but then the final version seemed quite good to me. I don't know if you ever wavered, but I know you also did support the final version at least. How big of a deal do you think it was that it was vetoed and how, or actually how big of a deal would it have been for it to exist?

I think it's, we're going to find out a lot more by February 1st, like by the time that Trump has had his day one and we decide, he tells us what he's going to do with the executive order and other immediate executive orders of his own that he issues on day one. But I think we're seeing that maybe this is a, this is a giant disaster. This was a huge clusterfuck. As a result of SB 1047 being vetoed. The bill is not being used as the model for right now, other legislation in other places, even though it is obviously the best model legislation that we have available. Instead, people are considering bills like the one in Texas, right? Which Dean Baal has written up very well, which use a use case targeting EU style, EU Act style approach. To regulation of AI. That approach has all of the costs you can imagine and none of the benefits to existential risk. That approach is a huge disaster. And without a better alternative that is currently viable to talk about, without any sense that action is being taken, that approach is going to become increasingly hard to stop unless we like quickly do a turnaround. And it doesn't seem like people like being Where others who understand how awful this is are willing to like get behind an SB 1047 style alternative revival and propose a compromise. They seem to continue to argue de facto, as far as I can tell, that everything anyone's proposed is always bad. And we can all line up and say the Texas bill is terrible, right? But I don't see how you stop. More and more states from adopting the worst possible kind of AI regulation unless you get to something better. And SB 1047 would have been something better out of California. It would have carried weight. It would have been copied extensively in other jurisdictions. Could have been complemented by narrow task bills that deal with things like deepfakes that aren't covered by SB 1047. We could have had a net positive, reasonable regulatory regime from the states. Pending Congress's decision and a much better chance that Congress adopted a similar policy. Instead, we live in a world in which we're looking at even a deeply red Texas looking to implement a bill that you would think they would recognize was like EU-style complete regulatory suicidal nonsense. And yet here we are, right? Like Texas should be the last place this should pass. Texas should know better. We should be like, why is Texas messing with this?

Yeah.

Like, it's like their entire shtick. They don't do that. And yet here we are, like, sure, Connecticut passing it makes some sense. I get that. But Texas? And yet that's the thing. You can't, no regulation was never on the table, right? Not for very long. This idea that we'll just keep yelling about every bill as if it's a disaster, no matter how well constructed, no matter how light touch, no matter how much the benefits are. Like we had this debate between, you know, libertarians. And nutcase extreme libertarians. Let's be honest. As to whether or not this bill was a good enough bill to pass. And now the libertarians and the nutcase libertarians are lining up to oppose the regular people who regulate everything into the ground. Right? But like those of us who still wanna regulate AI are trying to find this narrow path back to something reasonable. And the people who don't want anything are refusing to accept That their position is not viable, right? They want our help, frankly. Here's how it feels. It feels like I studied the blade and that's what variant is at the door. And now you want my help, but you're not offering it. You just want me to stop the variant because it'll kill all of us. And I will, but like, you know, somewhat, but like, this is, I'm just completely insane, right? Like you have to understand this. And it, but then it comes down to what Congress and, and the executive is going to do, right? If Trump repeals the executive order and doesn't replace it such that the reporting requirements in the executive order go away, or even worse, if he also tries to take down NIST and AISI in particular alongside it, which will destroy the voluntary commitments. If these things go away and we don't have SB 1047 to replace them and we don't quickly pass something to replace them. We are in a horrendously worse position, right? Like if we, if we can retain the executive order, we can retain NIST, Trump can simply revise the safety frameworks so that they don't have the woke stuff that he hates, but they still do the things that actually pertain to access of safety. We can reach a compromise. I think that's an entirely viable world. I think there's nothing stopping the new administration from understanding. That the things that we care about together, we still care about. Getting rid of the things that they hate, that they legitimately hate, that like, I'm not going to argue with you that you hate them because I'm not going to convince you. And also, frankly, these previous regulations did in fact go everything vague all way too far. It was impossible to implement requirements. Like you can't use an implicit bias. You can't use a disparate impact test on an AI algorithm. It will never pass, right? Even if you are fully committed to the DEI philosophy, it is politically impossible. You cannot do it. You cannot do it. So these rules were not going to ever work if they were trying to make them enforceable. But that is the future by default if we have nothing to work with, right? So I would implore the Trump administration the same way that I implore California to get its act together and give us something That fills that void, right? Like if you don't issue guidelines that people can follow that don't include the things you don't like, they're simply going to adopt the guidelines you hate, including the guidelines that you already were issued by the Biden administration. You need to replace them with something else. You need to be in this game to win it. And also we have the national security apparatus and whether or not they will take action against certain actions. to stop people from doing the things that they really shouldn't be doing and also impose visibility requirements that way. So there's various different ways this can go. But yeah, the answer is I think SB 1047 really mattered and it was killed and this is really bad. And I think that if it, if SB 1047 part due with a remarkably similar bill, but modified to around the edges or whatever. Gets passed next year because with Trump in the White House, it's a completely different political environment and AI is a year older and you have to worry about repeals and everybody understands something has to be done. Maybe. But also they can— let's newsom also to call for use-based regulation of AI, the worst possible thing you can do. If California goes that way, so goes the nation, so goes the world. The loss, and then as we did in '47, It's somewhere between substantial safety loss and the death of the AI industry. Like, I didn't want to use these arguments in real time because I would've been considered to be hyperbolic and crazy. But I think it was a very real scenario that's shaping up way worse than I feared, where a series of states passes these terrible bills and the AI industry is either crippled or forced to concentrate amongst a few big players. And like, SB 1047 would've stopped that from happening.

And the mechanism there is basically just that the burdens of demonstrating neutrality and whatever on all these different dimensions are just impossible for anybody but a big tech company to comply with.

Imagine a NEPA-style regime where anybody can sue you over any of a wide variety of harms by standards that are technically impossible for AI to complete from one of several different states with several different regulations. And you have to verify this every time you make any change to your AI model or something like that. Like imagine a complete nightmare. This is not that unlikely to happen. It's probably double digit for this kind of scenario to happen.

If you're like a real doomer, do you want that? I mean, not you, Z, personally, but there's some argument for sabotage of the industry there, right? I don't want that myself.

I am a virtue ethicist and a functional decision theorist. So even if I thought that was good. For the world. I would not, because of this reason, I would not want it. But like, no, it's just bad if we do that. The people who say we can't lose to China are like kind of warmongering and jingoistic, dangerous people quite often. But if we cripple our AI industry, we deny ourselves the benefits. We hurt America quite a lot. That's terrible. It's exactly what— let's try to catch up. Exactly what makes them into the threat that I don't think they are right now, particularly. That's how we lose. That's how we get this nightmare scenario. That's terrible. No, this is, I don't want that. I don't think that this ends well. Like we still gotta win.

I want to come back to virtue ethics in a second and get your take on virtue ethics in the time of Trump, because that's something I've been thinking about quite a bit. In the last week or so. Before that though, a couple more just roundup items. You mentioned OpenAI drama. Yeah, of course there's always more. I think we just did an episode actually with Dean and Daniel where they talked about their transparency proposals. And in that we got a kind of recounting of Daniel's personal story of leaving and not signing the thing and then posting online about it and that gradually coming to light and then I guess those policies mostly are reversed. He did say that, yeah, now he can basically, of course he's still bound by not disclosing trade secrets, but he is permitted to criticize the company and his equity is restored. And so that turned out reasonably well, I guess. They have also taken another $6 billion or so in funding with a notable clause attached that they must turn the thing into a for-profit or else the investors can get their money back. Presumably all that money's gonna be spent at that time on GPUs and lots of electricity. So it doesn't seem like there's any way that money gets kicked back. It seems like the, basically the train is now rolling on this conversion to a normal-ish for-profit entity.

$6 billion is not that much money for a company that's worth $150 billion. It's a very small raise, probably because they Would have had to take a much worse price to get a much bigger raise. But yeah, they're throwing their hat over the wall. They're saying we have to convert. We don't convert, then we're in a lot of trouble because who's going to give us the money to pay back the earlier investors at any reasonable price? They risk being sold off for parts or acquired or like other very disastrous scenarios in two years. They can't make this through. Look, they're trying to pull off the theft of. At least the millennium and quite possibly all of human history. They're trying to steal most of OpenAI from the nonprofit, which currently has most of the economic value of OpenAI plus the control premium. And they're trying to do it for a small fraction of OpenAI's stock. And the question is, will they be allowed to pull this off in broad daylight or not? And time will tell. I wrote articles about this and my position has not changed.

It seems like they're going to get away with it. What do you think?

I think that the president is listening a lot to Elon Musk and Elon Musk hates OpenAI. And it is, again, the biggest theft I've ever seen short of like, you know, the Bush tax cuts, for example, I could argue would be a bigger theft depending on how you perspective it is, but, or they could just be good business. I guess, you know, a little from Combi, but like the point being that like, this is a giant illegal Misappropriation of funds from my perspective, just very flat out in broad daylight. There are various actors who can do something about it. Many of them have reason to dislike OpenAI or distrust OpenAI in this situation. And also the board has to sign off on it, and the board's members could have legitimate worries about their fiduciary duties and potentially the piercing of the corporate veil if they sign off on a number like 25%. So I don't think it is at all obvious. They'll be allowed to get away with this at anything like the current number. But yeah, like by default, when this kind of thing happens, like the way it works is power just forces you into a corner where they threaten to blow everything up unless you give in. And then people one by one give in through some combination of threats and bribes and leverage and mindfuckery. So probably they get away with not necessarily exactly the deal they're proposing, but they get away with quite a lot. That's what I'm expecting to happen in my gut, but I put a contract on the table and I might buy something, right? If you gimme a cheap enough price.

The point about Musk being in the president's ear and the president potentially having a proclivity to use the Justice Department in, let's say, idiosyncratic ways definitely does change the analysis quite a bit. I would say in the absence of that, it would've felt overwhelmingly likely to me, and that There is a lightning bolt from, from a distance possibility that could change it.

But I mean, you can also look at it as Microsoft is attempting to steal quite a big tech companies are basically attempting to steal quite a lot of money from a nonprofit and Republicans control the Congress and they might not like that. Like big tech is not popular. Microsoft's part of big tech, very centrally part of big tech. Yeah. So like, I just think they'll talk them into the whole China.

I mean, right now it seems like. I guess a couple things. One is, I think there's always a creative argument and there's, I think there is an argument that, look, it's not worth that much as a nonprofit because if nobody invests the next however many billion to do the next round of scale, then we lose and you're just out of the game. You can't really compete in this way. So you can, you're entitled to some, but it goes to zero if this deal doesn't go through in a very—

So like, for example, like I put that, so like that's basically saying I put dynamite under your building. So you're going to sell me that Novik for a dollar because if you don't sell it to me, I'm going to blow it up. So it's only worth a dollar to you. So it's only fair that you sell it to me for a dollar.

And like, yeah, although I think they could argue in somewhat good faith that basically this is naturally occurring dynamite, right? Like I think the sort of Altman narrative, which I don't, I think is reasonably credible, is we didn't know when we started this thing that it was going to take $50 billion to get there. It turns out it does. No, no, no.

You don't get all that, right? Look, you still have to give fair value for the assets, and that includes the control premium. There is no world in which this is a fair compensation. You can argue, I think, for 49%. I think that's like the fair, aggressive, but not obviously blatantly unfair of the— they have to lose control, right? They have to lose control or this company's not viable. So we have to give the 51% to other people so that the company is no longer under their control. So they have 49 and it'll be further diluted by investments going forward and it'll get progressively less and less or something like that. But again, I just, I don't see any attempt to wrestle with the reality of the situation. I don't see any attempts to say, this is what OpenAI is worth. This is the distribution of future profits. This is the control premium. This is why this is a fair valuation. And if you want to make the argument, It's a fair valuation because if you don't give it to me, your company will collapse and therefore you shouldn't get very much. All right. That's your case. Let's have that discussion. Is that legal? I'm inclined to say no.

Yeah. The other thing that's pretty interesting about it is the intangible assets goodwill argument. I don't know what the law is governing this sort of stuff. I think it's probably not very well developed because we're only Recently moving into this world where so much of corporate value is the intangible goodwill type of thing. You know, this ain't the railroad era anymore, obviously. So I don't think we've really figured that out as a society, but the fact that they were all going to kind of shift over to Microsoft or that, you know, it's still not that many people, right? Like they could reconstitute themselves in a way where it's like, Unless you're going to totally change non-compete or sort of create new laws to prevent us from doing this, we could always just move across the street, bring our know-how with us and take a bit of a hit, but like they were willing to do it once, you know, and it seems like that—

You can try, you can use those arguments, but I think we know pretty well that if they had pulled the trigger on that, it would not have involved the vast majority of the company, Microsoft. Would've involved a large percentage of them moving to Microsoft, probably, but also a huge diaspora and a huge loss to OpenAI. And Microsoft, in particular, Microsoft did not want that scenario. By all reports, Microsoft was doing that because it was less bad than a full diaspora, right? And it was a credible threat, right? They did it strategically because they had to, but they were thrilled to just reconstitute OpenAI. Despite the board's pending control. So I don't know if that's a credible threat in this context or how credible a threat it is. It's a credible backup plan, but I—

It's a lot of hassle to go through that for sure.

I mean, there's a lot of value that gets destroyed if you do that. The bottom line is that there's room to cut the nonprofit in for quite a lot of the future profit flows of OpenAI while converting it to a B Corp. That's what they want to do. And if they manage to browbeat the nonprofit who it's for into accepting vastly less than their share is worth, then that is a theft as far as I'm concerned, flat out. And sometimes people get away with thefts. I'll say, Silicon Valley does this all the time. They just decide that someone doesn't deserve what they've been legally entitled to and they just take it. Not that weird. So yeah, I expect them to pull off some form of it and I And I would behoove the various attorneys general and other authorities over them to keep a very close eye and not let them do anything that's not allowed.

The ARC AGI prize. I don't know if you have any particular thoughts on this. People are generally familiar. Progress, I would say, was pretty good. We went from like 30-ish percent to 60-ish percent as the state of the art. 85% was where you were going to win the million bucks. So nobody's won the million bucks. But, you know, the gap has been closed by roughly half.

And I mean, everyone should assume that million bucks goes away in 20 next time around, right? Like, like this was probably not likely to last another year.

Yeah, it seems like can't last that much longer. Test time training emerged as an interesting theme. They had some pretty tight compute requirements and a totally unseen test set. So people started doing small model fine-tuning on the unseen test set to try to learn as much as possible in that new environment. And that seemed to be the big thing. I feel like that is likely to be, so I kind of a trend that gets exported elsewhere.

I found in discussing this with people who did the prize, and it's not eligible to win the prize, it requires things the prize doesn't allow. But yeah, it showed in principle you can do a lot better. On these types of tests with training time compute. That makes sense because these tests, these test questions are all kind of part of the same general logic that is deliberately different from the general logic that they're using to train. So it's in this very special case where a little bit of test time inference is gonna be worth a lot. So it makes sense to me. It paid off pretty well. Yeah, it's a, it's, it's an exciting potential alternative to O1's infrastructure for rooting in inference time. And we don't really know how effective it's gonna be in general. I think, I think it's kind of a best case scenario place to start. So try it on some other stuff. Yeah. See how it goes. I'm excited. But it makes sense that if you're doing a series of related questions, then you should do some form of temporary tuning of your model while you do that. There are various things in this general sense that fall under the things I would definitely be trying if I was like directing research at a major lab or otherwise had a lot of money to try, a bunch of programming time and compute to try and figure stuff out. Perfect Markov model.

It's like 2 things that maybe didn't happen. One that I would say pretty safely didn't happen, deepfake election chaos. Everybody was on the lookout for that. Of course, it didn't really happen. Did it not happen because people were just too savvy or the technology wasn't quite there or nobody really tried? What's your read?

They tried a little. There were a few incidents. There was like this fake accusation against Tim Walz. There was like a small, there was like a fake, a deepfake of Kamala that was shared by Hamas. There are a small number of them, but I think it's a combination of too tech savvy and not tech savvy enough in various different ways. Like people were just, I think the main impact was when people were suspecting real things of being fake, not when they suspected fake things of being real. And that's a common theme for now.

Mm-hmm.

My basic conclusion is the fake news is demand side. Not supply side. Like people of all political orientations demand fake news. They demand echo chambers. They demand stories that aren't true and conspiracies and tall tales and so on. So in a world in which they're already making up complete and utter nonsense with very basically no evidence and telling these stories, as they were this time around, like every other time.

Yeah.

And you don't want to know what would've happened if this election had been like Turned out differently, right? Been closer, gone in either direction, but we were very fortunate conditional on it. Whatever you think otherwise happened, conditional on a Trump win, we were very fortunate that he won popular vote and enough of the Electoral College by enough of a margin that everybody agrees he won. And there's no discussion about whether he won. We're all just agreeing it's over. And like, The nightmare scenarios for this election were not the true nightmare. The big, the biggest nightmare scenarios were the scenarios in which like both sides thought they won in a real way. And we didn't have that. But again, like 2020, we had the big lie, right? And it wasn't driven by deepfakes. It was driven by just flat out lies with no evidence, with basically no real evidence behind it. Just like making stuff up. So the issue is it just doesn't change anything to add. Deepfake pictures. It doesn't really help you. In fact, if anything, it hurts you because now people can point to the deepfake picture and with enough analysis figure out how to demonstrate that it's wrong. As it improves, there will be more and more problems, but I just don't think it's that central to the source of the problem of misinformation, the source of the problem of people having echo chambers and feeding each other tall tales from all directions and all sides. And AI analysis of information is going to become our ally, right? They can determine whether or not something is accurate and that will be very helpful. And so like, I've been optimistic about Deepfakes for a while now. Like it did not surprise me that there was very little impact. Obviously it was on the, the, the extreme left end of impact where there's like almost no impact, but it didn't surprise me very much. I, I expected this to go fine. I do expect other impacts, like 2028 is going to be very AI influenced in a lot of different ways unless I'm very, very surprised. But yeah, no, the people who thought that 2024 was going to be the AI election, I was always pretty skeptical of that. I thought too soon.

Something that I demand fake news on is the effectiveness of UBI. And another topic you've covered recently is the recently wrapped and pretty thoroughly analyzed Altman-backed UBI experiment. My general sense, and I haven't been super deep into the literature, was Most people are like, this doesn't seem to have worked super well, but I want you to tell me the story where it works great and I should expect that we end up with UBI. So please do.

Well, Nathan, we all want that, but I'm not, as you mentioned, I'm a virtue ethicist, so I'm not going to give you what you want here. Unfortunately, the UBI experiment in context was basically a failure. Not entire, not like 100% failure. The money didn't just disappear with no benefits, but it was clear that wasn't the way. As implemented. I do think that a future world where there is massive technological unemployment and there is near universal need for the universal basic income is just a very different scenario that this is not studied very well. So the good news, the good, the fake news that you want, the good news is that we don't know if this wouldn't be a good solution to a different world, or at least much better than not doing it. But basically, no, you can't just give people A bunch of money in the first world at random, some of them at random, a bunch of money and expect them to like permanently improve their lives by getting a bunch of money, uh, the way you would expect.

So how would you summarize? My sense is like people worked less, not so much less that they had the same money, but I saw some headline where it was like people worked enough less that they ended up with less money. That seemed very strange to me. I basically discounted that on priors.

I don't remember the exact details, but essentially at the end of the 2-year period, people were generally not substantially better off beyond the additional consumption and decreased work during that period. Like the resulting savings was mostly just exhausted without resulting in rents going up or acquiring permanent capital or otherwise like making their lives better, which is just not the thing that we were trying to fund.

Right.

Like we were trying to fund than getting into a permanently better situation, right? Improving their financial situation, improving their social capital, improving their productive capital, improving their skills, their educations, et cetera, et cetera. At least didn't see very much of that. There's no way this is efficient on that type of basis in this exact context. But again, like these things can be very context dependent. So the fact that it happened to a small number of people could be, for example, highly relevant because If you get a bunch of money in a relatively poor community, but the people around you do not, you are effectively in a situation where that money is being taxed very aggressively by your communal bonds and obligations. And you have a very hard time justifying why you have that right and other people don't. You have no opportunity to like move along with other people. It's very difficult. So I can imagine a thesis that says we've given it to everybody in Chicago. It would've been very different than giving it to a very small portion of the people in Chicago and saying no. And we just don't know. That's a, to me, a very valid thought.

Yeah. Threshold effects can be a big deal in, in a lot of these things, it feels. Was there any silver lining in what you reviewed around just like happiness, wellbeing, like time spent with family, kids going to school?

Non-zero effects, but again, nothing that would Okay.

We're probably not going to get it in the next 4 years anyway, because we know who was elected. Interested in your take on, you said you could argue which way we should update. I have found myself going into the election, I was like, this does not seem like the guy we want to be president if there's a decent chance we're going to have AGI. He seems just too unpredictable, unstable, inconsistent, not rigorous thinker, not trustworthy to people that might need to trust him if we're going to find ourselves in sticky situations. I still think all that's true. I've been at least tempted to some optimism by the Samuel Hammonds of the world who are like, look at all this, you know, super smart policy we might get out of somebody like an Elon Musk shadow president, whatever. What was your feeling coming in? What's your feeling a week on? And then—

I mean, prove me wrong, kids, right? Prove me wrong. Like I, You want this to work. We want, everybody should want this to succeed, this administration to succeed on AI especially, but also in general, and to make things good as much as possible. And Trump is very high variance, right? Like there are worlds in which Trump is a huge disaster by everyone's evaluation, potentially including Trump. There are worlds in which like it's an existential tail risk. It's really bad either to the country's economy or budget or democracy or literal existential catastrophe. It's all on the table with some probability that I consider to be higher than the alternative. But I also think there's more upside than there was under potential Harris administration, particularly for AI. Harris had already shown what path she was going to go down, right? She was going to go down a very Biden-style path. There are advantages to that, covering some of the basics, but the chance she would react decisively to events seemed low, which she tends to do as far as we can tell. There'd be a large amount of emphasis on, I don't necessarily wanna say woke or DEI, but like on left-leaning mundane concerns that I don't think are the right focus here, that were in fact reasonably ubiquitous amongst a lot of the things that she was in her administration. Collectively, her and the Biden administration pushing. She was involved heavily in the AI efforts, so I think we can in fact like read them as meaningful to her personal views on this, unlike some other policies. And like Trump and Vance, they could obviously choose like some of the worst possible policies in these areas. They could repeal the executive order essentially without replacing it. They could sabotage the ASI. They could actively encourage Open source, they could, you know, in ways that are dangerous, not in the ways that like are helpful. Like I wanna be clear, there are, there's a time and a place for open models and like obviously non-AI open source software, right? All the standard stuff. But they could also potentially go as far as to like attack or even break up some of the big tech companies, creating diasporas, creating a much more multipolar race in ways that would be very harmful. They could slow down immigration, illegal immigration in ways that are very harmful for competitiveness. You can impose tariffs, including on GPUs, in ways that are like obviously stupid and terrible. There's a lot of big risks. This room, I think, though, in addition to the obvious standard, he might just like very much disambiguate equilibrium and like cause massive inflation or loss of democracy or— there's a lot of really bad stuff that was on his proposal. But Go really well. It wouldn't strike—

Tell me that story. How does it go really well?

Okay. So all of the things you really fear, things like deporting vast amounts of immigrants and like printing tons and tons of money and imposing draconian tariffs that would wreck the global economy and et cetera, et cetera. That's all like big talk and negotiating positions and doesn't really happen. The Biden executive order gets replaced by an executive order that is essentially the same executive order without all of the woke, all of what Trump would call woke shit, right? Regardless of what it is, basically strip out those priorities and put the rest mostly back. The safety standards issued by NIST and AASI are modified in similar ways, but otherwise, like, he kind of continues the same general policies. He then brings down regulatory barriers, helped us build up our energy infrastructure. Generally, and most importantly, is capable of changing his mind. Like, it's clear that he understands, he's connected to people who understand that AI is dangerous, in fact, is capable of recognizing super duper AI and signs of the danger. So like when the time comes, he and Vance, who is actually plausibly like at least somewhat connected to the communities that we care about, potentially could be dialed in very quickly to the right people, realize what's going on, understand the problem. They pivot rapidly. They work with the national security apparatus. Things happen generally, just things get unstuck. We don't just meander forward. We have a unified government, at least nominally, can potentially pass things, potentially reach an AI. You know, it's possible that an AI bill can be passed under a Republican administration. I think the Democrats will be willing to support it under a Republican administration, whereas before, Democratic administration would've been basically like, the Republicans would've just mono-opposed it as anything in the Democratic caucus could've Drafted what it meant to rope for the Republicans to accept when they would've universally opposed it or whatever. I can tell stories, right? Very much tell stories where this works out perfectly well, including just the story of he makes the one big decision, right? Like when the time comes and these AGIs show up, he pulls the trigger on necessary outcomes in ways that Harris might not have. And that just overrides everything else that the entire administration does on every issue everywhere. You're within hours. Other than that, I think it's perfectly plausible. So like the story you can tell about the first Trump administration, right, is did a bunch of stuff until 2020 that was different from what Clinton would've done. And the impact of all those differences were Trump changed, it didn't matter very much. And then 2020, you did Operation Warp Speed. The result of Operation Warp Speed, the entire world economy and our lives More dramatically than ever. And this was just so much more important than everything else that happened in the Trump administration until Election Day, right? At least until Election Day and all the things that happened afterwards that you should otherwise, if the election had gone differently, at least you would be like ecstatic that Trump won over Clinton. Even if you thought his other policies were bad, he was a very, very, very reasonable thing to pick. And like, if AI is the one issue these people are capable of changing their minds and it's not an obviously partisan issue. And they can separate. Like the thing, the basic problem in AI is if Trump and the Republicans associate any action to keep AI safe or regulate AI with woke and DEI and therefore instinctively oppose all of it, slash they buy the open source, open model arguments such that they just sort of get this weird thing in their heads that nothing open can ever be dangerous or bad, like this Andreessen philosophy. The combination of these 2 factors in some form is the nightmare AI-specific scenario. Well, the other AI-specific scenario is that the prediction markets say that Taiwan invasion is potentially more likely now under Trump than it was under Obama. And obviously invasion of Taiwan is like a giant wildcard in the entire problem. But the specific problems in this specific area They might not come to pass. So, you know, you have both. And I don't know, from what I have seen, I, I try not to closely follow politics. I try not to pay too much attention. The reason I didn't worry about any of this, this is like as much as I've talked about politics in a while now, and I waited until after the election to do it. But the reaction to the election, I guess the right word for it would almost be normal. So far it's been like a week, but it seems like a normal transition. With a normal set of picks for running the administration, most of which are out of the Senate and the House, it seems like everyone has just been doing what we do. And if that's true, then there's not a partisan issue, right? And you can be happy, you can be very, very sad about Republicans having power. And that has nothing to do with the AI situation necessarily. The AI situation is its own unique animal. Again, they're going to make some decisions. We're going to find out pretty fast.

There's a couple of kind of keywords there that I thought would be worth maybe digging in on a little bit. And, you know, it's not a partisan issue. That was one phrase that I've been thinking about quite a bit. The other one being, you know, we, it would be bad if we got into a situation where they, you know, Trump, Musk, Vance, whoever, see anything related to AI safety as sort of Woke shit. Those points seem—

Musk specifically is not going to be like that. It's going to be—

Yeah, he's smart enough to avoid that collapse, I would think. Of course, he, you know, how long he'll be involved, at what point their relationship may have a breaking point, obviously could be any minute now. I'm one generally to think like, long may he remain influential, Musk, that is, but there could be a twist or a turn any moment, it seems. I want to get your take on what a person like me, and for the purposes of this discussion, I'm like definitely pro-technology, pro-progress, broadly libertarian, left-leaning on like a lot of issues that are more salient to most of the electorate, but very focused on AI. What is virtuous for me to do? How should I approach all of this strategically? I used to say that AI safety and AI ethics people should make much more common cause because it's all about getting the AIs under control and like getting good outcomes from them. Now I have to maybe reevaluate that if I'm thinking, geez, I don't want to show up at the White House with Sinvid Jayaprue, if I'm saying her name correctly, because they're not going to like that.

Yep.

And so Do I need to like reconsider my AI safety, AI ethics, you know, kumbaya attitude? How much should I be? I definitely am like pro more people coming to America. I'm pro even like people that are here technically against the law. I feel like for a long time we've sort of winked and said it was okay. And it feels like for the government now to like, Suddenly come down very harshly and say, well, you're illegal. You know, that to me feels like not a virtuous thing for the government to be doing. I'm really struggling to think like, where do I draw the line? Like deportations happen all the time. If they like tick up a bit, do I like go out into the streets? Probably not. But if police are coming door to door in my neighborhood looking for people, then there's gotta be some line. And I'm a little bit lost as to how to think about How much I should focus, how much I should compartmentalize. Should I be making like pre-commitments now? I realize I'm gonna have to take some pitches, but I, I don't have a framework yet to organize what pitches I take, which ones are sort of not my lane, but I still need to go outta my lane to address. And also like who I should kick outta my lane if there are people in my lane that no longer have a place.

Right. I think of this as a bunch of different questions. So one of the questions in classic, you know, the bad government comes, right? Call it fascism, call it mass deportation, call it whatever you want to call it. It just, your conscience cannot abide. It's just so bad. At what point do you defy the law? At what point do you work against the law? At what point do you take various levels of action against the law? And to be clear, I think there's a not that high a chance that like You face these questions in earnest, I don't really expect the actions that you're worried about to be that likely to take place on the scale that would, you know, cause you to seriously consider acting. But yes, I think that thinking now about how you would act in response to, uh, different actions that you strongly oppose is wise because you in the moment would, might potentially think poorly or in either direction. Um, it might be something stupid. You might decide to be afraid. They're both on the table. But I'm not gonna get into like exactly where to draw those lines or how to think about those policies. I'm gonna try to stick to questions like, what are the consequences, practical, of various policies? Because I wanna stay in my own policies I've decided on. And yeah, obviously mass deportations would be a severe shock to the economy.

Yeah.

So it'll be bad for America's productivity and competitiveness regardless of what else you may think about. And we have to go on perspective or whatever. So like, I'm hoping they don't happen. I don't expect them to happen. That's, and specifically how you spill the information gets like messed with versus expected, then that's very bad for our AI industry competitiveness as well. Certainly from that. But question of like the ethics people and how to deal with that. So I think that like we're now going into, among other things, a series of regulatory battles. In the States where there's going to be bad legislation that the AI ethics people are going to support and the AI don't kill everyone people are going to oppose mostly because like, oh, you know, I talk with people who think that AI is so doomy that they have different ethics than I do. And they're like, well, you know, it'll stroke out an AI. I don't care. But I think it's wrong. I think you should not do that. Some people of course will think that way, but I think the vast majority will not. But the reason to be allied with the AI ethics people is if we have common interest in passing either combined bills or narrow bills that are good for both of us, that drive home the points, that cause things to happen that are good, that cause better outcomes. If they're pushing for net negative stuff, there's not really much to talk about anymore at that point. I mean, we should continue to point out that the things that we want Advance their cause. They should be supporting what we want, but we shouldn't like support what they want. We don't think it's good in the hopes that they will support us because they don't look back. Like we need to understand that there is nobody we're negotiating with here, right? And instead there would be, they would stab us in the back anyway. We don't have any leverage on this right now that the SB 1027 window is closed. We don't get to. necessarily have that much impact on what they do. They're going to be who they are and they're going to gain strength. They're going to massively gain strength because these issues will become more salient. And, but also they will be, you know, strongly, strongly opposed by the current. So yeah, if you're trying to lobby Congress, you should be strongly, strongly supporting strong AI ethics calculations. That'd be a mistake. And in general, the entire may not kill everyone movement has a problem. Of culturally, so much of it thinks that like blue values are obviously just standard and correct on most of the standard issues. And that the orange man is horrible and bad and that red values are terrible and obviously wrong and is in terrible position to cooperate with a red administration, let alone a Trump, a literal Trump administration. And that is creating only a few Sam Hammonds of the world that can actually treat to that situation and usefully contribute. And that's a shame and it's too bad, but I'm not saying what I was rooting for on election night. People can guess, but we live in the world we live in and we should deal with what we have.

Yeah, I'm playing around with these like phrases or mantras. And one that I've been toying with is it's better to be a live player than to play Cassandra. And that's even like within the AI space. I feel like we are headed for a sort of general accelerationist vibe. And of course, even that could change. As we've said, Trump is high variance. He could get freaked out. Who knows? But it seems like we're more than likely headed for more like an AI Manhattan Project than a pause.

No, no, no. The Manhattan Project is Doge. Right.

So the, the AI Manhattan Project, a subsidiary of the Doge, but I like kind of reflexively recoil at the notion of an AI Manhattan Project. I'm like, didn't, you know, isn't there something we should have been taking away from the actual original Manhattan Project that would cause us to not want to rush into an AI Manhattan Project?

I mean, better, better security for sure. But beyond that, look.

I just, the sense that I think a lot of them regretted what they had done and maybe a little bit more thinking ahead would be a good thing. And just rushing to copy this dynamic of, we gotta get there before China just seems really bad. But if it's gonna happen, how do I calibrate where I want to focus my arguments, where I want to try to Have an influence. It seems like we're headed for an AI Manhattan Project. I probably can't stop that train. Maybe I retreat to, you know, some sort of China dovishness, or can we at least make the AI Manhattan Project like something positive? Can we like race for how many cancers we can cure or some sort of other metric besides strategic dominance? I don't know. Do you have a thought? Can you coach me?

This is a decision theory problem, right? So if you don't impact in any way, if you are not correlated, if your actions and your decisions and your perspectives are not correlated in any way with the decision to do the Manhattan Project, then once it already exists, you have to decide whether you would prefer to live in a world with a more advanced and more successful Manhattan Project or not. And I would conclude that conditional on the Manhattan Project coming into existence, I would expect your and my contribution to that project to be net positive because we could pull it in a safety-oriented direction on the margin. And also like once you are in fact on your race, no matter what, to the finish line, it would be better if you won. Due to the importance. And also if you won as safely as possible. And concretely, it seems strong enough to join the project. In general, it's a project, right? As Leopold calls it in The Push to the Edge, right? Just the project. does happen and there's no stopping the project, then like just based on consequentialism from there, like I think you have a pass. However, there is the problem that once you, like the decision that's being made now as to how to proceed partly depends on how people would react if the GANET project was done. And you shouldn't necessarily roll over when these things happen. For that reason, because that leads to everyone rolling over, which leads to the wrong plans, the bad guys, et cetera, carrying the day and the races happening, right? Like, you don't want to let them like be like, well, we're racing now, so you better help us win. But they wouldn't have started the race if they didn't thought you would help. That's a disaster. So you have to consider all of these possibilities, assuming you think the Manhattan Project is a mistake. Obviously, if you think the Manhattan Project is the right decision, you should support it. And it's not the worst option, right? A unified benign project by the United States, style of approach to the project, is, seems clearly superior to me to 3 to 5 AI companies racing to the finish line as fast as possible so the other guy doesn't. And that's kind of their fault. You should only consider not helping the project in that sense if you think that either your participation actually makes the situation worse. Because you would be accelerating the project without helping it be safe in a way that would be bad. Or you think that the project is something you shouldn't have been encouraging by your future willingness to help out in it, but that required it to be an error versus the alternatives that were on the table. And if the alternatives on the table were worse, it's not at stake. Like the project is not that unhelpful, right? Like it's not the best scenario, but it's far from the worst. And the worst is kind of the default. So, yikes.

So it sounds like you, you're, you're net favorable or tentatively favorable toward a nationalize the labs, have one. We know we, we know we're not going to have a huge number of live players for economic reasons as previously discussed. So given that there's a small number, corral them all into one. I mean, I'm choosing my words.

I'm choosing my words carefully because of the way that people have run the discourse these days and because it's genuinely confusing. And conditional and nothing is obvious and we don't know what our scenario is going to be or what our options are going to be. But if we are clearly rapidly progressing towards AGI and ASI and the situation is essentially a race condition between multiple companies that are under pressure to proceed faster and is reasonable under the conditions, then a unified project seems better than a non-unified project. And there are various ways to get the unified project, like the emergency assist clause, for example, is another way to get there, right? The government being involved is not a first best solution almost ever in any situation, but it's not the worst case scenario either. So it comes down to just, do you have a better option? There's often the thing similar to iterative development. Once the project is inevitable, waiting longer to start it is potentially just Worse, faster, right? So you only want to oppose starting the project if you can actually stop the project from happening in at least some world after that. So it, it's all very complicated, but I noticed that we're going to know more before this happens. We're not going to start the project in '24. We're not going to start in '25, right? By '26, we'll know a lot more. We'll be able to make better decisions, hopefully. I mean, we're starting administration, so we could either Any number of decisions could happen. We don't really know. And it's not like we can't influence them. We should try. But, but yeah, like we should deal with the situation as it is with the options that are still left on the table. And one way to think about the project to me is we should make decisions keeping in mind that we can start a, we potentially could start a project and we don't want to like make it impossible to start a project if we need to start a project.

Yeah.

But that like we would prefer to live in and steer towards worlds that don't Why are the product, right? Where the product isn't necessary. And then that raises the question of, well, what if the product's gonna happen in worlds where it wasn't necessary? What do we do about that? But yeah, again, we could talk forever about these hypothetical scenarios, but my guess is the majority of the time that the product actually happens, it was an improvement over the second alternative on the table that people in the room were talking about.

Yeah, absolutely.

Yeah. All right.

Last question. Where do you look for inspiration when it comes to virtue? You know, the challenge of course with virtue ethics is it's kind of tough to know what's virtuous, right? It's subject to debate. Maybe few-shot examples is the way to inform our behavior. So interested in what sources of inspiration for virtue you personally look to, and if there's any changes or what you're contemplating on the margin to change or evolve your own worker approach in light of new circumstances.

And I just asked on Twitter about, like, just in general, who is the person that you're trying to model after in some important sense? And my brain just shot back the answers Richard Feynman, to the extent that there is such a person, sort of embodies more than other people, I guess, who I'd want to be in these spots, but who's the people I can think of. But the truth is I don't really have a single role model per se here. I'm just trying to work my own way through these problems in a very unique mishmash of different influences and perspectives. I've got my Douglas Adams and my Robert Anton Wilson and my Aristotle and my everything else. Like, you just put it all together and you, you figure it out. And like, I don't think there's one particular person that I'm like, that is the virtue set that I want, including myself. I, I don't think it's always incomplete, but I don't think virtue ethics depends on precision in the Freudian sense, right? I guess part of the way it works.

So surely you're joking Mr. Feynman should be required reading. I have actually never read that book. Maybe I should.

It's required reading. Put it this way. I cannot really imagine an information diet where placing the thing you are largely going to consume as Mr. Feynman is a mistake. It just seems like very hard for that engine. Okay.

Well, I'll put that on my list. Any strategic updates for yourself? Have you found yourself thinking about any changes in the very recent past?

I'd say I'm definitely happy with the direction that I'm taking in a broad sense. I pivoted into doing more coding for now, primarily towards accelerating. I'm just doing self-improvement, recursive self-improvement. I'm just making my writing process faster. Um, so far I'm well in the hole in terms of time spent versus time saved, but—

Me too. We should compare notes on that 'cause that, that I've been in the same spot.

But I am, but I am considerably, I, I am like noticeably more efficient going forward than I was before. I expect to win in the long run 'cause it also gives me experience I can draw upon in various ways. So it should be good. But beyond that, like there's a thing that I'm feeling like I did That I could have improved recently. It's something like, from what I can tell, the Trump transition is kind of a scramble in many places, and I should have considered the possibility it was enough of a scramble clusterfuck that I could actually, like, by preparation, have influence on decisions made. But like, my brain didn't consider that possibility until after I started getting like reports from people that they were in contact with the administration and sharing stuff and like whatever. You know, in various ways. But if I had to do it over again, I would have gotten some stuff ready that other people could like at least pass along that was in better form, like prioritization. But also like I appreciated the break, like for mental health of everyone's focused on the election. So like we had a relatively quiet 2 weeks. It's nice in its own way, aside from the election part. I hate it. But yeah, it's like. Like no matter what you want to happen, whether or not you get what you want, like the election process always sucks, right?

It's interesting. One of the other things I've been wrestling with myself is like how much to think about trying to influence central decision-making versus maybe just going off in a different direction and trying to do something good. Like it does seem that there's gonna be fewer nos, you know, in society broadly, and there's potentially going to be more vacuums if, for example, the education department is removed or whatever, or like mandates are dramatically curtailed. There might just be a lot more like freedom of action in spaces that people like me have shied away from because we just don't want to deal with the red tape and having to like work through the sort of friction that was there.

Yeah, the good scenario, right? Like the good scenario involves maybe Doge is great. Maybe they actually do cut a bunch of red tape and a bunch of regulations and do things again. Like, but the Trump that I've always dreamed of getting is the real estate developer, right? Like, happy. He'd be the ultimate YIMBY president. Like, he'd build, baby, build. Like, potentially with my name on it, potentially getting a cut. I don't care. Just build, baby, build. And we didn't see that in his first term. And maybe we'll get, maybe, maybe his true nature will win out somehow, even though he didn't campaign on it anyway, because like, now that he doesn't have to run for reelection. He can, you know, let his true nature win and have fun and eventually make money. But yeah, like it's a different timing, right? I think you should definitely keep an eye out for opportunity going forward. It's just more of a narrow window, especially where like really early the changes haven't been made. You can't seize your opportunities yet necessarily because you don't know which ones are going to be available. It's hard to prepare for them the same way. You can influence what happens in that window potentially. I don't know. But, and I introspect, like the mistake I always have made is not trying for the high leverage move because it just seemed like it would never work. Like it was just like, that's impossible. Come on. That's ridiculous. But like when you look back at the, the track record, no, it's not as ridiculous as it sounds and you should try it more often. That's true of a lot of things in life. You don't take enough big shots. Like no one does. Almost no one does.

Yeah. The, if we learn nothing else from Peter Thiel, we should learn that a good long-term conspiracy slash plan, depending on how you want to position it, sometimes can really pay off.

Yeah. I mean, I, I took my one big shot, right, with Falsa, and it didn't really work as such, but it still had some positive side effects and got me a lot of contacts and got me a lot of knowledge. And we still have some assets that like Who knows what could happen?

And that Jones Act just might one day meet its end, you know?

Look, look, Trump, you have this great opportunity. Just come out and kill this thing. Let's kill this thing. You need it. You need it for your identity score. You want to, you want to like balance that budget. That's how you do it.

Anything else on your mind before we break?

Actually, I want to pitch to the Republican side is the Trumpian pitch for repealing the Jones Act is I want to just get this out there so people can pass it along. Is reshoring. Like we want to, in this philosophy, produce things in America and sell them to Americans. Right. But to do that, you have to be able to transport the goods to produce the thing and then transport the goods they were produced to the person who wants to consume them. And if you don't do this, if you have a situation where it costs more to ship things within the United States than to ship things from outside the United States, that's like an anti-tariff, right? It's like a tariff on yourself. You don't have to have tariffs on America selling to America. And obviously that is insanely terrible, right? So we should remove that. And yes, technically speaking, we are now going to buy chips from other people, but we're buying chips in the name of producing stuff in America. And we're not replacing American production because America doesn't produce the chips. We're supplementing it. And then we have more chips, which makes us greedy. So it should be a slam dunk pitch. It should be intuitive. It should be obvious. Remember, all the unions love it. Unions should hate it. Remember that, like, all the blue states, all these people, you know what to do. Just follow through and then do the Dredge Act, Maritime Passengers Act. I'll be happy. All right.

I'm sure those influential administration members who are still with us almost 3 hours in will be acting quickly.

Any— I'm kind of hoping for a forwarding. Yeah, but yes.

Any other closing thoughts before we break or calls to action?

I mean, you know, I think it's, we're, yeah, we're 3 hours in and I, I've covered a lot of things, but decide what you think is the most valuable thing to do and do that. Right. And I'm not gonna try and tell you what it is.

The rest is left as an exercise for the reader. Zvi Mashrut, thank you again for being part of the Cognitive Revolution.

All right. You're right.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.