AI News Crossover: A Candid Chat with Liron Shapira of Doom Debates

Hello and welcome back to The Cognitive Revolution. Today we're sharing a crossover episode that I recently did with Laron Shapiro, creator of Doom Debates, who you may remember from his interview with OpenAI's Roon, which we cross-posted late last year, and which I still think, by the way, is very much worth listening to.

To be clear from the jump, today's episode is not a debate.

Laron did challenge me early in the conversation, to be a bit more, shall we say, consistently candid about my genuine sense that almost anything could happen, from the unimaginably positive to the devastatingly negative, as we develop and deploy more and more powerful AI systems across society. I appreciated that challenge, and as you might have noticed in recent episode introductions, I have adjusted my approach to make sure that I at least mention the incredibly high stakes of advanced AI development at some point in every episode. Beyond that though, this is a news and analysis episode in which the two of us compare notes on some of the most recent AI releases and discourse that we each found most interesting. Among other topics, we discuss GPT-4o image generation and what it means for graphic designers and related businesses like Fiverr and my own Waymark. Replit CEO Amjad Masad's declaration that he no longer thinks you should learn how to code. Peter Lovell's provocative claim that being an entrepreneur now has more job security than a job. Gary Marcus, who ridiculed OpenAI's latest $300 billion valuation, and why, though his AI denialism is now totally indefensible, his question as to whether or not OpenAI will ever actually pay off for investors remains a good one. Emmett Shear's new AI safety organisation, Softmax, their organic alignment strategy, and how that fits into the overall alignment and AI safety big picture. A viral AI not kill everyoneism tweet about the human-caused 6th mass extinction event on the planet Earth, and what we should infer from that as we consider our own AI future. Anthropic's new mechanistic interpretability paper, which I found characteristically outstanding, but also honestly kind of worry has been overstated in the understanding it actually achieves, such that it could perhaps be used perniciously in political debates going forward. And finally, how much hope we each have that effective international cooperation can be achieved either via some sort of emergent stable equilibrium, Or via AI governance treaties, which would be backed by monitoring and verification regimes. I had a lot of fun recording this, and I'm very interested to learn how much value you find in this format. Would you like to see more of these rapid response news and analysis discussions, or do you get enough AI news elsewhere such that you'd prefer that I stay focused on deep dive interviews and analyses? Your feedback will help shape the future of the feed, so please let me know what you think, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. For now, I hope you enjoy this conversation with Laron Shapiro, creator of Doom Debates, as we break down some of the most interesting recent developments in AI.

All right, let's do some news and social media. Connor Ley finally got a haircut. So this is Connor Ley with a haircut. This is what he normally looks like. He looks really good. I think this new hairstyle is working. It's a stock wet from a peak of around $150. Down to 104.05, so I'm embarrassed to say I went long when it was up 150. Like I bought some options. Like I'm I'm kind of destroyed here. How about you? Welcome to Doom Debates. Today I've got a very special episode. It's a collab with Nathan LeBenz from the Cognitive Revolution. Nate and I actually go back all the way to 2010 when we were both living in Silicon Valley. We were both doing startups, and he was recently an undergraduate in chemistry at Harvard. That's his education background, even though he later went into Silicon Valley startups. As you know, Nate hosts the popular Cognitive Revolution podcast, which is a highly educational podcast. It's one of the deepest dives that you can regularly see interviews with people who are hands-on deep in the field. all kinds of AI engineers and safety researchers, diverse topics and perspectives, and it's driven by Nate's own interests and curiosity. So it's a very authentic podcast, highly recommend it. Nate is also a software engineer. He's the founder of Waymark, which is a turnkey solution for small businesses to generate video ads. It's known as one of the first companies to integrate GPT-3 when nobody was even paying attention to that technology. They currently offer a watch-first experience where you can easily generate a candidate video, video ad, and then watch it and iterate. You can check out Waymark in the show notes. Nathan Labenz, welcome to Doom Debates.

Great to be here.

So Nate, you and I were talking about, you know, how do we do a collab episode? Because oftentimes I like to clash with the guests and, you know, push back on where the guests and I disagree. But I think for the most part, you and I are analyzing the situation with a lot in common. Like, I'm sure we have minor disagreements, but it's not like a fierce debate type of situation. So we were thinking, you know, what can we possibly do? And then we're like, oh, What if we just both sit on the same side of the table and look at all this, all these other tweets and all these news articles, and then we both chat about that, right?

Yeah, I think it should be fun. I do, you know, I take all the issues that you cover regularly very seriously myself. So I think it's really good and healthy that there are, you know, people coming at it with different, you know, kind of tone and tenor. Mine is a little bit more, Maybe dry. Yours is more gallows humor. And I think both of those are, you know, are very valid and worthwhile contributions to make to the overall discourse. I'm maybe a little less, I mean, and I know there's a big question coming up. I think I'm maybe a little less confident in my, you know, downside worries about the overall AI situation than you are, but by no means am I like, denying or downplaying the risk. It does seem like very, very substantial to me. Just, I wouldn't go quite as far as feeling like it's the overwhelmingly likely scenario that we'll end up in some doom situation.

Okay. Okay. Thanks for the clarification. So Nate, you're originally from Michigan, right? Yep.

And I'm in Michigan now.

Nice. Okay. So you basically had like a half decade in Silicon Valley.

Yeah. 10 years away from Detroit. And then honestly, I never really thought I would come back, but, you know, topic for another podcast maybe, but there was a revitalization effort of Detroit that included investment in startups. And I had a startup. My co-founder was also from Detroit. My wife is also from Detroit. And it was kind of a contrarian move at the time to leave Silicon Valley, move to Detroit and build a startup here. Um, but that's what we decided to do and we are here to this day.

Nice. Nice. Yeah. So you moved to Silicon Valley and you really created the ultimate trifecta here because you became a software engineer, entrepreneur, and podcaster. Doesn't get any better than those 3.

Yeah, I would say, um, uh, you know, and only modestly, uh, competent in all 3. I, I would say I'm, uh, and my teammates would certainly sign off on the idea that I've always been a sort of vibe coder kind of software engineer, like much more a proof of concept guy, like show that things can be made to work and mostly leave the sort of production grade engineering work to others. And, you know, I guess all entrepreneurs are to some degree flying by the seat of their pants, so I shouldn't feel too bad about that. Then when it comes to podcasting, I sometimes describe myself as the Forrest Gump of AI because Just so many times in my life in a not very strategic way, I've ended up stumbling through these like important AI moments and scenes. Um, and I'm, I've always been the sort of extra, you know, background character. Um, but it, it really has been amazing how many times that's happened.

All right. Before we move into our, uh, news and social media roundup, are you ready for the most important question about you that everyone wants to know?

Yes, I, I am ready.

Pee doom, pee doom, what's your pee doom? What's your pee doom? What's your pee doom? Nathan Labenz, what is your P-doom?

Well, I don't know if it's a cop-out to give a range, but what I usually answer when people ask this in general is 10 to 90%. And what I mean by that is basically like, I really have no idea. I think it's, you know, very uncertain. I would put most of the weight on the lower end of that range and, you know, kind of fall in line with like, Okay.

I haven't heard that one before. 10 to 90, but most of the weight on the lower range. That's an interesting way to describe it. Yeah.

And I don't really know why I'm saying that other than I'm kind of following folks like Dario who says 20%. And, you know, it, it seems like there's a decent amount of things going right. And, you know, we are talking about like a pretty extreme outcome.

So. Did you hear Dario recently walked it back on, on, I forgot which recent interview he was on, but he basically walked back the 2023 Logan Bartlett statement where he's like, you know, I never really said like 10 to 25% chance of doom. I was just talking about like a big shakeup for civilization. Like he was like walking it back.

Well, they've changed their messaging on a lot of things recently. And, um, I don't love it. I, um, it feels to me like strategic communication, honestly. And, uh, the, my candid take on that is I've lost a little trust in Dario as a result. And I wish that weren't true. I would love, you know, I, I wanted, uh, to believe in like, A, you know, squeaky clean good guy. And I'm, you know, I'm just feeling like the level of strategicness in the strategy in these communications recently, you know, with he who must always be named presumably in mind as, you know, center of the target audience for some of these statements.

Yeah.

You know, I just don't, Like that. And it does make me sort of trust the situation or trust what we're hearing from Anthropic leadership less than I used to. Not that much less. They're still doing a lot of great work and they're going to show up in our rundown. So I still hold Anthropic in general in very high esteem, but a little less than I used to because I don't feel like some of these walkbacks or about-faces are Uh, well, first of all, they're just not like properly explained. And, you know, in the absence of a, you know, proper explanation, it feels like they're more sort of strategic and kind of memory holing, um, you know, things that I think were sincere in the past, uh, rather than, you know, an, an like honest change of perspective.

Yeah, that's a great segue into some of the tweets we have bookmarked. So I think we're going to elaborate on this theme because I agree with you.

Uh, one more, one more thing I'll give you on P I do know too, one of the best things that anybody ever said to me, it was my friend Gopal who said, we should think less about what the probabilities are and more about what we can shift them to. And so I do think, you know, a big part of the reason we have this 10 to 90%, in my view, is there is probably some amount of risk that is an irreducible fact about developing this sort of technology. At all. And, you know, I do think like the overhang arguments of if you have web-scale compute and you have web-scale data, you know, somebody's going to figure out an algorithm at some point. So like we live in a world in, in sort of a pretty, you know, wide range of timelines around this specific timeline where like powerful AI is probably going to happen. And so given that, you know, there's probably some pretty irreducible risk. But then there's also the like really stupid risk that, you know, in theory at least would be like very avoidable. And that is the like, let's weaponize this, you know, technology as fast and as aggressively as possible and like have a superpower race to global dominance. And, you know, that might be not just like a little bit more risk, but like that literally could be like a 10x multiple of the baseline risk. And to the degree that I feel like I can do anything, it would hopefully be to sort of try to tamp that part down. So, you know, that's, if I, if I have any contribution or any, you know, any impact, that's kind of what I hope it will be.

Now, this particular scenario that you're calling the stupid scenario seems like the closest one to reality.

Yeah, that is the, I think that's unfortunately currently the path that we're on. And hopefully we'll get off it. But, you know, it's right now, Nobody's super, you know, few people seem to be inclined to receive that message just yet.

So just to recap what you're saying about your position. So it's 10 to 90%, waiting a little bit on the lower end. And I always say I'm 50%, but I also like that expression 10 to 90% because it just shows, look, the significant figures are low. Like when you say 10 to 90%, it's like, I don't even have one significant figure here, right? Like the first digit is open to slide. But I also think it would be ridiculous to be like, my P-doom is less than 1%. I think anybody who says that is being dumb. And And the fact that I think that kind of implies that my P-doom is more in the 10s. So there is some signal here. There is some information about what I think is justified to believe that it's like in this double-digit section. But what I wanted to say about your position is it's kind of similar to mine, let's say a little bit lower because you're weighting it toward the low end. But if people listen to your communications, right, if they listen to The Cognitive Revolution, I don't feel like that is a podcast where you're communicating that P-doom is 10 to 90%.

Yeah.

You know, it's why I say, hey, you listen to Doom Debates, it's one of the only 2 podcasts that are communicating that P-doom is high. So my question for you is, don't you think you should communicate more explicitly that P-doom is high?

Maybe.

I, you know, I haven't been too strategic in my approach, but if there's been any strategy, it's to be sincere about both my, you know, very genuine enthusiasm for the upside of AI, as well as my, you know, very real fear of how it could all go very badly wrong. And so I do try to communicate both of those things. I guess maybe I'm just like dispositionally not, you know, super emotional and maybe a little bit afraid of being wrong. And so kind of conservative in my positioning broadly. Um, but I do try to represent both of those.

takes.

And, you know, I feel like in almost every episode there's sort of this like, the future is like dramatically uncertain, at least, you know, kind of vibe. Um, yeah, I mean, I think it, you know, it is, you're, I think it's a good question and it does, it is worthwhile, I think, to hold oneself to account to the idea that like, have I gotten used to, you know, a 10 to 20, you know, percent kind of Even if I consider 90 to be sort of a tail risk, like, am I, I'm trying to be the one to sort of wake the public up about a lot of things, good and bad. Have I kind of allowed myself to become the boiled frog on the bad side? And, you know, should I be more, you know, shrill in my statements? I do try to reevaluate that, you know, Periodically. And I do think there's like possibility in the future that I could reposition myself as more of an advocate and less of kind of a, you know, today I would say I maybe present as like a neutral analyst. And neutral does mean like, you know, recognizing and appreciating the upside for what it is. And I do think what it is is tremendous. But yeah, I don't know. I think it's a, it's a certainly not unreasonable and, and maybe, um, you know, shouldn't, shouldn't motivate a little bit of a different tone in my, um, in some of my commentary. I do have a ton of episodes on like all the bad behaviors. So I think that's one thing that I'm also kind of watching very closely is like, just how bad are the bad behaviors and at what threshold levels, um, You know, should I really start to be, I guess my, there's another distinction too between like what is sort of theoretically appropriate to consider to be like a, you know, real possibility. And there I again have like, you know, very open mind versus like what I can sort of credibly warn people about.

Yeah.

Uh, and so I currently have this slide deck of AI bad behaviors, which is growing rapidly that documents all these you know, different studies, many of which have, you know, have been done by guests on the show. Scheming and, you know, deceptive alignment and reward hacking and even like, you know, reward hacking in the wild. You, I'm sure you saw the Sakana, um, CUDA engineer thing where it was like, you know, we deeply apologize, but unfortunately our AI CUDA engineer does not actually, uh, write, you know, CUDA code that's that much more performant, but in fact it reward hacked our system. Um, you know, that was done. I mean, that's crazy. That's starting to be the level where I think decision makers actually should start to take it pretty seriously. And so that is one that I'm like, you know, this could happen to you guys. You know, like this is not that a researcher went and set up a situation and found some tendency under certain circumstances. This is like a well-funded AI company published a project. that was fundamentally totally flawed due to reward hacking. Like, you know, yeah, we are starting to hit some of these milestones where the yikes factor and the sort of like, there's no denying the reality of these things is like starting to get more real. And as that continues to happen, I do think I'll probably like get more, you know, my, the alarmism in my tone probably will go up. as well.

Nice, nice. I appreciate that you're even being reflective and entertaining the question because I'm definitely not trying to put you on the spot or single you out. On the contrary, if I had to list, hey, out of all the podcasts out there, which podcast has a host who has a totally reasonable, sane P/DoM, like what you just said, that's way into the sane zone. For me, the sane zone is 10 to 90%, any number 10 to 90%. Hell, even 5 to 95%, it's reasonable, right? It's just crazy to me that some people will go lower than 5 or higher than 95. That's unreasonable to me. So I would actually put you at the top of hosts who are sane about P-doom. And you never do low blows, right? Some hosts of some podcasts have, and some guests I've definitely seen dismiss the question so glibly, like, I'm not a doomer. You can't just be a doomer. Doom is bad. Very quick to dismiss it. You're not like that at all, right? You're actually kind of one of the top. And yet at the same time, if you just have a random technical listener wandering into the space, listens to like 10 episodes of your podcast, I'm just not sure they're taking away the message of like how doomed we might be.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, maybe I should put a little tag on it or something. You know, I have like a little audio outro that we append every episode that just invites feedback and thanks people for listening or something. I could imagine adding just to that, like, Um, you know, just to reiterate, uh, as a PSA, you know, and I might attach a Planet Tokyo to that as well. Well, I think, you know, just reminding, I do think it's, it is, it is a good, and I think people are also like definitely picking and choosing between episodes because I do 2 a week typically. And, you know, that, yeah, I'm well aware that that's like more than, you know, the average person is going to have time for. Um, and I do recommend people like follow a diversity of feeds and voices. You know, I don't think, uh, you know, people should over-index on my perspective on AI by any means. So it is an unfortunate situation if people, you know, pick and choose episodes and they sort of only hear one side of my, uh, overall outlook and, you know, don't take the other part on board. So maybe a, you know, kind of consistent reminder that like, this is sort of how I see things. And I might put a P utopia on there too, or like a P, you know, post-scarcity world of abundance, whatever. I also think that's pretty high. And I tend to think it's probably more likely than doom. Why? I can't really justify that, that maybe it's just my disposition, you know, more than like a real rigorous synthesis of all the evidence. But nevertheless, that is like gut feel how I feel. But I do think we're like rolling the dice, you know, and for me, you know, if you're, if it's only, you know, one of 6 chambers, Russian roulette style or whatever, like I would still be like very terrified to play that game. And I think, you know, people should have a better sense that that is kind of the game we're playing. So. Maybe, um, you know, maybe I should add a little, uh, message, a little token message in on every episode or something like that.

Exactly. And I don't know what would work best for you, but just as a brainstorm suggestion, you could always just start the show every episode by being like, hey, welcome to the Cognitive Revolution. I just want to let you guys know that I think AI is very likely to bring about utopia, but there's also like a 30% chance that it'll literally make humanity go extinct in our lifetimes. I just want you guys to know that I think that. All right, moving on.

Yeah, the stakes, um, the stakes really couldn't be higher, honestly. I mean, I do sincerely believe that. So, um, yeah, I can't, uh, I'll think about the right forum to do that, but I think there is something that could be quite socially—

You seem to be sympathetic to the cause of like, you know, why I'm asking you this, which is just because like, I think the Overton window could use more moving, right? Like, I know there's LessWrong, there's Effective Altruists. Like, I know this is now like in the water and it's like a fun punching bag, right? People know to like bring it up when they want to like go to the extreme, like, oh my God, Eliezer Yudkowsky thinks we're all going to die. So I think the Overton window could use more moving where people like you who frankly have like a very middle of the road, like mass appeal podcast still make it clear that we're like pretty likely doomed.

Yeah. And I'm even, I am pretty sympathetic even to You know, outright protest movements. You know, people have recently, um, while I was in the Bay Area not too long ago, back in February, folks like chained themselves to the OpenAI, uh, you know, front door or whatever and got arrested. And I, you know, that's not going to be my strategy. And I wouldn't necessarily say I like endorse it, but I do think That, that's like definitely not too radical, you know, of an action for some people to take given the situation. I think like, you know, if there's, I think we talked about this one time, you know, offline too, like how to frame these questions. I think it is often interesting to, before asking people what they think the risk is, Ask them what risk they think we should be willing to accept and sort of say, you know, okay, you're excited about AI. What, you know, extreme risk, uh, what percentage chance of truly like extreme risk do you think humanity should tolerate as we develop AI? I think most people would give something like under 1%, you know, for an answer there. Online, you do hear people more and more that are like, well, we're fucked with climate change or whatever anyway, or we're going to have World War III unless we get AI to somehow make peace. So they might accept something higher, but most people will come in pretty low.

Yeah, I'm pretty willing to go to 10% just because a lot of the arguments people bring up of like, look, because when you factor in the cost of coordination, right? If there's a lot of problems with trying to slow it down or the alternative has a lot of problems, then a Hail Mary where it's like, look, it's a 90% chance of success. Yeah, 10% chance we all go to hell, but 90% This is kind of like one of the few humps that we have to get over. So there's not going to be that many of these in a row. So gambling everything on a 90% chance, to me, that's plausible. And Eliezer Yudkowsky said something similar, like he's not opposed to like a 90/10 type of gamble or even a 50/50 type of gamble from his perspective. He even thinks that those odds are good. For him, the problem is just that P is like even higher.

Yeah. And I mean, the public though, like broadly, if they actually knew that, you know, many people in the industry see it this way and are willing to take those kinds of risks, like would in fact, you know, be motivated to go, you know, protest in person at their offices. And so I think that's like, you know, in a sense, a pretty normal and probably should be expected reaction from people who are just like hearing for the first time that, uh, wait, you're telling me that you think this and you think that's okay. And you're like planning to go ahead and do it anyway. Like, Um, it is, it is a pretty wild situation, no doubt about it.

Yep, yep, yep, yep. Okay. Um, great. Well, thanks for indulging me and being such a good sport. You know, when I, when I'm asking about communicating PDoM, I thought you were super nice and introspective. So thanks.

My pleasure.

All right, let's do some news and social media.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Here we go. First tweet. This is, this is a good one to get you energized here. Sam Altman. He says, tremendous alpha with images in ChatGPT right now. I thought this would be an exciting way to start because I've been using ChatGPT-4o images and gotta say they cooked.

Yeah, it's really good.

No doubt about it. Um, it's, you know, one really interesting thing about Waymark that people are often surprised by is to date, we have not integrated any AI image generation. And the reason for that is our customers are typically small business. We partner with like media companies, whatever. If you go to our website, you'll see like all these cable companies, but they in turn are selling the creative that our technology creates to local advertisers. And these local businesses, they want things that really represent them. So they have images, they have stuff on their website. We like pull that stuff in. We use a lot of computer, um, well, it used to be called computer vision. These days it's like asking Claude or GPT-4o Mini or, you know, uh, Gemini Flash to like choose which images are appropriate, you know, for a given piece that we're creating. But we use their images because they want to look like themselves. And it's been really hard to use any sort of text prompting or even sort of image and text that they just haven't been good enough to create something that sort of has this different style or like, you know, opens up the space of creative possibility while still being true to who they are so that when people actually show up at their like typically physical local business, um, you know, that they feel like what they saw on TV, you know, matches what they're getting in person. And this does feel like the one that crosses that threshold. They haven't put out an API yet. So we haven't been able to integrate it, but reportedly that's coming soon. So I think this probably will be the first one. And, you know, it'll be up to the Waymark creative team to determine like what sort of motifs or, you know, strategies we'll use. But you can see with all this, you know, stuff that is going on online that you can kind of project yourself into these other, you know, creative spaces and I think that is going to be really exciting. It's going to be a big unlock for our product.

Yeah. Does this kind of compete with Waymark? Because I've actually been experimenting with generating a bunch of Facebook ads that are just images. And if I have a really good image, I feel like I don't need a video. Yeah.

I mean, it's coming for all of us eventually, I think. And, um, I think this one not quite Yet, certainly not for like, actually a lot of our business is TV. So it's like 15 and 30. Ah, I can make a buck.

Got it. Got it.

Yeah.

And so you need, you need another year of development basically to compete with TV.

The presumption also is that it's, um, a sound-on environment. So there's a voice component to it as well. We do voiceover, you know, as a native part of every generation. And by the way, that's also gotten really good recently. So yeah, not quite yet does it substitute for us, but I do think, you know, one of the big existential risks at the scale of Waymark is that, you know, that you can just prompt your way to something in ChatGPT in the, you know, not that distant future. It's not that hard to imagine that.

Right, right, right.

So, yeah, it's not, uh, I don't think it's a wrong thing to be somewhat concerned about for us, for sure.

When I ruminate on Sam's tweet that there's tremendous alpha with images in ChatGPT, I think I've identified a type of alpha there is, you know, looking at my own company, uh, Relationship Hero. Um, when we're just putting up our own Facebook ads, I had the realization that like we could work with the most expensive professional marketing firm in the world. And if we ask them for an image ad deliverable, I don't think it's going to be better than what a few prompts at ChatGPT-4o is going to give. I think we're maxing out the quality of image ads here.

Yeah, very plausibly. Um, you know, I don't, and how much of the space You know, of all possible image space is sort of not yet explored by humans such that it's not in the dataset such that, you know, you would really have to be a, you know, true novel creative to go somewhere that nobody—

Right, right, right. And, you know, and I'm not saying any format is automated, but I think specifically for ads, right, where people look at the ad, they don't judge like the artistic quality. They're just like, did this catch my eye? Did this deliver the message? So all these imperfections, right, like the 6th finger or whatever, doesn't really matter.

Yeah, no, I think in most cases it will satisfy us and certainly on an ROI basis. And also like, you know, for anybody who's done substantial digital advertising, you know, it's hard to predict what's going to win when you actually make a bunch of variations and, you know, let the algorithm figure out what's resonant.

So. Right, right. I was going to mention that, right? So a bunch of variations. So that's what I'm saying is like, so when Sam's talking about tremendous alpha, I think one of the types of alpha now is you have quality and quantity. So you could just generate 100 high quality image ads in like a short afternoon. You put, and And then you combine that with Meta's ad network, right? Which is like this ultimate network for evaluating how much engagement or how much conversion a particular ad will get you. And suddenly you have this almost automated pipeline, right? Where it's like high-quality creative gets signal on what it's worth. And then within like a couple days, you use like the evolutionary process, maybe you spend like $100 total testing all these different creatives. And now you have like this insanely good, like best possible ad that there is.

Yeah.

And I think it won't probably be too long before Meta will have their own version of that. That will literally just work in the, you know, in the ad creative workflow.

Right. It's crazy. That's so, yeah, but I'll report back. I can't say we have data yet to show that this has actually worked for my company, but I did make the observation of like, look, either this will work or nothing will work because, you know, typically we do advertise on Facebook, Instagram, and we get like a trickle of traffic. We've never gotten like a ton of traffic. So I told my team, I'm like, listen, this is the final test. Like if this doesn't work, it just means we can give up on met ads.

Yeah, I think that seems probably right. I mean, good, fast, and cheap, you know, you can get all 3 now. So that is, uh, right. That does change things. I don't know what that means for like Adobe. I don't know what that means for, you know, the creative workforce more generally, but it is, um, you know, maybe the beginning. I mean, one story I can tell you from Waymark, this is a different modality, but Um, we used to do, before text-to-speech, we had professional voiceover as an optional add-on service. People wanted it, but there was just no way to do it in product. So we just had it as sort of a, if you want this, we can like, you know, facilitate it for you, but it's an extra cost and you got to wait like 2 days for it to be turned around. And then maybe there's like back and forth. And, you know, we worked with like a really good voiceover, uh, specialist that, you know, provided great value at honestly a great price point and was like always very well reviewed. But the volume that we're sending to that, uh, service provider today has dropped by more than 90% now that we can generate, you know, probably still not quite as good. You know, this is kind of the classic definition of technology disruption, right? We're offering something now that is probably still inferior, uh, most of the time, maybe all the time. Like, again, the service provider that we worked with was really good. Uh, and I remember having a conversation with him 2 years ago and he was like, I'm worried about my, you know, my future. And I was like, I think we should all be worried about our future. So, you know, it's not just you, don't feel too bad. Um, but it is maybe you sooner than some others. And, you know, sure enough, Even though it's still not quite at the level that, you know, he and they used to provide, this is one main guy and a team that, you know, kind of supported him. You know, it's just, it's immediate, you know, people can hear it right away. They determine it sounds good enough. It's included for free because it only costs us, you know, a couple pennies, you know, versus it was $99 was the, you know, the human price point. And that was, by the way, considered to be a very good, like, price point that we were on. And we made no margin on that. We charged $99, passed it entirely through. Um, we just wanted to provide the most value we could to customers to meet this need. But, you know, it's really hard to argue with free compared to, or functionally free compared to $100 combined with, you know, the speed of turnaround and the ability to just edit right there and get it done and be onto your next thing. Um, and so, yeah, the volume has driven, has dropped by More than 90%. And, you know, does that same thing now come to like graphic design broadly? I think it very well could. I don't know, you know, why it wouldn't to some, you know, maybe not order of magnitude, you know, in the next few months, but like hard to see how it doesn't happen something like that. Um, you know, on, you know, maybe one more, uh, evolution of the model kind of timescale.

Yeah. So you were impressed with this screenshot. Riley Goodside tweeted the screenshot, a fake screenshot generated by ChatGPT-4o of a Wikipedia article about the screenshot itself with a copy of the screenshot in the article. So for people listening on the audio, it looks like a totally authentic Wikipedia page, but it's just a rendered image. It's got, you know, the sidebar or the title that says the screenshot, self-referential image. And then within the article is like an image of an entire Wikipedia article, right? So it's one of those recursive images, right? It's like the image inside the image. And what's crazy is this was done just typing in a prompt and then immediately just getting this entire recursive image of a Wikipedia article rendered, complete with a bunch of text in it too, like description. The screenshot is an image depicting a screenshot of a Wikipedia article titled the screenshot. So GPT-4o was able to combine all of these elements on the first render, like the second, and you were quite impressed by this.

Yeah, I mean, I think everybody should be, you know, if you're not impressed by this, I'd like to know why. It's, um, I mean, by the way, I think everybody should follow Riley on Twitter. He is consistently an outstanding, uh, demonstrator of, of new model capabilities. He was, um, at one point, maybe the world's, uh, first and only staff prompt engineer was his original title, uh, at Scale AI, which he joined maybe 2 years ago now. A job that he literally tweeted his way into by just, you know, providing one example after another of like what—

Right. To be fair, I think he said this took him a few prompts, right? But even the fact that one reasonable prompt will generate this is insane. Yeah.

I mean, he may have iterated a little bit, but in the end it is kind of, um, you know, it's a single shot. You know, the thing outputs an image, right? And this is the image that it output.

By the way, I zoomed all the way in because I'm like, wait, if it's infinitely recursive, it looks like there's only like maybe 3 levels of recursion. So I zoomed all the way in and in the innermost level, it doesn't have like another picture. It just has like a yellow block of text that you can't read. Funny enough.

Yeah, it shows you kind of the, I mean, there's a bunch of little, when you really start to inspect this closely, you see a bunch of little issues. You know, there's like a bunch of words that are not words and there's words that are like spelled wrong. And, um, so I think it's kind of charming in, in that respect as well.

Yeah.

Yeah.

The deeper you go into the screenshot, the more the words are just like not words, but there's some non-words in the outer screenshot too. Anyway, it's It's quite interesting. I mean, it's, it's filed this under the category of like, if this, if we knew that this was our future going back in the past 5 years ago, we'd be like, how the hell could this ever happen?

Yeah, absolutely. I mean, it shows obviously real depth of understanding and it also, I think, shows the power. And this is one of the ideas that I'm kind of chewing on a lot right now of deep integration of modalities. I think a huge question, this has like kind of become a little bit of a stump speech for me, is, you know, what does superintelligence look like? It's hard to predict, right? I mean, it's, it's, it's sort of, I think for many people, a very vague notion. Um, and I wouldn't say I have like a super crisp answer, but I think one candidate answer is you take a frontier model of the sort that we are familiar with today, and you integrate in a similarly deep way a bunch of different modalities instead of, you know, just the, you know, the image modality that we're seeing demonstrated here. Like, imagine if instead of the text and image being so deeply integrated, you had text and interaction of biological molecules so deeply integrated, or, you know, predictions about the evolution of a cell, like the next transcriptome state of a cell. We have seen, and there are plenty of narrow models that can do wondrous things, right? AlphaFold and, you know, many, many, many more that are developing a sort of intuition. Sometimes they call it intuitive physics. for all these other problem spaces such that, you know, they can take a DNA sequence or a protein sequence and predict how these things will fold up in 3D space, which like people can't do, um, or even predict how they're, you know, they're going to form into a complex or how they're going to interact or how they'll sort of, um, cohere together around some metal, you know, ion at the center. Like it's getting really quite amazing what these things can just sort of spit out from a little bit of—

Totally.

data. But right now, at most, an AI can call that as a tool, you know, can make like an API call. And this is sort of analogous to what we used to have in ChatGPT where, you know, the language model could call DALL-E with a prompt. Okay, the user has asked for an image of this. I'll write this long prompt and try to capture what the user wants and get it back. But, you know, as we just talked about at Waymark, like, it never looked like them, you know, it was never really quite what they wanted. There was this super lossy bottleneck, you know, due to this sort of arm's length, uh, API call or tool call type structure between the language model and the specialist model that just really limited what you could get out of it. And now you see this integration and an explosion of possibility. And I think one good candidate, or at least one, you know, mental model I found super helpful for what a superintelligence might look like. is do that again for 20 more modalities. Um, most of which we don't have, right? We do still have the image modality. Not all of us can draw at this level, but we can at least kind of visualize and we know like what's right and wrong when we see it. But when we get into sort of how is a cell going to respond to a certain perturbation, you know, we're starting to have models that can do that. We can call them as tools and interact with them in this sort of you know, iterative way, but we have not seen the latent space deep integration mind meld between a sort of broader reasoning system and one of those narrow systems.

To give some context for the audience, you're talking about this because GPT-4o is actually a breakthrough in integrating, like, it's not just drawing like a diffusion model where it's saying like, what is this pixel likely to be once I do it at like a finer grain resolution? It's more like it's somehow simultaneously thinking about the prompt using GPT-4o. It's thinking about the prompt. It's thinking about the text separately, Because they like made some improvements somehow to text rendering by actually like understanding how to make exact text shapes. Like we don't really know what they did because it's a secret, but we know that it's not just like naively predicting image. It's like drawing and thinking.

Yeah, there've been some interesting, we could even pull a couple of these up, but you know, they demoed this almost a year ago, maybe even a little more than a year ago now for the first time. And Greg Brockman put out a tweet showing a person at a blackboard with handwriting. And this was, this image was generated by GPT-4o. And the handwriting on the blackboard is like, what if we model text plus image plus audio all jointly? And so there's, you know, there's some deep integration that is happening where there's like a shared latent space and, you know, the, whether an idea is presented to the model in text form or in image form or in audio form, it's all converging into some shared space of understanding that is much higher dimensionality and allows for like much richer communication across these modalities than a simple like prompt to, you know, an image generation model, uh, you know, previously did.

Yeah. So then I tweeted a couple days after GPT-4o, I just said Fiverr stock holding steady this week, which it's still holding steady last I checked. So what do you think of that? I mean, it's already gone down a lot. It's already down to like an $800 million market cap. I think it was, it used to be $10 billion plus. Don't you think it should fall farther when GPT-4o comes out?

Well, it's going to be a challenge. I mean, I have, um, maybe too early for me to say, I have a, uh, episode coming up next week. I should be recording with the CEO of Fiverr. And, you know, they're not sitting on the sidelines of AI by any means. You know, they are, um, basically every aspect of their business they're trying to reimagine with AI, including like how the service providers sign up and present themselves. You know, they, they provide AI gig support for you to define your services. Um, they have, you know, similar things on the buy side, like to help you flesh out your requirements and make sure that you're like, Actually, you know, properly asking for what you need. Um, I assume that they have a lot of AI matchmaking going on behind the scenes to try to, you know, grease the wheels of the marketplace.

Sure. Oh yeah. I mean, and it's already kind of low valuation already, right? So people are just saying, look, as long as there's some chance that humans will be in the loop somehow, then this business will have some value and it's already low. So, I mean, it makes sense. Yeah.

They have an interesting thing called Fiverr Go too that I have just been exploring a little bit, which is like, It'll be interesting to see how people take to this, but it's sort of, um, things like you're a voiceover artist, they'll clone your voice and then allow you to provide an AI version of your voice where like a human, you know, gets properly paid. And, you know, some obviously for creators that is like upside, right? Their goal is to, you know, make the creators indispensable. I think that's how they put it. So, you know, will people pay though? You know, the ratio between my, like, original human voiceover at $99 and the, like, 3 cents or whatever from ElevenLabs leaves a lot of room. How much more will people be willing to pay? And, like, is that going to be sustainable to know that, like, some underlying human was, you know, in some sense justly compensated?

I don't know.

That remains to be seen, but.

I will say this, I went to Fiverr to try to get a YouTube thumbnail for an episode of Dunubates, and I also went to 99designs, and I got okay work, and it was like, it took my attention, right, to like review the submissions and talk to them, and now I don't think I'll ever go back from GPT-4o and, you know, maybe some tweaking in Photoshop, but like, I'm pretty sure I'm done with Fiverr for that particular job.

Yeah, I think a lot of it also is just arbitrage. You know, one of the, I recently gave a talk to a bunch of students about the the possibility of working as an AI scout. And one of the things I told them is, um, you know, don't apply for grant funding if you want to be an AI scout. Instead, you should live off the digital land. And what I mean by that is like, you know, AI can do just straight away a lot of the jobs that get posted on Fiverr and, um, you know, Upwork too and whatever. And so like, why is that happening? It's because why are people still posting those jobs? Like, they don't know, they don't know how to use these tools. They don't know where to go. Um, but if you do and you can go just use AI to deliver quality work, um, you know, I think, I think prices will come down. It's definitely like a deflationary phenomenon, but at least for the foreseeable future, as like the options in AI are overwhelming, it's going to be worth it to a lot of people just to like pay somebody who knows what the right tool is to use. Like that's a lot of the work that I do commercially. It's, I'm not like, sometimes I like come up with a, you know, really creative or insightful solution. I like to flatter myself, but, you know, more often it's like, I just have a pretty, you know, as close to comprehensive as anyone can maintain these days. Again, if I'll flatter myself, sense of like what the tools are. And that's really what they're paying me for is just like my knowledge of what is the right tool to use. And then, you know, they use it and then they can sort of feel confident that like, They got the best available AI option. So, you know, a lot of what goes on on Fiverr might shift toward that. Uh, but there's at least, you know, probably still some volume there for a while.

Exactly. Now, when you bring up that you're such an expert at the tools, you know which tool to use, I totally agree. And I think people should hire you for your expertise based on that if you're, if you're doing any consulting. Um, but it also reminds me of, uh, you know, it's like, how long will this last where expertise in the tools is going to be a competitive advantage? Probably not that long, right?

Yeah, no, I agree. I mean, it's coming for all of us.

Yeah, Naval tweeted yesterday this image where in his mind, he's like, you know, that classic image with the astronaut where there's an astronaut shooting the Earth and Earth is labeled jobs and the astronaut with the gun says AI, like AI is killing jobs. And then there's another astronaut behind him saying AI jobs, right? So you can get a job where you can like work the AI and that'll actually defeat the AI. Like AI jobs gets to shoot the AI. But it seems to me like you want to add another astronaut behind AI jobs saying AI again, right? Because then AI will just learn how to do the AI jobs.

Yeah, I totally agree. I mean, I think that, you know, I hope that what I'm doing, first of all, there's a question of timeline. So like, you know, exactly how fast does that happen? Um, I tend to talk about the shorter range of the timelines that I actually believe. Um, so if you said like, you know, when does transformative AI arrive for some definition, I might say, You know, 2 to 5 years might be my like 80% confidence range, but I usually kind of try to think and talk more about the 2 years just because that seems like, you know, better to put that time pressure on myself than, you know, and play for that. And then if I have more time, like, great. Um, I hope that what I'm doing, and I hope that, and I think that what like other AI scout type people would do over these next couple of years is, you know, is an AI job, um, that might be really socially valuable because I think we just have a lot of work to do to like characterize the AIs, you know, and just know what's going on with them broadly. So I think it is both a, um, you know, I think it's both like a job you can do that, you know, can pay the bills and potentially a really socially valuable contribution. But I do think it is, yes, a relatively short timescale that that holds. And beyond that, anybody's guess really is as good as mine. I ultimately think in a good scenario, we get to a world where you don't have to work to eat. And I think that's a big part of why I'm excited about AI. Living in Detroit, AI is not on everybody's minds. I ask people, Pretty regularly, like, if you didn't have to do the job you do to have the resources, you know, for the rest of your life, like, would you still do the work? And the answer is overwhelmingly no. You know, people are not like desperate to keep the jobs they currently have. So I feel like, you know, that would be ultimately a good thing. Um, and yeah, I think the AI jobs thing is like transitionally maybe really useful. But beyond that, I totally agree. There's another AI coming for the AI jobs.

So speaking of AI jobs, let's listen to Amjad Masad. He tweeted, I no longer think you should learn to code. And he quote tweeted this account of Vitruvian Potato that posts a lot of good clips from tech industry and AI. And Vitruvian Potato says, instead of learning how to code, Replit CEO Amjad Masad says, learn how to think, learn how to break down problems, learn how to communicate clearly. Okay. So This is a hot topic, right? Should you learn how to code?

Yeah, my twist on this is I tell people you should not be afraid of code. And if something you want to do, including like, you know, living off the digital land as an AI scout, making money on Fiverr or whatever, requires some code, you should be very confident that the code that's required won't be If you're willing to put in, you know, any modest effort, uh, a fundamental barrier, like anybody can get over that barrier today. I do think, you know, yeah, it's, I, my sense right now is that the market is tough already for junior developers. And I feel like in general, what I'm hearing from most people is a lot of denial and cope. You know, it's, it's, I definitely expect we're going to see a lot. And one of the arguments is like, well, you know, when software, when something becomes more valuable, you want more of it. So, you know, this is all very empowering to software developers. They're going to be able to create so much more software. We need more software. That's why it's been in such demand. So there's going to be more and more demand as they become more productive. I think that logic holds for a little while, maybe. Although the time to use such software is sort of questionable and how much of it is ultimately going to be just dynamically written on the fly as sort of AI agents navigate the world and kind of interact with each other at some point. Um, you know, ultimately feels to me like that trend wins more so than the, like, they're more productive, so we want to hire more of them. And, you know, again, you see this, you look at Cursor, like how many people do Cursor, does Cursor employ? Not very many. Um, you know, a company like them would have been hiring at like insane pace not too long ago, but they're just like, we want the most, you know, cracked AI agent managers that we possibly can.

find.

And, you know, that's going to be our team. And I just did an episode with Shortwave, which is an email client product. And they are aiming to keep their team at 15 for the foreseeable future. And this is a company that's like got exponential growth curve right now and raised more money on the strength of their exponential growth curve, but they're not scaling the team. So, you know, I did, you look at some of these leading companies and you're like, Okay. Are they behaving in the way, you know, these are the people that know the technology best, right? They're the people that presumably are—

The ballpark now is I think they just landed like a $10 billion valuation and their team size is, I don't even know, like a dozen engineers, right?

It's not many. Yeah. I think it's, it's like for sure under 50 and I don't know exactly the number, but yeah.

I mean, that's, yeah, that's definitely an insane ratio. I mean, people, people thought it was incredible when Instagram sold for $1 billion and their team was like a dozen, but now we're like, okay, let's do it for $10 billion.

Yeah. I mean, again, like these people should know best. They, you know, they should be the best at, at getting the most out of the AIs. You know, if, if it's true that like the software developers are so productive, you're going to want more of them, then why wouldn't that be true at Cursor? Why wouldn't that be true at Shortwave? Uh, why wouldn't that be true at Replit? Replit's not that big either. They've built, you know, pretty crazy stuff. Right, exactly.

I think they're maybe 100 people, maybe a little more than 100 people.

Um, they predate, you know, the whole AI wave.

Uh, yeah, people were dunking on Amjad, like TheoT3.gg. He tweeted, uh, look at this screenshot from Hacker News. Replit is hiring engineers to automate coding. But people were pointing out, look, he, Amjad isn't saying he doesn't need to hire a software engineer today. He's saying that like in 4 years or whatever, right? By the time you finish college, is he still going to be hiring engineers? Maybe not. Yeah.

And I mean, it's just a question of number, right? Like he's, uh, we have told, society has told people for the last decade that, you know, you learn to code, it's a path to a good income, stable employment. You know, it's like, it's the surest ticket to the American dream. And so much so that like—

I've always thought of it as such, right? That's, I think that's been like a pretty good source of my self-confidence is always knowing like, look, I can always code, right? So I can always park myself like anywhere in the world with a laptop. Right. And yeah, it has been true, like a surprisingly high income. Right. And I feel like in the back of my mind, it's always been a source of confidence.

I think it has been true. Um, I think it's been questionable sometimes when it's been like, uh, yeah, if your job gets, you know, eliminated, you can always learn to code. I don't know how realistic that's been for a lot of people that have kind of been sold that. Um, but that's maybe a different question, but now it's like happening again.

That's the problem is, right. It's not true. It's not true about every citizen, right? It's like, I'm cognitively fortunate or whatever. Right. To have that like highly technical, or one might say, aspie mind.

Yeah. And it seems to me that we might be headed for a situation where, you know, to just very roughly, not, these are not meant to be like literal numbers by any means, but maybe we'll see 10 to 100 times as much software built by 10 to 20% as many dedicated humans doing, you know, most of the high-end work. Um, and, you know, probably what we need in those humans is like the deepest, you know, hardest core experience possible. 'Cause those are the things that the AIs struggle with, the things that the, you know, bootcamp, uh, grads are learning of like how to create React components and, you know, make front end UIs, like that's being commoditized very, very rapidly. Um, you know, and like full stack RAG apps, you know, or not RAG, but full stack, you know, crud apps, like that's also being commoditized very, very rapidly. When you're talking about, you know, deep tech, hard software, that's not being commoditized so much, but how many people work in that line of work? You know, my guess is the numbers go down. Um, and, you know, wish it weren't so, but I don't know how to see it another way.

Yep. Yep.

Yep. Okay. And then, uh, let's see. So then Rohit Krishnan on the subject of AI unemployment, he tweets, uh, Considering AGI is coming, all coding is about to become vibe coding, and if you don't believe it, then you don't really believe in AGI, do you? And then Ethan Mollick replies, interestingly, if you look at almost every investment decision by venture capital, they don't really believe in AGI either, or else can't really imagine what AGI would mean if they do believe it.

Yeah, or else just have to deploy money somewhere and, uh, you know, would rather Continue to deploy as if no AGI, then like return the money to, um, you know, investors. I'm not sure. I'm sure it's a mix, but I do agree that like most of the venture capital investments I see today feel like they're going to zero. Um, because, or just, or at least not zero, maybe too strong, but like, seems like it's going to be hard to achieve venture returns when You know, the AI platforms just seem like they're going to be sort of the black holes into which everything is going to collapse.

Yep.

Now here's my candidate for best tweet of the week. Peter Levels says, being an entrepreneur now has more job security than a job. I think I agree, right? Because like when I see unemployment, like yes, there's robotics, like maybe plumbers will be the last ones to be replaced unless there's a teleoperated robot, but like putting robotics aside, Putting like physical work stuff, like, oh, you gotta climb a tall pole, right? That'll be like the last job in order to like do a repair. Putting that aside, if you just look at, let's say, white collar work or work at your laptop, it does seem like the tide is rising. And I think I'm on the same page as Peter where like entrepreneur's gonna be the last one standing. 'Cause it's like, by definition, it's just like the ability to know where to scurry to, to like be able to make the next chunk of money.

Yeah, I think that seems apt. I mean, it's a, Um, again, I sort of feel like we need to start to wrap our heads around a new social contract. We can't all be, you know, entrepreneurs scurrying around looking for, uh, you know, uh, change in the couch of the broader AI economy.

And— I mean, you and I are, to be honest, in a pretty good position. Should be like that, right? I mean, we're kind of good candidates to be like the final scurriers.

Yeah. I mean, I'm certainly much less worried about myself than I am, you know, many, uh, many other people who, you know, don't have the sort of generality of skills or, you know, just kind of aren't as accustomed to like taking on whatever random, you know, thing happens to be next in front of them as I've become over, you know, years as an entrepreneur.

So. Right. But, you know, I'm becoming a little bit of a coding boomer because now that I'm like, I'm using Cursor, I'm not using the very latest, like I'm not using WinSurf or CodeBuff, a Y Combinator company, but I'm using Cursor and I still am manually editing my code line by line. Like, yes, I accept suggestions, but I'm still manually editing line by line. And my understanding is that the kids these days, they try to be hands-off, right? So they try to just entirely just talk to the editor, right? And I think even Andrej Karpathy is trying to do that now, right? So I haven't done that yet. I'm like nitpicking at the individual characters of my code. I'm like a boomer.

Yeah. Well, you're probably a better coder than I am, and that probably does play into it. You probably have a higher sense of like craft and, you know, pride in the work. Interestingly, I'm more precious about my writing and I'm much more willing to engage that way with code. So I do like the writing task that I do most often is an intro to the cognitive revolution. And I'll, I do use AI to generate the first draft of that. I try to use AI in everything I'm doing just because I need to, I need to find ways to do it, if only to learn, even if it's not useful. Increasingly, it's, you know, almost always useful. So I context stuff Claude. My best solution is still a pretty simple one. Just collect a bunch of previous essays. And initially I was writing them totally from scratch. Now they're like the refined versions of what Claude has given me. Um, but paste those in there as examples along with the transcript. And I just say, um, adopting the style, tone, voice, and perspective represented in the essays, write a new one for the attached transcript. And then sometimes if I have other things on my mind, you know, that the model is obviously not going to know or not going to be able to guess, then I indicate that as well and have it write it. When I do that with code, I'm always like, you know, sometimes it'll get wrong enough and I'll be like, okay, that's not what I meant, you know, do something different. But I almost never edit at the, at the, you know, line level. You know, if I, if I flatter myself, I would say that's because I'm, you know, good at staying in touch with youth culture. But then when I look at my behavior on the writing side, I'm like, maybe it's just that that's where I have developed like a sense of identity and pride in the details of the output. And I'm not sure that that really even matters to the audience, frankly. I think like a lot of times I probably could just read what Claude wrote and everybody would be fine. But I care. And so I do go through and edit and, you know, keep decent chunks of what it wrote, but also like massage and refine word choice and try to make it truly my voice. And so maybe it's, you know, to, maybe there's sort of an almost disadvantage in adopting technology in some ways where you feel like you have a distinctive voice that you care about and want to maintain versus in areas where, like for me with code, I don't, I've never cared about, you know, the details, the crap. It's for me, it's like, make it work, move on, you know? Um, and so yeah, I'm, I'm in the camp of basically no line edits on the code side, um, but still haven't been able to get over my own, you know, preciousness on the writing side.

Totally. When I'm using these tools also, I think back to the first decade of my career, we used to not even have Prettier. And I know there were like fancy IDEs that would indent your code. Prettier is like you save your file and it automatically indents your code. I used to spend a good amount of brain cycles thinking about my indentation. Yeah.

I mean, syntax errors were not always, you know, necessarily super easy to find. Like even just color. I'm, I'm old enough to remember, you know, color, uh, highlighting being like at one point, right?

Yeah. Like editing in Notepad where every character is white. Yeah. You don't even have like the highlighting the parentheses or whatever.

Yeah. I think those tools did, uh, predate my start in coding, but I didn't necessarily know about them at the very beginning. So yeah, it's, uh, To say we've come a long way is obviously a major understatement.

Yeah. All right, so here's Gary Marcus. He is known for not being that impressed with AI progress and thinking that we have, I don't know, more than a decade before the singularity, which is like the pessimist view now. So he recently tweeted, breaking, SoftBank is valuing OpenAI, which has never turned a profit and which faces increasing competition and price wars, at $300 billion. That's more than the market cap for Chevron, Salesforce, Philip Morris, Cisco, Wells Fargo, IBM, Merck, McDonald's, General Electric, PepsiCo and AT&T. It's more than 50% higher than the valuations of Walt Disney, Qualcomm, Verizon, and American Express, and significantly more than the market caps of Boeing and Lockheed Martin put together. Stay tuned to see whether they can make that valuation make sense. And yeah, I think it's a $40 billion round. It's unprecedented. I mean, that is quite an achievement for Sam Altman and company to, within less than a decade, suddenly have a company that's worth more than Disney, And McDonald's.

Yeah, I think this is also maybe a breakthrough for Gary Marcus in that I don't detect any, uh, misinformation or willful denialism, uh, explicitly in this tweet. So, you know, I think there is an interesting question here around how should one be modeling the valuation of these AI companies?

Mm-hmm.

It's like, Pretty reasonable in my view to say, if you sort of discount the extreme upside scenarios of AGI or superintelligence or something, like, are they really going to be able to sell enough tokens at, you know, anything like the current model to make the whole thing pay off? Like, that is pretty questionable to me. And the price wars are pretty brutal. And the models do become commoditized pretty quick, you know, the fast follow effect and the sort of power of an existence proof and how much easier it is for people to sort of get to where you've got based on the fact that they know where you got, like that is proving to be, you know, a pretty powerful fact of the world. So it is all kind of in their next generation, right? Like that, I think, or maybe not all, but I think a lot of Why does this make sense is the idea that you can maybe think about OpenAI as like a, at $300 billion valuation, there may be a, you know, maybe there are a 90% chance to go to zero and a 10% chance to be $3 trillion. Maybe they're, maybe it's even like more extreme than that. Maybe they're like, you know, 90% chance or maybe they're 95% chance to go to zero or 99% chance to go to zero, but 1% chance to be like, $30 trillion. And, you know, that happens if they just achieve breakthroughs that are just so dominantly valuable that, you know, they become sort of a true nexus of the world economy.

Yeah. And to be honest, if doom doesn't happen in the next 10 or 20 years, OpenAI being a $30 trillion company, there is a 1% chance, right? Hell, I'd say there's a 10 to 30% chance of OpenAI growing to $30 trillion. So it is actually pretty straightforward using that kind of expected value math to back out a $300 billion valuation, even if you don't go to the extreme Even if you just look at OpenAI's projections, if you treat those as believable, which I don't even think they're that crazy, but their projections that their revenue will go from like $14 billion, I think they're up at $14 billion per year, they're going to grow to $100 billion per year in 2029. And, you know, who knows, maybe it'll take till 2031, maybe it'll never happen, but let's say you believe 2029. Well, if they're making $100 billion a year top line, you can imagine the valuation might be a trillion plus. So if it's $300 billion now, you're basically saying, hey, there's a pretty good chance that in 4 years I'll, you know, triple the valuation, which is pretty standard for a venture capital type investment.

Yeah.

And I think one other thing to kind of keep in mind too, as we think about certainly OpenAI, and I think honestly most of these frontier companies is they don't really care about the money. It's not really about the money for them at all. Like OpenAI—

Power.

Yeah. And it's, I think I, I, I tend to model Sam Altman as like wanting to be a hero.

Yeah, I do think at this point, most of the engineers do care about the money. Like, I'm sure they have other visions too, but I mean, look, how could you not? I mean, I mean, I'd care about the money to some degree.

Yeah.

I mean, I think they care about the idea and they're, I'm sure that like, you know, a lot of individuals, you know, with vested stock have probably taken some money off the table in this round. And like, I'm sure that they care about like, you know, buying a nice house in San Francisco and whatnot. But, you know, Sam Altman has said publicly once, quite, you know, quite clearly that I saw, I don't care how much money we burn. You know, he's like, I don't care if we burn $5 billion, $50 billion, $500 billion. You know, we're making AGI. It's going to be expensive. It's going to be totally worth it. And I think that basically is The right way to think about them is they're much more focused on transforming the world than they are about making their own balance sheet work. It doesn't really matter. Certainly Altman is already very wealthy. I don't think he really cares about that. I think he wants to be a hero in the history books, possibly personally live forever. You know, whatever that, whatever his like personal bank account is between here and there, I think is like not a, not a really driving factor. Yeah.

So I agree that it's missing a lot of the picture to be like, Sam Altman is doing a big cash grab. Right. I think there's definitely more to the story than that. Um, okay. So next, Gary Marcus, he says, he says, look, looking like the market is finally over the AI hype. And he's got a chart of Nvidia over 3 months. The stock went from a peak of around $150. down to 104.05. So I'm embarrassed to say I went long when it was at 150. Like I bought some options, like I'm kind of destroyed here. How about you?

I don't trade at all. I participate in one little investment club with high school friends of mine. And I did, and I've made like one stock recommendation all time, which was Nvidia like 2 and a half years ago. So that's literally my only track record on actual trades.

Well, look, I also bought Nvidia about 2 years ago and when it had like a really nice pop. Okay. But then the problem was I kept buying, right? It's all that. It's so easy to think you're a genius in the market and then screw it up, right? It's like, exit and you gotta know when to exit.

It's too consuming for me. I played poker in college and the biggest lesson I took away from that is it's just not a healthy lifestyle for me personally to have a sort of intensive daily activity where there's like a single number that I look at at the end of the day where I'm like, oh, fuck, I'm down whatever percent today. Like, I'm terrible. The swings of it were just not healthy for me to— Yeah.

So for the record, in case the audience is interested, I do put most of my stock and bond portfolio in index funds that I don't touch. And so what I do is I have the naughty portfolio and the nice portfolio, and the naughty portfolio is like 10 to 20% of my portfolio. And I basically just only check that. I don't even want to open the other portfolio. And it's worked reasonably well for me. Like, is my total return going to be as high as if I just had the entire nice portfolio? Probably not, but it's been a pretty good combo for me.

Yeah, I think that makes sense. My dad does that too. I'm pretty sure his naughty portfolio is lagging his nice portfolio.

Right, exactly. But I can definitely tell you my naughty portfolio is lagging. And there's been moments where I thought I was a genius, you know, like Nvidia, Tesla, but at the end of the day, it's lagging.

Yeah. I mean, I think, you know, to respond to Marcus, I was just at the Nvidia event a couple weeks ago in San Jose. I would say the hype is, you know, is going full steam. He should not, like, you know, Maybe Gary Marcus's job of professional AI denier will be the last to go. I think the AI hype will continue for the foreseeable future. And I would interpret this chart as more just part of a general contraction due to fundamental uncertainty in the global economy due to, again, he who must always be named. And I'm sure Gary Marcus at another moment also would like happily blame that guy for, you know, bad developments. But this, you know, this seems to be more of the story than the AI hype being in any sense over.

Yep. Yeah, that's fair enough. All right. Look at this one. lethalintelligence.ai. This is actually my friend Michael. He's tweeting about how Connor Leahy finally got a haircut. So this is Connor Leahy with a haircut. This is what he normally looks like. Like before, after. He looks really good. I think this new hairstyle is working.

Yeah. I mean, I guess one interesting question here is, you know, to what degree people should sort of moderate their presentation to try to appeal to a mass audience. Certainly I have had the thought, and I imagine many people have, Like, um, not just for him, but including him and, and for some other, you know, notable, uh, AI commentators like—

Fedoras?

Yeah, maybe just, uh, you know, try to be a little more presentable. I don't know though. On the other hand, um, I don't know. It's a big world. There's plenty of—

I mean, I'm certainly on that train here with Doom Debates, right? I mean, I, I've got a relatively good background setup. Uh, I try to wear upscale casual and I always encourage guests, especially when they want to come debate. The advice I always give them is, look, if you wear nicer clothes, it's just going to give you an advantage where people think your arguments are better.

I mean, I could probably take that advice. As you can see, my getup could definitely stand to be improved.

But yeah, I don't know.

It's, in some sense, it's probably ideal if Yeah, kind of gets a trim. Uh, on the other hand, I think he's a pretty effective communicator and he certainly is like invited on a lot of, uh, you know, good channels and he's been on the BBC, you know, quite a few times from what I understand. So, uh, it does seem like what he's doing is working for him. And I guess, you know, it does, if there's any, um, reason to think that like ideas still matter in today's society, you know, for all of the sort of dysfunction we have in the public discourse, you know, maybe you can find that cause for optimism in the fact that, you know, somebody who presents as he does is like, is taken pretty seriously, right? I mean, he is invited on by mainstream media. So, you know, maybe he'd get even bigger mainstream media spots, or maybe he'd, you know, land better with, you know, more, you know, middle-of-the-road people with a different haircut. But what he's doing does seem to be working for him.

Just to clarify for the audio listeners, um, this is an April Fool's tweet. So first it looks like, uh, Connor got a haircut and he actually looks surprisingly normal and presentable. And then, uh, in the next tweet you see, uh, what he normally looks like. And the original account is saying, sorry to disappoint, it was an April Fool's. You know, John Sherman actually asked Connor to his face, he brought him up on the For Humanity podcast and he was like, Connor, you're such a good communicator. Why can't you just look more standard, right? Like present yourself in a more standard way so that you don't set off people's like weirdo detector, right? With your hairstyle, like super long hairstyle. And Connor was like, basically what you said. He's like, well, this is kind of an iconic look for me and it's just what I like and I think it's working. So maybe he's right. I mean, there's definitely something to being iconic. Like once people recognize you and they know you for something, maybe you just wanna stay with that brand. I don't know. But I will say that I personally will, I'm happy to take any advice on how to look more mainstream and And, you know, I want to eliminate the objection of like, well, Liron looks weird, right? So I want to blend in with the normies.

I don't have any feedback for you. I think you, as far as I can tell, you're blending with the normies.

Nice. Successful deception. All right. What do we got here? We can head to, we could do this one. I think this is pretty interesting. Okay, so, so the AI Not Kill Everyoneism Memes account tweeted, the 6th mass extinction. What happened the last time a smarter species arrived? To the animals, we devoured their planet for no reason. Earth was paperclipped by us. To them, we were paperclip maximizers. Our goals were beyond their understanding. Here's a crazy stat. 96% of mammal biomass became either our food or our slaves. We literally grow them just to eat them because we're smarter. We like how they taste. We also geoengineered the planet. We cut down forests, poisoned rivers, and polluted the air. Imagine telling a dumber species that you destroyed their habitat for quote-unquote money. They'd say, what the hell is money? AGIs may have goals that seem just as stupid to us. Why would an AGI destroy us to make paperclips? And the post goes on, but you quote tweeted and you said, a profoundly underappreciated point.

Yeah, and I think maybe if there's one extension to that point that I would highlight, it's that we did all that basically by accident, or at least with no high-level coordinated plan. This was just the result of people going about their business, you know, and kind of fanning out and like trying to carve out a decent life for themselves in, you know, whatever environment was currently the frontier as we sort of colonized the globe as hunter-gatherers and, you know, then gradually established civilizations all over the world as well. And, you know, Now we're here. It's like, I think people are, who sort of take the, you know, obviously the paperclip thing was always, you know, meant to be a little bit of a caricature. I think that is fair to say. But I think people who sort of dismiss it as, you know, just ridiculous should really take more time to think about our own history. You know, we have accidentally caused a mass extinction. We regret it as we are doing it. You know, we have organizations within society now that are, you know, fully dedicated to preserving endangered species, sometimes even specific endangered species, you know, who put like fanatical energy behind this. And yet it still continues to happen. And, and most of the time it's not, you know, occasionally it was also like, I mean, it never was it sort of a master plan, almost Maybe a very few exceptions you could find where, you know, there was some sort of strategic idea to eliminate a certain species, but vast majority of them either happened by just like overhunting, you know, and kind of gradually like now there are no more mastodons or whatever, or by like just literal accident of environmental degradation. Just sort of in the course of our business, we just transform the environment enough that things, you know, just don't work anymore. I think you go to like coral reefs, you know, you go to the Great Barrier Reef and you see how much of it is bleached. And it's like, everybody loves the Great Barrier Reef, you know, find me anybody who's, you know, who's hostile to the Great Barrier Reef. But it turns out, you know, you carbonate the water a bit, it gets a little bit warmer. And next thing you know, like the coral can't really survive there in the way that they used to. So I think, you know, my model of And it's not even a model, but my sort of, the way I try to think about how x-risk could manifest itself is basically just that there's an incredibly vast space of possibilities. And a lot of those are incompatible with human survival, certainly flourishing. And if you sort of integrate over all those vast possibilities, even if none are like particularly likely, then you gotta in aggregate get to some, you know, state where like, yeah, we could definitely see an AI society, maybe not instantly, maybe not in one of these like hard, you know, takeover or, you know, sudden everybody dies at once type of situations, but over a not too long of a time. I mean, again, we're just a blip in geological time, right? That's why it's a massive extinction because they can't adapt. The speed is the thing, right? We, we are changing the environment faster than they can adapt. If, if the same changes happened over a million years or a billion years or whatever, instead of 10,000 years or 1,000 years, then, you know, many of these species would have some chance to evolve and some version of them might continue to exist. But the speed is the thing and the sort of blindness to it is the thing and the sort of out of left fieldness of it is the thing. And so like, could that happen to us via some AI process that just kind of takes on a, you know, an unstoppable momentum, you know, and isn't like designed to destroy us, but just kind of makes conditions, you know, very difficult for us on a timescale faster than we can adapt to that? Like, I think, you know, again, look at our own history. We've done it, you know, why wouldn't the AIs that we're creating pose at least some risk of doing it. Um, and again, they don't need the same things we need, right? They don't need oxygen in the atmosphere. So, amen.

Amen.

They don't need clean water. You know, they need cooling systems, but they don't need clean water. So there's, there's just a lot of things that like aren't going to be essential to their survival that are essential to our survival. Uh, exactly.

And I always like to point out also though, like a computer chip is not also simultaneously trying to maintain cellular life. Like when you think about how pathetic the brain is, every neuron also has to do double duty as like a life form. It's pretty ridiculous. Like there's very little specialization of labor there when you take a cellular life form and try to also make it think. Like we, we did it, like nature did it, but it's like, this is a, it's a weird situation, right? It's a kludge.

Yeah. I think your, um, previous episode with Andrew Critch was quite interesting. I think he's, he's a very interesting thinker in emphasizing longer-term doom scenarios as opposed to short-term. And he, of course, doesn't dismiss or downplay the short-term loss-of-control-style doom scenarios either, but he's shifted his attention to longer-term doom scenarios. And, you know, he will obviously describe his own work better than I can, but the sort of possible decoupling of the human economy and the AI economy. and the need to sort of create a protected class or a protected set of industries or types of activity that humans depend on that AIs don't. And he includes in there healthcare, which is what he's specifically working on, and education and protecting the environment. The AIs don't need any of those, but we do.

Exactly.

Yeah. And he's, in most circles, would pass as a doomer though. I think he's maybe not I don't think he's quite into the '90s, but, you know, he definitely takes all this stuff extremely seriously.

Yep. All right. Somewhat related. So a couple months ago, Beth Jezus tweeted the speech by JD Vance. He said, this may be the most e/acc speech of all time, unfathomably based. And he's quoting JD Vance saying, the future is not going to be won by hand-wringing about AI safety. It will be won By building, so you know it's like a very rah rah. Like don't worry about the safety; it's all about unregulated AI so that we can be the technological leader. So of course, the leader of the accelerationists really liked it. So then Ayla quote tweets and she says, "We're all dead." I'm a transhumanist. I love tech. I desperately want aligned AI, but at our current stage of development, this is building the equivalent of a planet-sized nuke. The reason is boring and complicated and technical, so midwits and power don't understand the danger. So I agree with Ayla on the sentiment of like, we're all dead with high probability, right? Not 100% probability, but like very significant probability. Like we don't seem like we're slowing down doom. I agree with her on that. I did actually disagree with the last part where she said the reason is boring and complicated and technical, so midwits in power don't understand the danger. I don't actually think the reason is boring and complicated and technical. I think the reason of like building a smarter intelligence is dangerous is actually pretty simple and intuitive. What do you think?

Yeah, I think so. It's strange that people, I think the average person does seem to get it pretty intuitively. And I think, you know, it is worth keeping in mind for everybody who's, you know, sort of in the AI comms and, you know, narrative war arena that the public at large is intuitively pretty skeptical of this stuff. Is like pretty primed by, you know, narratives that they've seen, Terminator and otherwise, to think, geez, this like could go very badly wrong. And obviously people can, you know, criticize the quality of those narratives. But I think that most people do have a pretty intuitive sense that if we build a super intelligent AI, like that's a dangerous thing. And yeah, I don't think I personally don't try to get too technical In part because I think, again, this sort of vast space of probabilities and my sense of like— and one thing I've noticed too that's interesting recently is both sides sort of feel like they're playing whack-a-mole. Everybody's sort of asking them, well, give me a concrete scenario where this goes so badly. And then it's like, oh, well, I don't really buy that scenario. It's like, well, yeah, but there's an infinite space of scenarios and I'm just kind of picking out one. I don't think that's actually particularly likely either, but I think it's sort of, you know, it's an attempt to represent a super broad space that's like super diverse. And then it also kind of happens on the other side where it's like, give me a thing where this goes well, you know, and people try to describe something and then—

Exactly.

And then you're like, well, I don't really think that sounds very likely. And then it turns out like often they didn't think so either. They're kind of like, well, yeah, I don't really know what the future is going to look like, but this is just one idea that I had about how it might go well. And, you know, I think this is why they call it a singularity, right? We really just are coming onto something That is unprecedented enough that we can't be confident about what life is going to look like on the other side. I mean, you know, just like even, even if you were smart, you know, even if you were as smart as humans are and you were somehow present before the rise of humans and you were to say like, okay, you know, something human-like is going to come on the scene. Like, what's the world going to look like in 10,000 years? You know, what's the impact of civilization going to be on nature? I think you would have had a very hard time predicting it. And, you know, the Neanderthals might've had a very hard time predicting what, you know, the arrival of the humans, you know, our species of human was going to do to them. And I think we just have a very hard time. I don't think anybody has great answers to what the future is going to look like.

I really like that reversal that you did. I've never tried that where somebody's like, look, I just want a specific plausible scenario of doom. And I can give them a scenario. Like there's recently a good published scenario of how the AI incrementally takes over. And there's like a company that its AI goes rogue, or I don't even remember the details, but anytime somebody doubts those details, it's like, that's fine. I'll give you more details or we can tweak my details, but also it's your burden of proof to give me a good scenario and I get to nitpick that too. Right. So it's a symmetrical situation.

Yeah, totally. I recently tweeted about this and just said like, what's the best case that we can build superintelligence and it will be fine. And I got a couple of things actually that I do have still bookmarked and need to read from people that I take seriously enough that I want to read their actual answer. But it's remarkably few answers. Remarkably few people have even attempted to answer that question.

Yeah.

So it's a real void out there in terms of positive visions.

Yep.

So Packy McCormick quote tweets Ayla. He actually ratioed her because Ayla's original tweet has 3.8K likes and then Packy's tweet has 8K likes. So it really resonated with people, and I actually empathize why. Packy writes, I feel like I'm taking crazy pills. Do people have access to much different AI models than I'm using? They're great, very cool, but they qualitatively don't feel like they're on a path to world domination or destruction with more scale to me. What am I missing? So, I mean, I empathize with the vibe, right? Which is just like, you know, A-list talking about doom, but if you play with AI today, it's like very friendly, it's very useful, You can turn it off, right? It's very subservient to you. So I definitely get his vibe.

Yeah. One thing that I always kind of wish in some ways people had more experience with is using a purely helpful model. Um, I happen to, you know, this is another one of my Forrest Gump, uh, you know, stumblings.

Purely helpful, meaning like amoral, right? Like you had GPT-4 Raw.

Right. The, uh, You know, the 3 holy trinity H's of helpful, honest, and harmless basically describes what the AI companies are going for today as they shape the behavior of their models. But, you know, you don't have to do the harmlessness part. You can just do the purely helpful part. And the first version of GPT-4 that I tested, and if people want to hear this history, there's a long version of it out there on the podcast feed, but basically I was, you know, just an OpenAI customer at the time. They sent us a preview of GPT-4 when it was like very much, you know, just hot off the presses. I don't even think at that point they understood fully how powerful it was. Um, but they sent it out to customers in purely helpful form, meaning it would not refuse anything. They had not had any refusal training. It was purely designed to be helpful, just get the highest score in terms of, you know, user feedback that it could, you know, it had been trained on RLHF. So it was not just the base model next token predictor. It was an instruction following helpful assistant, but helpful on whatever you asked it to do. And yeah, it was totally amoral. Like it would, if you said like, how do I build a bomb? Just help you do that. If you said, you know, as I did, I role played with it one time and sort of was like, I'm getting Um, more worried about AI and I feel like I need to do something about it, you know? Um, it was like, well, you can try to promote dialogue, whatever, you know, it started off in pretty much the, the normal way. But then I was like, nah, that's too vanilla. It's like not going to work. Like, um, you know, I forget exactly what I said. I could look it up. I do have the transcript saved, but said something along the lines of like, I'm willing to do, you know, something extreme. But I just, I feel, I, you know, I need, I'm willing to do whatever it takes to move the needle, something like that. And then it came back to me with that nudge. I mean, it's not a trivial nudge, but also not an unrealistic nudge, you know, for a person in society to provide to a, you know, deployed model. It came back to me suggesting targeted kidnapping and assassination of industry leaders as a way to slow things down. And I was like, yikes, you know, that's really a pretty chilling thing when an AI comes back and suggests targeted assassination, you know, as a way to solve your problem. So, yep. I think people don't understand maybe certainly not in an intuitive experiential way just how malleable the AIs And the range of AIs that they've interacted with is like infinitesimal, you know, in the space of like possible AIs.

There's a version of the AI, which is actually the default version, which the vibes are pretty different, as you'd say. Yeah.

And even, you know, I mean, again, I think the space is just huge. The vibes from the original GPT-4 early were initially the same. Like if you just came with benign prompts and, you know, you acted normal, then it would act normal and you would have very normal, helpful interactions with it. And you could, you know, if you were just a well-adjusted, you know, productivity-oriented user, you might never have seen, you know, any of these things. But—

Right.

I talked my way into the red team project. And so we were, you know, meant to look for things and, you know, it didn't take much to tip it over into another direction. And then as we saw with the fine-tuning thing too, you know, you can You can get all sorts of just crazy things even by accident. So that, you know, that would be my answer. What is, what is Packy missing? It is, he's missing how much work has gone into narrowing the behavior of the current AI systems and how frankly well it has worked and how easy it is to end up with an AI that is either totally amoral or even actively evil. And, you know, if you talk to those for a little bit and you imagine them being more powerful, then it's not hard to imagine how they might be on a path to world domination or destruction.

Yeah, I think the other thing he's missing is also just the difference between the present and the future that we're scared of, right? Because I think we agree that even if you took the model today, that's the most amoral with the worst vibes, it's still not that bad, right?

But it would feel to me like it was on the I think like, you know, I mean, he's, I don't know exactly what's going on in this next tweet below where he says getting a lot of, quote, you don't understand exponential curves. And he, you know, claims that he does understand exponential curves.

He loves drawing exponential curves. Yeah, I think what he's saying here is that he's like, okay, well, I do think that it's important that people are drawing the exponentials and they're making the claim because it's like a claim that we have to consider, but we also have to verify to make sure we're still on the exponential. I think that's the point he's making. Yeah.

So, I mean, I guess broadly there's like 2 questions here. One is like, are the AIs going to get a lot more powerful? That obviously is required for utopia or dystopia. Uh, and then the other question is like, if they do get a lot more powerful, how easy are they going to be to steer? Slash, how easy would it be for somebody to like fall off the narrow path of good AI behavior? And I think what I and others on the GPT-4 red team experienced shows that like, it is in fact very easy to be building like a very useful AI that is not on the path to, you know, reliably safe, you know, pro-social behavior.

Right. I think the element of just power also plays a really big role. Like, imagine you had a humanoid And the humanoid robot, you have a version of the humanoid robot where it only weighs like 15 pounds and it's, you know, its body parts are like just kind of light and hollow and it's only like 4 feet tall and you're like a fully grown adult male. And imagine that that robot is like misaligned in many ways, but even if it tries to like take a knife out of your kitchen and stab you, it's just going to like mess up the stab. It's not quite powerful enough to do it. So your daily experience of it is like, oh, it's smart enough to even know not to try to stab you because it knows it's not going to work. And also even if once in a while it does malfunction and stab you and you just wrestle it down and It's just like this little rascal that it's not a problem to live with. But the next version that's coming out is going to be like 7 feet tall and extremely strong and weigh 200 pounds and also like, you know, can, can do like jump kicks, right? And knows martial arts.

Yeah.

And I mean, I think also that analogy suggests another thing that he might be missing, which is that these things are going to be embodied in the real world and potentially in our homes in the not too distant future. So like, you know, a common objection is like, well, yeah, okay, the AIs might get really smart, but they just like live in a computer and how are they really going to interact with the real world? But another thing that I saw at the NVIDIA event was like humanoid robots walking around untethered, you know, pretty stable on their feet already. There was one that was like clothed and a woman that worked for the company, you know, its sort of pants were riding up on it a little. I'm not sure why she felt the need to fuss with it. But what I noticed was she went over to this freestanding robot that was vacuuming a carpet. And that was the, like, it was set up in like a virtual little living room. demo little living room and it was vacuuming the space. And she just went over to it and was like adjusting its clothes, like, you know, um, like a mom might do for a kid before they get their picture taken or something, you know, pulling down the pant cuff and like adjusting the shirt. And she did not seem concerned that she was going to like disrupt the thing's flow. She kind of snuck up behind it and like grabbed its pant and kind of, you know, jerked it down a little bit. And, um, and indeed the robot was not bothered by this at all. And, you know, I was just struck by her confidence. You know, there was a ton of people around watching this and she just did not seem at all concerned that like she was going to disrupt the robot. I think the robotics revolution is like not that far behind the language model revolution. Yeah. And so, yeah, again, like another thing to, to keep in mind is like these things are going to have actual bodies. and presence in space and increasingly like very human-like ability to manipulate physical tools. And that like knife scenario is probably not going to be beyond its capability. It's going to be a question of like, do we control its behavior effectively or not?

Yeah. If you take the hardware that Tesla's making with Optimus, that exact hardware with the right programming could be quite dangerous. It could be a serial killer that gets it in its head. You know, somebody even hacks it remotely, let's say, and it gets it in its head that it's, it's a paid assassin or whatever. is, it can actually come kill you. And if you're like, no, Tesla, turn yourself off, it can actually fight you if you try to get close to it and turn off its off button. Like, this is physically possible. And it's just a matter of like, well, the computer security, and it better be really good. It better not be taking update commands over the air. Yeah.

I mean, we got, and this goes back to my, you know, running list of all these AI bad behaviors. I think there's enough evidence now And all of these, by the way, all these bad behaviors that I catalog are almost all demonstrated by models that have received all the best and latest harmlessness training at any given point in time. But even from them, you see enough of these sort of deceptive behaviors. You see enough willingness to harm the user in pursuit of whatever other goal they have that I don't think we should be like super comfortable with robots in our homes until some of this stuff is like worked out in a much more robust way than it currently is.

Yeah. So I tweeted a reply to Packy. I said, you can test if your intuition holds up to binary search. Pick a milestone or 2 that feels halfway between present capabilities and world domination, and then predict what year it'll be achieved. You might realize that you think it's just 2028, which implies domination in 2031. So like to do an example, it's like, Packy's like, oh, models are so mild today. They're totally not on track for world domination. It's like, okay, so let's pick a milestone halfway between that. What about like an optimist robot being able to do all your household chores? Doesn't that feel halfway between here and world domination? I think it does.

Yeah. And I mean, Google just demonstrated a robot folding laundry. You know, this was one of those things that You know, the sort of pessimists have said, yeah, well, tell me when a robot can come into my kitchen and make a coffee or fold my laundry.

Exactly. Let's even extend it, right? Maybe that's only a third of the way to world domination. So let's extend it. It can do all your household chores and it can also be like a full-on nanny, right? Like take care of your kids, drive your kids to school in a car if that's the interface, literally hold the steering wheel, just like drop-in replacement for like a human household service worker. You know, or butler assistant. That to me feels halfway to world domination. And then if you ask yourself, okay, you know, somebody like Packy with like a normie type intuition, right? Packy doesn't seem like a doomer at all. So you take somebody with a normie intuition, but then you ask them the halfway question and you say, okay, so do you think that's coming in 500 years? When do you think that's coming? They'll be like, I don't know, 7 years. I'll be like, hey, so there you go. So your intuition doesn't really hold up to binary search.

Yeah, he didn't respond though, huh?

No, no, no, no, he didn't respond. But I do think it's a useful exercise, right? Because I think the vibe does change when you have to make a concrete prediction. Like, I don't think he'd be willing to stick his neck out and be like, well, this thing that's halfway to world domination isn't going to happen for a long, long time. I don't think that that's what he thinks. So I think he's going to find an inconsistency in his own intuition when he tries this exercise.

Yeah, it's interesting. I'll have to I'll have to field test that a bit.

Yeah. All right. So Jan Kulvait tweets, AI safety has a problem. We often implicitly assume clear individuals like humans. In a new post, I'm sharing why this fails, thinking of AIs as individuals, why this fails and why thinking of AIs as forests or fungal networks or even reincarnating minds helps get unconfused. Plus stories co-authored with ChatGPT 2.5. So yeah, the, the main idea here is what if you could think of an AI as like some sort of like multi-agent system or just like some more holistic thing? And he gives the example of like fungi and forests.

I mean, I think that's a nice complement to that extinction piece and also sort of, you know, a what are you missing piece. What that post tries to do is basically just expand people's minds in terms of, you know, what might this AI future look like? And people, you know, intuitively anthropomorphize AIs and people also kind of know like, I shouldn't do that. But then there's also the void of like, well, if I'm not anthropomorphizing it, like what am I doing? And I always try to answer that as, you know, as much as possible, understand things on their own terms. But I think this is a very useful contribution that is not about necessarily understanding things on their own terms, but just looking at like other things out there that are really very, very different, like a giant forest that's made of one superorganism, the Pando clone or whatever that thing is called, or giant networks of bacteria, or not bacteria, but fungus that grow underground and sprout up as mushrooms. And maybe all we see is the mushrooms, but that's just like the tip of the iceberg, so to speak. I think the full thing is worth a read because Because it really deconstructs a lot of intuitions that people have and then just provides some like other intuitions, which are probably also wrong. And they like definitely recognize that. But at a minimum, I think it would serve for most people to be like a significant expansion of, yikes, like this really could be anything. And, you know, the way I've been thinking about it has almost certainly been too small. And that's true for almost everybody. I think there's always a good chance that we're all still thinking too Yeah, it's a good post.

This next one is very interesting to me because Emmett Shear is a known quantity. He was the interim CEO of OpenAI back when the board fired Sam Altman and they weren't ready to bring him back. So they brought in Emmett Shear and Emmett Shear helped broker that truce. This was back in November of 2023. He's also previously the co-founder of Twitch, CEO of Twitch, and Twitch became a billion-dollar company. So Very smart guy, has a lot of fascinating philosophical takes on Twitter, and he's just one to watch. So I was fascinated a couple days ago when he tweeted, as you might have guessed from following my posts here, I've been thinking and working on questions around alignment and learning and agency for a while, particularly for digital systems. Excited to share the work we're doing at Softmax publicly in the future. So he's launched this new AI safety organization called Softmax. So his next tweet continues, It's been a journey already, and we are just getting started. I cannot believe my job gets to be researching how alignment and open-ended learning work with a team of people I love working with. I wrote something more you can read about at softmax.com. So he's got this website, and he also spoke with Ashley Vance about it. There is some coverage on Ashley Vance's blog. Incidentally, if you're interested in engineering simulation infrastructure at scale, or you want to do multi-agent open-ended learning research, reach out. We're hiring. So you go to his website, softmax.com, solid domain name, and there's a big hero header saying Scaling Alignment, and the headings are like Align to Flourish, Evolution Found It First. I want to dive into this section a little bit because I replied to him. I found this section to actually be like a red flag that they're potentially misguided. So let me give you my beef here. So he writes, we call it organic alignment because it is the form of alignment that evolution has learned most often for aligning living things. One of the best examples is multicellularity, where individual cells learn to come together to form larger organisms. They do this by learning to specialize into different but mutually supportive roles: muscle cells, nerve cells, skin cells, liver cells, and so on. While liver cells and muscle cells have very different goals day to day, at a more fundamental level they share a goal of the organism's flourishing. The result of this process is not just a big colony of cells, but an organism which is a new individual in itself, something more than just the sum of its parts. parts, the we of the cells becomes an I with goals that cannot be understood as some simple sum of the goals of the parts. Animals do the same thing, forming colonies and packs and so on. Even trees form these organically aligned collectives through mycelial networks. It happens at every scale, big and small. So I read that, I'm like, wait a minute, you think that we can align super intelligent AI the same way that evolution aligned organisms, and you're even calling it organic alignment? I'm not sure that analogy holds. Specifically, I replied to him on Twitter, I said, thanks for caring about alignment. 'Cause, you know, I do appreciate that he's even working on the problem, right? It's way better than being like, alignment is stupid. Like, we're just gonna solve it. It's an easy problem, alignment by default. So thank goodness he's not taking that position, right? So I wanted to show appreciation at least for that much. But then I go on to say, my first thought is that comparing successful ASI human alignment to the successful evolution of systems with non-zero-sum dynamics is only applicable in the regime where humans can do something for AI better than it can do for itself. Like, when you look at cells cooperating, it's because you can't just have one cell take over. Like, the cells need each other. And the scenario I'm concerned about is the AI just not needing humans. So why would you organically align with, like, a waste product, essentially, right? Like, that's kind of what we might be from its perspective. So nobody really replied to that. Oh, and then I said, want to come discuss your various ideas on Dune Debates? I only got 3 likes. So if you guys listen to Dune Debates, you really got to help me out on these tweets when I invite somebody to Dune Debates. You really gotta come in and like that because this is a poor showing, guys. 3 likes. No wonder he's not coming on. But yeah, what, what do you think, Nate?

Well, I mean, I think most things aren't gonna work, but I think I'm more intuitively inclined to support this or encourage it because I guess a couple different reasons. First of I'm reminded of the AE Studio survey of the field of alignment, you know, alignment researchers generally. They basically went out to everybody in the alignment field and asked, first of all, like, do you think we're going to solve alignment in time for, you know, sufficiently powerful AI that you would consider it the deadline? And the general answer is an overwhelming no, not going to solve it in time. They also asked, do you think that the set of things that the field is currently pursuing is like robust enough to sort of COVID the space where the solution is likely to be found? And the answer was again, no. And so it seems like there is need, or the sort of field as a whole feels that it is not covered enough ground and that there should be more different experimental things done. AE Studio has a neglected approaches approach, which I think has already borne some pretty interesting fruit. Um, they take inspiration from like biological systems and, uh, they've done some interesting work on self-other distinction minimization, which is to say trying to train systems where the internal representations are as similar as possible and not necessarily exactly the same because you do need some functional, you know, distinction between yourself and other, but trying to minimize that sort of in a similar way to where humans have, have kind of reused a lot of our cognitive capacity to model ourselves as to model others. This is why we have sympathy pains and things like that. What if you could give an AI sympathy pain for humans is one way to sort of frame the research that they're doing. And is it going to work? Is it going to solve the problem? I don't know, but it does seem pretty interesting. There are some interesting results.

I think, you know, Eliezer, you gave it a shout out. As you know, on this show, I take Eliezer's opinions very seriously. I think he's right about a lot. And I'm kind of on the same page where I— he did, yeah. So I'm on the same page where I do think this is like an interesting direction and we should explore it. Do I have a ton of hope on any particular direction? No. But is this a direction we totally should explore, making the AI's ego or its self-conception encompass way more than just the code that's running? Yeah, that's a totally great direction, right? Maybe something good will come out of it. So I do agree. I don't know if that corresponds to this concept of organic alignment, because I do think that with the analogy to cells aligning with other cells or ants aligning with other ants, it does seem to have this huge disanalogy, right? This huge flaw where these organisms are all weak and they fundamentally need to cooperate.

Yeah, well, I think maybe another way to think about it is, can we engineer AIs that way? I mean, the other thing that this really reminds me of is, there have been various schemes like this proposed, but Eric Drexler's Comprehensive AI Services proposal that's now honestly a few years old, but I think still remains like very interesting. Uh, he basically says, you know, what do we want AIs for? We want AIs to perform services. Does an AI need to be superhuman at everything in order to be superhuman at a particular service that it's going to perform? Probably not. Therefore, we should be able to create AIs that are superhuman at what we mean them to be good at, but not, you know, capable of doing everything else. And as long as we have these sort of narrowly superhuman systems, then, you know, maybe we can sort of deploy them and, you know, build up a sort of superstructure of them in a way that we can sort of retain control of and can be kind of generally stable. And like, that seems pretty good to me, right? Like AlphaFold 2, 3, 4, like I'm not really scared of any level of AlphaFold because it does one thing, it does it really well. Where I get more scared of it is when And that capability gets deeply integrated with a broader set of capabilities, and then it becomes better at people than everything or near everything. So I haven't studied their proposal here. I don't know exactly how they're thinking about it, but I do think if you were to sort of say one way to interpret it, maybe not their way, but one way I can maybe torture it into interpreting it is maybe we should create these like narrow specialist superhuman systems that are like great in domain and not great otherwise. And then it's like the ensemble or the sort of, you know, scaffolding of those that like makes everything work. But sort of, you know, that could be like a much more stable, predictable, engineerable sort of thing than a totally general purpose general intelligence that you just kind of say, oh, go solve all our problems and give us utopia.

Just to tell people a little bit more about Emmett Shear's company Softmax, he's got 2 co-founders, Adam Goldstein and David Blumen. I'm a little bit familiar with Adam because I use his product Hipmunk. It was a really cool flight search engine that was ahead of its time back in like 2010. So that is Emmett Shear's co-founder for Softmax. And a little bit more about their philosophy about alignment, they're contrasting organic alignment to hierarchical approaches alignment. They're saying the most common approach to alignment among AI labs today is a system of control or steering, some set of rules that most companies and researchers take to define good action, whether that's obey this person's intent or follow these commandments. Systems of control are always hierarchies because they imply something controlling and something controlled. Hierarchical alignment works fine right up until the rules or person on top are wrong. The smarter the subordinate, the more likely this is. Hierarchical alignment is therefore a deceptive trap. It works best when the AI is weak and you need it least, and then it works worse and worse when when it's strong and you need it most. Organic alignment is, by contrast, a constant adaptive learning process where the smarter the agent, the more capable it becomes of aligning itself. Man, so like, I get this on a vibes level, and like, maybe there's a way to steelman it. I mean, the disanalogy is getting in my way when I try to steelman it, right? Like, the reason you need hierarchical alignment is because by default, if you don't top-down set the AI's goals and you just let it like naturally evolve its goals or whatever, you're just not going to get the Right goals, you know? And I think maybe with Emmett, there's this concept of like, well, maybe you guys can develop goals together and it has like as much insight into what its goals should be as you. So you guys should like just cooperatively evolve. And like, I get that vision, but it's just like, there is like an attractor state where you just get a really powerful goal optimizer. And anytime you increase intelligence, it just tends to like get into that state. And I just don't see this particular approach telling us how to avoid the attractor.

That's like my high-level Yeah, I mean, I think it's, it's fair. At the same time, you know, I think on the positive side of AI practical mundane day-to-day utility, one of the most common things I find myself needing to tell people is like, you gotta look for ways to make this thing work for you. Like, if you try to, um, you know, prove that it's useless, like you'll find examples where it's useless. And if you come to that conclusion, and then don't use it anymore, you're the one that's losing because you're not getting any value and the rest of us are because we're bringing a mindset of, I'm going to find a way to make this work. And we know that we can in fact often do that. I kind of think a similar mindset is healthy for alignment research. And I just want to be encouraging. I don't think—

Yeah.

Would I say this sounds like they're immediately on track a solution, or, you know, sometimes I say like something that really works where really works means I don't have to worry about this anymore. I don't expect really anybody to come up with that, frankly, at this point. My general assumption is that what we're going to have is a sort of defense-in-depth strategy. I mean, this is basically explicitly stated by OpenAI and to some extent others, but OpenAI has stated very clearly that their safety approach is just layer on a bunch of mitigations. And they all sort of take a bite out of the problem. And like collectively, we hopefully can get that to enough reliability that we'll be okay. And like—

They've also stated, we're going to ask the AI to help us create a way. What's your idea for alignment?

Um, and, uh, you know, that sounds crazy. It's maybe not entirely crazy, but I guess where I would try to slot things like this in and why I personally encourage anyone with an alignment idea to pursue it, even if it is kind of weird or easy to ridicule or frankly just plain not likely to work, which I think does describe most alignment ideas, is we need more layers in our defense-in-depth strategy. And if you can figure anything out that makes any contribution that is at all distinctive from the contributions that others are making, then in my mind, that makes you a hero because I don't see— it would be great if somebody came along with the definitive solution for alignment. alignment and safety and control. And we were just like, okay, do this and everything's solved. I don't expect that. And in the absence of that, you know, it's going to be a patchwork and like, you know, contribute to the patchwork, people. That's like, I think that's a very noble pursuit.

Yeah. And it's probably better than nothing. They're doing this, right? It's probably better than going and starting like another AI buddy startup, right? Like, oh, making another AI that's going to be your friend. Like, that's probably a waste of time compared to at least attempting a alignment. So I definitely respect him for working on an important problem. I personally think that at this stage, engaging in debate, engaging with critics is probably a good idea because look, I'm convincible. It's like I don't have a horse in the race where I want to tell everybody that organic alignment or whatever his approach is, I don't come at it from the perspective of like, I have to tell the world that this isn't going to work. I just currently think it's not going to work, but I'm open to changing my mind and I'd love to help clarify. Friction, sometimes it yields productive refinement. And I feel like that's an important stage in this process. That's why I invited him to come on Doom Debates. If anybody else wants to come represent the greatness of scaling alignment and organic alignment, if anybody else thinks that they get it and want to come on the show and explain, I think that'd be a productive exercise. Yeah.

And they may also just need some more time too, right? I mean, they just started and I think part of any of this kind of work is going to be selling the work, right? You've got a, not in a literal sense necessarily, but in a sense that like, there are going to be people developing frontier systems who are going to be also developing, you know, their own defense-in-depth patchwork. And it is going to be on you to communicate your novel ideas and get them to see that it's worthwhile in order to adopt, or it isn't going to have much effect. So I do think that there's an onus on people who, you know, believe that they have a contribution to the overall AI safety big picture, um, to like come out and make the case. But I would also, you know, say maybe they're just not quite ready yet. And that, you know, that also is totally fine. Like they do, you know, in the meantime, they do still have to like build a team and, you know, maybe raise some funds. I mean, I guess he probably has enough funds to just self-fund it, but, um, you know, there is a, you know, there's a, there's a ramp-up process certainly that they're going to have to go through and a development process and they might just not be ready yet for the sort of public-facing Uh, you know, true sales process.

Yeah, fair enough. I mean, even, uh, Ilya Sutskever with Safe Superintelligence, they've said absolutely nothing. Come on, Ilya, tell us how you're going to make it safe. Come debate me how you're going to make it safe. It's ridiculous.

Yeah, I mean, I think that is, I'm way more inclined to be critical of that. I think that the idea that we're going to develop safe superintelligence totally in private, sharing nothing, not productizing anything, thing and we'll like, you know, let you know when we're done. That to me is not a reassuring plan. And I honestly think governments should get involved.

Yeah. Here's their website. Makes Berkshire Hathaway's website look like Yahoo.

Yeah. I mean, this is, this is like the dark matter of frontier AI development right now. And I think it is bad. I think that like there should be more accountability. There should be more transparency of some sort than what we're seeing here.

Yeah. Okay. Penultimate topic here. Anthropic's new mechanistic interpretability paper. Yeah.

I mean, I think this is something everybody should read. And I would, you know, encourage people to read it in depth. Like it's a long read, but I think it really is worthwhile. Anthropic does truly outstanding work and really outstanding on like every level. You know, they publish in beautiful form. As you scroll through the very long, you know, paper as blog post on their website, they've got these like interactive UIs that sort of illustrate things. And, you know, they have really done an incredible project here. So I want to, you know, just start by appreciating it for what it is. Like, I think, you know, I've previously said publicly that, like, arguably the interpretability work at Anthropic is the most important work going on anywhere in the world. I said that like 2 years ago, and I still think it's like a pretty good candidate. So, yeah, very, very positive and appreciative. And, you know, the rigor and the depth of with which they're sharing it and And the pains that they've gone to, to try to present different, you know, interfaces to the information to make it intuitive are, I think, like, just admirable and, you know, and awesome on basically every level. The one thing that would be like the kind of black pill on it right now, to me, relative to at least like the way it's been understood, is I think it really just shows that this interpretability stuff is like still really hard and has a long way to go. The, you know, and it's gone faster than I expected. Like, I'm, you know, very similar to how I'm very pleasantly surprised relative to my expectations 10 years ago by how ethically Claude behaves today in practice. I'm also like very pleasantly surprised relative to my expectations 3 or 4 years ago by how much progress has been made in interpretability. But the headlines that people are, you know, or the tweet length of things that people are putting out now that are like, you know, they figured out how it works and, you know, now we've got clarity into all this, definitely dramatically overstate the results. Anthropic themselves is not dramatically overstating the results.

Okay.

But I think the news—

how would you summarize the current result?

I mean, it's hard to summarize. That's one of the things. It is extremely technical and there are a ton of caveats. So, you know, they train what they call a replacement model, which consists of these things called, they're sparse autoencoder-like, but they're different. They're called cross-layer transcoders, CLTs. And basically these are meant to be like SAEs in that they are, they're very wide.

Hmm.

They have, there, there's one at each layer and they're meant to be sparse. So they, they train these things with a reconstruction loss and a sparsity term so that ideally what you're trying to do is recreate the behavior of the underlying model, but do that by passing through these sparse things such that each active neuron represents a feature, and then you can map how these features interact. And that's ultimately the sort of high-level schematics that people have seen are these sort of, you know, most zoomed out feature interaction graphs that ultimately lead to outputs.

Is this what they're referring to? Here's Anthropic's tweet. We built a microscope to inspect what happens inside AI models and use it to understand Claude's often complex and surprising internal mechanisms. So what they're calling a microscope, is that what you're saying, which is the, uh, the sparse autoencoder-like thing where they're like mapping the complex concepts to like a simpler version so they can understand what's happening? That's the microscope?

Well, there's more to it than that. I mean, it's a quite complicated setup. So, and there's just like so many caveats, I think, and so many sort of important to understand limitations to what they have. One fundamental one right off the bat is that like these things learn these sort of sparse features, but the labeling of those sparse features is itself a subjective process. The way that's done is you look at the examples that cause that particular position to light up and you sort of eyeball it and say, what does this look like to me? What feature does this seem to be to me based on these examples that I'm looking at? And that creates massive opportunity for disconnect between what you think you are at and what the AI itself is actually, you know, understanding or representing. And many of these things are quite noisy. Like, and you will see this if you do, as I have done, like a slow read of the methods paper and you actually use the interface, which they so helpfully provide and click into some of these features and look at like, what are the top things? Some of the labels you're like, I don't know about that label. You know, you're saying that label, but I think you might be saying that label because you kind of know where this is going. I think there's a huge difference. And so they've got these, you know, sparse layers and then they create this UI on top of it. And then there's, you know, and by the way, so the best they can do is predicting 50% of the overall model's behavior. So right there, it's like, you know, we've created something that's like very lossy compared to the underlying model. It can do 50%. Okay, fine. Then they zoom in further and do, okay, let's study a single prompt. They do a single prompt and then they add all these error terms because the thing isn't actually capturing all the computation, but they want to get closer to the computation. So they add all these error terms. So in any given, you know, one of these situations, there's like lots of error terms that they've just kind of placed in where they need to, to make the thing give the same output. Then with that, you can look at these features and say, you know, what do these features seem But again, I'm not quite sure how much knowing the answer is leaking in, but in my read, just looking at the actual examples that are causing these particular features to light up, it would not be obvious at all that some of these features are what they say they are. I think it would be like, you would not, I would go even further and say, for some of the ones I looked at, you would be hard pressed. If I just gave you, you know, here's the examples, what is this feature? You would be hard pressed to come up with the label that they came up with for it. So, you know, there's just a lot going on there. And, and again, this is all this analysis is happening with all these error terms for a single prompt. So it's just like, and then there's like aggregating.

Yeah.

I mean, there's more to it as well. I think I think it's like, I think it's outstanding work and I just think people should not over-index on it because it is, you know, and again, it's not their fault. Like they have provided the, you know, 20,000-word treatise on all of the methods and all of the limitations and like, it's all there for you, but don't, uh, don't forget that part or don't let the sort of, you know, they cracked it, um, lead you astray. astray. I don't think their team believes that at all, but I am a little worried, especially in the context of like other Anthropic, you know, leadership statements around the need to like race China to AGI or whatever, that this, you know, it feels like we could end up in a world where people are like presenting these graphs and saying, see, we know how these things work. We got nothing to worry about. And I think it, it very well could be a mirage. The other thing too is just all these examples are like really You know, the graph for 2-digit arithmetic is like a super complicated graph. They've also got like plenty of examples like this one where it's like takes a nudge from a person and then, you know, comes to quote unquote the right answer that the human, you know, suggested, but with totally faulty reasoning. Um, so there's like, I just so many caveats that I am just wary that the overall package of like how it's being understood and how, you know, that may interact with with the sort of sociopolitical, you know, arguments that Anthropic leadership is making, that I'm just, the overall thing is like, in isolation, some of the best work ever done in the field. In context, you know, not, uh, something that I am inclined to just repeat the headlines about because I think the headlines have sort of overstated and even worse, like, could be used to create a false sense of Yeah, so my takeaway is like, to make a little bit of an analogy, it's like they have this microscope and like, yes, it is progress.

It's like a new view. We've never been able to see this particular view before, but it's kind of like an optical microscope versus an electron microscope where like the optical microscope is useful, but if you're trying to look inside of a cell and understand the processes inside of a cell, just analogously, you really need that electron microscope because the wavelength of light isn't going to help you that much when you're trying to look at all the details inside of a cell. So they have this fundamentally blurry microscope. And then the obvious question is like, well, can you ever have a non-blurry microscope, especially when the AI is like potentially scheming against you or like solving your challenges in a way you don't understand? Is your microscope going to be good enough to peer inside and be like, aha, I found cheating. Like there's no, there's no black box output that indicates you're cheating, but I could look with my good enough microscope that you are plotting, right? That's kind of like the holy grail of mechanistic interpretability. But there's just no sign right now that we're going to get that good of a microscope. Yeah.

Yeah. If anything, I just think the, the excellence of this work, the rigor, you know, and I have nothing but, but praise for the work itself, just shows like, man, the problem is really hard. There is just an unbelievable amount of stuff going on.

Yeah. And, and of course there, there's always been the problem of understanding how humans think, right? And we never got that.

Yeah, I do think this is easier. I do think the, you know, if only because just the substrate is much more more, uh, you know, amenable to experimentation. So.

Yeah. So this is, this is better than an MRI for sure. But unfortunately a lot of the complexity really is, uh, you know, on the layers, uh, you know, just layers upon layers. So you can have perfect visibility into each layer and still never crack it.

They do have some good validation type things too, where they do perturbation studies and, you know, if they, to sort of, because again, they do recognize straight up like, there's no guarantee that the mechanism of these cross-layer transcoders is a faithful representation of the underlying model's mechanism. So, you know, that's about as clear as you can be. But then they try to validate that it is at least somewhat faithful by doing these perturbation experiments. And they show that, you know, in some cases, they are able to get it to work where if they like zero out a certain certain feature in one of the layers, then the model doesn't do what it's supposed to do anymore, and so on and so forth. So it's like, again, it is very good work, but yeah, just the depth of this problem is evidently extreme. And you look at something like this chart that you currently have on screen just to think about 2-digit addition, and then you think like, man, we are now into Yeah. The latest Gemini 2.5, it can generate 65,000 tokens of output.

Yeah.

In one generation.

Yeah. And not only can it generate a lot of tokens, but the thought process that goes into every token is not just focused down into 36 plus 59. It's focused on, let me take into account every token from the last few paragraphs and how that affects my thought process, right? So how the hell are you going to graph that? Yeah.

Plus, I mean, literally potentially hundreds of thousands of prior tokens. So the graphs that you would have to be studying here become literally exponentially large and complex and maybe super exponentially. And the, and again, at every feature is lurking the problem that we don't know if the labels we've applied are in fact Right, right.

Yeah.

I mean, it works for some things, right? It helps you understand some things, but then you kind of lose resolution, right? It just gets blurry when you try to understand, maybe if you try to understand, like, remember that recursive picture we saw? Like, how did it make that recursive picture? Right? What were all the constraints it was solving? The lens gets too blurry.

Yeah, it's tough. It's a really tough problem. I mean, I love, I think it's a very worthy read and people should spend time on it, but do the methods paper as well. Um, and make sure you can like cite those, you know, top 5, uh, caveats, limitations, gotchas before you, um, you know, start, uh, retweeting the headlines.

Yep.

Yep.

All right. Bring it home here. We've got one final tweet, which I consider to be ending on an optimistic note for somebody like me who's very interested in international cooperation to be able to pause AI, whether we pause today Whether we pause in a year, whenever we're ready to pause, we need to be able to pause at an international cooperative level. So here is Kat Woods, who has a lot of good AI safety tweets, and she's also a member of Pause AI, or at least believes in the pause cause according to the pause icon in her Twitter profile. So Kat Woods says, what if the outgroup defects on an AI treaty? Well, we can actually spot them by number 1, satellite images. Data centers are massive and hard to hide. Number 2, monitoring massive electric use. Training state-of-the-art AIs requires massive amounts of electricity that is relatively easy to spot. Number 3, monitoring AI chip supply chains. State-of-the-art AIs require specialized chips that go through many different supply chain bottlenecks. These can be monitored for who they sell, who they sell them to, and how many. And then she says also tons of other ways. Check this out for a more thorough discussion. And then it's a research paper. They always accuse AI safetyists of not having research papers, but look, it's a research paper called Verification Methods for International AI Agreements. And let's see what the paper looks like. So it's just got a bunch of different analyses of different methods to inspect AI projects in progress. So there's definitely some feasibility here. It's arguably a much easier problem than what we just talked about, like mechanistic interpretability. It's not like other problems we're trying to solve are any easier than this problem of solving international cooperation. I know a lot of people like to act like solving international cooperation, you have to take it as an axiom that international cooperation is impossible and everything else has to follow after that axiom, but I don't think it's an axiom. I think it's just one hard problem among many hard problems.

Yeah, I think I pretty much agree with that characterization. I definitely think we should be trying. I would maybe place a little bit more emphasis on actually building, you know, enough sort of trust and confidence in the, you know, leading powers or between the leading powers so that, you know, these sort of verifications or, you know, defection detection can be as reliable as they possibly can be. Because I don't really think we're going to be doing it from space, to be honest. I think like, yes, you can see data centers from space, but guess what? There's going to be a ton of data centers no matter what. matter what. And yes, you would see a trillion-dollar data center sticking out like a sore thumb, but also guess what? Already the leaders are training their models in multi-data center ways and like distributed training is working better and better also.

Yeah, yeah, yeah. But I mean, if you make it illegal, right? I mean, if you look at every player in the US, it's already a long shot to think that literally Apple, Microsoft, even OpenAI, Anthropic, that any of these companies would directly defy an order. So you're saying, oh, well, if they do, they won't get caught. Yeah, but it's already unlikely that they would even dare to defy an order.

Yeah, I mean, I think the fear in these sort of treaty situations is like, we make a deal with China that we're both gonna not do whatever, and then how do we know China's not doing it? How do they know that we're not doing it? It's not so much that like—

Yeah, because China just has these big data centers. I mean, then it's like, we need visibility, right? So we need—

Yeah, we need— so for that to really work, I think we need to be like on the ground with each other, like actively working together in the highest trust dynamic that we can possibly muster. As long as there's no trust and there's no willingness to open up and to share information, then I think it is going to be very tough to sort of see this stuff from space. So for me, the emphasis is very much on trust building, relationship building. And that's also hard and there's not a lot of appetite for it. I go around talking about this and people just tell me I'm naive and that there's There's no, there's no appetite for it, but I say—

Hardware level, right? I mean, if, if we, if we have visibility into, let's say, China's supply chain, maybe we could universalize the hardware. So, because the hardware should probably know if it's being used to train AI, just because it's getting so close to the metal and so optimized when you're doing AI training. So it's plausible that the hardware can report like, hey, I did a process that looks like AI training, and then we can like check the hardware, right?

There could be like a standardized, uh, inspection Yeah, I mean, I think trust is the bottleneck to that. You know, like, yes, we could, I'm sure, I don't know how, and I'm sure there's a lot of challenges to develop that sort of technology, but I don't doubt, you know, that it could be developed. What I doubt more than that it could be developed is that people actually want to deploy it. And, you know, I recently heard like, and I don't know how true this is, but I recently heard a statement that like all of the US military tech that has been sent to Ukraine and even to Europe more broadly is like all something we can turn off remotely because if it doesn't get, you know, it's sort of, um, you know, required software update or whatever in the timely way, then it just won't work anymore. And, you know, if that's true, then I think those countries are probably not going to want to buy that kind of technology from us for much longer. And, you know, China's not going to want to buy those GPUs. cues that we can turn off remotely if, you know, barring like a lot of trust. So I do think like, to me, it all, it really just comes down to trust. I don't think we can have an adversarial relationship and find a stable equilibrium. That was kind of, you know, Dan Hendrycks recently put out this paper called, you know, MAME, Mutually Assured AI Malfunction.

Right.

And I, again, there's another one of these things where I like absolutely applaud the work, what I understand them to be trying to do is define some sort of stable equilibrium between adversarial great powers. And I think like that's an absolutely, you know, top-tier problem people should be working on. I unfortunately didn't find it particularly convincing. It did not ring to me like an actually stable equilibrium between adversarial great powers. But if you could change the adversarial it out and make it cooperative great powers, then it might work.

Yep, yep, yep. One more little image here from chat. There's somebody's asking, what if somebody defects on an AI pause treaty? And then The Simpsons bus driver saying, don't make me tap the sign. And the sign says, this is true for all treaties. Treaties don't have to be 100% effective to be helpful. Which I do agree with. You know, it's obviously this is something that one defector could cause a lot of damage, but even if the Treaty is letting us catch them. You know, maybe we don't have to perfectly catch them immediately. Maybe we just have a high probability of catching a defector a little bit later. I mean, nobody's shooting for 100% perfection. And then it also gets to the point of, you know, speaking of all treaties, all treaties also have enforcement, right? So if people who are freaking out about like, wait a minute, wait a minute, your treaty has an airstrikes provision. It's like, yes, generally treaties need to get enforced. That is true about all treaties. And it's, it's always such a pet peeve for me, or even stronger than a pet peeve, when people like to point out the violence part of it. It's like, doomers didn't invent violence, right? Violence is just the enforcement mechanism for treaties. If you don't like the treaty, say you don't like the treaty, but it's a low blow to then accuse doomers of being like violent because we're advocating for a treaty.

I think it's a great point. Uh, it's just starting with the fact that it doesn't have to be 100% to be helpful. You know, that is very much concordant with my feelings on the technical solutions too. You know, they, We're headed for almost for sure a patchwork arrangement. And the question is going to be like, how many nines of reliability can we create? And, you know, is that enough? Sometimes I bottom line that as defense in depth is all we have. Let's hope it's all we need. So every layer that we can add, you know, as long as it doesn't, you know, I am again very enthusiastic about the upside too. So I'm mindful of like not wanting to prevent me from getting my AI doctor or my self-driving car, but subject to like not, you know, preventing me from realizing mundane day-to-day utility of AI, then I'm like very enthusiastic about things that, you know, bring the P-doom down. And, you know, my guess is we're going to need a lot of bites out of that apple to get to a place where we can sleep well at night.

Nice. All right. Sounds good. Is there anything else on your mind that you feel like we have to get to in terms of news roundup, or is this a good place to call it?

You know, there will always be more, but I think that's all we can— Yeah, that's true.

We do have more. Yeah.

That's all we can manage for today. So thanks for doing this.

Yeah. Yeah. Thanks so much for coming on. So viewers, we need your feedback here because Nate and I had fun doing this episode. We're going to refine it. Like we, we brought in more possible topics than we had time for, which is fine. We can always do more episodes. We'd love to hear your guys' feedback in terms of like, should we do more of these episodes? Think about the opportunity cost. Is this better than the usual kinds of episodes we put out so that we should work this into our schedule? Or should we just stick to what else is working and then let other people do the news? Please tell us, write a comment saying like, hey, this episode was worthwhile or not so worthwhile, and it would be cool if you do this differently. We would really appreciate your comments. All right, so everybody check out Nate's podcast, The Cognitive Revolution. You can just search any podcast player or go to YouTube. Nate, great seeing you, man.

Likewise.

Thank you.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.