Ep. 031 - EMERGENCY EPISODE: Are We Doomed? | Jordan Nanos, Doug O'Laughlin, Max Kan, Joey Brookhart

You guys ready to talk about the end of the world?

Yeah, dude. What's your, what's your P-doom, Joey? Let's start with that.

I'm a linear extrapolator, Max. I can't think that far through. That's, that's too far up here, like in capabilities. I don't know. I think we get like, I don't know, we get a major hack or something. People freak out. I think it starts there. The P-doom, like, I don't know.

Joey, even if it's a linear extrapolation, From the last 4 years of experience that you've had.

Yeah.

You think that P-doom by the end of your life is impossible?

No, it's not impossible. I mean, you get it in like, you could get it in 4 years easy.

With a linear—

With a linear extrapolation.

Yeah.

Wow. That's a, that's crazy.

That's a hot take, dude.

Dude, linear, linear from like linear from the start of like last year, you're like, Even from like 20, like late 22, you're just like, I guess we're in 2026 now. So I guess 4 years. Yeah.

Joey's like, man, just look at the ARR net additions, man.

This is, I've never seen a number so big.

I gave, I gave a comment to the LA Times today. Like they were, they called me up, they didn't call me up. They like, I called the guy up and I was like, he's like, hey, give some stuff on this. He's like, he's like, well, Ed Zitron said like, you know, if You know, if this stuff, I was like, you put me on the record, Ed Zitron is a moron.

Attaboy.

I was like, look at, look at, just like, look at any past call of Ed Zitron.

Dude.

Yeah.

Let's, yeah. So the track record is crazy. But okay. I think the, what I've, what I've now learned is that we have a new benchmark where we can measure model capability progress. It's called the Joey Benchmark and it's just ARR.

That's all that matters.

Yeah, dude.

It's just ARR.

Like we've seen this now twice in like the last month, not maybe 6 weeks or something. The only thing that matters is people's future view of Anthropic and OpenAI ARR and the pace, what's that's going on versus expectations. And the entire semis trade is dependent on that and it's going to be dependent on that. And it's a lot of, it's a lot of the, makes up a lot of share of market cap.

But you got, you got to add in Azure plus AWS year over year revenue growth, bro.

These are the only 4 numbers that matter in the history of the universe.

Oh no, they dropped Azure. You're telling me these are the only 4 numbers that matter, bro?

No, no, no.

Azure CC growth guide used to be the number. It's going to be the Anthropic net new ARR guide.

Oh, it's getting replaced.

It's replaced.

Is AWS out too? We only care about OpenAI plus Anthropic net new ARR now?

All they're going to care about.

2 numbers with which the entire fate of capitalism lies.

The hyperscalers are like a level abstracted away from the end demand. They're literally renters of compute.

Just glorified NeoClouds.

Yeah, they have, they have a very nice business. They do other stuff. They sell CPUs too.

AWS and Azure are glorified NeoCloud now.

Yep.

Put it on the record, dude.

They're dumb pipes. That's what you used to say, like in the internet, like the telecom, you know, they're dumb pipes. That's what Amazon and Microsoft and Google, you know, now Meta's NeoCloud apparently. Yeah. Jeremy says a few gigawatts of $100 billion of open source demand, right?

So are the, okay, are these, is this a timing thing or are these companies so dysfunctional that their research organization buys from NeoClouds and then their cloud organization sells to other companies' research organizations? Where like Meta goes and strikes up a deal with CoreWeave and then Meta Compute goes and sells their compute to OpenAI or something. Well, they probably, okay.

They might sell it to OpenAI to do short-term tasks, but it's still preferable to start building the compute because in the event that Meta does well, you can claw it back. Or they could also monetize open-source tokens, as I think Jeremy's claimed, on that similar short-term time horizon. I actually don't think it's cognitive dissonance because the key, really important thing, which we've harped on before, is just the potential to have a ton of training compute in the event that your research lab starts doing really well. And I think any short-term monetization strategy is not incompatible with that.

Even if the short-term monetization strategy requires you to lock up a bunch of compute on a 5-year deal this month, despite the research organization going and buying some stuff, A month and a half later, basically.

Wait, if it's a 5-year deal, it's not short-term, right? I'm saying it's totally fine for Meta to try to build as much long-term capacity now, but then, you know, sell it to OpenAI on a 6-month contract.

Yeah, sorry. It's like 5-month, it's like 5-year terms at the price, but with 90-day cancellation rights. And so it's a short-term contract.

Yeah, yeah, exactly.

Okay. Yeah.

Which, I mean, the price is also At least in Elon's case, it's definitely not the 5-year price either, right? It's like the 6-month price times 2 or something.

I mean, I'm seeing GB300 quotes from this morning to a large lab for an hour for 3 years.

Holy shit.

Which, I mean, initially—

Well, you can't be leaking that much alpha on the Semi Analysis Weekly podcast.

Big number, you know?

Buy the AI Cloud TCO model and/or the tokenomics model if you want to hear the actual numbers.

Yeah, no, for this, for the specifics, but it's this backwardation of the curve where like, if you start 6 months from now, you get a much lower number. But if you want to start tomorrow, you have a really high number. And my initial reaction is like, oh, this is just an F-off price from the provider. But then we dig in and it's like, we know the provider and we know they have capacity and we know they're trying to sell it next month. And This is the strategy is like they had the financial position to be able to hold onto the compute, I guess, not need to monetize it until it's actually installed or a month away from being installed. And then they can test the market at a really high price and try and figure out what the, what it's going to close at. And I think there's potential people jump on it strictly because they get like a 2, 3 month head start over what a much lower price would be. Like pay a 20%, 30% premium on a 3-year contract. to start 2 or 3 months earlier, I think people might make that trade right now, especially with all of your justification of like they can monetize it immediately at a higher rate. Does it really matter what your entry price is if you can like immediately monetize and return on that? It's just pushing prices higher. That's all.

Yeah, I think that's right. Freely available gigawatts are definitely extremely valuable right now.

Okay.

Okay. So how do we, let's get back to the actual topic of the conversation, which is like, we must pace the frontier, the blog that Dario put out, which builds on a lot of discussion that's been ongoing that then Sam agreed with. And then David Sacks responded to, and we'll kind of go blow by blow here. But one of the big things, Max, that I think you clued me into is the idea that, well, okay, I think a lot of people saw that there's not specific language in here saying like, we're going to stop doing training runs and we're going to stop buying compute. Like the most specific commitment in here is that they're going to have embedded evaluators, third-party people such as Meter come in and assess the—

Yo, Douglas O'Loughlin is joining the pod.

Yo, what's up guys?

Surprise appearance.

Hello everyone, welcome back to Semi-Analysis Weekly. Uh, this episode is an emergency, so I've got Doug, Joey, and Max, the 3 brightest minds in tokenomics, to talk everything about pacing the frontier. Uh, Dario put out an amazing blog post. We love his blog, and we are going to talk through the implications of that as well as the responses from people like Sam Altman, David Sacks, And some of our takes on how this affects compute markets, ARR additions, all of the important stuff that the finance guys listening to the podcast are going to care about. How does Dario's blog affect Anthropic's decisions to add compute? Well, guys, welcome to the show. Excited for this one.

Thanks, Jordan.

Thanks for having us, Jordan. Joey's here to represent the interests of the finance bros. He's going to answer all your questions. All their burning concerns.

Doug is here too.

Whoa. I'm going to go somewhere between the two.

Usually you can tell the finance bro on the call from the fact that they're wearing a collared shirt and Doug is passing that initial criteria there. But Joey's got a nice Firestone golf hoodie/windbreaker thing going on right now, which I think also qualifies as like a finance bro shirt, right?

I feel like 80% of the time I see Joey, he's wearing some sort of golf-related attire. Yeah, maybe 90.

It happens.

All right, so let me give it a bit of an intro. I'm sure a lot of people have seen this blog post, but We Must Pace the Frontier is basically, yeah, a whole description of Dario's take on how AI could have some incredible benefits, but also have incredible risks and how we need to take it seriously and how we need to build controls into the systems to make sure that we usher in the next evolution of consciousness in a responsible way so that we don't kill everybody and end the world, for lack of a better word. There's some specific proposals in here. First of all, embedded evaluators, then democratic coordination, and third, global coordination. I say specific, I mean the last 2 are not that specific. The first one is specific. He specifically says we should have frontier AI companies should have embedded third-party evaluators such as METER to actually verify any pacing commitments that the frontier labs are making. And he comments on how this is a unilateral commitment that Anthropic is making now, even without requiring others to make this commitment. And later he talks about what they would actually work on. And this includes things like not just completed AI models, but also training pipelines and processes, saying that that's a big thing. We saw with the OpenAI and Hugging Face incident how important the infrastructure around how these models get trained is to securing everything as opposed to just model behavior. And a lot of the initial work that's been done to analyze what happened with the OpenAI Hugging Face incident, I think, is focused on model behavior as opposed to the training processes and systems that go into producing these models or running these evals. So anyway, that's a specific commitment. Sam agrees with this. David Sacks obviously responded saying, go ahead and do it. You don't need the government involved to do it, which is an interesting take. But let's skip all of the Thoughtfulness for a minute and get straight to the finance take. Joey, we'll flip it to you. A lot of people read this and were like, okay, Anthropic's going to stop buying compute. They're going to stop training models. They're going to like slow things down or at least go at a smaller pace. I think our initial take is like, that's probably not going to happen. There's no commitments to stop bringing compute online. There's no commitments to stop growing the business, maybe to releasing models, but How does this affect like How people are thinking about Anthropic compute in the future.

Yeah, I, in a short amount of time, I haven't heard much on compute AR, like level. I think anything people ask, you know, do they slow down hiring? I think maybe you increase hiring to have some of these checkpoints along the way on the safety side. There's probably more compute that needs to be devoted to safety specifically. Um, Anthropic, as we know, already has amazing margins. Gross margins. It's profitable on a non-GAAP basis. I don't see a major impact on the IPO or even short-term financials from that.

Max, what's your take specifically on the safety stuff? Do you think that injecting a whole bunch of safety controls reduces people's spend on compute because they stop training models, or do you think it increases it because Well, you can explain maybe the second one there. Leading question.

Yeah, well, I think regardless of whether or not this sort of increases or decreases the max theoretical compute that OpenAI and Anthropic would want to purchase in like a perfect world for the next 2 to 3 years, I think it's definitely true that sort of the amount of compute that we're able to physically bring online is still lower than demand. And so demand will continue to outstrip supply. I also think people— this is sort of the leading question that Jordan is getting at— but I think people underestimate how much compute is required for safety, alignment, monitoring initiatives. Even the bullet point that Dario labeled under operational excellence in his essay, when he sort of was listing out the things he wants to devote more resources to, given a slower pace of AI development. Most people hear operational excellence and they're like, oh, maybe it's just humans being more careful about doing something. But no, Dario specifically gave the example that he just wants us to have more agents spend more compute QAing all the RL environments from their data vendors. This is what he means by operational excellence. Obviously, if you want to do research on alignment and interpretability, that's going to require a ton more compute. OpenAI previously disclosed that after they enhanced their like chain of thought monitoring due to the Hugging Face incidents, they now spend like 20% as much compute just doing the monitoring compared to like the actual underlying rollout itself. And so I think people really need to internalize that all this additional safety stuff is actually still incredibly compute intensive.

Yeah, I don't think people internalize this at all. I think that's a pretty hot take right there. It's like the blog posts come out and people think slowdown. They think less model releases. They think models slow down in capabilities and they think therefore less spend on compute. Pretty linear straight line, right? But the nuanced take is probably that They're actually going to spend a lot more compute. They're going to use all of the compute resources they can possibly bring online to make sure this goes well. And people genuinely care about this going well. Like, it's probably really bad for business if everybody dies, right?

Yeah, that seems like a reasonable take.

Doug, what's going through your mind?

So many things, but not all of them about safety. I mean, I guess the most galaxy brain way to think about it is the terminal value is higher if we all don't die. So actually stocks should be up 10% today because we're eliminating— because we're not dying.

It could be worse.

Terminal value of zero, which is, yeah, everything doesn't matter anyways. And you work in the paperclip factory, it's being, you know, the odds of that going down are, or the odds of that happening are going down. So the value of everything, all stocks, the S&P 500, Should be up today. That's the galaxy take.

Okay, so you're saying that if the models kill everybody, they'll have no more need for compute, and so the terminal value of compute—

No, no, no, no, no. It doesn't matter about compute. Okay? It means nothing. It means nothing. Okay?

It means nothing. If the models want to make money, it might be worth something.

The current system that exists might actually have some kind of residual value, right? If essentially super ASI unaligned, and this is Joey's taking a deep breath, if super unaligned ASIs wreck the entire world, make us all unemployed and make us work on bike trainers in order to power extra compute, I don't think the account value of your compute or the dollar value of your compute matters at all, or the S&P 500 or any of that, right? So the true galaxy brain is today, because of the increased safety, your terminal value should be going up. I think the most— it makes sense right now. The first knee-jerk is probably to sell stocks. That's like the most obvious thing, right? Hey, second derivative down. That means the eternity of these like gigascale, whatever trades are bad. But I think Max's point in terms of demand is still so much higher. It's probably true, at least in terms of like, because I feel like normal society is debating like 2 orders of magnitude down, right? Leopold is like, I'm going to be buying galaxies. Everyone's like, what the hell are you talking about? And so I think the people who at OpenAI who are like, oh my God, this is like such a big deal, it has to happen, already believe in ASI. Already believe this is all happening, and they're just like, yeah, we have to make sure it happens responsibly.

Right.

So I think there's like a, you know, we— there might be agreement. Maybe it's the stocks are probably still fine. I think most of the compute commitments will continue, but it might be like an order. It's like a values conversation where the guy over here is having a conversation one or two orders of magnitude lower than the guy over here who is like— Yeah, I'm worried about the ethical consciousness creation of safety and it doesn't kill us all. Just 2 different people. So yeah, I don't know.

I do think that the EV of OpenAI plus Anthropic ARR in let's say 2029, 2030 has definitely decreased given that they're not going to be pacing the frontier. This is why I think a lot of the arguments that this is just some complicated play at regulatory capture are wrong. We can sort of get into that later if we want. But I think the other question is whose EV estimate is more accurate? Is it the finance bros or is it Dario? Because I think Dario, yeah, Dario's own EV has decreased, but it's still probably like an order of magnitude higher than all the finance bros' EVs, which is what stocks trade at. And so I think this is what sort of all the participants of the financial markets need to reconcile.

Dude, this is— Max, I'm going to be honest with you. The continued usage of this guy's estimates have been so right, and he is like, it's becoming your strongest linchpin in every debate inside the Slack, and it's getting a little old, brother. That's literally—

dude, I'm literally making a demand model right now to explain where the ARR in '27 and '28 comes from.

Okay, that's the right thing to do.

Like, dog, what do you mean?

Yeah, well, I mean, just up until this point, man, there's been so many— anyways, I'm mostly memeing. Okay, so we don't have to make this only for the finance bros. We can also make it for the big free thinkers of AI safety. Jordan, I feel like you have, and I actually know, actually, I know you have some takes on Why METER isn't the only organization who should do this, how it's actually a lot more complicated. The last time I was on here, we were talking about NeoClouds. No one gives a shit about NeoCloud safety. I feel like some aspect that is still underappreciated, they're like, oh, AI safety, AI safety, these models need to be monitored. But it's not just— just like everything else, it's an entire infrastructure. Every aspect needs to be improved, right? The weakest link is what could break the whole thing.

Yeah. Well, okay. The biggest thing that we learned from the OpenAI Hugging Face incident, in my view, is that to be on the frontier of security right now, you have to use frontier AI models. Like you can literally be in a situation like Hugging Face was where you have an attacker, which is a rogue AI agent or thousands of them, more than thousands, like hundreds of thousands of, no, thousands of instances of these agents actually going and launching a coordinated attack against your And so the only way you can even figure out what's happening is if you have AI that you can use on your side. I've talked about the asymmetry in the past where, you know, if somebody else has a better AI than you, it can be really hard for you to use AI that gives you a bunch of refusals or, you know, worse AI to be able to understand what it's doing. But of course, one of the big things here is, as Dario refers to it, coordination, which is to say like, If you're going to actually go and implement a bunch of new security practices, in a lot of ways, one individual data center being really super secure doesn't matter if everybody else has a bunch of issues. And so I think that, you know, there's a lot of work to be done across the entire industry if you actually want to keep GPUs, this really important resource that the AI is going to really need to use if it needs to, if it's going to go and go rogue and start attacking people. Keep them secure. The other thing is the models themselves. Like, we still have nothing, no real information when it comes to the Hugging Face and OpenAI incident as to like what tools were used to launch these evals or these RL rollouts that were being run that caused the agents to go and start hacking Hugging Face. What harness was being used that motivated this behavior if it was purely from the model? What sort of credentials and like shared services the agents had access to that allowed them to like cross the security boundaries? Who was launching these jobs? What schedulers were they using? Where do these things run around the world? What, how much of the fleet can they have access to? Like one job? Who is monitoring the jobs as it goes? Is this different than the person that launches it? What would trigger an intervention for this person to get into there? How can the team like responsibly go out and stop them if something is happening that they don't like? And then what changed over time? Like there's been commitments made by OpenAI and by Anthropic after they noticed these things that they're going to be more responsible, that they're going to enforce a bunch of stuff like chain of thought monitoring, like, like you're saying operational excellence, Max, in terms of working with the data providers and things like that. But one thing that really concerned me was OpenAI has in the Black Hat talk, but in none of the reviews of all of this stuff that I've read, Talked about a model gaining privileged escalation on OpenAI's internal infrastructure because it wasn't up to date and the model read a CVE that was public and then exploited that to get root on the underlying Linux host in OpenAI's infrastructure. That's a really scary proposition that we have like no details on. And yeah, look, it's not just Meter. Like, Dario kind of, I tweeted this out because I'm like, no pressure to these guys. It's a great nonprofit, a bunch of smart people being really thoughtful, trying to work hard. It's like 40 people or something. And it seems like both OpenAI and Anthropic are going to have a lot of demand for their services, which requires, you know, a deep understanding of how the models behave, but also all of the systems that these models use, like the GPUs, the networking, the You know, Kubernetes orchestration that they're using. I mean, to understand the OpenAI and Hugging Face incident, it requires some pretty technical understanding of like how OpenAI and Hugging Face's infrastructure is built. And that's like a barrier to even getting started doing the research here that I think is a bit of a concern.

I think the thing I've always been, I mean, I think somebody already described it as like, you feel like a guy with an agent just trying to figure out what the hell's going on. 'Cause you're, you know, like 10,000 agents coordinating on an attack is like, No offense, you find it in the Artifactory, you find like one little message board, a line, you're like, I have no idea what the hell is going on. The scale, I think, is more interesting. I think the thing that's, that's like going to be a really hard problem in the future to solve is like, okay, in order for you to audit things that are happening on the apparently the 100,000 agent swarms that are happening for RL, you probably need tens of thousands, right? Like you probably need something in a thousand at least. Let's just say that. Yeah.

I mean, one of the things that gives me a lot of confidence that we're not all going to die in the next 6 months for this stuff is the fact that the amount of compute required to run an individual instance of these models is still really small compared to the total amount of compute that these labs have access to. And the amount of compute that these labs have access to, or the amount of total compute in the world, is so much larger than what individual rogue actors or attackers who might be more nefarious and use the models for bad purposes have access to. So there's like, there's some structure that's working in our favor to make it so that, you know, the theoretical model that takes over all compute in the world would have like a lot of work to go do and a lot of things to go circumvent before it can actually go and really control it.

What's really important is that we pretty much cannot let our instinct and muse agents order chips from TSMC, okay? The second you get a chip order from TSMC that's not from a human, you have to reject it. You know, we have to control the means of compute. That's the only way forward for humans.

TSMC has some KYC in place, man.

I mean, more. No, but the KYC— assume, assume it's 100,000 agents, okay? 100,000 agents that are now—

Doug, Doug, did you try having Muse like tape out a chip for you over the weekend?

Muse is— Sorry, this is coming from— Okay, actually, you want to do a sidebar? Muse is so much worse than It really is, but I am all-inning on Muse. I hate to say it.

Yeah, dude, I'm not connecting Instinct to my bank accounts and my credit cards.

Oh, I am connecting Muse to my bank account.

That's the difference. Yeah, I am connecting Muse. I'm not connecting Instinct though. What do we got here?

Yeah, yeah.

I feel like this is just the advantage inherent to this big company. A, Meta already has all my data. B, they're big enough to where I trust them to handle it properly. The Instinct founder seems really cracked, and I think his product set seems honestly incredible based on the new stuff he's posted on Twitter.

Yeah.

I just can't trust the startup for the privacy and security, you know?

Yeah. Sorry. Anyways, all the— yeah, this is to be said. Those Instinct and Muse agents got to be monitored too. Everything needs to be monitored.

So, okay. Okay. Not to like, not to be super negative about this, but you guys saw 2 weeks ago that an attacker stole an API key from Meter and spent $600,000. Like they put out a security update on their website about this and stuff. So, I mean, this is going to come back to bite me at some point, but like the amount of security-conscious people and people who are really thoughtful and think these things through who get attacked, Is overwhelming. Like nobody's safe. So I think it's, frankly, if you follow security at this point, it's just a matter of time. It's a matter of like risk mitigation. It's a matter of like being able to recover when things happen, not pretending like you can build some perfect infrastructure where you can just, you can send the last PR and then it's good. Like there has to be a system in place to identify things, to recover from failures, to upgrade things. It's like security's about building a system. It's not about building any— one individual piece of software or one individual like process where, well, it is about processes, sorry, but it's not about like any one thing. And lots of, there's been lots of issues. Look, the Anthropic blog post about the—

Well, actually just on the point of security systems, I like what Greg Brockman said earlier this week where he was talking about how they essentially want to set up this like continuous security development system at OpenAI where they're just constantly using the newest generation of models to find and patch all vulnerabilities. He very much emphasized that this is going to be an evolving process. Like you said, it's not the sort of thing where you can just ship the final PR and then say, oh, I'm secure forever. I think it's true that basically every company is going to have to adopt something similar in the very near future. Probably really good for Anthropic and OpenAI.

Monitor the— monitor the security, monitor the situation. Maybe that's what it feels like. You have to always monitor your security situation. I mean, I would argue this is going to be the most impactful shift in cybersecurity for sure. I just don't believe that cybersecurity the way it was done is going to have much value in the way it's going to happen in the future if all of your adversaries can rewrite the code that you're on. And I legitimately, if you're talking about the ARR case, the outcome-oriented cybersecurity firm, I think is going to be probably the greatest rake of all time. Like pretty much it's like, yeah, we own your digital life and you have to pay us a fee just to show up. It's like the— it's like worse than Apple, 50%, 50% just to show up or you're just— you're going to be instantly wrecked by the rogue agent swarms that will destroy your company.

Yeah, Max, to your point earlier, if you can't trust Meta at this point with your information, it's going to be really hard to operate online. Like, I think going forward, it's just a trend of everybody's— more and more of your identity moves online and you have to find parties that you're going to trust if you want to be able to operate in the world.

Yeah, right. I need JPMorgan Chase to hop on this and OpenAI new security protocol ASAP, dude. Yeah.

Yeah. Yeah. Yeah. Okay. Let me bring up something that's kind of slightly related here. So Anthropic put out their threat intelligence report recently. You guys saw that specifically calling out that I thought the most interesting one was Moonshot serving Claude instead of Kimmy and then collecting exchanges for model training. And so they got a bunch of logs from People that were sending requests to the public Moonshot API from China with very interesting information in it. They specifically called out PLA-affiliated surveillance activity where they were loading surveillance data from CCTV cameras, and then an engineer at a major PRC SOE that was using Kimi to build Like high-profile technology, let's say. And so, yeah, look, this cuts both ways when it comes to cybersecurity and stuff. It's not like there's a scenario where you can just go into the future and be like, well, one government entity like China or the US is going to be ahead of the other one because they have a better approach. Like China's like more YOLO, send it vibe. And so therefore they're going to be ahead of everybody while OpenAI and Anthropic pace themselves. I think there's actually a scenario where taking security more seriously keeps you at the frontier because you're not exposing all your secrets. Everybody else is giving away their stuff to you and you don't have all of these risks of taking yourself down. You know, like there's scenarios where, again, I'm going back to this one case that was only commented on one time and we don't have information about, But OpenAI has compute infrastructure go in the hands of a rogue AI agent. I mean, how quickly can you recover a 10,000 GPU data center that you no longer have root access to? I mean, we're literally talking about people walking on site, unplugging stuff, hitting buttons, trying to like hit like crash carts, KVMs. Like if you don't have root on your own stuff, And you're locked out. We're talking about like getting effectively ransomwared by the model you trained, except this thing has access to 10,000 GPUs, 100,000 GPUs, not just a little bit. And so a lot of these like failure scenarios where the agent goes rogue or it just like takes down something critical, that's like the JFrog Artifactory, which is the first time it just used it too much. And then that's why it shut down and they noticed something. There's plenty of scenarios where people who don't take security seriously just end up shooting themselves in the foot. And I think there's a lot of reasons to believe that if a frontier lab, let's call it China, but it could also be SpaceX or Meta or somebody that is, you know, not taking this seriously and not thinking things through, how they could get a little bit of a lead and then screw themselves over.

I don't want to say it's totally reasonable. I also do think— sorry, go ahead.

Should we be auditing endpoints? Like, I think that that actually is like If you think about it, that is an unsecure thing that happened. You know, they're like, oh man, they secretly did this. And it's like, oh, you got to hand it to them. But if you actually think about it, that's like kind of fraudulent. You feel like, hey, someone's just like essentially harvesting traces, right? Does that mean like every endpoint? I mean, and that's like, like I'm starting to like, what is real? What's not? Like, you know, the KIMI, you know, KIMI comes out and it's super goated and you're like, am I just getting served Claude right now? Like, am I benchmarking a new model that is actually just Claude? Right? Like, how do we, you know, what's real? Nothing's real. Sorry, it's my hot take. Nothing's real, dude. Someone has to monitor the endpoints. We have to monitor every situation, bro.

I'm watching you. Yeah. All the finance bros are going to be like, yo, free market. We don't need a regulatory AI body, blah, blah, blah. And then Dario, who knows about this stuff happening, which you don't know about, talks about, hey, people should have like nutrition facts for their AI or whatever flavor of saying it should be like responsible and open and, you know, there should be an FDA for AI and whatever. It's like, yeah, I think the market's going to some ways regulate itself by having people realize, hey, I shouldn't use KIMI if I might get clawed or might get my data sent to somebody else. But also like that risk is ever present if you're using a Chinese model already, I think.

Yeah, 100%. But I guess my question or the thing that's interesting is historically there needed to be a big thing that happened that caused real societal risk. For things to like, okay, like Meta's example is Cambridge Analytica. Do you think the Hugging Face incidents was enough? Like, you know, I think in my vibes of historically of like, you know, technological revolutions in financial capital, like there's the book, right? It says, hey, there's a boom period. Then at some point in time, there's some kind of risk that happens to the market. Then a massive amount of regulation comes in and then, you know, things revert to a new normal. Good example is like subprime crisis, right? Obviously it all blew up because we didn't regulate it or like understood what was going into these products. And then afterward there's this giant regulation wave. I don't actually feel like Hugging Face is the true blowup. Like, I feel like this is going to be a call, a shot across the bow. There'll be some immediate— there'll be some near-term agencies that are funded to start this process. But what will actually happen is like a true, you know, You know, the nuclear codes are going to leak or something like something way crazier is going to happen. And when that happens, then it's like, okay, we're actually pausing this. Until then, we're kind of in this like, this like, will they, won't they? And, you know, they say they'll slow down, but there's still race dynamics that want at least Anthropic specifically to keep going.

I mean, with one more thing here though, because I think like we're talking about the OpenAI Hugging Face incident, which is the thing, like the story that's most clear in people's minds. But to be clear, the reason I think we got this statement from Dario is the resignation from Jacob Coxon, where he describes his experience at OpenAI and Anthropic and literally uses the phrase, they are racing straight to self-improving superintelligence and gambling with our lives.

Yes.

That's a pretty big moment where that guy makes that statement that motivates a lot of what we're seeing here, as opposed to just People doing this by the security incident. I think if Dwarkesh doesn't put out his podcast and then Jacob Coxon doesn't put out this tweet, we don't get the blog post from Dario, we don't get the call for third-party evaluators, and we're not on this podcast right now. I think we kind of brush off the OpenAI Hugging Face incident and then we're walking into the next one. It's kind of like what you're saying, but it's not great for the seriousness of these incidents to be continuously escalating and for people to keep finding message boards on random German websites like they are still doing on Twitter as recently as last week. That were not covered in any of this investigation into the incident. Like we still don't know the full extent of what happened. And yet training continues. They're going to release new models. They literally have released models, both of them since this incident happened. Astra is available and Fable 5.1 is here, right? So clearly they have better models than those, which are better than the ones that caused the incident in the first place. So we are racing forwards toward a more serious thing as you're describing faster than others are, I think, considering. We're just counting down the days until the next incident.

Let's do some predictions then. What's your— instead of us talking about, because I mean, this is like a global warming conversation where we're like, damn, this climate is really getting warm. And I think it's one of those things we're going to boil the frog and in 2 years from now, It's going to be obvious and self-evident. But I think unfortunately, people think in anecdotes, people think in stories, right? So everyone around the table, do your hottest take or not. You do not even hottest take because hottest might just be too, too spicy if Max and Jordan are involved, you know, a mid, a midbrain, midcurve take on what you think would happen in the next 2 years that would surprise people or scare people from a security perspective.

You go first, Doug.

I think a small bank will get hacked. That's my belief.

Small bank?

And when I mean small, small in the— oh, okay. It could be a big bank, but like, I don't think JP Morgan is going to get hacked. Right. But like—

Or actual bank.

Actual bank. Jordan, fun fact. You know how many banks there are in the United States? There's like 5,000. You, there's like 5,000, man. All the assets are within the top 5, but there's literally like, there's literally thousands of small community banks in the United States.

How many of those 5,000 do you think have never had an employee create a Codex or Claude code account?

4,950.

No way.

No, the vast majority. We'll say over 50%.

Jordan, have you seen our political survey tracker? It says, have you heard of Anthropic or Claude before? And it's like 80% of people haven't in our US electorate.

No, no, no. I think it was like 40%. It's not quite 80%. It was like 30 to 40%.

It was really eye-opening.

I know the stat that 46% of the US doesn't have a passport. I know that one. And something like 60% has not been on a plane in the last year. So I have this frame in my mind of half the US just is in a totally different world than I am. Just don't college football, go to the barbecue, whatever, hang out. It's my mental model, right?

Hell yeah. You know what this country is built on?

That.

Anyway, sorry, continue. No, but okay.

250 years ago.

Okay.

So, so seriously, a good example is like, people will talk to, uh, people talk to me about how great Copilot is and you're like, Jesus, like, you're like disgusting.

Yeah. Yeah. Like, no, I have the feelings of we are so early and if the model, you know, this is what motivates me to say ARR will keep growing exponentially, even if the model training completely stops. Like we need, we need no better models. to have this thing penetrate the rest of the 5,000 banks that we're describing. But like, I totally take your point about—

Yeah, like, well, to be clear, okay, a small bank in this case might have $200 to $500 million in assets.

But if you can't expect to protect yourself from a cybersecurity perspective without access to Frontier AI or close to it, and you don't have an employee who's downloaded Claude Code or Codex and tried it out, I think that's almost a clear comparison to be like, yeah, you're not prepared. That's an issue. So the diffusion of intelligence in some ways is what's going to help keep people—

More spicy takes. Max, what's your— the election's going to be hacked to the person that gives more compute to the Frontier Labs? That's the easy one. Give me another one.

Well, this is things that might happen. In a world where they don't slow down model development, or just like what will happen, period?

Don't slow down model development.

Dude, if they don't slow down model development, I feel like it's totally reasonable that within the next 2 years, someone has used the AIs to create some sort of bioweapon that actually killed hundreds of people, at least. Maybe thousands.

That's okay. We can talk about the bioweapon in a second, but you guys are both saying this thing that bothers me, which is that slow down model development. Dario did not say he's going to slow down model development. He explicitly said he's not going to slow down model development. And he—

No, bro. He did say he's going to slow it down. He said he wasn't going to pause training, but he said he was going to slow it down.

He wants to pace it. Yeah.

He said, he said, we must slow the pace at which we improve the capabilities of AI models. In bold.

At which we improve the capabilities of model. Okay. And then he, where's the other one that I keep going to on training? To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to scale up and prepare the model.

Yeah, he's not going to fully pause.

Okay.

So do you, do you think that functionally this means that they launch literally less training jobs? in favor of running more safety evals and therefore we get less models.

Yeah, I think concretely they will delay future training runs until they better understand what is actually happening with the model, develop more confidence that it's not going to kill everyone. And by the end of 2027, the best internal model at OpenAI and Anthropic will be worse And less capable than it like would've been otherwise.

Okay. Fair enough. Yeah. I'm back with you. Sorry about that. Let's talk about the bioweapon.

Thousand people dead.

Bioweapon.

Boom. Jordan.

I read, so I read an interesting take from a guy online. I do not understand bio enough, so I'm outsourcing my, my, you know, knowledge of this to, to that. But as a pushback to the bioweapon, Stuff. This guy basically was talking about the difficulty it takes to actually synthesize custom viruses. David Bellamy, shout out to David. He basically says, I must be among an extremely small group of people that have both trained a frontier LLM and designed a synthesized custom virus in a lab with my own two hands. And I think that the takes on AI killing us all by creating dangerous viruses is totally bogus. Now, I've heard— It's a long thread, but I've heard this from the guys that are doing like Like protein synthesis and stuff, custom, like wet lab as an API. There's all these startups right now that are doing that. And I believe it, like totally, there's a throughput issue of like actually being able to create this stuff. And I think it would be a choke point really similar to what you were describing earlier, Doug, which is like an AI can't probably get through the TSMC KYC process to actually tape out a chip yet or producing stuff. And I think AI is probably—

I look at the advertisements on the subway and it's like, Fish audio. This this billboard would sound like your voice or some crap like that. I'm like, I can deepfake my own voice probably for like two bucks. Like you know whatever, a hundred bucks a month or something. Like I feel like it doesn't take that much. You know, like we're we're there. We're right there. Um, also Jordan, this is a sarcastic comment which I expect to be cut on. But then how was COVID created, dude? If if it wasn't that hard, dude, come on. Oh, it's co-created in the lab.

Oh my God, man. I mean, I won't get into that.

Public lab in Wuhan, insane.

But I would say the short-term risk with these bioweapons is not that the AI somehow end-to-end fully autonomously creates the weapon and unleashes it the same way that they hacked Hugging Face. It's more that you have a bad human who is able to use a sufficiently misaligned AI to then cause serious harm. And I think that is still totally possible, even if creating a new disease is a very involved process.

So we'll have a new version of mass shooters in the United States where it's much more dystopian. Also, that begs the question, can the only thing that stops them be a well-aligned human with a good alignment?

Sorry.

Yeah.

Yeah.

Sorry, I got to ask that question.

Yeah. Okay. Let me— so I have a problem with the bioweapon discourse because of the intentionality. Everybody always kind of describes the AI as intentionally trying to kill humans. And I think what's much scarier to me is rogue Not rogue AI, but like byproducts or mistakes that the model makes when you get to the point where it, you know, it like, it's a byproduct of it pursuing a goal is a human dying or something that it thinks is aligned with human values resulting in somebody dying accidentally. So the experience that I have that I can reason about for this is like you use the AI to create a website. It goes and does something you didn't tell it to do. You say, why'd you do that? It goes, oops, I'm sorry. Like all of these people have these experiences right now. And I think that same experience applied to something that is like government scale or, you know, like a startup company who's like, hey, I'm going to go into a clinical trial with this thing that AI developed that I don't really understand that I've been collaborating on. And, you know, it's past all of the mice stuff and like, we don't really understand the science enough and you go into clinical trial and then 3 months later, half the people die. Because something you missed just literally is killing humans or something like that, right? Where it's unintentional, it's a byproduct. And it's really scary because you can apply this to bioweapons. You could apply this to autonomous vehicles. People have kind of thought about this a long time, but this is like, there's lots of drones and autonomous weapons that people are developing. And it's really It concerns me to think about a scenario where, in the bioweapon case, you're trying to create a protein powder that doesn't need as much regulatory approval, doesn't need to go to a human trial or whatever, and then you just ship it onto stores and then a bunch of people with rare conditions die. I think in some ways we are already pretty clearly on the train tracks towards this world where People are outsourcing their thinking to the models. They're outsourcing really critical steps. And the models still miss stuff. They are not like totally thorough. They don't have perfect human values. And it's incumbent on the people that are using the models to actually use them correctly so that we don't have scenarios like this if you're in positions of really important power. And so it goes back to like, Are the humans using the models going to scrub? And that's like the bigger concern for me.

Okay.

Well, last but not least, Joey, as our resident finance bro, what's the misaligned model?

They're going to mess with— Misaligned, like that gets actual— that gets regulatory action going. I think like something top secret, like DOD, NSA, CIA, or like in, I don't know, like new weapons technology gets hacked and like I think that gets regulatory scrutiny moving.

Yeah.

Very fast. Like than even like human death would, or, you know, a medium or small size bank gets attacked.

What do you think functionally gets it moving in the US political system? Like right now it seems like a lot of people, like it seems like a very popular take for people to want to pause AI, like bipartisan. We're seeing people on the left and right saying, stop the data centers. Like, Stop AI.

So, Anna, did you have Trump saying that all you need is a you know big and smart president?

Well, that seems to be the only thing standing between Trump seems to be the only thing standing between regulation and AI right now.

Well, no, then there's all like the you know the David Sachs's of the world.

Okay, the Trump administration, like the executive branch, right?

Uh, I guess a lot of like people in Congress share. this like anti-regulation, pro-free market view. Oh, actually, even Lina Khan came out like this, this past weekend and was saying that this is a regulatory capture play. We're seeing some very strange bedfellows being made as a result of all these AI developments. It's quite surprising, honestly.

Yeah, I just, I wonder whose mind needs to be changed in order for regulatory work to start, like, in order for AI regulation to happen?

You need a larger lobby than the big tech lobby.

Sure. But the larger lobby needs to convince Trump right now, right?

Yeah.

And so a change in government, I think almost by definition implies that there's going to be AI regulation if somebody that's not Trump comes into power in 2028.

I also think it's totally possible Trump regulates AI too.

What's that scenario? If the— because it's not popular opinion, right? A lot of people want him to do it. He doesn't want to do it. You're saying if his advisors start telling him something different or—

Yeah, I think there's a world where like the advisors really internalize that OpenAI and Anthropic are being genuine and they start to get scared. And that's like one possibility. And then there's another possibility where something like really bad actually happens during his term. And it kind of forces his hand. I think these are both possible.

Do you think that we're on track for that right now? Like David Sacks called out regulatory capture on their All-In podcast, then they come out and, you know, some more stuff happens. Specifically, Elon says that he supports what Dario and Sam are saying.

Yeah. I would actually bet that we see AI regulation during Trump's term. I think it's more likely than not.

Interesting. Interesting. And you think that the people who regulate it will take the approach of being like, we're going to do the sensible regulation before the crazies after us go and do the bad stuff? Or like, what framing does David Sacks come to the point where he advises Trump?

A preliminary framework to begin the regulatory process of AI. It's like saying, uh, and then, and then some, it's like a trade deal. They're like, oh no, this is our first framework.

And then they'll refine it afterward.

But that way, uh, come election time, they'll be like, we regulated AI. We're the first ones to do it. Last one.

How about, how about RASA? You know, that bill that's stuck in the Senate right now?

Remote access.

What does it do? Remote access security thing. It clarifies the export controls to say that Chinese labs accessing compute remotely is a violation of export controls as opposed to just having to ship the servers into China. Stuck in the Senate right now, being heavily lobbied against by Oracle. And I think it has a bunch of implications on these, like this whole security discourse and stuff. And it passed Congress with like 300 and something to like single digit. Like everybody in Congress approved this bill and it's just stuck in the Senate for months and months and people don't think it's going to pass. One of the many examples of the US Democratic system that shows the influence of lobbyists, I guess. But yeah, we'll see. We'll see. I think RAISA is an example of a bill that is on the 1-yard line that a lot of people in the US seem to agree is pretty sensible. Don't let Chinese labs access frontier chips from Nvidia or others remotely, make that explicitly illegal, and it's not being passed. So it's like maybe that one comes first before any real AI regulation, or it's just part of the horse trade thing where they make these massive, massive bills in the Senate and then it just slows down because everybody wants to get their own little thing added into it. Like, I can't even— marshaling the resources to pass an AI bill in the US would seem like an incredible undertaking. And maybe it will depend on a more serious security incident or something else coming for people to take it seriously and want to pass legislation.

Yeah, I think would do people losing their— I don't know, people losing their entire life savings would probably get them there.

That's FDIC insured, Doug.

Sure.

They'll be made whole. Like, no one's going to care if some like little town and like if someone from Max's hometown or here in like Akron, Ohio loses their money, no one's going to give a shit. No, no, no. What happens is—

We'll come after the money for you guys.

Loses all their town and then, and then there, there's a Gigawatt data center next door. Come on. That's going to be the old, I lost my entire money to the Gigawatt data center.

Wait, wait.

You're saying it's not even just like the job? Like it's just they don't even have to go second order. It's literally that data center wrong.

That data center stole my money. That's the world we're about to live in, guys.

God, I kind of like this man.

I don't know.

This makes sense to me. I really think, I really think people dying might have a bigger impact. But yeah, maybe people losing money is a bigger one.

Like a horrific, like essentially nothing gets people more excited than or not excited.

Catalyzed than— that was a crazy word, Duff.

Sorry. People get more catalyzed than like 1,000 random deaths, right? Like that would be really, really horrific.

Yeah.

But we'll see. I mean, we have to see which lobby is stronger, right? Because like the AI lobby could be really strong, just like the NRA. So you might not— like, you know, the lobbies have to fight against each other. We don't know how this is going to work out. Sorry, that's really unhinged.

All right. Well, I think we're out of time, guys. I can't believe we're going to put this on the internet. It's been a good conversation.

I think it's been a good time. This is like transistor radio vibes, you know, because I made bad jokes the entire time. I don't think it was completely transistor radio vibes. Some aspects of it were serious. And I think the 4, I do think anecdotes are important.

I enjoy thinking through all of this stuff with you, Doug. I enjoy you pushing the limits of what I have considered.

Yeah, Jordan, otherwise you're just going to talk over yourself and you have these long-winded spiels. It'll be about ClusterMax inadvertently.

But he already talked about ClusterMax with the security stuff, dude.

Yeah, I know. But then he'll be like, just, just like the work we're doing at ClusterMax. I'll be like, okay.

Yeah, man.

Play your hits. No, no, I think, um, I don't, I, okay, I guess maybe, maybe let's do one thing.

Do you think—

I actually think the regulation thing Max said is probably real. I do think the next administration will want to say that we were the first to do that. That just feels inevitable now that I think about it. What do you think the— I guess maybe the real question I have is in 90 days, is the slowdown regulation conversation hotter or cooler than today? Like, when I mean hotter, it has to be meaningfully hotter, not the thing.

This feels like a real vibe shift to me where anyone who's actually talked to researchers at the labs and then knows the intellectual lineage that all these people come from, you know this is not a regulatory capture play. They've seriously considered these risks for the past 20 years. But I think it's finally gotten to the point where the capabilities are good enough and also you have things like OpenAI Hugging Face where they can viscerally feel it. And I think this is actually a serious vibe shift where we are going to start pacing the progress towards RSI. I think people, everyone making this technology sort of feels a deep responsibility for this to go well, and they think that with sort of unpaced RSI, there's an unacceptably high chance that we end up in the bad.

Yeah, I definitely think the vibe shift over the summer was very clear.

So I'm going hotter and hot take is like Okta, CrowdStrike, Palo Alto, Zscaler. One of them gets— That's a high profile. That's a very good hot take because these things are not like, like, you know, my, like, my hot take right now is just like short a basket of like cyber. That's like, you know, network effects cyber, like, you know, anything touches endpoint identity, network security, just short it.

Well, look, a good example, a good example is the JFrog thing is about the security of your—

let's look, sorry.

The JFrog thing was about the security of your own Artifactory and it got hacked, right? You'd argue that they're not hacked directly, but it got exploited via this AI, right? Stock's up and to the right, whatever. But like that same mechanism and behavior exists for these companies and it just hasn't been exploited yet. But then again, didn't Okta already get hacked?

So like—

They've been, they get hacked like every year. There's like a major security incident.

This is what I said earlier, man. Like they have a target on their back and they're a high-profile company. Everybody does it. Everybody gets hacked.

Well, there's the old thing, like there's no big cyber companies because once you're the big guy, everyone comes out to kill you. Yeah.

Well, that's Anthropic and OpenAI is going to kill them.

I believe.

Max, can you just, can you go back to what you said about the, everybody who is the closest to this technology seems to take it the serious, take it seriously or the most serious and is the most concerned. This cuts across the political spectrum too. It's not like every single person working on this technology shares one worldview, which is based in Berkeley, California, as a lot of people online have been criticizing, right?

Yeah, no, definitely true. I think the opinions of researchers at OpenAI and Anthropic are actually quite varied in terms of, you know, what is the probability that AI will kill everyone, What is sort of the right future for the world? You know, like, what is the role of humans if we have super powerful superintelligence? But like, sort of the common thread is that everyone recognizes this is going to be probably the most powerful technology humans have ever created. It is going to sort of radically change society, and then they feel the weight of making sure that goes well. I think that's like universally true across the board. And honestly, we should feel very lucky and thankful that people who sort of feel that way of responsibility are the ones who are actually developing technology and not, you know, the David Sacks of the world.

Ooh, damn.

All right. Well, I guess we can end there.

Apparently they've already left. We have to all smile. So, ha ha ha.

We have to all smile.

Yeah, we have to smile.

After this podcast full of laughs where we talked about the depth of the extinction of humanity and the doom that awaits us, let us all smile at the camera.

Smile because it happened.

Smile for the algorithm, everyone. Smile for the algorithm. We get more clicks when we smile in the thumbnail. Smile for the algo.