Washington Hones in on AI Safety, What Are AI Doomers Proposing | Diet TBPN

What are the AI doomers actually proposing? That's the question I was trying to answer this morning. For a while—

20-year sentences.

That's one. AI 2027 predicted that this month, you know, late in 2026, Congress would wake up, and that is in the journal. Congress is suddenly waking up to the AI doomsday threat. And so this is happening all over the place. Was it Matt Damon who was caught on TMZ being interrogated about his thoughts on AI risk? There's now protests. I believe AI 2027 predicts a 10,000-person anti-AI protest by the end of the year. I was trying to figure out how big this protest in Chapel Hill, North Carolina earlier this month was. I think it came in Sub 10K, but certainly tracking to it. It was about 150 people at this at this protest in Chapel Hill, but just two ooms away from the prediction from AI 2027. So I wanted to dig into like what the actual proposal is because the joke for a while has been just like everyone when they're pressed on this they just say we gotta talk about it. We gotta talk about this, and the actual proposal is super concrete in AI 2040, and it's it's very interesting just to hear about how they want. to slow things down. So, the thing that I think a lot of people online who are like, yay, an anti-AI sentiment that's going viral, are going to be depressed about is that this is not stop AI at all. AI 2040 is like, keep the inference flowing. The current models are great. Also, let's keep doing capabilities research, but we just want to reach superintelligence by 2040 instead of 2028 where we're not necessarily prepared. So they want it to be highly controlled by governments, nation states, highly secured. And there's a whole bunch of very, very tactical recommendations that they make around that. So the first mechanism is an AI pause. They wanna pause training. They don't wanna do any more new frontier training runs or R&D experiments. And to enforce this, they're calling for to apply inference-only verification to essentially all major AI data centers. So anyone who has more than 10,000 H100 equivalents, roughly $100 million of equipment. And that is like pretty easy to figure out. You're just like, big building over there. Let's send the inspector inside. Oh, says NVIDIA on all these chips. Count them up. There's over 10,000 of them. You got to apply for this permit. You got to do— you got to tell us what you're doing. Right. Very easy to enforce, at least in the United States. And with an international body, you could kind of do the same thing internationally. So the whole goal, you can only inference the current models and you got to verify your workloads with an independent audit Probably the government. Maybe there's some sort of non-governmental organization that's doing this. You know, there's a whole bunch of different solutions that you can pull from across nuclear nonproliferation work that's happened in the past. Major countries, they want them to declare AI compute inventories. Tell everyone, not just your local population, but also the international community, how many warheads you got, how many H100 equivalents do you have, where are they? Everyone shares this. That's gonna be a really tough sell because international agreements are really, really tough sells. It's much easier to have a groundswell of support for something that happens in America. America changes, we have a system that we don't really have the international rules to quickly implement that in a way that doesn't allow for a lot of defection. They want to physically remove high-bandwidth east-west networking inside of data centers so you can't do large distributed training runs. But you can still do inference. For any new AI R&D data centers, they want entirely new facilities built from scratch with nation-state-level physical security and verification. But for the next run and the one after that and the one after that, as it gets crazier, we want it in a new building built inside a Faraday cage so you can't communicate it from the outside. We want a bunch of physical controls. It's like going to a nuclear facility, highly verified who gets in the building and when, air-gapped communications. This is One interesting proposal that they have that really shows how deep they thought this through. They want the R&D data center to be connected externally. If you want to communicate with it and you want to tell it what to do, okay, train the next running or do whatever, they will have a bandwidth-capped connection at 1 meg per second. So you can send little instructions, but if you say, send me the weights because I'm taking them somewhere else, it would take you like 5 years to exfiltrate it. So interesting, like hardware solution to this. I, you know, how do you actually go and implement that? There's gonna be a whole bunch of other things, but interesting that they're thinking about like the width of the pipe. So it'd be very obvious if you're stealing the model weights because it's like, wait, this 1-meg pipe has been at full tilt for months. What's going on here? Someone's taken the stuff out of the data center. They want, when Frontier model weights move from an R&D facility to an inference facility, They want it to be placed on physical storage devices encrypted independently by both the US and China. So both countries have to sign off and physically escorted by representatives of both countries to the destination. It's a tall order. One's a tall order for sure.

Yeah.

And they actually want frontier models to be made deliberately larger than compute optimal. So they want the weights to be 100 terabytes instead of honing them down to something that's just 1 terabyte that could actually be moved around a little bit easier.

Yeah. I mean, first of all, this— I mean, a lot of this tracks with The kind of regulation that, you know, we had pushed for, you know, beginning about 2 years ago around podcasts, wanting podcast studios to be air-gapped.

Yes.

Wanting, you know, Faraday cages around podcast studios.

A locked briefcase with an SM7B in it. And in order to unlock it, Patrick O'Shaughnessy and David Senra both need to give you codes to independently verify that this podcast is worthy of being recorded. Yeah, I like that one. Yeah, it just makes sense. High-level value valve they see as being like the most effective in controlling the speed of capability improvements is compute caps. So there is a world and we're going to the Bernie Sanders thing because it's already getting like sort of twisted. But the big hammer is just chip controls and data center buildout slowdown. That's the easiest thing. And I think that's like the biggest valve that they're going to— that we're going to see twisted around to actually slow down capabilities. And so the goal is to allow models to get better mainly by adding hardware rather than inventing better algorithms, which can leak to secret projects. So the goal is like, okay, well, we know that this model is capable of this, so we want this much compute over here. Okay, you've done well, we're allocating more compute as opposed to this one weird trick that AI doomers hate.

Yeah.

The goal here is not to go back in time.

Yeah, it's so interesting because when you look at, I mean, anytime you have, you know, really, really hardcore government regulation and international coordination around issues like this, you're gonna have a bunch of unintended consequences.

Yep.

And one thing that feels obvious around this, if these policies were to be rolled out, is that you would effectively create an incentive for millions of individuals or groups globally to be in secret, like trying to find entirely new breakthroughs that are—

Yeah.

And again, this incentive already exists, but it kind of pushes a lot of the idea that humans are just going to be like, oh, I'm no longer going to try to create the God model because there's this big global organization that's sort of policing it.

Yeah.

I just—

I mean, that's the same thing we see with nuclear nonproliferation. There's always a discussion about what countries are getting the bomb and how far along are they and wars break out over this. And yeah, like the game's not over just because you create a framework.

But ultimately, GPUs and computers are much more wide, infinitely more widespread than nuclear materials.

Yeah, but you still got to marshal them all together. Yes, there's some weird scenario where there's a Python script that's AGI that can run on your laptop, but I think most people, are convinced, at least in this crowd, that—

Scale.

That scale is a prerequisite. And I mean, we were joking about, like, it would be so— like, SSI, Ilya Sutskever's new NeoLab, recently got a big cluster from NVIDIA. And we were like, the most bullish thing you could do if you're this secretive NeoLab would be like, we're actually selling our compute because we've discovered a more compute-optimal way to reach AGI and we don't need a lot of compute. But of course, even Ilya is like, it's time to scale up. I need more compute because it's seems like even if he's taking a completely orthogonal approach to innovation and research, he still needs a lot of compute. And so, it does feel like everyone is sort of with the consensus that it's going to be a big building with a lot of energy, big heat signature, definitely visible from space and pretty simple to track, at least in the short term until people start building crazy underground facilities. And then you're back to— You know, nuclear nonproliferation. But their goal is at least to, like, you know, try.

Yeah.

And then the other side of this is—

Not a panacea.

Does AI development actually become something closer to the Manhattan Project where, you know—

That's what a lot of people think.

100 researchers are working with the government in secret because you can't just assume that other countries are going to slow down or—

Yeah. But in general, I think the proposal is don't go back in time. It's definitely not stop everything in its tracks. It's a slowdown with the goal of scaling gradually. They do actually want to reach superintelligence. They just want to do it by 2040, hence the name of the project. So, the goal is gradually scale into top human expert capability around 2035. Now, a lot of people are saying, oh, we might get this by 2029, 2028, 2027, and they see that as too fast, so they want to push that out to 2035, then wait 5 years with AGI, and then unlock superintelligence in 2040. This is their initial proposal. Of course, there's a lot that could change over the next decade. And I think, like, to zoom out overall, if you're worried about x-risk, the AI 2040 plan does feel like a concrete path towards slowing down. The conversation definitely gets dragged down into PDoM estimates and trying to narrow down exactly how a human extinction scenario plays out. And that can be— that can— I feel like that's almost a sideshow, because in a democratic society, amongst humanity, like, it doesn't really matter the mechanics of getting to 10% PDoM or any of those. It's just like, if everyone feels that way, something will happen. This is a concrete plan of what that might look like. And that's valuable to understand in this case.

Yeah.

So, for the safety sceptics, it's easy to see how this level of control over what you can do with computers is authoritarian or anti-libertarian. Even if we're talking about $100 million computers, there's a lot of people that say, like, I should be able to do math on my computer. I can do whatever I want. Let me do cool things. I'm excited about this. That limits your freedom. It might create regulatory capture. for a few major players. It might crash the stock market or delay economic gains that come in the good ending where alignment is solved and x-risk plummets. You can imagine a situation where in a few years, if x-risk fades into the background, you're like, yes, there's still a risk, but it's the same risk that we face every day with an asteroid hitting Earth. It doesn't really change anyone's behavior. That would be sort of the good ending, in my opinion. So, it's a balancing act. And for most of these slowdown proposals. I personally have a hard time blackpilling about them in the sense of, like, if all of this gets implemented, how frustrated will I be? Like, the models are good. I would like better models. I want safe models. But at the same time, like, there is this massive capability overhang. The current models can do a lot of interesting work. We're finding new uses even for, like, non-leading-edge models. There's a lot that can be done. So I don't believe the doom doomers who are dooming about what the doomers are planning. I find that unconvincing right now, but people are starting to lay it out more. Brad Gerstner, Jensen Huang, David Sacks are talking about the other side of this equation.

Yeah. There's also a big question around what— do the 2040 people have a point of view on robotic and physical AGI, because it seems like if you, even if you pause efforts towards RSI, well, if we add billions of robots into the world that are just running on today's model, that also presents today's—

I don't think they're worried about that. Yeah.

But to me—

But to me, that—

a billion robots with GPT-6-level intelligence and years of alignment work, That is currently happening, that's fine. It's the next, next, next thing, the superintelligence, the thing that might have its own goals. I think, I mean, we're talking to someone from OpenAI's robotics team, hooking up Astra to a robot, a paintbrush, and a camera. We talked about it earlier, painting. I don't think we're at a point where that poses a risk. It's It's the next model. It's the model with its own volition, basically.

Yeah.

Which a lot of people still aren't seeing. They're just like, yeah, like the models keep getting better, but they seem to follow your instructions, sometimes too much, and then you need to worry about the paperclip scenario. But it's not that they want to do their own thing necessarily. I don't know. But people are going back and forth on this. Clem over at Hugging Face says, sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change. Not saying it's necessarily uninteresting or wrong per se, but let's keep things in perspective and hear from the full range of expertise across the ecosystem. And Nathan Lambert says, banger. What was our take? AC guy might be right about cold temperature.

Well, my AC guy would be like, I don't really know about that, but I just want to make sure when you're hot that we can run this AC cool.

I guess it's— yeah, the AC seems—

I don't know about all that. I don't know about all that mumbo jumbo, but, you know, when it's a hot summer day, I don't want you to be worried about the heat, brother.

I like that. Yeah, this is a— this is kind of an unnecessary shot at AC guys. You know, AC guys are important. I don't know. It's funny. What did Bernie Sanders have to say? Is this— is this real? Entities will— shall be subject to the corporate death penalty. And persons shall be subject to not more than 20 years in prison if they don't pause AI development. That seems pretty easy to comply with. I don't know. I guess, how do you define AI development? Is prompt engineering AI development? And then you get caught because your model sort of did a little prompt engineering in the final stage, and then you're Guilty of this, like that. Yeah, that could be a negative knock-on effect, I guess. It does seem aggressive. But his overall proposal is banning artificial superintelligence. So, no person or entity may develop or deploy superintelligent AI systems. He defines artificial superintelligence as an artificial intelligence system that exhibits or can easily be modified to exhibit capabilities that match or exceed human cognitive performance and capabilities across a broad range of domains or tasks. It sounds like the mission statement. It sounds like the explicit goal of like 17 different companies right now. AI system or AI systems that have sufficient capabilities to plan and execute the disempowerment of humanity. Okay, that's a good one. I like that. I don't like overthrowing or undermining the US government. So strongly in favor of banning that. Pausing advanced AI development until a new federal AI regulatory body is up and running. And then the new cabinet-level federal agency will monitor frontier AI systems at all stages of the lifecycle, supervise the removal of dangerous capabilities, and supervise the destruction of artificial superintelligence. We're coming for it, which is similar to the—

Corporate death penalty is a line that you don't hear a lot, right? Usually, usually these companies just, you know, go bankrupt. and wind down. But corporate death penalty goes pretty hard. Kind of metal. It's kind of metal.

Yeah, you kind of got me with that one. Yeah, rough, rough, rough situation. We'll see where it goes. Jamie Cox over at FluidStack, the co-founder of FluidStack compute provider, shared his convictions sort of pushing back on a lot of this, saying that he thinks America should build more. They're pro-freedom, pro-democracy. They believe AI will bolster human flourishing. We support simple, clear, enforceable regulation frameworks that set simple requirements proportionate to capabilities and risk with clear responsibilities and no unnecessary barriers to competition. Yeah. The real— you're going to see a lot of pushback from people who are like, the regulatory stuff is going to be like these 10 companies and I'm going to be number 11 and I'm basically getting the corporate death penalty then because I didn't make the cut to be one of the regulated, one of the approved companies. I'm still early in my stage. So, there's a lot of nervousness, I'm sure. But Will Menaides said, we believe AI will make everyone rich, healthy, and free. His novel and interesting comms from the frontier. He's endorsing this. And I agree. I like this. I like these convictions. I think it's generally like a positive direction to move in, not a direct response to the proposals that are going out, but we're going to get a whole lot more of them. Where do you want to go to next?

Over on TikTok, they're sharing a photo of the whistleblower. And saying in every worldwide disaster movie, there's a dude that looks just like this that nobody listened to. And he really does look like an actor. He does look like an actor here.

Yeah, he looks good. But people are, people are all—

And please stop calling him Scary Potter. I've been seeing people. I've been seeing people over on X calling him Scary Potter.

That's the goal. The goal is to wake up China, wake up Congress, wake up everyone.

DoorDash has entered the conversation. They say, 2 years at DoorDash, I do not say this lightly. We are extremely close to the burrito arriving before you decide you want it. We are not asking for a ban. We are asking for a pause. I don't know why they would ask for a pause.

Yeah.

That seems like very, very aligned to humanity and to their business, which I think is fantastic.

The doom is very much contained to the frontier lab work. Everyone in the application layer who's applying the models, diffusing them, they're like, I can't get this thing to work right. I got to get forward-deployed engineers to teach people how to use this thing. Everyone deeper in the stack—

Yeah, even Jim Rainbow over at Jim O'Reilly.

Sure.

Or Jim Riley.

Yeah, Jim Riley.

Jim Riley.

O'Reilly Auto Parts.

Jim Riley over at Charleston AI says it's a whole lot of mumbo jumbo.

That was your words. He's just happy to get You know, 8 hours back. Yeah, yeah, yeah, yeah. And then, yeah, everyone deeper in the supply chain like Jensen and all the different semiconductor manufacturers are not particularly on this side. And then you also have Wall Street who's just like, what's the enterprise acceleration? So lots of different groups around the table that need to be brought on board to this movement, the discourse. Truly is fascinating. So what is Tyler Cowen calling for?

He's banging the table saying bet on this.

Tyler Cosgrove, are you named after Tyler Cowen? Is that your namesake? Tyler Cowen says if you have very pessimistic fears or predictions about AI, name the market prices that will support or confirm them. This is what taking this seriously means. And I love Tyler Cowen. I'm not sure this matters because if you, if you, you're not going to be around to collect it, right?

Yeah. This is what Deep Dish Employer says. Tyler, why would short-term existential risk affect market prices in any meaningful way? Spell out the exact mechanism. Contracts that pay out if everyone dies aren't worth anything to me.

Yeah, I don't know.

Yeah. I think he's just, he's calling for like, he wants to like make a falsifiable claim. Like, is this Am I able to tell if your claim is true or false? And so it's very hard with these scenarios where—

Isn't it an unfalsifiable claim though, just by definition? And you just have to accept that and move on?

Yeah, but then it's so hard to have any real discussion.

Was nuclear any different? The threat of nuclear apocalypse, the threat of World War III, this was a very motivating factor for decades, most of the 20th century. People made real decisions based on it, based on where to do business and where the conflicts were going to be and the motivation for nuclear treaties and nonproliferation and how we treat—

You can make financial decisions, like thinking about nuclear war, right? You have a bunker. That's like a decision you make.

Bunky?

Is anyone making like AI bunkers? No, because they think it's going to be so totalizing that the bunker actually doesn't do anything.

Exactly. Yeah. So, if you think it's so totalizing, then you don't make the bunker. And so, saying, hey, you don't have a bunker, is not proof that the person doesn't believe what they're saying. There is the other side of this, which is Paul Christiano, who has been worried about risk. He recently just joined the board of OpenAI. And Tyler Cowen has this quote, if you're a doomer, why aren't you short the market? And Paul Christiano is 2x levered long and he's short the— isn't he short the bond market or US Treasuries? Right. So he has 5% of his net worth in Tesla, 90% of his net worth in AI bets, and 100% of his net worth in normal investments. No Tesla options. That sounds like a scary place with lottery ticket biases and the crazy Tesla investors. And then Eliezer Yudakowsky says, am I correctly understanding you're 2x levered? And Paul Christiano says, yeah. And so he says he's personally short the US, the US 30-year debt. I think that just means you have a mortgage. This doesn't seem like a doom-based bet. This seems like this bet also pays out in just like the good ending and like AI is real and delivers value. So your AI bets perform well, the market performs well, and money slides from US debt to data centers and AI buildout. debt or something like that. So, he is putting his money where his mouth is, but it doesn't feel like a representation of, like, Doom by any means. People have been creating AI agents of fruit flies. Have you seen this?

Yes.

My understanding is Google basically mapped out all of, like, the neurons in a fruit fly.

Fruit fly.

Yeah, like the entire nervous system. 3D.

They got the entire structure and now people have been able to recreate that in software, in a simulation.

Yes. I think in theory, you can replicate all of the fly's decisions or whatever, its movements.

Yes.

And this is notable because many people have said I have the mind of a fruit fly. They've said I have the intellect of a fruit fly.

Yeah. So now they're going to put it to the test to see which performs better, the simulated fruit fly or just Jordi Hays who we got right here.

Organic, farm-to-table Jordi Hays.

So now that this is out and the code is out and you can run this fruit flying simulation however you People are having— they're doing all sorts of experimentation. So Kevin said he trapped the fruit fly in his Rabbit R1. When he shakes it, he can see its brain's escape circuit light up.

Yeah, I think what ends up being unsettling and weird about this is if it just purely you being human and doing this is is probably not good for your own soul. And imagine you have a fruit fly in a box and you're shaking it and you're like, look, it wants to escape, right?

Yeah, totally.

I'm not a huge fan of small insects. I don't really want them around that much. But when they're in my house, I try to— if a spider's in my house, even if I know it might want to take a nice bite of me, Yeah. I'm still going to try to transport it out of my house and put it back into the world. I think it's not good for your soul to be, you know, a merchant of death. Sure.

In that situation. And yet, if you're playing a real-time strategy game and you highlight a bunch of soldiers in this simulation and you send them on a charge that will result in their virtual death, You might not feel—

Those soldiers opted into riding and dying with you.

Okay, okay.

Right? Okay, what about— The fruit fly is just sitting there, John.

So, they made the fruit fly play Doom. What about if you play Doom, you are killing a demon who's simulated? Is that immoral? Like—

Well, that— well, is that demon trying to kill you?

A lot of it comes down to like the fact that it's Simulating the actual representation of the fly makes it a lot more concrete than just, oh yeah, it's a 3D model and a Python script that just says, if you see a character, shoot at them in the simulation.

Yeah.

But we're clearly starting to grapple with these odd moral questions of if you're simulating something, then the next step is a couple of orders of magnitude, but you get there and you can simulate a human. And you could talk to that human and it would do everything the human does. Does that human have rights and agency as an ethical moral agent, or is it merely just a simulated, just a really good computer simulation? It's just existing on transistors, so you don't need to feel any moral weight about anything that you do to it. I agree with the, just the vibes-based analysis that like torturing a real fly, torturing a virtual fly, probably just don't be in the business of torturing anything. You don't need to overthink it. But people are— the real debate here is like the question of, you know, are LLMs sentient? Are they moral? Is there a moral weight to synthetic intelligence, to artificial intelligence? That's what people are debating here. And I'm sure the debate will continue. Who knows if it will ever be ended. But they did teach it how to parallel park, which I think is cool. Interestingly, last night I had Astra use Computer Use to play a video game that I very much enjoy playing called Balatro. It's sort of like a modified poker game. And I don't feel like I was torturing the LLM by that. I feel like I was giving it a treat. I was like, hey, instead of doing my taxes, you get to just chill and play a video game. It did very well. It won. SOL was not able to win. And it was really fun because I would like pop in while it was using the computer inside, kind of armchair quarterback and be like, is it making the right decision right now? Felt like a coach coaching like a kid on the soccer pitch or something.

I had a funny moment last night. I was on my racing simulator comparing, asking ChatGPT to compare my times at Laguna to just other, other, you know, what, what would best in class be like, what's beginner like, etc. Um, and it said, if you want, I can help you, uh, you know, cut, cut some seconds off of this time.

Yeah.

Uh, and it was like, why don't, why don't you take a video of a full lap?

Yeah.

So that, and I'll analyze it for you. And I was like, yeah, yeah, dude, I bet, I bet you'd love to just watch, chill back and watch track, hang out and watch track footage.

Pretty soon it's gonna be like You want me to just get in the seat? You might just take over because like I'd really—

Yeah, no, computer use in iRacing is something I'm, I'm gonna experiment with this week.

I'm down, I'm down. Take, take half my quota, my monthly quota, just play games, chill, do whatever. It's a nice treat.

I've got some banked resets. Yeah, that I'll put to work. Wonderful week.

Wonderful week.

Short week. Yeah, but Monday, 11 AM, do us a favor and Go ahead and have the best weekend of your entire life.

Have the best weekend of your entire life.

Let's do it. Put the pieces together. Make it happen.

We'll see you on Monday. Leave us 5 stars on Apple Podcasts and Spotify. Sign up for our newsletter at tbpn.com. Goodbye.