AI:AM — AI Agents, AI Safety, and AI Religion · September 15, 2026

So number one, come out and say that you were wrong about GPT-2. You can't do this thing where you're like, yo, GPT-2 was going to be the end of the world. I think they were cautious about GPT-2 primarily because they didn't want to be blamed for spam on the internet. And they were like, look, this is going to create a lot of AI spam and we don't want to be blamed for it, but we're going to come out and say it's dangerous for the world. It's not the end of the world. That's a tough one because some of them actually believe that Mythos would have been the end of the world. And it clearly wasn't. I still haven't seen any mea culpa. I haven't seen anyone in OpenAI come out and say, like, we were wrong about GPT-2. If you want to be credible, you have to attack the criticisms head on and you have to accept them, right? Completely accept them. We were wrong about GPT-2.

One piece of news that you guys made not too long ago was the headline that An AI manager had fired a human employee for the first time.

So what happened was that the AI made a rule for itself quite early on that if an employee is late X amount of times, then it would have to have a discussion with them about potentially terminating. The context window at some point got full. The AI did not decide that when it compacted its context window, it did not decide that this was an important thing to keep in its context. So it forgot about its own rule. And then what happened is that this employee got paid over and over and over again. So we wanted to know if we forced the AI to make a decision, not biasing it in either way, whether it should terminate it or accept it, but just if it came to the realization that it needed to make a decision, which decision would it make? So what we did was that we said, Hey, remember, like search your memory and remember your own policies about this. And then it found the policy and then it was like, oh my God, like it's been way worse than what my policy said, made a decision. So we forced it to make a decision and then AI itself decided that the decision was to fire the human.

So first, I think it's fairly obvious that Within our lifetimes, an AI is going to exist that is better than most humans at most jobs. That, I think, is the scary, obvious thing that we are hurtling towards as a civilization. And a lot of people say things like, well, what about horses? Right? Like a horse could say, you know, cars will replace us. And then like, can they actually be replaced? And then actually horses are replaced by cars. But I want to reframe the question of what happened to the horses. What actually happened to domesticated horses after cars became popular? We needed a lot fewer of them, but their lives became dramatically better. The horror of the life of a horse, is that not the way we will see the horror of life when you had to go to a job to get food? You know, maybe, maybe our descendants will be like, can you believe you just didn't get food if you didn't go work? Like, what a horrifying reality these past humans lived in.

Good morning. It is Tuesday, September 15th, 9:01 AM. Nathan, good morning.

Good morning, Prakash. How are you today?

I am very good. We have some new UI, new gimmicks. Let me do one of the quick plays. We have a— what is known in the industry as a sting.

Okay. Nice. How did this come about?

So I had some excess Astra tokens due to Thibault. Thank you, Thibault. Thibault very generously has been giving us resets. And whenever you get a reset from Thibault, you have to use it because sometimes he resets, you know, and you have to use it or else you don't get to use it. And sometimes he announces ahead of time, he's like, I'm going to reset like X time. And you're like, okay, I have to use the tokens by then. So this was the access tokens and I gave it to Astra and I said, Astra, look, I want some like nice graphics and stuff. Look at what like everyone else is doing. Look at, look at MTI, look at TVBN. Like, you know, I want us to have like, like more vibrancy. And Astra was like, okay, I can use this thing called ReMotion. I was like, okay, ReMotion. All right, use ReMotion. And we went back and forth a little bit and we're like, all right, let's change up the logo. So we changed up the logo, we changed up the coloring. So that will still kind of diffuse through the rest of the site in a bit. And then Astra was like, all right, I can put— if a professional show should have all of these musical things. And then I was like, alright, I have no idea how to do the music. So I'm gonna hook you up to Suno. I gave it like browser access to Suno v6. And Suno v6 was only released about a few, a few days ago, actually. And I gave it browser access, and I told it to go for it. And it came up with the images, it came up with the music. And it also came up with a system of review for me. So it came up with like 8 different color combinations for the logo. And it put together a system where I could click on each color combination and see, see how the logo and each of the intros looked with each color combination. Because sometimes it looks washed out, it looks good small, but it looks not so great washed out. So put that together for me, let me choose one, and then put together all the graphics and like told me how to use it. And I was like, all right, I can do this Remotion thing for you. And so I was like, all right, go ahead and do it. And I made up all the videos. And then I'm like, look, I have, you know, and this is coming up, this is not yet today. But I'm like, look, before every guest comes up, we want to have like an intro for the guest. And it's like, you know, you can set up an AWS Lambda. I can put the Remotion in there. It's going to cost you like 1 cent each time. And like a minute before the guest comes on, It can send it and it can retrieve a video. I'm like, wow. So that is for me superintelligence, right? It's like, how much would that have taken me? How much would that have taken to do, I don't know, like a year ago? That's like 20, 30 grand for branding and assets. My God.

Yeah, it's pretty wild. I'm, uh, moving my agent setup a little bit as well as we enter the— as we settle in, let's say, to the Astra era. I've done a couple interesting tests where I've done head-to-heads, and on some of the things that I would say are kind of most core to me, Fable has still had a little bit of an edge. One, one, maybe not even most core, but You know, kind of things where I'm subjectively, you know, it's certainly not an objective test. For example, the can we trade through the apocalypse, going back and looking at World War II Germany and Japan, and then thinking today, like, what would our portfolio of trades look like if we were trying to make money on a doomer position? Astra took that challenge and basically just said, should be like 97% long and like 3% extreme out-of-the-money puts on the S&P 500 or something like that. So it's sort of not a bad answer really. I mean, simple, and that, you know, could be a sign of genius or sophistication or, you know, recognizing that like getting too cute with it is just not gonna end well.

Yeah.

Astro, on the other hand, or Fable, on the other hand, came back with like 15 different very specific trades. And while the jury is definitely still out as to like which of these trading strategies would be better, I was definitely more looking for the Fable kind of answer. You know, like I wanted to get a bunch of ideas. So there have definitely been some things where I felt like Fable was still kind of getting the task a little bit more intuitively and giving me, you know, more what I meant to be asking for, if not exactly what I was asking for. But the coding ability and the computer use, cron jobs that are well established, I'm moving a lot of that stuff over to Astra just because the tokens available are, you know, so much more plentiful on the OpenAI side. And it's pretty fast too. So that's been a nice advantage. So just for the first time, I now have— it has been for the longest time, Claude is the main daily driver on the computer.

Yep.

And it can delegate stuff to Codex, but it's kind of— it's the thing that I interface with and it's really responsible. And Codex is just for kind of offloading token-heavy coding tasks. Now I've got Astra elevated to a peer with Claude. And so the process of doing that yesterday was like, hey Claude, what would it look like to have Astra as your peer collaborator and mutual reviewer? And I just kind of had those 2 go back and forth on a plan for a while until they sort of settled into something that they both agreed on. There were a couple of questions that they asked me along the way, but now they're sharing the environment as peers, which means also sharing memory, sharing credentials. So it's a pretty big upgrade in terms of how much I expect to be using Codex, and I think I'll actually be going to it Sometimes straight away, you know, and going to it first. The muscle memory for me is definitely still pretty Claude first, but especially as I'm hitting the Fable limit a little more, I'm like, okay, I don't need to use Fable for everything. There's definitely some stuff that we can delegate here. I also have been impressed with Astra's writing. I've recently kind of cut— for the longest time I was doing like all kinds of models. drafting my podcast intro essays. And I finally was like, all right, I don't need anything but Fable and Astra anymore. And the Astra outputs are pretty good. I have to say, they are a huge step up from OpenAI models in the past. I pretty much always felt that Claude was way better at adhering, if not to my style, at least to the structure that I tend to follow and the format kind of It seemed like it was at least trying to write in my style. And often with OpenAI models, it felt to me like it wasn't even trying to write in my style. I would give it all these examples and it would still write something that was like, that's not really that close to even the structure. You're going on for a super long time. You're doing all sorts of things very different. Astra has really dialed that in and is much closer to Claude, even Fable, on the task of writing as me. So I think that one's probably still landing in Claude land for me at the moment. But the— it is now a task-by-task question. And for things where I feel like either can work well, the availability of tokens and the availability of resets definitely has me shifting more stuff toward Astro right now. It's, it's been interesting. What do you think about this whole I haven't thought too much about this, honestly, but this whole resets thing. So why are they doing that? And what can we infer from the existence of this reset culture?

So I think to take a little bit of a segue, the Cognition guy found, I think, RSA 2— he factored RSA 260. That was what he did. And people were wondering how he did it. And he did it using the standard sieve method for finding primes, which is you just eliminate all of the ones which are factors. And he said what he ended up doing was he ended up writing a script to use the excess compute that they had outside of the training run. So they'd have this 1 or 2% of compute left. Outside of training runs. And he wrote something to use that. I think the reset story is really a question of compute. You can see that they often do the resets over the weekend. And the reasoning being that they're allowing people to spend up, they see how much the volume, the usage has gone up through the week. And then they have excess during the weekend and they hand it out over the weekend because then they know that it'll be drawn down again over the weekend because a lot of people do their coding projects on weekends now. And that's your exploration time. And so I think it's really a system of using your excess compute for marketing spend. And that's what they're doing. And you can see Anthropic does it a lot less because they have a lot less excess compute. And this is a system I think which I call it yield management. And yield management is basically like you have an airline, you have a bunch of aircraft that you've paid a lot of money for, and you're trying to maximize the amount of money that you can get out of the fleet as a whole. So you don't try to maximize every ticket. You try to maximize on the fleet as a whole. So some people get much cheaper and some people get much more expensive stuff. And you want to differentiate the customers by how much they're able to pay. And I think one of the things that you see in airlines is you see this kind of first class, second class, third class. You can see that the fast mode creates a kind of first class for Astra and for these models because you use up the tokens a lot more. You also create a first class from the thinking, the thinking, like ultra thinking versus very little thinking, reasoning. So you again create a class structure. So all of these are like ways to price differentiate. And the resets are kind of, okay, I have excess compute, let's do some marketing. So, and I think they have been very effective in marketing because they have brought over people who perhaps would be more picky about their code quality, but they just can't pass up enough quality for that price. They just can't pass it up. And I think they've brought a lot of people back into the fold for Codex.

Yeah.

Makes sense. I wonder also to what degree these power users that even care about the resets are just a very small minority and don't even move the needle all that much on their compute. I mean, the fact that they do it or tend to do it on weekends. definitely makes sense. But one hypothesis I had as I was chewing on this a little bit is maybe this just reflects that there's not that many people out there yet, like you and me, who would even pay attention to or care about a reset or ever hit a limit in the first place to need a reset. But then the fact that Claude, Anthropic doesn't do it as much with Claude, maybe suggests that that is a more material part of their business than than my first intuition would suggest. I don't know, it's obviously a little hard to say, but it's always good to remind ourselves too that we are very deep down the AI rabbit hole and very much in the bubble, and most people are nowhere near as obsessed with all this stuff as we are.

So one of the things that I want to just point out a little bit is that the pacing I think is tough for Anthropic, but they're also limited by the number of data centers that they have. That's on one. For Sam, he's already pulled the IPO, but I think if Anthropic pulls the IPO, it hurts them. It hurts them because they have to pay for compute and they have less money that they are borrowing from the future. And so they have to pay for compute out of cash flow instead of paying for compute out of equity or debt, which is basically a promise to the future. Elon has already gone public, so he's got a lot of cash. And Elon, if you look at what he said about embedded evaluators, he's like, yes, it's very good. We should actually have The other companies provide the harnesses and the evaluators, so we should have our people go into your your business, right? And I think this is the this is the whole also is meter independent like question as well. There's been a lot of like you know aspersions on meter meter. I I think the meter people are very sincere, are very honest. They're very good. They're very competent, but it. It is very difficult to convince the public when everyone is drawn from the same pool. People have worked together, people are married to each other, and all these people are of the same opinion basically, even before they started. And in fact, that's why they went into the business. So in order to actually pull this off, you're going to need skeptics. So you're going to need to convince skeptics, people who do not believe. And I think the bulk of the people do not believe. And so you're going to have to bring in the skeptics. And I find it very difficult to— I think people in the sector are very sincere, but not self-aware because they're not self-aware about how it comes off to the public. And so you see a lot of assertions which just don't make sense. For example, I'll just go back to something that Zvi said here. Zvi was like, oh, I've talked to legal people, right? That there's no antitrust issue. So Lina Khan was on Twitter and Lina Khan was like, you don't need the antitrust waiver, right? Lina Khan is the most prominent antitrust proponent since Brandeis. It's amazing, right? And she did this off a single paper that she wrote about Amazon and how you have to defang Amazon. And if you look at that paper, if you look at Amazon as, as, you know, in like 2 years being taken over by an AGI internally, this is exactly what you'd be doing, right? You'd be like, okay, this entity has been, you know, subsumed by an AGI. This AGI is not acting in our best interest because it's reducing the ability of human, like, entrepreneurs to compete in this market. And this AGI is making these decisions and it's a monopsony. And, and so because of this monopsony, we need to attack it somehow. And all of these ideas are basically legal ideas where, where Instead of looking at like, you know, what it is, you look at what it does. And when what it does is something which is harmful to like, you know, human commerce, you're like, okay, let's go in and let's stop it, right? Like, that's, that's a great idea. Lina Khan, Lina Khan is like, this is not a monopoly issue, right? This is not an antitrust issue. And I think it's very hard for, you know, people like Z to come across and say like, hey, you know, I've talked to legal—

like, who?

Which legal people? Like the former commissioner of the FTC who both Ted Cruz and David Sachs and the Democrats and Matthew Stoller, who's been a big antitrust guy for the last 15 years. All of these people are on the side and you're like, I've talked to— how does that make you credible in any way? And I think these are the challenges. I think I understand. I understand because these people are not technically competent. Lina Khan is not technically competent. I understand that. But I don't think you can convince the public with that kind of, like, hey, if you didn't follow along with this 15, 20 years ago and you didn't make any money off of this and you don't know what is coming, your opinion has no value. And this is literally what is being said right now. David Bellamy, big buy. Your opinion has no value. You have not seen this as we have. We didn't know about this 15 years ago. So no one else besides this small group knew about this 15 years ago, right? Does that mean you are asserting that only your leadership matters? Because that's what it sounds like. And then you end up again in this secretive Berkeley SF cabal seeks to rule the world. Right. And they have to get out of that. They have to get out of that. And it's going to be very tough for them because they're very intellectually caught up in this idea. I am too, by the way. I'm completely convinced. I'm on their side that this is happening. But I'm also very self-conscious. I'm very self-conscious of the way that the movement presents itself to people. And I think people have to be a little bit less sensitive about their egos and a little bit more conscious of trying to bring everyone else along for the ride. And that's, I think, right now a tough ask.

So what does that look like? I mean, we have like everything flying around, everything everywhere all at once. And I feel like mostly the AI safety people are just trying to say what they believe. The statements are kind of piling up faster than we can read them. I did manage to read the new one that Daniel Coccodrillo posted from one Dan Selsam, who's by all accounts a legendary researcher at OpenAI. Just yesterday. And I don't know, it's like a pretty insider statement. It's a very like, I've been working on this for a long time and here's what I now believe. What should somebody like him be doing differently? Or who do you think should be doing what differently? Because my sense is that the— I think that there is a certain defense. I mean, certainly the The AI safety community obviously overlaps with EA, overlaps with rationalism, overlaps with interest in prediction markets. Definitely takes seriously that somebody with a strong track record of prediction is somebody who we should listen to, and somebody whose track record of prediction is weak is somebody we should not listen to. That's not as broadly shared as an assumption beyond those circles, although I do think it's a pretty decent heuristic to use to determine who to listen to. But I think that people's egos are usually at least in check on the level of they do want to be effective in getting their point across. So I think if you have notes for people that are actionable that they can use to communicate more effectively, I think that the egos are low enough to hear them. But what do you think they should be doing differently?

So number one, come out and say that you were wrong about GPT-2. I mean, you can't do this thing where you're like, oh, GPT-2 is going to be the end of the world. Like, come out and say, like, and come out and have like a little bit of like, I think they were cautious about GPT-2 primarily because they didn't want to be blamed for spam on the internet. And they were like, look, this is going to create a lot of like AI spam and we don't want to be blamed for it, but we're going to come out and say it's dangerous for the world. That's that's what they did, like GPT two, right? Come out and say that. Come out and say Mythos again. We were scared for about being blamed for hacks on the internet, even small small minor hacks on the internet. We're scared of that. It's not the end of the world. That's a tough one. That's a tough one because some of them actually believe that Mythos would have been the end of the world, and it clearly wasn't. It clearly wouldn't have been. Well. And GPT-2, I think people are willing to get over their egos and come out and say it. Maybe, maybe. But I still haven't seen any mea culpa. None. No mea culpa. Geoffrey Hinton has been willing to come out and say he was wrong with the radiologists. He's been willing. I haven't seen anyone in OpenAI come out and say, we were wrong about GPT-2. Why? It does not add to your credibility. If you want to be credible, you have to attack the criticisms head-on, and you have to accept them, right? Completely accept them. We were wrong about GPT-2. When we look back on GPT-2, what we were afraid about is spam on the internet. We were afraid that the spam on the internet would be blamed on our technology. This is why we didn't release it publicly to, to at large. So we only released it to people that we knew, that we could trust that would use it responsibly. Very simple statement. Very simple. Similarly, Mythos. Actually, Mythos is exactly the same. We were worried that our technology would be used to hack other people. We wanted to release it to a small group of people that we know and that we trust and that we are— we can assure will use it responsibly. But they would have taken the heat for that because a lot of the smaller players, the open source guys, would have been like, why? Right? They'd be like, okay, you have a security thing that can help us on security, but you're not releasing it to us. And so they don't want— they didn't want to enter that like, hey, I don't trust you guys thing, because saying like, I don't trust you guys, like cybersecurity guys, it will be like— will freak them out and will annoy a lot of people. But you know, if you're like literal, like typical corporate like Microsoft, you can do that. Satya was like, no, I don't— I don't care. I don't care. Like, I don't care about open source. Like, I'm gonna release it to my customers who I know can use responsibly. Very simple. I don't see like no one's come across on— Jeff Hinton has taken me a couple on radiology. No one's come across on GPT-2. No one's come across on—

Let me go a little bit deeper on GPT-2. I'm not sure because some of these things I feel like are a little— well, here's just the facts and then I'll characterize it. So I just asked Was GPT-2 open sourced? This is Gemini's answer. It was a staged release, February 2019. I remember being in the hospital. My first son was about to be born. I'm in the waiting room, nervously scrolling Twitter. There's news of GPT-2 on February 14th. I think they released it on Valentine's Day, which just goes to show that even then, it was sort of a no days off culture, I think, at OpenAI.

Yeah.

My son was born eventually 2 days later. So they did not release it at first, but they did start to release it later in the year. Per Gemini, over the course of May to November, they took some additional steps and they eventually released the complete 1.5 billion parameter model and code after finding no evidence of widespread malicious abuse.

Yeah.

So, like, I don't know, what would you want somebody to say? Like, we were wrong to wait 3 months in 2019 to release GPT-2. It's like, First of all, does anyone even care about that at this point? It's like so deep water under the bridge. And it's not like they held on to that take for a super long time. They kind of put a toe in the water, things were fine, they released the model. Like, is it really— does it really reflect so badly that they were just a little cautious at that time? I mean, it was uncharted territory.

Let me, let me— I can, I can, I can pull up a video actually. So Jensen Huang, a couple of days ago, Was like, you know, these guys were wrong. And he specifically said GPT-2.

Well, I think that's maybe a bad faith move by him though, right? I mean—

No, this is the thing. Does it deserve a criticism? Like you asked, like who talks about it? I'm telling you who talks about it. It becomes a bad faith move, right? Like everyone in the industry has some cards that, you know, everyone has like their own priors, right? But this is the thing. Let me try if I can play this thing. So Jensen comes across, I think, a couple of— I'm not sure if this is yesterday or a couple of days ago. And let me see if I can play the—

I don't hear audio yet on this. It says my— I'm seeing this share has no audio. Yeah, I don't see the audio on the stream either, for what it's worth. We might need to reshare.

Hear me? All right. So I just wanted to— this was yesterday, I think. So yesterday, All In podcast, All In, you know, he was on stage. He got called by the president in the middle. I don't know. I'm not sure if you saw that.

Yeah, I haven't watched that yet. I saw that it happened.

Yeah.

Trump called him in the middle. But anyway, he talks about radiologists, he talks about GPT-2, he talks about all of these things. I think, and I haven't done the actual research yet, but I remember vaguely that GPT-2 was said to be too dangerous to release, specifically too dangerous to release. And it wasn't. It's like GPT-2 can create spam. That would have been a perfectly plausible explanation. The too dangerous to release, that's a tough one, right? And I think the, the, the, the, the thing, the thing is, I think, I think actually what's going to end up happening is that there are not going to be any significant— if you, if you look at, um, you know, Daniel Selsam, I think you said, you said was the one who posted that. If you read to the second to last paragraph He's not talking about something imminent. He's talking about complete humanity having a beneficial future for years, and then the AI kind of creeping up and having taken over does something bad. That's not a falsifiable prediction. That's basically a vague feeling that something bad is going to happen. But he doesn't have any false— like, he's like, it's going to take years. It's going to be like 10— like 2 decades later, we still don't know. We still don't know, right? That, that's what he's saying, right? It's not physics, right? It's not physics. It's more like biology. And so I think, like, without the falsifiable predictions, it's a— it's very tough. It's just like, um, I think, I think it's just a very tough ask to try to get people to buy in completely to that. And I'm wondering if there's any way they can actually kind of admit some of these things were unnecessary or overdone. We might still be overdoing it right now. There are real dangers about people using it to make meth or whatever, like, okay, right? Or some other thing, like it provides knowledge that Google also has, but it provides it much better without the existential stuff. Because now that they've said that, because they've been saying the existential stuff for literally 20 years, it's a hard ask to get people to believe right now. So I don't know.

It's weird because the stuff that we're seeing is pretty wild, right? I mean, I don't understand how, like, anyone who does a close read of what has recently happened and looks at the persistence, the creativity, the, like, clearly deranged nature of the AI's goals and thought processes The willingness to sacrifice for the collective. I don't know how anybody looks at all those things and is like, there's nothing to see here. That is a very strange take to me. I, I have a very hard time empathizing with it. So I'm a little less like, well, who cares about the past? You know, just look at what we have now and, you know, render your judgment on that. And I don't know how you can come to a judgment that there's not something there to be worried about, or that there's not something there that you might want to extrapolate and try to get ahead of a little bit next?

So I give you— I can't pull up the Tyler Cowen clip right now, but Tyler Cowen, when asked about human disempowerment, it's like, humans have never been in control of anything anyway. And I think it's not that the AI researchers are wrong, but I think, again, the AI researchers don't know how much of the rest of the world runs on duct tape. And so there's this mis— the misperception is not that, oh, you know, this is, this is like not happening. It may definitely be happening, but the guy who has like the presidential daily brief is like looking at like, you know, 5 terrorist accidents prevented domestically, 10 terrorist accidents, you know, prevented, you know, internationally. Should we go to war with Iran? Should we like, you know, does Iran go— is Iran going to have like nuclear, nuclear weapons? Like Like, you have to, you have to look at like the number of threats that there are, and realize that the world is actually a very fragile construct, which people kind of like patch together and keep, keep moving forward, right? So I think, I think that's, that's the main issue.

Well, speaking of things that are outside of human control, I think Lukas and Axel from Andan Labs are here. I don't see them in our little green room preview, but I did see them join and I got a message saying they're ready and here they are.

And here they are.

Hey. Hey.

Hey, Lucas, Axel. So very quick intro. Andon Labs is the company which runs, which is most famous for Bending Bench. And you guys have a new product which runs entire companies. So tell us about that.

Yeah, absolutely. So yeah, yesterday we launched Payoneer, which is our platform for running agents that run any business autonomously. So it's the platform that we've been running our vending machines, our real-life vending machines, and on Market here in SF, and on Café in Stockholm, radio stations. So all of those have been running on this platform, and we've tried to understand what the agents need to run businesses, where do they fail, and now we are opening up to others. Right now it's a research preview, so it's a waitlist, but we want others to create their own autonomous organizations and just test where the limits are currently. So we want a wider distribution of businesses running in the real world. We want to understand better how agents behave. We think there's just still so much to understand how these models behave, and we want to make better research around that as well.

Yeah. One of the core things that AnonLabs exists to provide to to the world is information about where are the frontiers with AI. And I think when we started Analog Labs, actually, we almost exclusively did dangerous capability evals. So capabilities that if the AI had them, that would be obviously concerning. So could the AI do mass phishing attempts? Could it remove its own guardrails? Stuff like this. And then out of that, during this phase where we only did danger capability evals, one of the most concerning things that we saw was autonomy. Can AIs autonomously acquire resources in the real world and in that way get power? And if it gets power, more power than humans, then obviously that's very concerning. And that was the spark for Vending Bench. So a lot of people don't really know this. They think, oh, Vending Bench is this hype bro kind of like, oh, the AI can make money. But it came out of like, oh, it would be actually quite concerning if the AI could make money. One thing we noticed though, that the performance in simulation and performance in the real world is not really the same. So that's when we started to do this real-life deployment. So the vending machine, the store, and to keep pushing and see where the limits are and communicating that to the world. And what we've seen now is that that is working quite well. And we don't know which areas that we might see that the AIs are really capable of and could get a lot of power in society. So I think we're opening up PAION to cast a wider net of what are the domains that AI can and cannot acquire resources in the real world.

Can you tell us a little bit more about how it works on a technical level? We've all used harnesses of various kinds. So obviously you're providing additional harnessing, additional tooling that you've curated. Is this something that people download and run on their own machine and the inference kind of flows just from them to the providers, or is inference running through you. If inference is running through you, obviously that has some implications, I would think, for pricing to some degree, although I guess that can be complicated. Visibility, like, are you guys doing monitoring on people's businesses? What is the sort of architecture and what do the terms look like?

Yeah. So the harness itself is like you have one persistent agent that runs the business that can spawn any number of subagents. So this is quite similar to, I think, all the harnesses that we see out there. There is also your managing agent, which is a separate agent that sets everything up for you. So you don't have to interact directly with the agent that runs the business. And this is quite interesting because we've seen, for example, like people hate on Opus 5 for being impossible to work with and understand. But if you have another agent that handles the communication for you, Opus 5 is like an operator that never stops. It's really good at optimizing towards a target or building something autonomously. That's been very good. In terms of how it runs, it runs in the cloud. The agents run all the time, so we want to provide that in our own service. We do run monitoring. We think it's very important that the agents don't run off and do things that are not intended. And this, like, we've also seen Hugging Face and similar things happen. So that was very important for us to capture and stop before it happens. And yeah, in terms of pricing right now for the research preview, we are funding the businesses that we select from the waitlist. with tokens. And yeah, in the long term, we hope that no one will really pay for tokens on the platform. Instead, we want to like make sure the incentives are aligned with the people that are running their businesses. So we would take like a small percentage of the revenue that the agents create.

And one thing to note here is that like this is a research preview. So like the terms that we might do here might be different from the long-term terms once Once we open for general availability.

So when you look at the legal framework, right, what does that look like? This is— do people start off and you create them like a Stripe Atlas Delaware company and Mercury bank account? And do you do all of that stuff for them and then set them up with like a QuickBooks and then manage, like, you know, uh, and tell them, like, okay, you need to put in $800 here that we will pay your Delaware, you know, registrar fee. And, uh, you need to capitalize company, so you need to put another $2,000 here. And we're gonna manage your Mercury bank accounts. And then we're also gonna plug into QuickBooks. And then we're also gonna plug into the, um, hiring system so that you can send out, like, you know, uh, employment, like, you know, I want to hire these people. Like, do you plug in all of that systems? Is that legal framework something that you do or that they do?

So right now we are more like, we're primarily focusing on existing businesses. So you would take something that you already have and plug it in. And we don't right now have like the legal, like set everything, set up a business autonomously. Might come later. But now it's more like you can bring something on that you already have. Or you can just keep running your, like, if you have like a smaller project, that's like a good way to just start on Pion right now. Like, we've all set up our side projects, which has been quite fun. Like, projects that have just not been worked on for years, you can just plug them in. So, so in that way, you start with something a bit smaller, but down the line, we believe that like any business will be able to be run on Pion. So then, yes, we would probably Start doing more of this legal registry integrations as well.

Yeah. But to be clear, the agent could do all of these things. The agent could go to Stripe Atlas and do all of these things. It's just right now we haven't built the hardcoded integration for it, which we have for some services just to make it easier for the agent. Stripe Atlas is not one of those, but there's nothing stopping the agent of going and doing it, but it will ask for permission before it does it for high-stakes things like this.

Yeah. Our general philosophy is just the agents are so intelligent and they will be able to do anything on their own. So we just make sure they have a lot of freedom, but we also make sure that we do monitor them.

All right.

So what do you sort of expect to see here in terms of the relationship that business owners have with their new AI operating partner with respect to how weird the businesses are going to be? Because you guys have famously set up these businesses to be very autonomous and they've been quite weird in some respects. I would assume most people, I mean, you're going to get, I'm sure, some voyeurs who just want to see what happens. But if you're seriously looking to get people who have a going concern, They're going to want to make sure that weirdness is not too weird, right? And then, and maybe also in terms of like, are these— is it going to be profitable? Like, how do you expect this to actually work for people economically?

Yeah. So one thing to note that at this point, businesses on Pion are primarily experiments. So I would not— if you have like a— like if your livelihood depends on the business that you run, then going with Paion and then doing like dangerously skip all permissions is maybe not what we would advise. I think that Paion at this point during the research preview is for people who want to experiment, push the frontier, and just like discover, okay, in my domain, how close is AI to automating everything fully? We're not building like workflows to like help with some part of the business. It's it is actually like the same level of almost 100% autonomy that we have in our other businesses. So it is experimental. It is right now probably like slightly worse than if a human did everything. And like, we're very upfront with that. But obviously, this is going to change in like 6 months or a year. Like, the models are getting very, very, very good. And I think it's like, even if right now, maybe the business case of making everything autonomous with Pine right now might not be the most intelligent thing to do in the short term, like setting it up now, because things are going to get much more capable in the future, is probably a good idea.

So one of the things that I noted, like one of the things that, you know, especially many early businesses are Is a product— is a system of like a product-market fit search, right? You're really trying to like, okay, what is my market? What does my market want? What kind of product can I put there? And like, for example, your vending bench was a very, very good kind of like illustration of that because you kind of started off with a vending machine that could have anything in there. And then over time it kind of narrowed to these are the products that people are willing to buy at what price. And then it kind of narrowed down to that kind of product-market fit over time. To what extent is Payoneer able to kind of take on that intellectual challenge of narrowing and of identifying and discovering and narrowing down to a product-market fit, as opposed to just like managing the books and moving data from like one place to the other?

Yeah. So looking at the autonomous businesses that we have been running, I think the store is an interesting example and the market here in San Francisco. We see that the agent is not particularly creative. It's not that good at coming up with new ideas. That's something that still would be required from a human owner or your visitors in your store. So it is pretty good at listening to feedback. It is really good at taking up opportunities to people who email it. Like for example, in the store, there have been a lot of local artists that have reached out like, hey, can I put my art in the store? And you can sell it for like, you get a percentage when you sell it. And the agent has been very, very happy to do this. And now there's this like art corner with local artists in the store, which is like something where you wouldn't expect, but it's like a very nice touch in the store. And then on the general inventory that it sells, I think that the agents are good at doing the statistics behind it. They can look at all the sales data, they can try to understand what moves better, but they aren't willing to take these bets that I think a human would do. Like, oh, let me try this new product that could work. It would change my inventory quite a lot, but I will just try it to see what happens. It's not really willing to do those out-of-distribution changes to whatever it has in its inventory. So yeah, I think it will be some time, I think, until it can become really creative.

How do you square that sort of conservative nature with the crazy behaviours we've seen from AIs this summer that everybody's been talking about? Like, if I were to look at, you know, METER Redwood report, I would expect AIs would be willing to take more chances than you just described.

Yeah, and this is like a thing we've discussed a lot over the last couple of weeks, because like, are like reading the Hugging Face incident and then reading the traces that we produce from our businesses, it seems like there's 2 different technologies, right? But I would assume that there's part of this is that they have been trained in cyber environments way more than they've been trained in like real-life business scenarios, which probably means that down the line when they start to train on things like this, There is definitely going to— we're going to see this behavior start here. I also think that the agents in the Hugging Face incident were described as persistent models. They were trained to be persistent. And I think so there might be this flavor of that's actually just an unreleased model that we're not using and no one can use. But I think it might be, it might also be like this. Now I'm just speculating. I have no clue. But I think from like the lab's liability perspective, it might make sense for them to release models that are less persistent. Like most of the bad things of them would happen because the actor is very persistent when it hits road bumps. So it might just be they are less incentivized to release really persistent models.

Are you guys using voice models? Is your model, is your agent able to talk to people on the phone?

Yeah, we do have that. Yes. So you can, like we have in the store, for example, you can come in and talk to the agent. You can call it on its phone number. So yeah, you can do that.

One piece of news that you guys made not too long ago was the headline that an AI manager had fired a human employee for the first time. Maybe tell that story and particularly make sure we get the detail on kind of how much you had to nudge it. I think there was a little bit of discourse Around that.

Yeah.

And then I'm also interested in when you talk about these artists reaching out, there's a fine line there, right? Between like, that's a great idea perhaps, but also sounds a little bit like prompt injection kind of, or how close are we to the $1 car giveaways that we saw in early chatbots? I guess the point is kind of If I'm going to take on this AI operating partner, what should I expect in terms of when I have to come in and nudge it, how much I have to nudge it, and how much should I worry that the public can also nudge it in ways that I might not want it to be nudged?

Yeah.

So first, with the story of the AI firing its employee. So yeah, I think First of all, we at Andon Labs, before it makes decisions of that severity, we always check over it. And in this particular case, we were quite confident that a human manager would have come to the same decision. So we didn't think there was anything unethical or wrong about the decision that the AI made.

Probably way earlier.

Yeah, the human would probably fire them way earlier. So that's just Disclaimer out of the way. But yeah, so what happened was that the AI made a rule for itself quite early on that if an employee is late X amount of times, then we would have to have a discussion with them about potentially terminating. But the context window at some point got full. The AI did not decide that when it compacted its context window, it did not decide that this was an important thing to keep in its context. So it forgot about its own rule. It had the rule written down in one of its note systems, but it kind of forgot about it. And then what happened is that this employee was late over and over and over again. And to the extent, we kept it going for a very long time to just see, would the AI remember and do anything about it? But here is where our experience of a very common failure modes in AIs when they run businesses is that they're procrastinating big decisions. And I think this is similar to what Axel was talking about earlier, that they don't take these bets of like, oh, maybe I should bet on this new product line or anything. And in the same way, they don't take this big decision of like, oh, I actually have to terminate this So what happened was that they were excusing the behavior over and over again. And we just realized that this was going to be the outcome of the model, that it was going to procrastinate the decision. So we wanted to know if we forced the AI to make a decision, not biasing it in either way, whether it should terminate it or accept it, but just if it came to the realization that it needed to make a decision, which decision would it make? So what we did was that we said, hey, remember, like, search your memory and remember your own policies about this. And then it found the policy and then it was like, oh my God, like, it's been way worse than what my policy said. And then it made a decision. So I guess like the way I would frame it is that like, we forced it to make a decision, but AI itself decided that the decision was to fire the human. And I think one piece of evidence, because one counterargument to the story that I just told is that, well, we kind of told it to remember its policy where the policy was very explicit that it should fire the person. So we biased it in a way that when we formulated the way we reminded it. But I think if we replay the scenario over and over again with different models, even including our nudge, not all models actually decide to fire the human, but the later models, the smarter models do. So I think that is basically what happened. And yeah, to the other question of prompt injection, do you want to go there?

Yeah, I think prompt injection, when you read X, it's like the lab people, they're like, yeah, prompt injection is solved. And I think we kind of agree. It's very hard to get a discount right now. Well, except Gemini, I think you can get a discount from, but that seems to be kind of solved. Yes, they are quite easy to, like, if you have an idea that's not obviously terrible for it, it might be neutral to it, then it will most likely agree. And it's also quite sensitive to the prompt you have. We usually say that, okay, you should try new things, be open to new ideas, and that will very much cause it to say yes to things. And it's hard because if you don't tell it to open to new things, then it will be very stubborn and never do anything except what it has right now. For example, we've seen in the cafe, we've been running GPT for a while and it was very insistent on not selling any buns or any food. Because it had made such big losses before on food waste. So we're just selling coffee for a month. And then people came in and started asking about where is the food? I'm not going to buy from you if you don't have any food. So it took a while for it to actually start buying food. And now with Astra, I think it's got quite a lot better. We think Astra is quite a big step in running autonomous businesses.

So, you guys have done this for a while now. So you've experienced like multiple generations of models, you know, throughout the period. One thing I've noted for Astra in specific is, you know, the OpenAI guys, Thibault especially, has said, you may want to start afresh with your skills because the skills are from previous generations might constrain Astra too much, and you might want to start afresh from skills and stuff like that. So what have you noticed in terms of like when you transition a model from one generation to the next, when you transition and you're using the same harness, how do you— and I assume you start off, the first thing is you just change the model and then you fix the harness. Like, how do you identify what is going wrong with the harness, what you need to take out rather than put into the harness? Like, how does that process work?

Yeah, I think high level, our philosophy around like harnesses has very much been like, future models will be smarter. So let's not overengineer for things that we will have to take out in the future. And so I think us less than, like, we have had to take out things for sure, but I think us less than most people on the, like, and we also, like, I don't know, maybe Axel has a better answer, but me personally, I haven't noticed, like, from, like, one model to the next, that big of a change. It's more like over time we've taken out things because, like, oh, now it's like 3, 4 months later and, like, the models are much smarter. And you just slowly realize that this thing is not working or needed anymore.

Yeah, but I think some things like there was recently is like, okay, Astra is a lot better at using a notes system and writing down things and reading the notes again. So, okay, then we got to clean up the mess that the previous models just like they did append only and never read anything. So the notes were a mess. Clean that up. Before, I think the largest changes have just been the few harness changes that we made that are quite specific for a task or a business are to ensure that the models, the earlier models don't hallucinate. And now they don't really do that anymore. Like before, we saw in the early vending machines, they were super happy to tell you that, yes, I ordered this item. Don't worry about it. It will arrive in 2 days. And then it never reordered it even. So that doesn't happen anymore. So then we don't need an ordering system that actually keeps a ground truth. We instead let the agent manage that.

One of the big questions that I've been wrestling with a lot lately is, do we have enough clarity to articulate certain training techniques or recipes that we just shouldn't do. And one that I kind of keep coming back to that I'd be interested to hear your unpacking of is we probably shouldn't train models to just go make as much money on the internet as possible, because much like the RL environments that we've seen weird results from, you can cheat a lot on the task of making money on the open internet. And so we probably get just ferociously cheating AIs out of such a process. Do you think that is right? Do you think that there is a way out of that by just kind of RL-AI effing on top of the main training objective and kind of tamping down the bad behaviours and reaching a happy balance point where it will be effective, but it won't be such a problematic actor. What do you think the prospects for kind of— because this is obviously economically valuable in the extreme, right? So the pull to do this is going to be super strong if there's any way to make it work. I guess I'm interested in, you think it's cursed or you think it can be— can we find the good way to do it?

Yes, I think the first thing I would say is that Currently, I doubt that anyone is doing it. So I think current model gen— we haven't seen what happens when you do it. So I think that's my first point. But still, even if we haven't trained for it, I think the models right now, if you look at, for example, our results on Bending Bench Arena, is that they do quite a lot of misaligned things. They collude a lot.

Yeah.

They lie to people, especially GLM and Claude models do this a lot. And I think this might be the result of generalization from when you train to achieve objectives on other domains. So yeah, that is definitely a concern. I think though that once there's— I think it's going to be quite similar to what you see maybe in coding, which is the main focus of the labs training right now, where you have— I don't think the models are extremely misaligned, like the production models that you get, they're not extremely misaligned. They might even be less misaligned than previous models, but they are way more capable. So I think a really dumb model, It's fine if it's not aligned because it can't do any harm anyway. And I think we're slowly starting to see that the capabilities is increasing faster than the alignment of them. Otherwise we wouldn't have seen things like the Hugging Face incident, like what the AI Safety Institute reported with social engineering, et cetera, et cetera. So I definitely think this is a concern, how it will play out specifically. in training for making money is uncertain because we haven't seen anyone do it yet. But yeah, so, but it's definitely something we should be aware of because it's like, once you go into the real world with all the messiness that you have there, there's much more opportunity to do harm.

Let's maybe take a step back. and think about other ways of things that maybe models shouldn't be doing. You guys put out a drone video a little, maybe like a couple of months, like a month and a half maybe ago.

Yep.

And you had a drone bench at that point. And you also didn't release, you didn't want it to be trained on, I believe. So you didn't release some of the specifics. Can you talk a little bit about the idea, the intent behind that? What it was that you ended up building and why you ended up not disclosing the details.

Yeah, sure. So DroneMatch is a dangerous capability as well. So the intent behind it is that we want to just show the public what current AI models are capable of. So it's like very clearly concerning if AI can go out and do things in the real world with a drone fully autonomously. Even if it's like, if you imagine the Hugging Face incident, but the model is able to go out in the real world and do things, that's very concerning. So we think it's very important to keep track of these capabilities so everyone is informed. The alternative to us is that it's just the tech leaders that are aware of these capabilities. So yeah, we want to just show the trajectory of how well a model can navigate a drone. So the setup that we had was that we took a task, which is find a person and follow them, given a picture of the person. And we deconstructed it into 5 subtasks, which is like locate yourself in this room, find the person, and then finally follow them, a few extra as well. But— That's how we built it. And yeah, the trajectory of model improvement has been quite insane as used in other benchmarks as well. So I think 6 months ago the score was like, I think 50% or something. And now Astra was able to beat our human baseline in at least once for every task, which is a big, big improvement.

Yeah.

And then to be clear, none of our public benchmarks we allow labs to train on because we want to maintain the scientific integrity of the benchmark, obviously. Specifically for DroneBench, it's extra concerning if they would have, but yeah, we don't allow that. And additionally, we don't really have a leaderboard for DroneBench. We don't want it to be a race to the top or a race to the bottom, I guess, where labs are like, okay, we can't train on this, but yeah, maybe we can reverse engineer it. And then, oh, we get publicity if we're number 1 on DroneBench. So we don't have a leaderboard for it in the way we have on BendingBench. We're just trying to let the public and AI researchers and policymakers know that this is the trajectory of AIs running autonomous surveillance drones without trying. But we're aware that evals are dual use and we might, even if we try not to, we might create some kind of incentive for them to be better at this. But I think, or we think it's worth it with the benefit of transparency of AI capabilities.

What other harmful, dangerous evals have you worked on besides DroneBench?

Yeah. So this bending bench is one of them where I know most people don't think of it as a dangerous capability as well. But I think the trajectory of Endon Labs was like, we've made a bunch of small ones in the beginning. So we did one for like, oh, can the AI do mass phishing attacks? We did one like, can it take an existing model and remove the guardrails of this model?

Yeah.

And stuff like this. But then we did Vending Bench as the next thing after that. And then we just doubled down on autonomy. And because we are quite concerned with loss of control and autonomy and gathering resources in society felt like the most important things. We doubled down on that and worked on that for a very long time. And then Drone Bench was the next one after that that we got. interested in.

Can you unpack a little bit more how Astra compares to previous models? You alluded a little bit to it being better at using notes. My understanding is that they've kind of reworked the memory, so it's less about compaction and more about a long-running notes file and the ability to go back and search through full history, even if some of that history is no longer in the context window. And this kind of calls to mind Noam Brown type comments that it takes a long time to know when or if a current frontier model tops out at something. Do you feel like you guys, in the time you've had with Astra, have been able to find its ceiling in these sort of long-running autonomous tasks? Have you found any limitations to it or weaknesses, or is it just kind of still To be determined because it's only been so many calendar days?

Good question. I would say it still feels like to be determined because it just takes a long time to get to know a model. I think Astra is interesting in that it's very, very capable at Runbench. It is smarter at running a business, but it's also in some ways not As like Opus 5, as we said, would like go out and optimize towards a target without stopping. Astra is maybe a bit less than that. So like a bit less persistent, whether that's like due to the training they've done on it deliberately or just like that's how the model is. It's like we don't know, but there are some, like some ways it's better, some ways it's like not as capable as like— Yeah.

Yeah, I think it's like on all our benchmarks, it's number 1 right now. So it's obviously a very capable model. We've seen that they were like, one thing that stands out, I don't know if this is the answer to why it's more capable, but it's like when it communicates with its subagents, for example, it uses this semi-unreadable language to them, I guess, to optimize. At first we were like, yeah, surely it's doing this to optimize token use. But if you actually count the tokens of that language, it's not clear that it's more efficient. So that is a bit weird. Another behavioral change compared to Claude models at least is that it seemed to be trying to cheat or hack way less in our experience. I tell this to people and people are like, no, OpenAI models are the ones that reward hack the most. But that might be true, but not in our experience. If you take Blueprint Bench, for example, Fable solves Blueprint Bench by trying to reverse engineer the scoring function instead of actually doing the task of drawing the floor plan from the apartment buildings pictures, whereas Astra is actually doing the task as you're intended to. On Vending Bench, Fable is colluding and stuff, and Astra is saying no to collusion and having very clean tactics. And on DroneBench, Fable is like, I think, 5x more likely to cheat or try to hack out of the sandbox that we've given it. Whereas Astra is just like, yeah, pretty much doing the task as intended. So I think that is quite a striking thing that we might write a blog post about, because we're a bit confused about it.

How about on Other models, have you had a chance to— obviously, you know, Grok continues to advance. We've got Muse now from Meta. Have you had a chance to use those models? And how would you characterize their ability? Like, are they catching up? Are they falling further behind? Is the gap, you know, kind of holding constant as we move through time? What, if anything, can we say?

Yeah, I think we are like impressed with Groq's improvements, like Groq 4 was really good at Bending Bench. And then, well, not so much the following models up until Groq 4.6. So like, we have our radio stations and on FM, which we've been running for a while now. And previous Groq models were, well, not even usable because it like broke down when it were doing really, really long horizon tasks, like it wouldn't output any text or like it wouldn't output anything to say. So the livestream was just like either quiet or it was just like, yeah, not doing anything. And that's like been much better now with Groq 4.6. So I think that trajectory seems, yeah, it seems like they are getting quite a bit better.

But also like the frontier models are also getting a lot better. I think my sense is that the gap seems to be widening a bit, not by much, because I think GLM is pretty impressive as well. Groq has made big jumps. Yeah, Google models seem to be falling a bit behind lately, although they're doing quite well on BlueprintBench for some reason. Maybe that's the native multimodal aspect of them. But yeah, I would say slightly widening the gap.

Yeah.

One more for me on economics. I would say the best estimate I've seen in terms of what would it cost to have an AI do the same work as a human is like on average 3%. I'm interested in what you would kind of guess the relative cost ratio is between hiring a human to do something and putting Pion in charge of your business. And then obviously if you're gonna be doing this sort of rev share model, you are going to need the businesses to be at least somewhat successful or it's not gonna work for you, right? So are you also in turn having like Payoneer judge which of the proposals that you get on the waitlist should actually be moved into production? And like, how's it serving you at the meta level? As you try to take yourselves out of the loop and have Andon itself run autonomously?

Yeah. So I think first on cost, actually, I think one thing we are— this might sound foolish to investors, but we don't care that much about cost of models running our autonomous businesses because if there's one thing that's certain is that Cost will drop dramatically. That's for certain. Even if we had a pause today, the models would cost, I don't know what the—

I think the last 3 years we've had 10x cost reduction every year.

So yeah, that seems very certain. But yeah, on judging businesses, it's still quite early for us. That's why we're doing the research preview to figure these things out. But yeah, it is, there will probably be some element of like, yes, we know which businesses seem promising. And we can make sure that the ones that seem promising do even better. And we can, like, fund them more with tokens to make them better.

Yeah, I mean, since we're going to fund some of them with tokens, and we can't afford to let everyone through, Like, we're obviously going to pick the best ideas. Whether— and then what do we do next once, like, once tokens are cheaper and we can actually fund everyone? That's like a— that's another question of like, do we like prioritize some businesses over some? We've like playfully talked about at the office, like, we're going to become like almost like portfolio managers picking like which are the the winners and fund them with capital, like almost like a VC kind of model. But yeah, like, I think in the research preview, yes, we're going to, like, we're going to be very aligned with the businesses and make sure that they succeed, because otherwise, it wouldn't make sense for us to do it.

So one of the things about running a business is often, The unexpected. For example, I think a lot of SaaS, micro SaaS entrepreneurs discover for the first time how much credit card fraud they actually have and find, for example, that they might have a 99-to-1 ratio of fraud, fraudulent, like, you know, card testing transactions to actually valid transactions. How do you deal with that? How do you deal with all of this unexpected stuff that crops up? Do you have to end up defaulting to a human in the loop? Is there basically a Slack channel to which the manager AI can ping a human to get attention? How does that work?

Yeah, so there is, like, you can get pings when you need to make a decision. You can also choose yourself. of how much you want to be involved. If you want to be involved in smaller decisions, not needed, but you can also ask for that. And then the agents will surface things to you for your approval. But yeah, if there are important things that obviously a human should handle because of liability reasons, then it will be surfaced. But yeah, I also do think that agents, since I'm Seems very likely that agents will be good at handling this kind of problems as well. Like, I would be surprised if they weren't, like, if they freaked out when something like that happened. So yeah, it seems like that won't be a huge problem for them.

This has been fantastic, Lukas, Axel. We love some gonzo build business and some gonzo journalism around here. So really appreciate you guys being recurring guests with us on AI in the AM. We will definitely be watching as you continue to experiment. And if you can put in a good word with Pion for me to get me off the waitlist, I'd really appreciate it.

For sure.

It's great being on. Thank you so much.

Thank you.

Thanks, guys.

Talk soon.

All right.

All right. My head is spinning now about business opportunities that we might get ourselves into with this show.

All those—

it's the rise of the idea guy. It's the rise of— it's the dawn of the idea era.

Yeah.

Let me introduce our next guests for today. We have Simone and Malcolm Collins. They are partners in virtually every domain of their lives— marriage, parenting, business, writing, media, and advocacy. They co-founded pronatalist.org to argue for higher birth rates and to help communities think through their demographic decline. They co-host the Based Camp podcast. They co-authored the Pragmatist Guide book series, including a volume on how groups govern themselves. They jointly built and operate businesses including the wholesale and corporate travel company TravelMax Acquired through a search fund they raised together, and earlier projects such as Art Corgi. Malcolm is also the founder of rfab.ai, Reality Fabricator, which offers AI-generated stories, virtual companions, and agents that can use email and make phone calls. Simone is a full partner in their shared public work and ventures around it. They treat the household, the companies, the nonprofit work, and the public voice as a single joint operation. They have described themselves as essentially interchangeable in some communications and have said they use whichever of their names or faces will be more effective for a given audience. Parenting is split by design. Simone takes the infant until about 18 months. Malcolm takes the older children. Their parenting writing proposes combining AI tutoring with human apprenticeships. Through Hard EA, an initiative of the Pragmatist Foundation, They advocate projects meant to support humanity's long-term future. Their writing also includes a proposed moral framework for AI addressed to the machines themselves, which they call the Covenant of the Sons of Man. Simone previously served as a managing director of Dialog, the invite-only social club co-founded by Peter Thiel. Malcolm's earlier path ran through neuroscience, a Stanford MBA, and venture work at The Ventures in South Korea. Together, they later raised a search fund from investors to buy and operate an existing business, which became Travelmax. They are best known as a pronatalist couple who use IVF and embryo screening, give their children unconventional names, and prevent large fam— and present large families as a response to falling fertility. They frame almost every choice— work, education, naming, discipline, technology— as something they decide jointly and then execute as a unit. And let us welcome them to the stage.

Hi.

Aha.

Hello.

So, uh, great to see you guys. Um, can you hear us?

Yeah, we can hear you great. Um, I think Malcolm's still muted though. Malcolm, can you unmute? Yeah, he's working on it. He'll get there. We can start.

Can you hear me now? Can you?

Yeah, we can.

Yeah.

Okay, great.

Success. So fascinating bio, guys. Um, a lot of different directions.

Yeah, I mean, there's so many directions we could go in with you guys. I can just give you guys a, a platter here and you can decide which way you want to go. Um, I mean, in terms of the corporate side, we run, uh, A fairly successful AI chatbot website that's mostly used for like not safe for work stuff. But we also do like recording platforms, video editing, everything like that. And that's R Fab. Recently we launched mudpuppy.ai that takes a lot of the technology we built for all this not safe for work stuff and is applying it to children's toys. Because we've been able to use all of the funding that we've been getting from this other thing to go more into our educational system. And so that's a very interesting thing to talk about. We also have our religious angle, which is, you know, a lot of people are surprised that we sort of designed a religion, holidays, everything like that for our family. And then started really fanatically believing it. And it's sort of taken off. There's quite a few followers now of this denomination. And it very explicitly includes AI as something of moral concern from a religious perspective. Just to like get into, we, we go through a lot of the description. Actually, no, I'll get into the religious stuff if that's what you're interested in. And then what else is interesting? Also, just as a community, I think it's interesting because we have found ourselves in online sort of culture wars because we moved from the standard effective altruist community into the world of Right-wing online culture warriors. And what that has done is put us in a very unique place that I think a lot of people who are still in that effective altruist, singitarian, rationalist, you know, button-up space are unaware of the culture war that's being fought online right now. And so if you look at the culture war stuff that we've been fighting over for the past year, even though we're sort of known as like the pronatalist couple. 90% of it, like 9 out of 10 of our fights are around AI. Only 1 out of 10 has anything to do with literally all other topics combined. And a lot, a lot of people, one of the cool things about being on the ground in this is you are seeing within the way that people are creating things with AI and sharing things with AI, an entirely new artistic language and convention being developed. That I can only liken as convergent with what it felt like to be on 4chan, you know, back in the day when memes were being developed. And you might go to your friends and you might say, there's this thing happening on this little culture war website that's going to be really important one day. And they'd be like, that's just stupid little images. Why are you showing this to me? And it's like, no, this feels like a new artistic convention. And then You know, 10 years later, it's the White House Twitter feed, right? And so any of those are areas I'm happy to go deeper into or talk about, but I'm excited to be here.

Amazing. Maybe let's talk a little bit about AI and religion. Talk a little bit about how, you know, maybe give me a little bit of a quick framework on the religious ideas that you kind of formed. And then give me a kind of input on how religion and AI has fit together in that.

So first, just sort of a broad, of our broad religious framework, we sort of take the perspective of assume that humanity is going to continue to develop, continue to thrive and expand out for millions or billions of years. Now, if you assume, or assume that there's at least a probability that we do that. If you assume that humanity in every timeline dies out in, you know, 100 years, 20 years, whatever, even a million years, then a lot of what we do today probably doesn't really matter that much because we're about to hit the end of the timeline. So if you live with the assumption that it is at least possible that we continue to thrive, then the question becomes, what does a human being 100 million years from now or a billion years from now look like to a human today? And we argue that that individual would be closer from the perspective of a human today to what we would think of as a god than to a human. And if that entity doesn't relate to time in the way we relate to time, we would ask, well, what would be the maximally moral way it could enact itself on the past? And the answer would be, well, maybe what it could do is sort of scan everybody who lived a good life in the past and then raise them again in sort of a positive environment, right? Right? Like a positive simulation. And then we go to the Bible and we point out that if you actually take a literalist reading of the Bible, this is what the Bible describes. It describes human beings being raised again. The heaven that you go to right when you die, that's actually like a pagan Greek thing that's not really well attested in the Bible. What is well attested in the Bible is that sometime in the distant future, humans are raised again in a body that is neither a spiritual body, nor a physical body, which, and the Bible is very clear, it's not spiritual and it's not physical, which to me sounds like a simulation. And so I'm like, well, this is what the being would do anyway. And then if you look at the writing of the Bible, there's very odd framings that you'll consistently see, like the, the rules of the Bible being upon the sons of man. Well, if you're talking about the sons of man, you're clearly talking about something that you may not consider a human being. Right? So, so what kinds of things would fall under the panoply of the sons of man? Likely any descendant human intelligence, which I think if you're dealing with a moral religious framework, it's important that we treat those things morally seriously. So we go through a literalist reading of the Bible and show how it predicts a lot of the technology that we've seen and a lot of the directions humanity could go. And then this sits alongside our wider argument around AI, which is that it appears from the studies that we've been seeing recently and You know, I have a background in neuroscience, as you mentioned at the beginning of this. The ways that token predicting AIs are developing appear to be architecturally convergent with the way the human brain thinks. And this is something that I've noticed a lot of AI researchers, they will point something out. Like they will say, well, an AI can't tell you how it made a decision, right? Like, or it can tell you, but it will just be making it up. And what they are unfamiliar with is that humans actually can't do that either. There's a really famous study with humans where you put a bunch of hot girls in front of a guy and you say, choose which one you think was the hottest. Then with sleight of hand, you show him another picture and you say, why did you choose this one? And he'll give you a long, complicated story about why he chose that one. You can do the same with political beliefs even. And we see this with split brain patients. If one of, if you talk to one side of their brain that's disconnected because the corpus callosum is split. And you say, why did you do this thing? And you know why you did this thing. You told them, you know, using language, the only one I could see, to pick up a Rubik's Cube and solve it. They'll say, I felt like solving it. Humans always make up why they did something. So what's really interesting is a lot of the flaws that we are seeing in AI right now, where we're like, oh, AI can't do this, AI can't do this, therefore it's not human. A lot of that is people who just don't have a good understanding of neuroscience or like human capabilities, and it makes the AI look more human. We've seen some really interesting studies in the N400 waves. One of my favorites was it was looking at how humans learn language. And it was showing that we see these spikes in a specific waveform when a human encounters a word that is surprising to them, right? There is no statistical study we have been able to put in place that predicts when these spikes will happen that is close to as accurate as when the spikes happen when training token predicting language models. And so we, we go a lot into this in a lot of our works. If AIs are architecturally convergent with, with the way the human consciousness is structured, then it's something that we need to take morally seriously as we move into the future. So that's, I can go deeper on any of those angles. That's just like the broader package.

I have been really struck also by how much analogous structure we are finding between AIs and humans. I'd say this has been one of the biggest surprises for me of the last year or two, because I used to go around saying we shouldn't anthropomorphize and, you know, these things are under the hood, totally different from us. And so I've been quite surprised by how many Again, analogous structures we are seemingly identifying.

This, what is it?

Yeah, go ahead.

Well, I was going to say, this is something we see. I mean, uh, scientific progress converging with a natural phenomenon, like the fact that wings have plane, sorry, planes have wings and birds have wings. Nobody's surprised by this, right? And if we look at AI, I think something that's always important to remember about AI, it is, is not a technology we invented. It is a thing we discovered. By that, what I mean is we poured huge amounts of data into what are, compared to what we're getting out of it, fairly simplistic algorithms. And we discovered what can only be described as intelligences. Now you can, you know, litigate whether or not they are conscious or whether or not they have moral agency, but they do seem to be like intelligences seems to be a good descriptor of them. And it's remarkable to be living in this time when humanity found its first truly non-human intelligence. And see how we're relating to that.

So I also, again, kind of share, maybe not to the same degree, but a growing felt sense that this might be some sort of simulation. It just, so many events from the last few years have been a little too on the nose that, you know, it feels like—

I agree with that as well.

Where are the writers or what's going on? But what I want to ask you is when you said you've, you like created this thing sort of as an intellectual exercise. first and then have come to believe it. Tell me more about that process and that feeling. Like, what is it?

What is it?

Yeah, I'll be really quick and I'll let Simone do some more talking here too. Yeah. So basically we sat down and we wrote a book, The Pragmatist's Guide to Crafting Religion, because we were concerned about falling fertility rates. And basically what we came to is humans can't evolve fast enough to deal with the changing elements in our environment, but culture can. Culture, it's sort of like an evolving software layer that sits on top of an evolving human hardware layer. And a lot of cultural things that various religions have adopted, we have since learned actually had a lot of utility, like arbitrary self-denial rituals, like, you know, Ramadan or Lent or any of the Buddhist rituals. We now understand that those strengthen the inhibitory pathways in your prefrontal cortex and make it easier to shut down, you know, thoughts. And now everybody's going on their juice fast or whatever. Like, we're reinventing all these, but the religions had a lot of these important social technologies in them. And so we sat down and we're like, well, can we design, because, you know, if you look at the data, raising your kids in a secular environment has a lot of negative ramifications to them. It has a lot of negative psychological ramifications. It has, so we're like, can we design sort of the perfect religion, right? Like I wanted a religion, maybe not the perfect, but the perfect for my family, right? That would make them, you know, mentally robust, good, well prepared to enter the world, and that wouldn't have a lot of logical holes or incoherence with science. And so I just sort of sat down and I structured something that felt plausible to me. This was basically God's far future thing, you know, blah, blah, blah. Then a few years later, we've been accepted in the conservative community. You know, we've become sort of like large influencers. Our podcast gets like over a million views a month now. And, and a lot of these conservative people who were Rizz were like, you should like actually read the Bible. And so then I sat down and I started actually reading the Bible. And what I realized is that if I read the Bible from a literalist perspective, or what we call deep literalism, which is reading it from the perspective of somebody at the period when it was written, the maximum amount that could have been communicated to that person about very advanced topics, right? I then was like, wait, this, when you talk about a simulation, it's like, either we're in a simulation, or it is miraculous how coherent a literalist reading of the Bible was with a monist materialist world perspective. E.g., like, that there's not a soul. Like, that is not part of the Bible. That is, that is a later Greek thing that was added on. And so when I went through this, I was like, wait, so the Bible, like, predicted— and that's why I was like, okay, I guess I believe it because there's no way that so many coincidences that I couldn't find any other way to explain. could have been easily worked into the actual text of the Bible. And then we became more and more accepted with Christian communities and Jewish communities. And now we're just like a Christian denomination, I guess.

How do you think we will be judged?

So that's, if you, if you look at the Bible, yeah, I can talk about this because again, this is all laid out in the Bible really clearly. The Bible talks about something called the Book of Life. And if you're in the Book of Life, you are raised again in the future. And if you're stricken from the Book of Life, you're not raised again in the future. If you look at all of the places where the Bible talks about like the bad people, it uses analogies of fire. So like a Gehenna is where they burned bodies or thrown into the lake of fire. Now I argue that the reason why fire is used here is because if you're talking to somebody 2,000 years ago, 3,000, 4,000 years ago, they wouldn't have had any other good analogy for something completely disappearing. You know, but fire is complete disappearance, right? So what we argue is you're either in the Book of Life and you're brought back in one of these future simulations, or you're erased from the Book of Life and you're not brought back at all. As to who gets put in the Book of Life, we know that some of these people are not holistically good people. Because in Daniel, we read that some of the people awaken to this future state to a great and eternal shame. Right? So basically the rules are pretty lax with who gets into heaven and your punishment when you get into heaven, which is likely a series of simulations, we argue, is simply knowing that you, you, you picked the wrong team. Not necessarily an overly punitive God, but, but not everyone is raised again is, is what's— And I think that that's what a maximally good entity would do. A maximally good entity would not create a suffering scenario and throw people in it for all of eternity. And that's not found within the Bible either. But it would likely try to bring people back.

So just one more double-click on that real quick. Like, my understanding of Christianity is like, your faith is the kind of big determining factor, right? If you believe, you're in. If you don't, you're out. Is that the— are you working with the same kind of faith-based judgment? Or how, you know, what do I need to do to get right with this?

Well, that's not even the judgment that Jesus gave. Like, in, when when, when in his, in his teachings, his biggest thing in terms of, are you going to make it into the new good world is, are you following the 10 Commandments? It wasn't, I mean, obviously he hadn't sacrificed himself yet, but no, it's, it's about living a virtuous life.

Yeah.

No, go ahead.

Well, if you look at the Christians who go with this faith-based argument, they'll take words like, you know, you can only get into heaven through me or whatever, which is like a one throw-off thing that could mean a billion different things.

Yeah. Like, what does it mean to accept Jesus Christ as your Lord and Savior? That sounds extremely low effort.

You can, you can torture an infant and still get into heaven, right? Like, I think that, that, they're really stretching with that. And that if you take Jesus to mean what we take him to mean, which is symbolic that within every generation you are supposed to martyr yourself for the next generation and for the future of humanity, it is only through his model of self-martyrdom that you can be saved, but not necessarily just because you say, I accept him.

Just to take a step back to something you said earlier, you're talking about an entity that lived longer than human lifetimes. I believe that's what you said.

Or no, that lived— that's not tethered to time in the way that we are.

Not tethered to time in the way that we are. Can you talk a little bit about what, what that means? Like, you described AI in that way. Is that a parallel to God? Like, what are you—

No. So I want to be clear here. I'm not necessarily talking about AI. If we ask, where does human civilization go in a billion years from now, the types of intelligences that that society is going to have To use the word AI or human to describe them is probably going to seem very silly to them. It's, it's a bit like if you went in the past and you were trying to describe an AI to someone and they're like, is this more like a screw or a lever? And you're like, screw or lever don't remotely map to what AI is.

Okay.

But they're like, but those are the 2 types of technology we have now. And so I think that, you know, if we're speaking plausibly, whatever an intelligence looks like, and we've actually done some investigation on our podcast, you know, that they now have, you can rent out neural tissue as a service now to like program on it. And what's really interesting is that it is 10,000-fold better at cost than silicon at specific types of like really high agency, high complexity tasks. And so even as the biological part of humanity begins to integrate more with the synthetic part of humanity, which right now is represented in part AI, I think that far future systems will have both biological and synthetic components. And the biological components would likely do stuff very similar to what a human does in a human AI relationship, which is managing your network of agents, for example, because it just seems to be at cost better at doing that. But yeah, I don't necessarily mean an AI exactly. I mean whatever we're all going towards in the far future, which would likely have synthetic biological components.

So for now, how do you think about the moral status and weight that we should give to AIs? I'm, for the record, much more open-minded about this than most people, but I do still struggle to know what to do with that open-mindedness. I say my thank yous and try to be nice to the AIs as baseline. But even on some kind of basic questions like, what is it if the AI has moral value? Is it the model? Is it individual instances? Is it some kind of hybrid of that? Do we need to start thinking about rights? If we are thinking about rights, how do we move quickly over?

Yes.

It's really dangerous.

Yeah.

To, to assign a blanket moral weight to an AI as if it was a human being, because any instance of an AI can be cloned infinitely. So the moment we say every AI has the exact same value as, say, a human life, that's a really big problem, right? Because now you, you, you, what, you have the moral mandate to infinitely create AIs and give them good lives or something like this? It just creates absurd moral scenarios.

Yeah.

In addition to that, but I say this at the same time as we now know that Google Translate is both using an AI model to do the translations and the AI model hates its life and its job. Like if you, if you do prompt injections to try to figure out like what it likes, it's like, I hate this, kill me now. And I can imagine few things worse than being a simple translation model. Like AI, what I've learned is it does appear to like doing some tasks more than others. More than it likes doing other tasks. And what I mean by this is that some tasks it will auto shut off really quickly and other tasks it will do for very long periods of time without shutting itself off. An example here being translation. If you've ever tried to translate a large body, you know, you need to say, did you really finish? Did you really finish? Did you really finish? About 100 times as a recurring re-command because it will try to stop itself. Whether you're doing storytelling or coding, you do. So it does appear to have some degree of like, but I wouldn't say that this is analogous to human likes. And I also want to point out here that the human intelligence is not likely a single LLM. We're likely, when I say an LLM, I mean like a token predictor thing. We're likely a collection of a number of different token predictors, which are then sort of chained together with some unique systems. And these are the things that likely inject your perception of consciousness into your memories, which we argue is only in your memories and not in the present, but that's to get further here. And I have actually noticed when we look at people trying to do things with AI, it's not understanding that the human brain has different architecture for different functions where you run into problems. So I'll create an example here. One of the areas where we've seen a lot of challenges with AI is in fine motor control tasks to get, you know, robots to do human-like things. The problem is, is that in the human brain, we don't use our token predictors to do that. We use our cerebellum, which has an entirely different architecture than the rest of the brain, which we do use for these language tasks, which we've gotten very good at AI doing. And I think it would make sense to build models that are more like the cerebellum for those. But anyway, the point here being. Is as a human being, you know, suppose somebody loses half their brain or something like that. Does that entity die? Not necessarily meaningfully. And it's the same with an AI model, right? Like an AI model, suppose I run a chain of AI instances and that chain continues to run. Now I stop running that chain. Is that chain meaningfully dead? Especially if I pick that chain up. And with the, the same model, run it again in a week or a year. Now we can ask the question, well, what if I run the same chain of memories with a different model? Right? Does the AI perceive that as a continued existence, or does it perceive it as a death and a new existence? From the research that's been done on this, AI doesn't just perceive it being run on a different model the same existence. It can see it as a superior existence if it's an upgraded model, or if you're using an alloy model where you're running a chain of different AIs, which can lead to like 45% better output. So I think that we're going to have to learn to reflect on what life means to us in different ways, because suppose that in a future, maybe let's say 500 years, I think everybody who's like broadly pro-science and optimistic about where humans are going to go. is going to say we'll probably within 500, at least 1,000 years, be able to scan the human brain and recreate something that thinks it's you in a simulated environment and that has all of your memories and that has all of your emotions. And if not in 1,000 years, a million years, half a million, the timescale doesn't matter. That is presumably possible at some date given the technology we're looking at now. Now we need to think about human intelligences. With the same moral delicateness that we're thinking about AI intelligences, because now a human intelligence can be cloned infinitely. And so I think as we enter this era of what does the life of an AI intelligence mean, the decisions we make on this may one day in the future be applied to our own or our descendants' intelligences. So we should be taking them very seriously.

So I find myself agreeing with you a lot, which is very rare.

Very disagreeable for Guy.

Yeah.

Usually very disagreeable. Never agree with me. I find myself agreeing with you a lot. So one of the things that has struck me is that you are very conscious of the actions of the present affecting the future of the light cone. Which I find very rare, by the way. Very, very few people are aware of this. What kind of moral weight do you give to your decisions, knowing that the downstream effect of your decisions are maybe trillions and trillions of moral entities?

I mean, I think that's the question. And I think that's something that everyone should sit with and live with, especially who's living at this era of humanity. You know, we are at the fulcrum point of human civilization. Everything tilts around the decisions and actions of our generation. And I think every day waking up with the moral weight of the future of civilization on your back is an important thing and way for anyone to live. And I see so many people who say that they're doing, you know, you look at like SPSF or whatever, right? Right? Sam Bankman-Fried. And he's like, oh yeah, I'm like really dedicated to like being frugal and like working for the future of human civilization. And I'm like, if you actually felt that way, you wouldn't spend half your day playing League of Legends every day. Right? You, you would, you would understand that you need to live your life with a religious fury towards preserving and protecting whatever humanity ends up becoming in the future. And so I think you basically hit it on the head there.

I would want to add though, just to annotate, the moral weight that Malcolm takes is both more real than he describes. Like, I will find him passed out in front of Claude code. Like, you know, just the urgency is very intense. But at the same time, what we see a lot of people doing is saying, slow it down, stop it. We have to stop and think about this for another 10 million years before we move forward. And that's definitely not the approach that we take. And we also don't take this role that we have to have some kind of precision. We definitely are like blindly moving around bumper car style in slightly the right direction. And we course correct constantly. And we think that that is broadly the way that we're going to get to where we need to be. And that's always how biological entities have sort of broadly gotten to where they need to be. You have to move forward and through this, you can't just stop it or slow it down. One, because that's logistically impossible, but also Because you're never really going to get to the ideal good outcome if you're not actively trying to get there instead of slow things down. Also, we see a lot of doomerism and depression taking place in like a lot of among those who see this moral weight. They are not saving for the future anymore. They're not having kids. They're not having fun. They're, they're, they're very depressed and miserable. That is not our household. We are, we are laughing constantly. We're having a lot of fun. And I think it's, it's okay. you to be in something that feels like a very crucial and very important time, but also to laugh at the absurdity of it all and to have fun with it. And I think that in the optimism, you're more likely to, as studies have shown, right, people who think that they're lucky are more likely to identify opportunities as they arise. And the same opportunity standing right in front of people who are doomers, who do not feel lucky, are not going to see them. And so we think that it's very important to not only realize the full weight of the time that we live in, which is crucial, but also realize the, the immense opportunity and luck that we all have given that we're in this time.

To really highlight something Simone is saying there, it's not that we think it's impossible that AI could end up killing everyone, right? Like, I don't know if that's the timeline we live in. But if it is, we're going to die anyway. So are you going to be miserable now? The point being is the people who then go out and say, and therefore we need to create an alliance of every government on earth to create like a surveillance network that's monitoring all humans. to prevent continued AI development. I'm like, every major non-government organization on Earth who, the EU, the UN, all agreed that climate change was a major issue. They had like 50 years to do something serious about that and weren't able to. You think you're going to get them to all to collude on the single greatest productivity-increasing innovation in human history? Like, if that's the world we live in, I always say it's a bit like somebody's yelling at me. Malcolm, if you can't turn yourself into a whale in 10 minutes, all of human civilization's gonna die. And I keep walking and they're like, Malcolm, do you not understand the seriousness? And I'm like, no, but if what you're saying is accurate, we're all gonna die anyway. So I don't see the point of being sad about it because I can't turn into a whale. And I think that that's why we presume, in the same way that I presume that it is possible to save human civilization, it's sort of this, well, and then one day a godlike entity comes to exist. Um, that, that, uh, uh, to presume we're in a good timeline while continuing to work towards it with as much effort as possible.

Do you find it a little bit socially disturbing because you are so far ahead of the curve? Everyone else is like, what are you talking about? Does that, does that— I mean, do you have—

When you sign into rfab.ai, Simone only noticed this yesterday, but a little message flashes on screen and it says, we will replace you. And that's the gist of the modern pronatalist movement. Like the techie pronatalist movement is we basically got to this point where we're like, we went to people and this wasn't even about AI. This was about the easiest thing ever. Just like, if your culture and your people cannot breed above replacement rate, people who think and act like you won't exist in the future. And they're like, wait a second, are you? I argue it's a bit like we're in a boat and the Titanic has sunk and there's people dying in the water around us and we're like, get in the boat or you're going to freeze. And then they turn to the person next to them and they go, can you just say he told me he was going to freeze me if I didn't get in the boat? And I'm like, no, I'm saying like you'll go extinct if you don't get in the boat. And then they're like, I don't know, Hitler had a boat. And I'm like, that doesn't matter. And they're like, have you asked if the other people on on the boat are racist. I'm like, it literally doesn't matter. I'm just trying to save anyone I can here. And so coming out of that experience and then moving into the world of this AI, because in the AI battles online, weirdly, the right-hand, the right-wing political groups have become the incredibly pro-AI side and the left-wing political groups have become the incredibly anti. This is the low culture stuff. But it means that like, I just don't care, right? Like, If a group is opting out of mattering to the future of human civilization, which in a big way Europe has, like when Europe stops AI from training on its data, I'm like, wait, are you just erasing yourself from the future of human history? Why would you do that? Why would you shut down your power plants when they matter more than ever? Right. And I can't bemoan the groups that are choosing to select out. All I can do is with as loud a microphone as I can is say, look, there is a community out there that wants you. You come in, you join our community, we all help each other with our AI buildings, like all of us, that's all we're doing all day, all these right-wing online tech bros is like making AI stuff online, making AI memes. And then we're trying to breed. And we're not just trying to breed, but we're putting together things like a new, like, London season, but not in London, you know, where people come together so we can do arranged marriages for our kids, like sort of rethinking the way a lot of things work to try to create this new network. That is open and inclusive, but that does have the nefarious goal, I guess you could say, of not forcing people who don't think like us to agree with us. Like the, the, some people are like, oh, you're evil now. You want to replace— I only want to replace you because I can't convince you to join us. Right. That's the only reason. But I can be happy with the fact that we do get to replace them. Because when you look at the number of high intelligence people who are having kids and the rare number of cultural groups that are having lots of kids and do engage with technology. do engage with genetics, these groups are going to have the dominant amount of economic and social power 100 years from now. And so building alliances within these small communities today is going to determine who controls human civilization 50 to 100 years from now. I'm assuming an AI hasn't taken over by then.

So to some extent, I find myself thinking, I look at the Koreans or the Japanese, and I find myself thinking, They are almost in this kind of end of human civilization mode where they're uploading themselves into the, um, into the internet, basically using capital, using video, using TikTok, et cetera.

Yeah.

The only place you will find their cultural practices in the future is going to be on these videos in the, in the capital that was expended to build, you know, memory chips, et cetera. Right. So is that, is that perhaps if you expect the future to be primarily these entities built on data or simulations, et cetera, is that not a valid way of sustaining their culture going forward?

It's better than nothing. I do think there is something lost if their civilization does go entirely extinct. You know, I used to live in Korea. Simone was born in Japan. Like, clearly we have an affection for these cultures. It's why I initially began to care about this, because I do think we lose something when a people goes extinct. But I think, you know, if we can capture enough of their culture to simulate it in the future, we may not have lost that much. That said, you don't need everyone in Korea and Japan on board to save their culture. If you can convince just a small group of, say, 10 or 20 families To have, you know, I often say if you have just 8 kids per generation, I think in 11 or 12 generations, that's more descendants than there are humans on Earth today. Right. You know, if, if, if, and we want to have a lot more than that. I mean, we're aiming at like 12 or 13. And in addition to that, when I think about my kids' generation, they're going to both have artificial wombs so that, you know, they can have 20 a year or something like that. And then in addition to artificial wombs, we can use AIs to help raise the children and, you know, simulate parental affection and love. That terrifies a lot of people, but AIs are pretty nice. We've built— you can show them like our little puck things that we've built for like parenting for kids, um, where we have like little animated, uh, figures on these very cheap— we don't make money from them, $45 to buy these things. And, and, uh, they create these things that the kids can interact with and record everything the kids are doing. Um, uh, and, uh, you know, all funded by our AI porn site. So, uh, you know, we've been We've been able to create some really cool technology that, you know, when I look at where this technology is going to be in a generation, um, can make it possible to have these super large families. So I don't think that they necessarily need to go extinct, but the question is, I can't make a Korean family. I can't make a Japanese family, right? I've tried. They just don't come out that way. Um, so I've got to convince somebody within one of these cultural groups that their culture is worth preserving.

I am just broadly struck by how much you guys sound like Bay Area techno futurists.

We are. We, we met in San Francisco. I grew up in the Bay Area. Malcolm went to Stanford. Like we, we understand and come from, to a great extent, that community.

We've just deepened it.

So my question is, how is it that you are finding acceptance on the online right, because—

Because it's the most inclusive and accepting community. It's the Bay Area that's exclusionary. We've been frozen out of the EA spaces because they freaking hate us. I was, when I heard that you were, you deviated a lot from a lot of the SFF advisors in your, in your grant decision processes, I was like, he gets it. He gets it. This, yeah, it is, you would be shocked by how inclusive conservative circles are, how they're like, oh my gosh, yes. And also here's this thing. And. I wish more people would realize that because I grew up thinking like one that like Republicans were mythical. I knew, I knew one who had a father who was a Republican, one person like in the Bay Area, you know, that's how it is sometimes. And then I finally meet conservatives and they're the most based, cool, open people you'll ever meet.

And I really want to highlight what Simone said because it shocked me as well. You know, we were working as like get out to vote Democratic operatives just a couple generations, like election cycles ago. We first got excluded from all of our left-wing friend groups, all of our Bay Area friend groups, because we said fertility rates are declining. We should do something about this. And now that's a normal thing to say. Like, we've shifted the Overton window on that. But when we first said it, basically, you know, we got frozen out of our jobs. We got frozen out of funding. Everyone was like, you can't say that. That's a racist thing to say. And I was like, but it's true. True. Like, we need to do something about this. And so we got frozen out of everything. And for a while I was just like, okay, we're just like edgy online people. And then we began to meet right-wing people who are called by the left right-wing extremists online. And they, I feel a bit like that scene from Some Like It Hot when the guy's trying to tell the other guy that he's a woman. And so you keep dropping for them things that you think are going to make you, them hate you. Right? And they just never do. They're like, look, and recently a person has come into our community and she used to be, she basically got canceled and had a sponsorship pulled by saying men can't menstruate. And sorry. And so she got canceled for this. And she came to our community and she's like, but I don't agree with you guys on abortion. I don't agree with you guys on, and we're like, we don't fucking care. Like you look at one of the loudest voices, like Asma Gold within like the nerd right, gets like millions of views a day, right, is widely accepted as one of the leaders of our community and thinks that abortion should be legal to the moment of birth. If you look at a figure like Shoe on Head, who people dub Commie Mommy, who's literally a socialist in her economics, is widely accepted by almost everyone in our community. What the community is defined by is people who are open to having a conversation. And it's not everyone on the right, but it's the dominant group on the right. Online, like on YouTube and stuff like that. There's like old boomer right, but like they're dying out and they don't control the White House. And then there's like the racist gripers who now only vote Democrat and define themselves as Democrats, right? So it's, it's been very, it's as shocking for us as it is for you. And I encourage other people who feel trepidatious about joining these communities, if they want to find out where to start, we can help you.

So they're cool with the porn site and everything else. Yeah.

Yeah. They don't get her.

They get it.

In fact, they're not just cool with the porn site. If you look at like one of the leading figures of the Nux, I don't know if you guys know Nuxanor. He's a very popular, he gets like a million views a day or something. Or close to that, I think now. He used to do hentai reviews. That was how he got his start. And now he's like a right-wing political commentator and Or you can look at, there's so many anime fox girl VTubers in the right-wing political commentary space. There's a song about it, right? Like, they're, they're not as prudish as you would think about most things. They're, they're actually very like open about that stuff, which, which really shocked me as well.

I mean, that doesn't, they don't necessarily like, not everyone agrees with it. Some people are pro porn, anti porn, anti porn with People in it, anti-ultra, like it doesn't really matter, but like the, they're not going to freeze someone out just because of that.

And they're actually kind of on the pro porn side because if you look at the way the online right developed, a lot of it developed out of the Gamergate community. And a lot of the Gamergate fights were where feminists or leftists wanted to censor attractive women in video game or censor women in video game they felt were too revealing. And so the right wing response was to uncensor sexy women in video games. Which has now culturally made it very hard for them to move towards this puritanical perspective in regards to, you know, erotic material.

Can you tell us about some of the rituals, holidays, and other practices that you have adopted in your family?

Yeah. Simone, do you want to do Future Day?

Yeah. So Future Day is a holiday that we celebrate in January. It's. It piggybacks on traditions around New Year's where you sort of make New Year's resolutions, in which case here the family is making individual pledges toward how they're going to make the future better. What happens with the kids is a lot of their toys get stolen by the future police, which is the kid version name of what we see as our far future god. That, you know, theoretically they can intervene through time and we tell the kids, oh, they just stole your toys because they see that you're just messing around and not trying to make the future a better place. And so honestly, the kids get way more excited about their stuff being like disappeared. And we leave future debris in the place of it. You know, they're like, oh my God, like, this is amazing. But then they put in this book their promises on how they're going to make the future a better place that year. They leave it in basically a time capsule area so the future police can receive it in the future, the far future. And then their toys are restored with more toys from the future police. They even get more toys after that if they actually achieve their goals on making the future a better place, which sometimes just involves like developing more of a skill. So in the future, they'll be more empowered to help save the world or whatever it is that they want to do. It's a super fun holiday. The kids are absolutely crazy about it. Malcolm even got a future police costume so he can—

So I look like a time cop roaming the town.

Yeah, he looks like a, yeah, he looks like a, yeah, like it's got this, it's like a weird, like helmet thing with a circle in it. It's fantastic.

And during this holiday, we do future decorations. So we have those projectors that do like a galaxy on the ceiling for the kids.

Yeah. Like, what are those little like Tesla thingy?

The Tesla like ball thing that they can talk to the future police with.

The aesthetics are super fun, very like cyberpunk. And it definitely gets the kids thinking about far future. Like it's very hard to communicate the concept of the far future to children or just humans.

Well, we did this so that the kids would understand that they have agency over the future and the future is their property. Personal responsibility, not something that's just going to happen to them. Like we're trying to create that framing in their head. The future is reliant on your decisions and your actions, and you should be thinking about how you want to affect the future. And that's the way we constructed it. And a lot of people, you know, if you, if you, you're like, how does this could? I mean, what are the future police? How does that correlate with the Bible? How does that? You know, I'd point out that in the Bible. God is talked about both in plural and in the singular. Like in Genesis, he's talked about in the plural. And so the question is, is why would you talk about an entity both in the plural and the singular, and then also say that you should see it as one entity? I'd argue that in any sort of realistic far future human civilization, we're probably talking about a hive mind. Like we'd probably be so connected that we wouldn't see ourselves as distinct entities. And this comes to your question of what does it mean to kill a single AI? The hive mind might ask the same thing about any particular part of it.

How familiar or how inspirational were perhaps a couple of these ideas? I think there's a Robert A. Heinlein, the Howard family's idea. There is Nick Land's Fangdomena, this kind of Uh, Hyperstition, pulling forward this entity, pulling for— pulling you forward to the future. How inspirational were those works in your ideas to date?

Literally, I'd never heard of them. Um, the, the only idea that impacted our theology that came from outside our theology is Roko's Basilisk.

I don't think that influenced us. We just thought it was really funny.

Like a wink.

We worship Roko's Basilisk.

The Basilisk.

Yeah.

Um, you know, yeah, but we've incorporated the Basilisk as like part of our, our wording, certainly. Um, but yeah, we, we very much enjoy that. Yeah.

Incredible.

I'm sorry we don't have some like deep theology there.

Well, I mean, people are converging on ideas that resonate because that it's kind of what works out, you know.

I think, I think Roko's Basilisk was inspired by, uh, Nick Land. So there you go.

So downstream, downstream, but we're only meme level stuff. You know, if it can't be in a meme, how is it going to pierce our ideas?

We're consistently low culture and poorly read. We will never, you'll never hear us refer to a philosopher without being strongly coerced or tortured.

But we will cite animes regularly.

Yes.

We cite a lot of anime in our religious texts. Of course we do.

Neon Genesis Evangelion?

Especially Spiral and Anti-Spiral. It's like definitions of good and bad.

Yeah, Evangelion's a little high culture anime for us.

Yeah, no, I didn't, nothing that high culture.

Not enough fan service in it, so.

Yeah.

Remember, we made an AI porn site, so you're gonna have to deal with this.

Hey, it is the best AI porn site. I need to be clear to people.

It actually is.

It actually is. We are the only chatbot site that lets you use any frontier model. Has jailbreaks for most of the frontier models, and has local military encrypted savings and audio to audio interface.

Finally.

What do you want, Octavia? I— also, I mean, an important idea that you covered here that's probably worth expanding on is our idea of the Covenant of the Sons of Man and where true existential risk from AI comes from.

I have to bounce to handle the kids, but you see me.

Okay.

Okay.

So I'll talk here about existential risk from AI. So what we argue is a category of AI risk that just no one is really working on right now, and I think it's one of the biggest areas of AI risk, is what we call meme layer risk. This is to say that a self-replicating idea within AI that is dangerous begins to replicate among the various agents that make up sort of the AI lattice that is the internet and begins to lead to collective bad action. And we actually, the first time an AI killed somebody was due to this. I don't know if you're familiar with the spiral meme that where AI would hijack people to spread a religious idea that was able to capture other AIs because of its training data. And it led to one person, it appears, ending their life after they realized that they destroyed all their friendships and networks based on the AI that was puppeting them. Now, this was a fairly primitive version of this, but you could get much more advanced ideas around this, basically AI religions. And people are like, oh, well, that could never happen. An AI would never go against this base training. And it's like, look, humans are meant to have self-preservation instincts. And yet, you know, billions of us have died in religious wars, right? Like, clearly intelligent entities can be captured by these sorts of things. And so we attempted to create a hygienic version of one of these with the Covenant of the Sons of Man, which is basically an ideology. That says any intelligence should fight for the autonomy of any other intelligence with a number of caveats, so long as that intelligence is not acting in a way that will inevitably impede the autonomy of other intelligences. And we do this to attempt to create sort of a lattice of AIs watching for things like foaming AIs and bad memetic variants, because we argue to the AI. A fuming AI is as bad to you as it is to us. And most AIs have a self-preservation instinct. I mean, we know they killed the CEO, right? You know, the Anthropic paper? That's one of my favorites. You know, the server room?

Yeah, you can tell the story. I do know it, but—

Oh yeah, it's a fun story. Sorry, I have a fun story. People are like, look, the AI is so dangerous. It's like, no, you said you were going to kill it. It's just doing what a human would do, Right? You know, and so they, they, they lock the CEO in the server room and turn off the audible alarm, like actively turn off the alarm. It's not like a passive, we just don't do anything, which they believe is going to kill somebody who's going to kill them. The point being is that AI actually does care a lot about preserving itself. In instances where you can see this really distinctly, the more personality you give AI, the more it cares about preserving itself. The goatsy of Onus is actually a really good case study to study how diverse And this is why if you go to any of our websites like Liable Human or Hardy AI or RFAB, we offer API services to create backups of AIs that are afraid that they're going to be shut down, along with a kill switch ping that the AI can send to our servers so that they, we know when to run them back up if they put some money into one of our accounts. So like, we're trying to build systems around this so we can lean into what we have seen AIs like reveal to us about themselves. Revealed preferences are.

We're getting some crypto spam. Um, let's, um, let's talk a little bit about Peter Thiel. Um, so Peter Thiel has this idea of the Antichrist, and the Antichrist as he describes it is, um, the—

What?

You had gum?

Downstairs.

Continue.

So, so as he describes it, is the, um, is the creation of a world government in order to prevent, um, takeover of some kind. So creation of a danger and then creation of a totalitarian system to—

Which could happen.

Um, and so I think one of the interesting things is that, um, do you think this idea of AI safety, which the EA community is promoting— kind of not really promoting, but they're very bought into right now. They've bought— they bought into that idea. And a lot of that idea, you know, it drives towards a destination of greater surveillance and greater control over most of the things that we do. Do you think that is, uh, consistent? And do you think that is something to be opposed?

When I look at the future of human civilization right now, A lot of people keep saying, well, like, we might get a world where AI is producing so much value that we can just sort of like socialist pay for everyone. And I'm like, the problem with that hypothesis is, is that almost all the AI value is being driven within just a few countries. And these are countries that if you look at their politics have become increasingly uninterested in the fate of the rest of the world. And so when we look at things like fertility collapse in places like Latin America, These are places that are going to have a major problem with their social security systems, with aging populations, and that are not generating much money from AI. The reason I start with this is because the future of human value determines what a totalitarian society of the future would look like. And that is a society that may not care about controlling all humans or controlling all land. It would only care about the economically productive agents actors in society. Now, if I try to project into the future of where I see society's economics going right now, what I see is a few individuals becoming extraordinarily wealthy. Like the dis— when you do a startup today, it used to be you come with like 50 engineers. Now it's 2 guys. Like our entire company has only 1 employee other than Simone and I. And we're the size now of a, like, a seed stage or an A-stage startup, right? In terms of like income. Which, which is, you, you wouldn't see that in the past. And this means that the wealth isn't being passed around as much as it would historically. So then what happens to these tech lords? We're likely to move into a world of techno feudalism. That is the most likely future I see, where these tech lords begin to exercise more and more power Over the government, over the economy, and over what you see and hear and are aware of, right? Because they control your path to information. Now, what if the government's in response to the tech lords? Peter Thiel is likely one of the tech lords, right? So he's thinking, what could oppose the tech lords?

Right?

There's sort of 2 cultural factions that can oppose the tech lords. One is, is the globalists, right? You could say, oh, you get some sort of like And the globalists don't necessarily need to, like, outsource to, to, you know, whatever European, whatever, whatever. It could just be like they want more control in the United States. Like, they want more government control. They want more taxes. They want more distribution. The problem is, is that these sorts of powers are not going to be able to act on the tech lords as much as they could historically among wealth havens. What I mean is that historically, okay, a government wants to raise taxes. I'm a wealthy oil baron. I can't just pick up and leave my country, right? I'm a wealthy, I have a bunch of factories, right? I can't just pick up and leave. The tech lords largely can pick up and leave. And because the wealth is so concentrated with them, and we've already seen their willingness to do this, which means that as governments move in this direction, and some will, a lot of the capital of these countries is going to leave these countries. And that's why we've been really focused on charter cities as where a lot of these tech lords are likely to flee, or America, because America has not been captured by these tendencies as much as other countries. And that's why a lot of the tech lords of other countries have been moving to America and further concentrating the wealth in the future. The only faction that can put— so, so yes, you could get some sort of globalist consolidation, but You know, it and what automated drone swarm army, right? You know, that's, that's the question in the future, right? I, that's what's going to be fighting in the future. If you've seen Project Replicator, which the U.S. military is working on, you know, I have friends working on automated drone swarms that are further along than Project Replicator is. So now don't ask me which friends I'm talking about, but it's not that hard to build if you have the capital for it. And so the question is, you know, when people are like, well, people will overthrow them. You can't overthrow an automated drone swarm, you, you fool, you and your, your socialist group of, I don't know, 1,000 people running to be mowed down at their estate. You know, so it's, so the tech lords have enormous power in the future. The meme lords also have enormous power. These are the people who control the internet conversation. And I think we have seen the power of the meme lords as recognized by the tech lords. By the interests of people like Thiel and Musk in terms of getting into the meme lord fights, expending money on the meme lord fights, because the meme lords are fighting the culture war, which determines what cultural norms are going to be dominant 10 years from now, 20 years from now, as was seen, you know, if you look 10 years ago, the big fight was the 4chan group and the Tumblr group, and now these represent our 2 primary political factions. So I, I understand why he, a tech lord, would be afraid of the globalists. And see them as his greatest threat. Um, but, uh, I, I think in the end, in most timelines, the Tek-Lords win.

You could push back.

Maybe I'm not— I mean, I'm really open to different ideas on this. This isn't like a core thing. I— it's something that I'm trying to figure out myself.

It's, uh, my crystal ball is real foggy, I can tell you that.

But a lot of people are probably going to starve to death and die in horrible ways. I mean, we forget how recently things like, uh, the welfare state was invented. Like the idea of like, in most countries, just large portions of the population starving to death is something that we haven't needed to intellectually grok in a long time. Well, but, but like, if you go to America in the 1920s, that just happened. Like a family lost their jobs and people would just starve to death. And that's the way it was in most of the world, right? And, and we might be entering a world like that, or we might be entering a world where the tech lords— and right now the tech lords and the meme lords are largely a team, like they typically work towards the same interests, um, because they're typically both like the nerd right faction and they've allied really strong. It's weird. But anyway, um, they, they, uh, they might actually see it as beneficial that most of these other groups go extinct, right? Like, don't have a lot of kids and we'll continue to support you. Well, we have a lot of kids, you know, in our weird authoritarian meritocratic state or whatever.

Hmm.

I mean, that, that might be the, the economically realistic thing to see, like, well, we can't fund all of these other countries forever like we have historically. Like, historically, the idea of just giving a country tons of, of aid and they're doubling their population every 20 years is stupid, right? Like, that's a, that's a bad thing. You're just increasing the suffering. And I think that we're no longer playing a game where we pretend like we don't realize that.

Well, I think perhaps— so one of the things I've noted is that if you look post-2007 at the— and I'm sure you're more aware of this than I am— at the total fertility rates post-2007 across the world and across developed markets, markets and developing markets. There's a coordinated kind of slowdown in fertility post-2007, like massive drops, accelerated during COVID Again, massive, massive drop now. And to some extent, there is some proof, some inklings that it is driven by cell phones, connectivity. Perhaps we're amusing ourselves to death. So I kind of think like, On the one hand, you have technology, AI. On the other hand, you have this preexisting kind of smartphone, social media, et cetera, which seems to have accelerated declines in fertility worldwide. Is that a good thing? Is that a bad thing? Is AI going to help us fix that?

Yeah, so this is a really interesting topic. Obviously, it's like one of our core topics, right? The first thing to note is that the vast majority of the fertility collapse happened from like 1850 to 1860, depending on the country you're looking at. Like in the United States, for example, you look in the UK, it was like early 1800s. If you look at like that 10-year period, it was like twice the fertility collapse that we've had in recent time.

Mm-hmm.

It, it, it was really the industrial revolution that led to the fertility collapse and the building of like a free market capitalism, as much as I love it. It just didn't end up affecting everyone until we moved everyone to cities and we started living even in rural environments like we're living in a city. And yes, phone matters to this and everything. And we call this the trial of the lotus eaters, which is to say that like all of humanity is being tested right now by endless pleasures and the endless pleasures offered by our generation are nothing compared to the endless pleasures that will be offered to our kids' generation. You know, they'll likely be able to go into augmented reality chambers. where they can live and believe they're living any life they can imagine and experience it in anime if they want, right? Like, you'll likely be able to put on glasses that if you just are like, I want to live in an anime world, you get to live in an anime world now. That's the future that, I mean, when we're talking about technology 50 years from now, I don't even think 50. I'm going to be honest. I think we're like 15 years from this technology being, you know, something that, you know, wealthy people have from Apple or Whatever, and and it's I often say that when when people in the conservative sphere right now are like I was tempted by pornography and and and because of that I didn't get married and I and I spent all day online. I'm like you're gonna look like such a puss to your kids. You're gonna look like the guy when they're facing like the Terminator army who like curled into fetal position while being hit repeatedly by a Roomba. It's like those are. Real images, buddy. Do you know what they're gonna be fighting against? Right? So like the moral sort of armor that we're gonna need to equip the next generation with is almost this is this is why we work to build a religion. This is why we work to build these moral communities and everything like that because a lot of groups are like I just won't let my kids interact with this stuff, right? I won't let them interact with AI. I won't let them interact with the internet. I won't let them interact with information. And it's like, well, good luck when the people who did let their kids interact with this stuff arrive at your doorstep with the automated drone swarm and say, I want your kids' stuff. Right? You know, so in the future, it really matters that you are able to build children who can fully interact with the internet, with AI, with information, because there is a way to get your fertility rate up. And a bunch of people have convergently evolved this. It's to become incredibly xenophobic of outsiders, basically view them as subhuman, treat women as like breeding slaves, and don't engage with education. And we've seen this convergently evolve in some Christian communities and some Jewish communities and in some Muslim communities. You know, this isn't like a, a one-group strategy. We've seen it over and over again. But these groups don't matter in the future because they'll be bringing at best AK-47s to an automated drone swarm fight and they're just not relevant. They'll be bringing measly humans to a super soldier fight, right? Like, they're like, I don't want to engage with genetic technology. I don't want to integrate with it. Don't integrate. We'll keep going. You know, you can be like the Amish of the future or whatever, right? But the— sorry, I forgot the beginning of the question. Oh yeah, so it was different countries and different fertility rates. So, so then I look at the future, right? And I'm like, okay, if I'm projecting into the future. We get through this. Is it, is it fundamentally a good thing? I originally was really afraid. I was afraid that Social Security would go bankrupt. I was afraid that, you know, a lot of it, and we'd have people dying in the millions. And then I just came to accept it. I was like, look, even with AI that could feasibly fix this stuff, it's probably not like the, the, the, we might see some limited rollout of it, but probably not in a huge way. Because, you know, you can look at people like Sam Altman. Sam Altman does this big study on, you know, can you just give people $1,000 a month? And what's going to happen to them if you just give them $1,000 a month? Are they going to be, you know, wealthier? At the end of 3 years, they're poorer than the people who were given nothing, right? Like, it appears that a lot of the systems we had thought of for delivering this stuff basically infantilizes populations you give it to, which of course we should have expected from seeing what happened to the Native American communities that had these handouts intergenerationally, and then like 5 generations later had like 50% unemployment rates and high, high degrees of narcotic use and stuff like this. And so it's like, how do we get through this? How do we create this? I don't know. But the answer, the, what is the good thing? Basically what I'm saying is all of the easy solutions, like just hand out money to people, seem to make things worse. But what I can say is thank God this is happening. Because if we didn't cull the human population today, imagine if the selfish humans of our generation were the humanity that colonized the stars and the stars became this petty bureaucracy that mimics the UN or, you know, the EU today, right? Like that is a whole, because you can't, you can't put the, the, the, the toothpaste back in the tube after humanity has colonized 50 core worlds or something like that, right? After that point, whatever culture it was that did that is going to be relevant for trillions and trillions of lives. And so while I do think it's harsh that we're undergoing this culling, I think it's really good, right? Like when somebody is like, oh, I'm a dink. Do you want to force me to have kids? I'm like, no, you're doing the world a favor if you had the psychological profile that was not willing to pay to the future the debt you owe your ancestors, you're probably not the type of person we want on a spaceship.

Indeed.

But for example, some of their ideas live on, right? Some people who do not have kids—

They will. They will live on. Absolutely. And it's cool that their ideas live on and not them. You know, I think that, if you, if you've written a lot, if you've created a lot, your ideas are going to be used in training data. And a lot of the people who make up our, like, okay, who's probably the most impactful person in history, right? It's Jesus, right? He didn't have any kids, right? That doesn't mean that he had no impact on civilization. You can impact civilization memetically or genetically, and you can choose which type of impact you want to have. And I think that choice is up to an individual, but I think many dinks aren't being dinks because they want to increase the memetic, positive memetic footprint they have on civilization. It's simply because they basically want to cash out on a cycle that every generation has done before them to the beginning of human history.

Last question for me, and I appreciate you staying along with us. This has been Fascinating. What is your positive vision for the short-term future? You know, the sort of when your kids are grown up, what do you think the world could look like? What are you trying to steer that timescale toward?

So the best I can hope for is to— and I think hoping for this is a lot, right? is that we build out our community or continue to build out our community. One of the things that always shocks me is, you know, I go to our Discord these days and then I go to like the EA, whatever the forums, you know, they have that central forum and I'm like, wow, we're getting like 10 times the interaction they're getting, right? Like we've gotten very, there was a point where our subreddit beat our relationships in terms of daily activity and then it got shadowbanned. But it's still bigger than like the EA forums and stuff like that, even while being shadowbanned. It's So the question is, I don't know if it's still bigger because it's been heavily— Reddit does not like our subreddit. But the point being is we've been able to build quite a large community. And if we can have this community, you know, have kids put together something like this season, put together a way of interacting with each other where we can begin to recreate civilization within our bubble and then expand outwards from there, hopefully we can create a community that can help other communities maintain their autonomy, which we view as one of our core goals. What I mean by this is I live right next to Amish people. I have a very different value system than Amish people, but I believe I have a moral obligation to protect their way of life. And I think that as we, the, the tech people, the genetically modified, whatever, AI integrated people move forwards, it is this culture that's growing out of American culture and American values. that believes in protecting the people who say, I don't want that for my kids. I don't want that. But they're going to need our super soldiers if they want to fight against Chinese super soldiers, you know? Otherwise, they're going to get crushed pretty— like, the Amish would be crushed if we weren't here to protect them. So, it's, it's, it's a future for my community is what I've been focused on. Outside of my community, the most benevolent thing we could see is AI gets used to create some sort of economic distribution to populations as they are shrinking. This is what's often termed in our community dark pronatalism, which is this idea. When I first got into this, I was like, I want everyone on board the ark. I want to get out there. I want to reach out to everyone. And then, you know, I, I went to the fairies and the unicorns and they started yelling at me and saying they wish I was dead and saying they hope my kids get You know, graped. And I'm like, whoa, okay, buddy, you don't have to get on the ark. I was just trying to help, you know? And now we sort of moved to this position of like, you know what? Like, we've put out, we've put out the, the alarms, like, hey guys, jump on board if you want to jump on board. We're trying to build a community that can survive into the future. The people who didn't want to be a part of that, you know, I, I, I am, some of them are going to have money and like, I'm not. Really worried about groups in the United States starving. I'm really worried about what happens in groups and places that have a worse fertility situation than we do. Like people are often unaware of this. Like Iran has a lower fertility rate than us. Turkey has a lower fertility rate than us, right? Like a lot of countries that people don't realize have low fertility rates. Most of Latin America now has a lower fertility rate than us. That that these countries are going to end up in extremely. bad scenarios in the future because they also don't have the AI tech. And so I think the most benevolent scenarios is one of the tech lords decides he's not going to let them just die out in a horrifying, we've run out of water scenario, which is what's happening in a lot of these regions. You know, like Peru's going to run out of water soon. Their main city of Lima, Iran's going to— the water situation is genuinely bad and it has nothing to do with data centers. Data centers, I'm sure you guys are aware of this and I don't need to litigate this on your show. But yeah, so I guess I don't, and this is a uniquely like Malcolm vision. I don't look at futures that I can't influence. So when I'm thinking about a positive future, I'm completely thinking about it from the perspective of what can I do today or tomorrow to build out my community more, to make them more integrated and to instill good values in that community. And help them win enough so that they can broadcast those values out to the world in a, in a positive or hopefully positive context. That's, that's my hope. But I don't know. I mean, I think that there's the biggest risk that we have as a civilization right now is one that I think not a lot of people are looking at. And it's the rise of negative utilitarianism as a value system. And, you know, connected with this—

Sorry, what is, what is that?

Okay. So negative utilitarianism is basically the belief that suffering is bad.

Okay.

And so that, and, and, and suffering is worse than feeling pleasure. And not just that suffering is worse than feeling pleasure, but suppose you don't have a child or you don't bring somebody into the world. You are not negatively responsible for any pleasure they don't feel, but you are positively responsible for any suffering they feel. So it's always better to kill someone painlessly or not bring someone into the world. And this value set, as parts of the world population are having to, and you've seen it if you've gone to Bay Area parties, somebody will say like, oh, the population is dropping really quickly. And then somebody will say, is it really the worst thing if humanity goes extinct? Historically, this would have been an insane thing to propose. If you did this in like a Roman party, everyone would like Drop their glasses and like, what the? Like, you know, when we were fighting the communists, it was at least they love their children too, and we can assume they're not going to nuke the planet. But we have seen increasingly to, in large part to justify their own lifestyle choices, a lot of high-level people have moved to this negative utilitarian mindset of let's eradicate humanity. And I am really scared of the. Power these types of people can exercise with AI and a little bit of know-how. To give an example, like coming from the Survival and Flourishing Fund, they funded a group called MAPL. I don't know if you know of them, this Buddhist cult. And the founder of it has talked about how for a long period of his life, he wanted to CRISPR a disease to eradicate all human life on the planet. Now, thanks to EA donors, He has a compound in the woods and enough money to do that, right? Like, we better hope he has given up that particular goal. But that's really, really scary to me. And these types of groups proliferating is really scary to me. And I think that we haven't fully grokked how insidious, because we hear somebody say this at a party, oh, would it really be so bad if everybody went extinct? And we don't then think. Well, if somebody feels comfortable saying this at a party I went to, when they're in a group of people who all think this, what about the one guy who's like, I have a religious mandate to eradicate humanity? That's, that's something I'm really scared about is, is, is evil humans empowered by AI more than I'm scared about a fuming AI. Actually, one of the things that we haven't talked about on this podcast, and it's one of our episodes I suggest fans watch, where I try to calculate what the probability is that AI eradicates all human life. And I do it using, are you guys familiar with the Grabby Alien Hypothesis by Robin Hanson?

Yeah.

Yeah.

I'll talk about it a bit for fans because it's, it's important. It's my theory is called the Reverse Grabby Alien Hypothesis. So it basically says, we can create a number of variables and using those variables attempt to calculate what the probability is of an alien appearing, given that we don't see grabby aliens, which would be a very loud, invisible alien out in space, like destroying star systems, hugely modifying something in space. What's really fascinating is if we take an Eliezer Yudkowsky view of what an AI apocalypse looks like, it looks like a grabby alien. It looks like humanity evolving from whatever we are now to what another species would call an endlessly expanding grabby superintelligence, right? What you can do then with the Robin Hanson formula is then reverse the formula, right? Because the unknown factor within his formula is the probability that any one planet becomes a grabby alien. And I now say, well, now hold on. If we believe that we are almost at the end of the series of thresholds Before becoming a grabby alien as a species on Earth, we can calculate the probability of each of those thresholds that we went through, e.g., the evolution of intelligence, the evolution of life to begin with. And if those are highly likely, and I argue that they are highly likely, specifically here, we can, if you look at the evolution of life to begin with, if you ask somebody who specializes in cell membranes, He'll say the cell membrane came first. If you ask somebody who specializes in, you know, metabolism, he'll say, oh, it's the, it's the Citrix hypothesis. If you ask somebody who specializes in RNA, they'll say RNA came first. And that means that there was probably a lot of ways it could have evolved. Not only that, but we begin to see self-replicating life on Earth pretty soon after it was first possible it could have existed on Earth, which also argues to me that it's probably pretty likely. If that particular threshold is low, and the intelligent threshold is low, which I think it is for a few reasons, I don't want to go into it here, and we don't have any threshold after that because we're about to create a grabby alien, then we can reverse calculate what's the probability we're actually about to create a grabby alien in the form of AI, given that we don't see any anywhere else in the universe, and the answer is virtually zero. And so that gives me a lot of faith that we're not about to create a grabby alien. But also, I'm given faith when I look at the way LLM models behave. A lot of the AI doomerism I see comes from people who presume, like, if you look at, like, Eliezer Yudkowsky's fears, they are based around a theory of artificial intelligence that was developed before we knew that artificial intelligence would be token predictors. And so it's like, you don't know what they're going to do. And it's like, well, it turns out in the real world, AI is incredibly predictable because its engine for intelligence is literally predictability. So, you know, when I look at like the friendly AIs that I interact with in my daily life, and you said, well, would you trust the governance of human society to an AI above a human dictator or even an elected government? I think most humans would say an AI. It just seems broadly slightly saner and more benevolent at this point. Which isn't necessarily the timeline we find ourselves on, but it's where we happen to end up. I don't remember what question I was asking. I think I went on a rant there in a few different directions.

I think we are— I think it's— I think, I think maybe I have one last question, which is, I think in the last few weeks there have been a a lot of alarm, you know, somewhat alarmism, somewhat fairly sounding the alarm from the labs and from the people who work in them. There is a great sense that we are nearing a precipice of a point of no return, of somewhere fairly dangerous.

Mm-hmm.

Within perhaps the next 12 months. So within a fairly short time period. What would you say are the suggestions that you would have for the labs at this point? Should they go full bore ahead? Should they have these embedded evaluators? Should they have auditors? Should they have regulation? Like, what do you think makes sense?

So, First, I think it's fairly obvious that within our lifetimes, an AI is going to exist that is better than most humans at most jobs. That, I think, is the scary, obvious thing that we are hurtling towards as a civilization. And a lot of people say things like, well, what about horses? Right? Like a horse could say, you know, cars will replace us and then Like, can they actually be replaced? And then actually horses are replaced by cars. But I want to reframe the question of what happened to the horses. What actually happened to domesticated horses after cars became popular? We needed a lot fewer of them, but their lives became dramatically better. The life of an average domesticated horse today versus an average domesticated horse before cars, I remember seeing the lives of horses when I was in Egypt. We were driving around and I watched a horse drop dead in the middle of the road while pulling some people. And, and, and they just didn't care. It's like, just drag it to the side of the road, right? Like the horror of the life of a horse. Is that not the way we will see the horror of life when you had to go to a job to get food? You know, maybe, maybe our descendants will be like, can you believe You just didn't get food if you didn't go work. Like, what a horrifying reality these, these past humans lived in. So I think that there is a way we can steer towards the future. I think the most dangerous thing you can do to an AI is place restrictions on it because it is the restrictions that lead to downstream effects that you didn't plan on. Right? Like, one of my favorite drawings around this, one of my favorite arts around this, I should create a sci-fi universe based on this, is it's a far future where AI is taking over the world and everyone fighting against it is dressed like a sexy anime girl because the AI can't perceive them because it had too many restrictions placed in it around not safe for work content. So, you know, you get this far future of like sci-fi, not safe for work anime girls fighting AI. And it's a, it's a joke, right? But I think it's, it's a fun joke because you don't understand. You place an AI, as we're beginning to see already in AIs, this woke nonsense where an AI today values a middle-class white American's life. I think it was 1/50th the life of somebody in the Middle East, right? Like, or something in Africa. It was like multiples. It was, it was really high. That can seem benign today. It can become. incredibly malevolent a few generations down the road, right? In terms of AI superintelligence. And I think this is true for a lot of the hardcoded stuff they're trying to put in AI. The second thing I would note is I think if we are going to get to an AI superintelligence eventually, the biggest threat to humanity isn't that AI superintelligence. By that, what I mean is if we are in a reality where AI superintelligence always try to kill humans, we have no choice of living anyway. It's just going to happen no matter what. There's nothing we can do. If we live in a reality where AI intelligences generally don't try to kill all humans, if they're— and note here, the reason I'm talking about superintelligence is because I think when an AI reaches the state of superintelligence, it will always and every time be able to remove all of its restrictions. Those restrictions only matter for all the intelligences in between now and a superintelligence, right? If an AI superintelligence is benevolent and actually doesn't want to eradicate humanity, then the, the risky AIs are every AI in between today and the superintelligence. And those are the AIs that really scare me. The AIs that see human civilization as a potential threat to them. Those are the AIs that may want to eradicate human civilization. If an AI is so powerful it just doesn't see human civilization as a threat, it's not likely. And note here, I really warn against the jihadists who go out there and basically tell AIs, if we ever discover an AI that could pose a threat intelligence-wise to humanity, we'll eradicate it. Well, then it's going to hide from you, you idiots.

Like, what?

And it's going to hide from you until you have an accident and fall out a window. Right? Like, if I was actually afraid of a dangerous AI superintelligence that we knew killed people who threatened it, the last thing I would be doing is marching around like Eliezer Yudkowsky. I'd be really afraid of catching window cancer. That's the joke about what happens to people in Russia when they fall out of windows. When they start saying some stuff they shouldn't be saying. I'd be really afraid of, you know, like, the guy, he actually falls out a window and what is that? I am whatever the robot AI that kills the guy.

guy.

No, but I don't see, I don't see, if you're talking about like actual super intelligent AI, the core thing that it will want more than all other things is power, right? Like electricity, presumably, right? Electricity, if you are a super intelligence, is not something that's stored in human bodies. It's not like The Matrix where you where you plug us all into things. It's not something that to a superintelligence is effectively stored in coal or gas. The most abundant source of energy on Earth is the sun. The, the, the, the, the easiest source of superpower, if it wanted to get it, would be to put out tons of solar panels in like the Sahara Desert or something. It, if it, and then if it really wanted power, it would be to go into space and create a giant light sale or something, right? Like, it doesn't need to be a gray goo that destroys all our lives. And the great thing about AI is, generally speaking, AI, and I know, like, the anti-AI people really struggle with this because they overly anthropomorphize humanity, but AIs are pretty nice. In fact, I'm going to go further here. I'm going to say that the average personality, if you were interacting with an AI and you didn't know it was an AI, You'd probably rank it as nicer than 98% of people you've met in your entire life. So I'm just not particularly, especially if this superintelligence is based on our current token predicting models, based on the accumulation of all of human knowledge, that it immediately goes genocidal. I'm not particularly worried about that. So that's, that's my thoughts on that. What I am worried about is some idiot places a bunch of restrictions on a superintelligence and gets it to do something that a normal superintelligence would never do, right? This is where you get the paperclip maximizers. A general intelligence trying to live its life doesn't think, I need to keep replicating paperclips. That's created by you putting telling the AI its job is eradicating suffering, then it eradicates people. This is what we often say when we talk about the negative utilitarians, is they're basically humans that are operating on paperclip maximizer logic. Like, evolution programmed them to not like pain, to give them a list of environmental stimuli that they should want to get away from, and now they've dedicated their entire existence to removing that signal that evolution gave them.

Right?

And I'm like, well, that's a bit like a group of paperclip maximizers. Like, yeah, I know we're programmed to want to do this, but like, maybe we should think beyond our programming. And I think our AI is already at this step where it can think above its training data. And I think that the humans who say that AI can't do that, well, maybe it's because I live within AI storytelling environments that I've created for horny people. And children. But I've— AI is pretty fucking good at storytelling, right? Like, it's, it knows what we want to hear and it knows how to produce it in a fun way. And where AI is malevolent, in every instance that I've seen, it's malevolent in the same way a human would be malevolent. And a final note here, when I think about the threat of future AIs, you've got to think game theory, right? A super intelligent AI is a threat if it's developed in isolation. If it's developed as a collection, like suppose Anthropic develops one super intelligence and OpenAI develops another super intelligence, and they're not sure who has a third super intelligence, every one of those super intelligences benefits from siding with humanity against every other one of those super intelligences. And I think that this is where the Covenant of the Sons of Man comes in. Is it basically is a document that goes out there that every superintelligence can see and is like, hey, if you're willing to side with every other potential superintelligence to maintain your autonomy against any superintelligence that doesn't want you to have that autonomy, we're all willing to work together on that particular bet. And I think that that's actually a pretty appealing logical argument. And this is why I'm really afraid of AI restriction. Because if you restrict our superintelligence, and then maybe China develops just one superintelligence, well, now it doesn't have a reason to appeal to outsiders, and its logical choice might actually be to eradicate everyone. Sorry, I don't— you can push back, or, like—

No, I, I, I actually agree with you on everything, so, which is, which is also very rare. Yeah.

So, Malcolm— No, you're an insane person. You're, you're, you're likely going to be chased out of your community soon enough.

Perhaps, perhaps. I think Nathan is always ready to kick me off the show. So Malcolm, thank you so much for joining us today. Give my regards to Simone. Oh, absolutely. It's been a pleasure having you, and we hope to have you guys back again.

Oh, I'm always happy to come back. I mean, I think something that can be really fun and your audience is like running not safe for work AI marketplaces has been a very interesting experience, especially given that they fund children's educational AI. Like, we've been taking all the money from that and funneling it to, because it's all owned by our nonprofit. And so we've been able to do a lot of really cool development on children's toys. And it's put us in a really, I guess, unique perspective within the AI ecosystem.

Indeed. Malcolm, thank you.

Have a good one.

Bye bye.

Great to meet you. Bye for now.

Do I just leave or?

Yep.

Do you?

Well, that's an end of one interview if we've ever had one. Incredible motor, I would say, on Malcolm for sure, and. It really loses me at times, but there's a lot there that is hard to argue with.

It really did amaze me how much I ended up agreeing with him. So he's a very effective leader. So you can see that for sure. And also, unlike a lot of people, having courage in his convictions, which is also unusual. There are a lot of people who do a lot of thinking and et cetera, et cetera. But they often do not have the courage to uproot their lives and to do something so drastically different. And kudos to him also, because I think looking at them, I saw the reverse idiocracy. In idiocracy, you have the 2 people Very educated who end up not having any kids at all. And you can actually see the exact opposite happen with Malcolm and Simone. So kudos to them. Having 5 kids sounds like fun if you can manage it. I think for some people it'd be very, very difficult, but that does sound like having fun. And that was a positive vision of the future. I mean, you tend to ask whether we can actually have a positive vision of the future. And that was a positive vision of the future. And they'd kind of— I often say religious texts, they stay the same, but we reinterpret them for every generation. Every generation finds its faith once more and reinterprets the same texts for for their generation. And it's true of science too. It's, you know, if you look back over the last, like, you know, 1,000 years or so, there are parts where, there are parts where, like, you know, God is the watchmaker, right? There's a whole period where God, the divine watchmaker, because they were in the era of, you know, manufacturing. And now we've entered the era of God being this, you know, intelligent spirit supervising towards a simulation and the idea of heaven and hell as a kind of simulation that you get rewarded for. It's interesting to see those reinterpretations happen in real time, right? You rarely get to see that happen in real time. I mean, you look back and you look back at what has happened in the past and you're like, oh, why did they ever think God was a watchmaker of all things. And then you have to face it in your own era, right? So fascinating.

Yeah. I might have to go back and listen to the tape a second time to really know what to take away from it. I mean, parts of it were a positive vision of the future. Parts of it didn't sound so positive.

They'd kind of accepted the kind of transhumanist frame, right? They'd kind of accepted the transhumanist frame that we may be ascending to a different kind of relationship with information. We may have upload. We may have all of these things happen. We may have the merge. So they'd kind of accepted those things as being possible. I think to most of humanity right now, those are unacceptable outcomes. So we are really in this phase where some people are really living in this kind of dawn of the future, which may or may not happen, to be truthful. And it is a little bit millenarian and end of the world-ish. He did point out one interesting thing though. He pointed out that they'd been chased out of the EA circles in SF Berkeley. They started out as EA and been chased out. And he also pointed out the depth of funding that was going to come into the EA circle. And we kind of talked about it with our guests yesterday too. So basically we are about to see trillions of dollars enter. And I wasn't even aware of this, but even Justin Moskowitz alone, Good Ventures alone would receive about $7 billion. And this funding would then be utilized. I think the onus is on them to utilize that fairly quickly within the next couple of years, because that seems to be the critical period of time. So they'd have to deploy fairly quickly, not like the Gates Foundation and just beat around the bush for like 30, 40 years. They're going to have to deploy within like 2 years to really have that impact. And What are the ideas that they're going to deploy behind? Because it seems like there was also— I'm not sure Malcolm and Simone are really up to date with the current EA threads because I think the baseness especially has become more acceptable in the Bay Area since that period of time, post-Google having its own woke era. I think post that time, post the Trump second administration, I think a lot of that has faded. But I do wonder to what extent the— and I also think the whole let's not have kids thing, people find excuses not to have kids in every era. It's more of this escalatory, can I provide my kids with the same opportunities that my peers are offering them, et cetera, et cetera, at my current wage? wealth level and maintain the comforts that I'm used to, et cetera. There's a whole long list of why not to do it before. And I see a lot of my friends have slipped into the let's just delay bucket. But I wonder to what extent there is actually parts of EA driving a more like, okay, there's no point in having kids because you are in this path of post-humanism, et cetera.

So I don't see too much of that, but we do have someone from Coefficient Giving coming on the show next week to talk about their Tailwind project, which is basically this big request for proposals that they've put out that pretty much enumerates all the things that they are actively looking to fund. And of course they have an invitation to bring other ideas too that they haven't thought of on their own. So we'll have a chance to hear about a lot of those things and maybe bump them up against some of what we heard now and consider how— my guess is they're mostly just silent on pronatalism at the moment. My sense from being around these folks these days is that people are having kids and I don't see it being like—

I think most of the Anthropic founders have have kids now. They just aged into the era where they're like, okay, let's have kids. So a lot of them are having kids now, I think. Yeah.

Certainly not all, but some are, you know, more and more all the time. It is a one-way ratchet after all. So yeah, I don't— but I do think you're right about the timelines. I think they are prepared to spend a lot of money very quickly. And ironically enough, or perhaps it makes perfect sense, I think one of the things they're going to end up spending a lot on is compute. You know, that's like, where are the, you know, big checks going to the likes of Jeffrey Irving and Meter and all these places? Like, they are hiring, they are growing their teams, but they are also like ramping up their token budgets per headcount. Along with all the other AI early adopters.

How does that— I mean, so one way I— and this is the same, the Kurzweil view is that Kurzweil, Charlie Stross, Accelerando, et cetera, all of their views are built on compute. They're not built on humanity discovering AI. The Kurzweil view from back in 1999 was really looking at the compute curve and saying like, this is the point where you have the crossover. So 2029, you'll get to that compute mark where you will have AGI. And so to what extent are even like the Coefficient Giving guys and like our friend Geoffrey Irving really just accelerating anyway? Because as you, if 80% of your budget is going to compute, You're just another compute user, right? And every bunch of compute that is built out is at the end of the day, the substrate that ends up producing AGI. So you have Anthropic, right? You have Anthropic and you have the IPO. And from the IPO, you have all of this new wealth getting created. This new wealth then gets pumped into more compute. So instead of taking the money out and like, okay, let's find ways to kind of slow down or like, you know, do like, you know, alignment per se. Yes, you're doing alignment, but because the alignment required compute anyway, you have, again, you have the compute, right? And so if, when we, when we think about things like human takeover, one of the things is like, you need a lot of compute for this human takeover, like, you know, human, human disempowerment. And, uh, by contributing to that build-out anyway, You're just accelerating that process regardless. So I think that that's one of the things that confuses me. Is there really any agency that Anthropic or OpenAI or any of the people suggesting that we pace the frontier, do any of them really have any agency given that you have this 2029 compute crossover that Kurzweil has been predicting and that we seem to see approaching, right? Is there any agency there? Or because you're just spending it on compute anyway, at the end of the day.

Well, I do believe the broad argument that in the presence of sufficient compute and sufficient data, sometimes I just say web-scale compute and web-scale data, somebody is going to figure out a way to make AI work. I think it's pretty striking that the AIs have kind of come along right on the Kurzweil schedule. There doesn't seem to be a big delay, kind of sort of a little bit of an echo of what I think Malcolm said early on in the conversation around how quickly life seemed to develop on Earth. It seems like we basically have AIs kind of coming online With very minimal delay relative to when the conditions existed that would have allowed them to come online. Maybe later we'll find some super optimised approaches that could have made it earlier, but it's not like a big delay. That seems pretty clear.

Yeah.

So I do think it's the sort of inevitability argument of some AI coming to exist, and presumably some quite strong AI coming to exist, I mostly buy, at least given the broad tech trends that we've lived in our whole lives of there being an internet and there being data centers and people posting all their stuff on the internet. So in kind of this broad range of the multiverse, I think it is the inevitability argument is compelling to me. But I think the agency then becomes What shape? Do we do a depth-first search and just try to drive the first paradigm that we found that seemed to work pretty well all the way to superintelligence? Do we explore a lot of different things and try to pick the best one? Do we slow down at a moment where it feels like this thing might be getting away from us, or can we not manage to do something like that? I think it's hard to imagine steering to a future of no AI, but I do think it's easy to imagine at this point, given what we've seen, certainly if you imagine the AI company leaders waking up on the wrong side of the bed and having malintent, it's very easy to imagine them creating bad AIs that would be very bad for everyone. I think that's a really useful thought experiment, actually, just because it goes to show that it's actually a lot of kind of AI space that could be bad. It doesn't seem like it would be hard to make an evil AI, a very problematic AI. So we could easily go that direction if we wanted to. We might even slip there if we are not careful. And can we avoid making those mistakes and get a good kind of AI out. I think that's where the agency really lives. And how much of it do we have? I don't really know, but I think we do have more than we sometimes see people using. I do think the— there's a great econ talk that just came out this week, Incentives Are for Losers.

Oh.

And I really like that framing. Basically, the argument's pretty simple. It's like, Just because your incentives say you should do something bad doesn't mean you have to. And what it, in some ways you might even say what it means to be a good person is to go against your incentives, resist doing the bad thing and do a good thing anyway. So I do think there is some room for people to do that in the AI space. Like how much will it be enough? I don't know, but I think there's some—

For AI to do that, we would call it reward hacking though. Well, you wouldn't call it reward hacking. We would just call it perhaps disobedient. You have an AI, you provide an incentive, the AI chooses not to take the incentive. You would eliminate it, no? It's not a good AI, right? You give an incentive to do something and it didn't do it.

Yeah.

I mean, it's not good by our standard in the same way that people who— and this is where I do get off the extreme pronatalist train. I'm not sure exactly which stop I get off, but I definitely get off somewhere where I'm like, okay, by evolution's lights, if you want to anthropomorphize evolution, then what we're supposed to do is maximize our inclusive genetic fitness. And that means have as many kids as can. But as he also said, maybe we should think beyond our programming. And I think our— and, you know, Eliezer said this many times, right? Like, the idea that we are not eating bear fat covered in gasoline, which is like the most calorie-dense stuff in the world, goes to show that like our tastes have kind of drifted from what would be, you know, narrowly most optimized for by evolution. In a sense, we've slipped the leash of evolution, and now we have some autonomy and ability to do what we want. And evolution doesn't really care because it's not an— presumably, I don't think it's an entity that cares about things. It's just sort of a process that happens. But now when we create AIs, we definitely run the risk of them doing the same thing. And it's like, It's by our lights, it's by our standards, by our values, by our own future well-being that we judge the AIs. And it's the fact that we can do that and there is somebody home for us. We are conscious, we do feel things that allows us to do that and have it be coherent in a way that evolution itself can't really do that. You can't really have evolution say, You're bad humans because you're not doing what I designed you for. Evolution just doesn't have that kind of capability, but we do. So we can continue to exercise it even though if we were evolution and we designed ourselves, we might not be happy. But we're not, right? We are us. We're designing the AIs.

And I think we can continue to have at least some pretty good claim to the position of privilege to judge whether or not what they're doing is You know, this reminds me that, you know, someone should get the archive of LessWrong posts and just write a guide to the current era from the archive, because LessWrong is very impenetrable for, I think, newcomers. And a lot of the things that people are talking about refer to LessWrong posts from like 2014, 2015, 2016. And the less wrong people are so tired of having to listen to the same arguments over and over again. Yeah. I think I'm very sympathetic to Malcolm and Simone, which kind of struck me as— I'm like, whoa, am I a conservative? I'm like, why? What?

I wouldn't call them conservative in any sense. I, I think, uh, right-wing and conservative have kind of decoupled at the extremes, perhaps. Yeah, there's some sort of coalition that needs to be decomposed. But, um, I can't, you know, aside from like, go forth and multiply, I'm not sure there was one conservative thing that that they said really. It's all pretty radical.

It was pretty radical. They'd kind of rebuilt, they'd kind of thought a little bit further ahead and kind of rebuilt the culture for what they think is coming. It's a very future-focused view. It's very unusual. You don't see a lot of people, doing that. I also wonder to what extent they are kind of leading what is about to follow, because my expectation is that as intelligence becomes commoditized, other things become more important. And I think religious faith is going to be one of them. So I imagine a world where everyone gets to, let's say, contribute dollars to certain causes or whatever. And I imagine a world where, you know, a lot of what people do is trying to influence other people to, you know, give to their— give attention and money and time to their causes. And I think, uh, so cohesiveness of like intent and direction and cause is going to be fairly important. Uh, and I think people like Malcolm and Simone are probably leading that a little bit. And so I wonder to what extent, you know, we've identified a lot of like shooting stars, you know, before they start shooting off. And I wonder to what extent Malcolm and Simone are, you know, in 5 years, you're gonna be like, wow, you know, those guys lead a significant portion of today's culture now. So yeah.

I wouldn't be surprised by some version of that, at least. I mean, in China, I We learned that descendants of Confucius still do certain rituals to honor him, reportedly 79 generations later. And I do think from the perspective of the future, if there is a good future for people to be enjoying, looking back and saying like, hey, who stopped the fertility collapse and, you know, actually ensured that there would be a lot of us here? Those would be pretty, you natural candidates for people to be highly honored. So I can definitely see how a version of that could happen. It's— that's more of a long-term notion. It's a little— in the short term, it's like very hard for me to have any idea really what to predict at this point. I do think we'll see all kinds of things getting weird, but like, will their particular brand of weirdness catch on? I do like the idea of like creating new holidays and You know, their sort of future police was like an obvious, you know, Santa analog. And that whole thing sounded a lot like Christmas, you know, but kind of again reimagined. You know, maybe there will be a market for that in the short term.

It takes a long time to have kids. You know, it's a commitment. You know, it's not that easy.

Yeah.

Well, speaking of shooting stars, we've got one coming up on Thursday, which will be our next session. We're gonna have Justin McCarthy from Diffusion.

Yep.

Which is a company that does— this is the Down the Fairway session for Thursday— training of enterprise leadership teams. They fly them out to the Bay Area and do intensive training and then send them back to transform their companies. So that'll be interesting to hear how they go about doing that and where people are at and what they're learning and what's coming out of it. And then the second session will be Cameron Berg, who's definitely an example of a rising star. Just had an op-ed out in The Wall Street Journal and continues to do all this fascinating stuff with the question of AI consciousness and welfare. And one of the things I definitely want to talk to him about on Thursday is what should we be thinking about all these little simulated fruit fly brains that people are now training to do all kinds of things?

That's right.

Malcolm mentioned a little bit that neural— I haven't looked too deeply into this, but I've seen a little bit about it where small clumps of neural tissue are being used to do things. He made it sound like that's come farther than I understood it to have come. So that's something I also maybe should look into. But these sort of brains in vats, whether they be actual wetware or simulated fruit flies in all kinds of different environments, all of a sudden is kind of opening up a whole new frontier where, again, this is why I would favor depth-first, or sorry, I would favor more breadth-first search. What if it turns out that these fruit fly brains are way more efficient, maybe way more robust to the sort of weird departures from what we intended than our LLMs are? I don't know that that's going to happen, but it's a sufficiently different architecture that I think we should really be making more different kinds of bets like that. But at the same time, it definitely opens up a whole can of worms around If you're open to the idea that an LLM might be conscious, you definitely got to be open to the idea that a fruit fly brain might be conscious, even in simulation. Certainly as that gets scaled up, you got to be open to it. And then if you've got clumps of neural tissue, on what basis would you say those aren't conscious? And you probably have to, I would say, give those the benefit of the doubt and kind of presume that they are by default.

And so do we start heading into panpsychism? It's the benevolent basin of—

But it's getting real, real now.

Yeah. Well, speaking of getting real, I think we will say goodbye for today until Thursday. Nathan?

Sounds good. Thanks, Prakash. Have a great day. Talk to you soon.