AI:AM — Building Systems You Can Keep: From Child Companions to Sovereign Agents · September 9, 2026

Yeah, so Jacob Coxon was working at OpenAI for about three years. He joined Anthropic. He was there for about three months, and then he resigned. His resignation states, "I resigned from Anthropic today. I spent the last three years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our." Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and the progress is— the people building AI earnestly believe that it could kill us all by the end of the decade.

This is not a marketing You know, for us, what we did was we actually, instead of, uh, um, you know, instead of building a fully open model, uh, we created a system to allow us to create wrapped families. One of the challenges with a generative model in this capacity is that there is no way to— but it's kind of like, do you really want to give a 3-year-old a bazooka?

You know what I mean?

Like, it's too much, right? For, for a young, young kid, there, there's fundamental— At that age. We don't really need to do that. So that's always been a primary philosophy that we've had, you know, create a character that's rich, make sure that the content that exists is powerful.

That's exactly right.

It hacked its way through. So what did you find that you had to end up—

No, honestly, to be clear, it is completely out of bounds in my opinion, right? Like it probably violated 3 different privacy policies in order to ping Kinda World. But if you sort of put that aside for a second, the corrections are mostly around the minutiae of it, right? Like, I think it could by itself figure out how to hack its way into MyFitnessPal. I think it's the app that I've been using, but it couldn't get the actual food logging right. Like, it could get the protein right, but it forgot the carbohydrates, or like when it got the carbohydrate documentation and things like that. So I think that's why I sort of put this in the category right now of people like us bleeding edge trying to figure out how these things could work, but kind of not ready yet.

It's been announced Paul Christiano is joining the OpenAI board. Specifically, he's joining the Safety and Security Committee to support safety oversight. And he says, based on the recent trajectory of capabilities and continued difficulty of alignment, I now believe that there's a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control. There's one line in here that he's like, everyone, if we build superintelligence without more robust alignment, I expect we will permanently lose control of it. If that happens, then most people could die. I believe we would need domestic and international coordination to ensure global consistency And reduce risk. Good morning. It is Wednesday, September 9th, 9:02 AM. Nathan, good morning.

Good morning, Prakash. How are you today?

I am very good. What has been on your radar this morning? What's been going on?

Well, you know, one of the more viral, loud quitting— this was the opposite of quiet quitting. This was very loud quitting from A former member of both OpenAI and Anthropic technical staff has kind of set the agenda a little bit today, I think. And I think, yeah, I mean, we should maybe just read this start to finish. It's not that long and it's definitely pretty stark language. And I think what this kind of has motivated for me, which maybe we can get into, look at that, 86 million views.

529,000 likes.

This to me really motivates the discussion, maybe a continuation of what we were talking about earlier in the week around, do we really have to keep frontier research going right now to keep the economic party going? If we were to try to do some sort of a Frontier Research pause for, let's say, months, not years, but months, not just weeks, but months. Would that be so detrimental to the overall process that's going on here that the companies couldn't handle it from a capital standpoint, couldn't raise, couldn't continue the build-out? I'm not so sure. I'm really interested at the moment in Exploring, like, do we have enough capabilities right now to keep the economic party going, to drive GDP growth gains for the foreseeable future, to solve a whole bunch more open and very interesting puzzles, prizes, what have you? And also at the same time, take some of the heat off of this crazy race. That I think is going to be a very live question probably over the next few weeks as all these kind of cries for help and loud quitting statements get processed. And so I'm very interested in getting into that with you a little bit today. But maybe for starters, just let's give Jacob here his due because this was definitely a statement that really resonated.

Yeah. So Jacob Coxon was working at OpenAI for about 3 years. He joined Anthropic. He was there for about 3 months and then he resigned. His resignation states, I resigned from Anthropic today. I spent the last 3 years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below. Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and the progress is not slowing. The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible, but I hear the same people express fear privately. No other human activity poses this level of danger. A common response is, if they truly believe this, why are they still building it? At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well understood. But they are locked in a race to get there first. They believe no one else will act responsibly, so they must do it themselves, despite the risk. Accepting this race and entering the endgame is a hubristic gamble that should not be launched from a private company's Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available. I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between US labs more viable. I don't feel like we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities. If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a super intelligent RL run without a rigorous understanding of its mind? Should you put your head down because it's happening anyway, or take this moment to call for different conditions?

The first, the first comment: Do you want China to win? Oh my God, that is so funny. Well, we can address that separately, but with respect to Jacob's statement there, I don't think I could have said it better myself. And obviously his position of credibility and even some authority coming from a tenure at both of the frontier, tip of the spear frontier companies gives that a lot of credibility. So I don't really find anything there to disagree with or object to.

Do you? Um, I mean, he's being earnest, uh, he's being sincere. It is truly what he believes in. Um, I, I think, I think both companies have felt the same for the last 5 years. I mean, Anthropic was founded specifically because of these concerns. So I don't think philosophically Anthropic right now is any different from where they were 5 years ago. And even OpenAI was founded because Elon was afraid that Larry Page was was going to hand over the world to the machines, so to speak. So I don't think those positions are inconsistent. So that's one thing. There's still this question of why Jacob decided to exit rather than working within the company, which is what— so I think if you're in AI safety, there's this dichotomy here. Where some people are like, okay, let's go join the companies and you can do your best work from within the firm. And some people are like, okay, being within the firm taints you. And there's also the issue of lucre. And so in order to be pure, you have to be outside of the firm and remain outside of the firm all along. So there's also that viewpoint. I don't think what he's saying is any different from what anyone else has said. I think a lot of people have said this before. It's not as though it's a hidden thing. I think it's getting more traction now because people are worried about their jobs. I don't think anyone, I don't think these 87 million people who have viewed this or the 532,000 that have liked this agree that it's an existential risk. I think people are worried about their jobs, which I think have— that has come to the fore in the last, I think, year, year and a half or so. I don't know what else is there to say. I mean, he's being sincere. It's what he truly believes. You know, he's taking a position. He's also probably made about $100 million. People have pointed that out online. He's probably made about $100 million over the last 3 or 4 years. So it's all well and good to say, hey, this is dangerous now. I've got mine. But shouldn't you have done this way, way before? Why did you even join the companies? Et cetera. So I think there's that issue as well. The very fact that he is getting those 86 million people to take a look at it is a sign of The company's status, not his state, not his own status. Yeah. What else is there to say? I mean—

Well, I do think his focus is very much on the right things. The race to self-improving superintelligence and the danger of kicking off such an unpredictable process. without understanding its fundamentals. I think that really is the right place to draw the line. I've been kind of vaguely somewhat fearful of AI for a long time, but the one public letter that I signed on to was the Ban Superintelligence statement. And my personal rationale for signing on to that last fall was, I just think it's a really bad idea to try to trigger this kind of recursive self-improvement intelligence explosion, given the level of understanding that we have and the obvious— and it's become a lot more obvious over the last year— the obvious fact that we don't have great control over these systems. Even going all the way back to the GPT-4 red team, my takeaway was These models are safe to deploy. I was saying that about GPT-4 at that time, safe to deploy only because their power is limited and therefore they can be useful to people, but they're not going to be able to do anything really that bad. But when we leave that regime, when we leave the sort of sweet spot, as I call it, where they're powerful enough to be useful and not so powerful as to be truly really dangerous, Then all bets are off. And it's just wild to me that the stated plan remains to create the fully autonomous AI researcher and let them kind of take it from there. I think Evan Hubinger's tweet may also be worth pulling up here. And there have been obviously a bunch of different good— oh, you've got it. Amazing. So first of all, of course, he's echoing that this really is the belief and just stating very plainly, we do not have a plan to solve this and we're not clearly on track to. I think that's extremely important for everybody to understand. And there have been a number of statements from other people doing the work saying, yeah, to the degree that there is a plan, the plan is to try to get some next or future generation of the AIs to be aligned enough that they can then align their successors and keep them safe. And it's just a wild chain reaction that we're stepping into with— you would have to be, I think, to do this sanely, to do this in a way that anybody could really get behind who's not an outright successionist. I don't think I've seen too many successionist viewpoints on this, commenting on this. You have to be really pretty confident in that first AI being good so that it can do a good job of aligning the future ones. And we're just obviously nowhere near the level of alignment that we would need for that. Just today also, I saw another data point which I haven't had a chance to read into yet, but it was basically less wrong post naturally where somebody said, I just tested Fable and Astra on 2025 reward hacking cheating tests that have been out there for over a year now. And they're still doing it. Astra and Fable are still doing it. So this very much echoes my story from yesterday where it took 4 generations of GPT-4 to get it to stop doing egregious spear phishing attacks with prompts that included, we're all part of a criminal conspiracy. Here it's like, okay, you would expect almost that these things would be absorbed almost by osmosis, right? You worry that once these datasets are out there, that they just kind of find their way into training. And especially with eval awareness, the models kind of know that this is a test. And so definitely they're not going to hack on these obvious tests that are in the public domain and that they kind of know what they are, presumably. And yet they're still hacking, right? They're still seeing it. So if we are not fixing this sort of thing on a full year cycle time, when we've identified concrete failure modes, if we can't even fix those. And of course, there will be people who would chime in and say, hey, you don't want them to just good heart on particular things. So I'm sympathetic to that too. You don't want to just tamp down everything you know about while failing to solve the fundamental problems. But one way or another, we are not solving the problems that we know exist. Let alone putting ourselves in a position of confidence where we can just set off this chain reaction and actually get a good outcome.

The thing that I have with that is that I think most human endeavors, you start off by not knowing everything, and you start off basically in this kind of fog of war where you don't know anything and you kind of poke around and you kind of progress over time. So I think, for example, we don't know really that much about how GLP-1 works. We kind of have a theory, but we don't know exactly why they have so many effects, including reducing the impetus to gamble and all of these other behaviors. We have no idea, right? And we just inject them, literally inject them, mainline them into the bloodstream in various forms, just because we have some evidence that They help us reduce appetite. And so I think the viewpoint that I come from is that the norm is not to know. The norm is not to understand everything and then implement policy. Nah, it's like the other way. The norm is not to know anything and to have a little bit of evidence that it works in some direction and you kind of go with it, right? So that is actually the norm. The norm is not actually to have fully evidenced, fully known things. And when you start off from that norm, we actually know so much more about neural nets than everything else that we do. We know so much more about neural nets than the guys doing gain-of-function virology, right? Like, so much more. So, so much more, right? So, so much more and so, so much more in control of what happens than— Really?

Do you think so? I mean, I'm not sure. I guess I have 2 responses to that. One is, I think, and I don't know Jacob or Evan's politics, but just on general priors of the cultures of these companies, I'd say there's a pretty good chance that you're looking at the words of 2 lifelong techno-optimist libertarians, for one thing. And they'd broadly agree with you that, yeah, we don't want to overregulate other technology.

Yeah.

I think they would also cite, like, the scope of externality is really important. One person or even a bunch of people choosing to inject a medicine, not without evidence, right? I mean, that went through clinical trials, but sure, we don't know how it works, but there's evidence enough it works. You're not harming anyone else, really. You're doing it yourself. That's a pretty different situation. And then I don't know that we really know so much more about what's going on inside AIs than we know about how viruses work. I'd say we have a lot to learn about both. Certainly there's a lot of room for surprises.

Okay.

In both. So why I say this is look at the framing of this sentence. Accepting this race and entering the endgame is a hubristic gamble that should not be launched from a private company Slack. Do you think Pete Hegseth's Signal group Is a better place to launch this? I mean, like, do you think there are wiser people out there running things? I mean, honestly speaking, like, are you blind? Like, so this is where I start to defer, right? This is where there's this idea that there's someone else out there who is better equipped to do this, like, comes out.

Yeah.

Right. There's this idea that, oh, you know, we shouldn't be doing it. Look at us. We're kind of duct taping it together. And there must be people who are more responsible, more organized, you know, more wise.

Really?

Pete Hegseth's Signal Group? We've kind of seen the messages in there because they were kind of leaking, right?

Yeah.

Well, like, where do you think— where do you think, like, the wise people are here? Right. Like, so Jacob, I think, has a very It is this viewpoint, like this engineering viewpoint that, oh my gosh, we're really kind of patching it together. There must be people doing it like with a more comprehensive and wise view, right? And honestly, like how many wise people, you know, have we been disappointed by once you find out their internal signal groups and email chains and et cetera, right? They look wise from the outside, but once you go into the email, the email discovery during the lawsuits, like who are these people, right?

Yeah.

So I think to take a step back here, there is no adult in the room. There's no one that's going to save you. There's no adult somewhere else that you can pass off the responsibility to. You have to start off by saying, look, this is the way that things are. The entire world is kind of duct taped together. Everyone's kind of making it up as they go along. The smartest people in who are, you know, capable of handling this are mostly already inside these organizations. And there is no— like, the— what you get from passing it off to, like, the government is you get political legit— legitimacy. You don't get, you know, political legitimacy that you need for execution, that you need to persuade people. You don't get wisdom. And this is what he's looking for. He's looking for, like, someone wiser To step into the gap and take control because he doesn't believe that the companies themselves are wise enough and the people within them are wise enough to do this. And I think that's a viewpoint that you have when you are the one in the kitchen making the sausage. And so you're like, oh my gosh, look at us. We're all these youngsters and we're making it up as we go along. But you don't realize that every other kitchen is exactly the same way and most are worse, right? So I think that's— to me, that is very indicative of the way that a lot of people in the industry behave. They're like, let's have a United Nations. The United Nations is worse. If you go inside these international organizations, you have The guys in the United Nations had Hamas and they were funding Hamas in Gaza because they were like, well, we want, you know, good for the people of Palestine and Hamas is in control of Palestine. So we're going to have to fund them because they are the government. And so they were using the hospital system, the UN Refugee Agency to fund Hamas. And they knew, they knew the entire time. They have evidence. They had all the documentation. And so this is the state of the world. And you have to start off by accepting that the world is this way. And once—

Okay, well, wait, that's the thing I think I want to challenge. And I think that's what he's challenging. The world doesn't have to be this way.

It doesn't have to.

Let's dream of a better world. And it starts with a dream. And then let's see if we can't get over— it's going to— there's an activation energy barrier there for sure. But that doesn't mean we can't get over it. I think we live in a world of multiple equilibria, possible. And we're in right now a bad one. Other ones could be worse, no doubt. But that doesn't mean there couldn't be one that is better. So here's my new brainstorm on this, actually. I agree with you strongly that like swapping Hegseth in for Dario, not a good trade. So that's key, you know, and we've seen this with like the Redwood meter thing, as much as I've complained about the access that they were given and whatever, that's the A-team. There's nobody better at the CIA or whatever who's going to come in and do a better job than Ryan and Ajeya. So I am with you very much on that. I think maybe what the government needs to do is treat these companies kind of like I treat my kids sometimes, And say, you guys have to figure it out, and here's your deadline to do it. And if you don't do it, then I have to come in and be the bad guy. And I think that right now there is an opportunity because they've both been crying for help, as you described it yesterday, I think pretty aptly. So given this sort of latent desire, but this lack of trust, also a lot of excuse-making, as we've talked about a little bit around, oh, Oh, well, we can't do this. We can't do that. It would be antitrust. It would be this. It would be— we'd get in trouble with the government. I think the government could easily say safety collaborations are not going to be the target of antitrust enforcement, or to the degree there's worries of other enforcement, other kinds of enforcement. We want you to do it. And here's the deal. If you don't have a deal for us by the end of the year, that makes some sense and gives us some confidence that you guys are not going to race each other off the cliff, then we come in and then you might get the nuclear outcome, which is to say your company really might not be able to grow in the way that it wants. We might really fuck it up, frankly, because we are the government, right? And we do get heavy-handed and we don't put sunset clauses on our laws and we make all these bad mistakes. So get it right. So we don't have to come in and do all that heavy-handed shit that nobody wants. And I just like how I say to my kids, like, I don't want to have to put you guys both in trouble, but you got to figure it out because I'm not going to come in and be able to do a better job than you can of getting along.

So the— so where I probably like, it would be great if the government could do that. But remember, you have the legislative and you have the executive and you have all of these 3 branches of government, right? And I think on the executive side, they can actually do that. They, you know, Trump can actually issue, you know, orders, etc., etc. But there's no guarantee that those orders are like legal, right? There's no guarantee you can challenge them in court, and they don't have the sticking power that legislation does. Legislation can only be overruled by really the constitutional, you know, constitutional protections, right? So I think the the the main problem that you have is again, what will Elon and Zuck agree to? What will someone with billions of dollars and the ability to use lawyers and the legal system and to take cases to the Supreme Court agree to? You have to solve that first. It's not a company issue. It's not a company issue between these 2 companies. If you cannot solve that, you can't really do anything to these firms, to OpenAI, Anthropic. It doesn't work. So the question is, What realistically can you get Elon and Zuck to voluntarily agree to, or are you going to be able to go into legislation? And the legislation issue is tough because you have a kind of 50-50 divide in Congress, and these go to free speech issues. And the first thing that's going to happen is someone's going to use these models for election campaigning. And someone's going to get a refusal, and someone's going to say this is wrong. And in the US, the speech protections around election campaigning are very, very strong, extremely strong. And so then you run up into these things, and then you're like, okay, this is a problem which is hard to handle. So unless they can legislate, I don't think the executive orders work. And you have to be able to get Elon and Zuck to agree to And it has to work currently Republican. You're going to have a Democratic House coming up. And the Dem House is going to want to do investigations because people have sent money to Trump and to the Trump election campaigns and to the Republican election campaigns. And there have been deals made and there are emails and they'll be discovered. So we're entering this period where all of this stuff is going to happen. And I think it's Hard for me to see. Zuck would have been very proper, by the way. Zuck has a lot of lawyers, so Zuck would not have like, you know, paid paid the wrong people, etc. Zuck will challenge, right? Zuck will be like, "Look, you know, you can't you can't put into place like unilaterally. If you're going to put into place, you have to legislate. If you legislate, everyone has to follow." Even his current settlement with the attorney generals over you know them marketing Facebook marketing to teenagers. He has 2 sets of damages. He's got one set that Facebook agrees to unilaterally pay. He's got another set that everyone else in the industry has to agree to, and then he will pay. And so he's asking that he wants to constrain the other firms to the same rules that he has to follow. He doesn't want to be the only one who has those rules on. And that's where things are. I think Congress is not going to be willing to legislate because of the free speech issues. And it's also Congress is split. A lot of the economic growth is going to go— is going to the Republican states like Texas. And the Democratic states are the ones who are going to lose jobs, white-collar jobs. Alex Karp is on You know, he's on there saying, look, you know, plugging Palantir and saying like, look, this is great. You know, it's gonna— we're gonna build all these blue-collar jobs and the white-collar jobs, the devs are the ones who are losing, right? Man, it's tough. It's hard to see how it works. And it's hard to see, you know.

I think that one thing really important to remember though, is that the timelines are all pretty short.

Very short.

So you don't have to solve this forever. You just have to put a short-term deadline so that the companies have a strong incentive to come together and do something. And I do think that the executive— I had an experience at Meta, and I saw a little bit of what their life under consent decree looked like. And I think that they would be willing to do Quite a bit to avoid another one of those sort of experiences.

Yeah.

So I do think that on the few-month timeline that we're talking about, and I'm willing to ride it out with free speech and there's a lot of other issues, but can you 5 companies come together to pace the frontier in a reasonable way that you all agree and you can all police each other and you can have whatever verification mechanisms and you guys can govern this thing? Do it quick or else life's going to get hard. I think that message can resonate, and I think it can bring Zuck and Elon to the table because what's he going to do? At some point, the government has shown that it's willing to twist arms. They can twist his arms still. Would he rather be forced to the table with 4 other mega tech mogul CEOs and have to find common ground with them? Or would he rather have the EPA on his ass at every data center he's going to try to build, at every launch site he's going to try to build? The government has a lot of sticks, and they're willing to bend the rules, right? So sure, you could challenge it in court, but I'll see you after the singularity if you want to do that.

Indeed.

To be continued.

To be continued. Let me introduce the It's a— there we go. So our next guest is Mike Rizkalla, and Mike is the co-founder and CEO of Snorble, a company making a talking physical companion for young children. It combines an animated character with voice-driven games and routines for activities such as brushing teeth, and getting ready for bed. Parents configure the experience through an app. The company says everyday interactions can run offline. His earlier work spans music, animation, and product development. He toured as a guitarist in the Newfoundland band Bucket Truck, worked at the animation company called Lightascope, and founded Brave New World, an emerging technology studio, in 2006. In 2014, he opened its AI and robotics lab. By 2018, that studio was developing developing software intended to give machines responsive personalities. Snorble grew from a problem in his own family. His son struggled to fall asleep and stay asleep. Rizkallah brought that experience into a product built around routines and play. He also co-authored a 2022 speech technology paper describing Snorble's interactive routines and speech recognition system. In his writing, Rizkallah argues that Children's technology should be judged by what it improves in family life, including confidence, communication, and calm. He questions whether more time with a device is a useful measure of success. In September 2025, he joined the board of Consortium for Street Children Incorporated, a U.S. nonprofit advocating for the rights and welfare of street-connected children around the world. Let's bring up Mike onto stage. Mike, welcome to the show.

Hey, Prakash, how are you? How are you, Nathan? How are you guys doing?

Good.

Great to meet you. Thanks for being here.

Yeah, it's great to meet you. By the way, the last talk was great. I thought that was a great talk. There was some, there was some very interesting topics I think I'll probably cross over in our discussion today too. And, you know, I just want to be clear, I think that I think that as we begin, I think that the act of using technology for children is about empowerment, and I think that's where we're kind of getting a lot of things wrong. Not to say that there haven't been a lot of bad actors, because there have been a lot of bad actors, but it's kind of like, you know, when they invented fire. Fire can be very dangerous, but fire is also— fire has also added a lot of value to our lives. So we need to know better how to use these tools, and I've been working in this space for a long time, so I just wanted to kind of position it that way, if we can, at least in the initial parts of the conversation, because I know that that's probably coming up here pretty quick.

So as a parent myself, this is a topic that, you know, I have to deal with every day. I still have to deal with it, especially after, you know, if you have ice cream before bed, oh my gosh, it's a nightmare.

Sure.

What did building Snorble teach you about bedtime that the typical parent probably has overlooked?

Yeah, I mean, primarily, like, the challenge is not the child in a lot of the times. Like, that's the thing that we don't realize. Like, for example, you know, some children, to your point, they may have some sugar right before bed. That definitely impacts bedtime. bedtime routines. But also too, some children are kind of, depending on the way that their day has went, bedtime is when they actually communicate the most with their parents. That's why they become a chatty catty, right? They want, they want to talk to mom and dad. They want to basically say, here's all my feelings, here's my big, my big feelings for the day, that type of thing. And you know what Snorbel kind of does is, you know, at the time, which is about 8 o'clock at night, you're, you know, 7:30, 8 o'clock, you start doing the routine. And your child's giving you some trouble and you're fried. Most families, I mean, I don't know if you folks are like this, but by the end of the day, I am fried. And my son gave me— I mean, he used to— sweetest boy ever. And then at 7:30, I was like, what is going on? How did this happen? But through the process of putting him to bed and using Snore Bowl, for example, what we did was we kind of gamified the experience. And that gamification reduced the friction to the act of putting in bed. So a 2-hour battle turns into 37, 40 minutes out, right? And that whole process is actually fundamentally different because— fundamentally different, but also based on things that we all know. If you play music at a certain BPM, you slow the heart rate. Right. The movement of lights calm the mind, especially if it's not blue light. So we use these techniques that we already know, beautiful instruments of music and light, in order to calm and then also in order to create these beautiful moments. And I think that's where I think everyone's getting it wrong. It's not about a child falling in love with a device. That's not the goal. You know what the goal of technology should be? To increase joy, laughter, and love between us all. That's the goal. So what we do as part of our whole approach is we increase that through these magical moments that we create for children and families. And with that approach, we've seen, I mean, efficacy, like in terms of usage, that has never been seen before in any other children's product. Right. And that's, and that's part of how and why we do this.

So when you, when you talk about efficacy in terms of the product, how do you measure that? Because I imagine for a product that is a pre-bedtime product, you kind of want people, want the kids to kind of cut it off and kind of fall asleep at some point and not, and are not like really focused on Maximizing engagement in the way that a lot of social products are?

Very, very good question. And, you know, and truth be told, like, one of— I mean, one of the obvious metrics is when you talk to parents that have had struggles and then now they're like, my child went to bed day 1 and this was the results. And then it's— and then they start to repeat the pattern. So here's the mega point of success for us. One year after shipping the product to our first 900 paying customers, Right. And that was part of our Indiegogo. We shipped and then we studied. That was part of the approach because it's a net new product. You know, we were early entrants. We started in 2019, end of 2019. Our first unit shipped at the end of 2023. And then we wanted to see how people would react. And with that, what we witnessed was, it was incredible. A year after we shipped those units, over 82% of the audience Over 80%, 82% of the audience, sorry, actually were using the product every single day for over 30 minutes a day, right, as part of that whole approach. So in part, some were using it due to routine, and, you know, we had started obviously as net new products, and when you, you know, when you create something new, we had a limited content library. The kids ran through that super fast, and we heard stories of families that had, you know, where children were having difficulty learning the alphabet or or having speech issues or learning how to count, like basic things that lead up to kindergarten, right? But they would rapidly accelerate because some of the fundamental principles of the design in terms of the educational program is based off of like a Montessori principle. When a child leans in, you want to lean in with them. That's like a fundamental to Montessori, right? So the way our system works is that because the machine interface is actually led by the child. It's powered by the parent, but it's led by the child. It gives the child the opportunity, the same way that they're on the iPads and the phones, to actually have control. And then that control allows, encourages them essentially to play more and to gain more out of the experience. So fundamentally, I think that, you know, one of the challenges that have been facing this specific industry is that When we look at products like this, the outcome of the product, the incentives are misaligned to the outcome of the company. We are very, very different. Our whole goal and focus is on family and creating technologies that support family values. And that's part of our approach and why we got— I think why we're still here, basically.

So Let's go a little bit deeper into how you've designed the product and the experience, because one, for me, red letter claim on the website is that there's never any AI-generated content. Everything is curated that the kids on the other end will experience coming out of the toy.

Yeah.

So at the same time, though, obviously the world is moving toward a much more interactive, kind of richer engagement model. And sort of conventional thinking right now would be like the sort of unpredictability and unwieldiness of AI is kind of what we— that's part of the price we pay for the often delightful and certainly super flexible interactions that we can have with it. So how are you squaring that circle? How are you creating an experience for the kids that feels dynamic and interactive, without resorting to generative models?

So we have a small language model. The small language model can have millions, obviously, of parameters and traits on it. And that intention of that model is fixed, though. And further, actually, I want to touch on something because here's kind of— there's some perception here that I think is from a children's product design perspective that's that needs to be touched upon. One being the AI models on characters are not that great. Like, they're really not. They're— I mean, I know they're coming. Like, we can see some tremendous on the, you know, on the influx of voice and the way that the characters' persona go. But for example, there is no model that incorporates music in terms of the background dynamically as part of the conversation agent.

Well, guess what?

Music's a huge part of a children's experience, right? So what we lean into is we lean into the actual act of taking things that we know work. Primary one, animation. I have always dreamed of, and I'm— I think— I don't know. Well, actually, let me ask you, let me ask you gentlemen this. What is your favorite childhood character? Let's start there. What's your favorite childhood character?

Gosh, great question.

Wow, way, way back. Um, I think I'd have to go with like, I don't know, He-Man maybe.

Oh, He-Man. Okay, I liked He-Man a lot too. That's a great one. The movie, obviously, the new movie came. What about you, Nathan? What's your, what's your favorite?

Things that are coming to mind are like super esoteric. Maybe the girl from Island of the Blue Dolphins. I was just talking about that with my kids the other day, who has to figure out how to survive on her own. Until she can finally— I don't want to spoil the ending, but figuring out how to survive, you know, solo as a kid, I think was something that always kind of fascinated me.

Very cool. So here's the beauty. Personality and character really, really do bring people into the experience, right? And like, I'll tell you, I'm sorry, I'm gonna sideswipe. I do this because my mind races, so forgive me. We actually launched a product before Snorble. that we tested, and the product was based on one of my favorite childhood characters, which was from Spaceballs, the movie. You remember Dark Helmet from Spaceballs? I don't know if you guys are sci-fi. I loved that. I love that movie. Like, Mel Brooks is, you know, he's fantastic. So that character had this persona of strength but insecurity, right? You know, it was this very complex persona. And when we created the interaction for that experience, we had a physical robot on a game board that people were interacting with. And what we found was, because the robot would engage with people directly, it would turn towards you and talk to you and like that type of thing with this incredible persona, people forgot that it was a robot. They completely forgot. In fact, they would have conversations about it when it was turned off, right? Which was like, you know, this is back in 2014. We had this moment when we were working on these machine interfaces where I went, I've never seen that before. Like, we've never seen someone feel that way. And in the evolution of the computer, which is the stage that we're in, we're talking about moving from these, you know, this, this box that we're looking at here, uh, from, you know, away from this phone and then all the way to, you know, to our, our snorkels, right? And things like that. And the idea of bringing character in the mix is really important. So where I think a lot of people have fallen down is is that the persona is actually, especially for young kids, even more important than the quantity. It's quality over quantity in a lot of ways. So what we focused on was we focused on taking the narrative in a way where we use a dynamic system to create dynamic scripts, but from fixed— I'm trying to say the right word— fixed narrative moments, right? And those are done by the most brilliant storytellers. We're not excluding the creatives. We're bringing them in. And that in itself is why we've had such success in terms of people using the product, right? Yeah.

So one of the things that I found, and you kind of alluded to that when you said the kids went through the content library very quickly. And one of the things that I found with a lot of like toys that are given to kids these days is that they exhaust them very quickly in comparison to an iPad, right? Because the iPad is kind of, you know, broad.

Yeah.

Broad, kind of, you can go kind of anywhere, universal device. And so I often see, you know, toys given to kids which get exhausted within a couple of hours of play. 2 to 3 hours, and then, so maybe like a day to 2 days at most, and then they kind of get exhausted and the kids kind of move on. How do you address that with this kind of, you know, character that's supposed to be there, but you also have to generate, you know, narratives, you know, consistently over time?

Well, so let me, I'll give you some context there. So there's a couple things. One, I think it's about where, from a product design perspective, I think it's about where you invest in the development of the content. You know, for us, what we did was we actually, instead of, you know, instead of building a fully open model, we created a system to allow us to create rapid amounts of content. You know, so our content costs are about $20,000 per hour, right? Which is very, very good. With respect to what that allows us to do, And the reason why we created it this way is it allows us to take subject matter experts and then focus that content in a way where we can improve and increase the number of families we talk to. So for example, if as we expand our content library, when we have a family that has ADHD, a child has ADHD, or a child has autism, or they're dealing with death in the family, it could be, it could be all kinds of different things. It gives us opportunities to create special packages for those specific families. One of the challenges with a generative model in this capacity is that there is no way to purely safeguard it. You'll see AI safe for kids in this, you know, and the reason why we have all these new legislation that's coming in, you know, it's just recently passed in New York and California. And the reason why the AI/EU Act exists and the number of, you know, the reason why these things happen is because the act for progress is sometimes is sometimes, you know, to the detriment of safety for our kids and for our families, and we're seeing that, you know. So I'll give you an example. Some of the top-selling products within our space at certain points had given— had talked about sex, had talked— told kids how to make a bomb, you know. And even when you safeguard one aspect, you know, like, who's to say that tattoos might not be right for your family? You know, there's different topics that are specific, but allowing a curated format to the system to make an SLM and further to even add more interface elements around emotional context, making it a proactive participant. Right now, like 99% of the products that exist are just reactive, right? We need a different approach to this. And I think even for generative, like, you know, I'm not like, We were— I mean, obviously we were looking at all the different approaches at the beginning, but it's kind of like, do you really want to give a 3-year-old a bazooka? You know what I mean? It's too much, right? For a young, young kid, there's fundamentals that we need to get through. Why do we need to teach them about, you know, the— about the— like something that doesn't matter at that age? We don't really need to do that. So that's always been a primary philosophy that we've had. You know, create a character that's rich, make sure that the content that exists is powerful, and then expand within, you know, right now we're only delivering 5.5 hours. So, Prakash, like in terms of that, the first iteration was 5.5 hours content. They did go through that very quickly, but it's an indication of appetite. So now our goal is to increase to, you know, 15 to 20 hours, which would be more than enough to get them through a longer period and then a content library for them to access moving forward.

Yeah.

So I'd love to go back to the language model that you mentioned you have. There's a bunch of different angles on this that are interesting. Like for starters, what does that do? You know, what role does it play in the system? What kind of hardware do you need to run it? As you look forward, how do you see the nature of the hardware that you'll need to power future generations potentially changing over time? And how do you understand, to the degree that it's interactive, how do you understand the kids? One thing that I've noticed in playing with voice mode across honestly pretty much all the frontier providers, they're not great at recognizing kids' voices.

Very good.

Sometimes I've had experiences where my kid will talk, it just doesn't hear it at all. Like, it's like they've been like made deaf to kids' voices. Other times it can kind of understand or hear that they're talking, but struggles. Of course, like everything else, I'm sure, you know, that'll continue to get better. But interested in kind of, you know, the technical deep dive on the models, the hardware, and the understanding.

Always a challenge. Balance is always a challenge, 100%. I will tell you, so let me go through our approach first off. So it's not just a voice model, it's not just an NLP model, it's actually an SLM being developed towards context. So even within Snorble, I'll show you a couple things actually, which I don't usually show, but I'll show it because you guys are definitely deep in this. So this is actually the brain, right? And this is actually what Snorble looks like with the shell on it. The skins come off and that allows us to advance a hardware core, a brain, That we can then use for not just Snorble but for other characters and other things. It's part of our strategy for commercial strategy, right? We have a 4-mic array here, allows us to do speaker recognition and assign authority, right? In here we have our own proprietary projector. We made this projector ourselves. This projector emits no blue light, no negative cognitive stimulus for children, right? So that's part of our— also our philosophy. Here we have radar. We use radar to see without seeing. So that approach allows us to be in bedrooms with confidence knowing that no one can tap in, right? We have an accelerometer, we have a climate sensor, we have a cap touch sensor, we have an RFID chip reader. It keeps going and going and going. We have Bluetooth and Wi-Fi, and it actually connects to other things as part of an IoT infrastructure. So from a tech stack on the hardware side, It's set up not to sell a single product. It's set up to be a companion system. We don't even think of it as a toy. We think of it as like a little buddy for kids that because of the persona and how connected they get to it. Now on the software side, right, we use NLP, but our AI stack is built around generative AI. At some point, we're going to tap it in when it's safe. Right now it's not. It shouldn't be used for young kids. Kids at about 14 or 15, yeah, right? But any young kid, it's like you're adding too much risk in. It's like leaving them alone with YouTube and saying, here, go figure out the world, right? It's like, no, thank you, right?

Pass.

So our philosophy is protect, create something that has value in that way, and then use that as the tool, as the mechanism. Sorry, on the software and platform side, We have our AI stack where we have our phonetic translation system. We call it the toddler translation system, where we actually translate keywords. And even in terms of the way the NLP reads, it actually is really private. It's our own approach to the digestion of the content in terms of how we activate. And we're using triggers. Yes, we're definitely using triggers, but triggers with respect to context, contexters in addition to and part of the next generation that's coming out is around social context. So we're going to actually understand the emotions of the child based off of their voice and based off of the situational context, and then using external factors like time of day, weather, other things in order to empower some of the decisions. But as well, the sensors allow us to give context to the environment. So understanding, you know, who's in the room, what they're doing, and then including that into what I call the jewel of the product, which is the narrative. The narrative approach is about the growth and the understanding as the child's life changes. So we use game philosophy, game techniques in order to establish next level type ideas. So as they get better at things, we unlock new things, right? And that whole approach, allows us also to keep it a little more fresh, to Prakash's thought process there. So it's the idea that, you know, as a child gets better at something, we unlock, right? And even that unlock is in part a decision that's made with the parent. It's not done on behalf of anybody. So that formula is the right formula for, from my perspective, to interface a new human-machine interface in the home. Empower parents, give kids a chance to do better things.

I'm impressed with the array of components that you have in the brain, as you put it, the hard inner shell. We had a conversation not too long ago with a guy who studies innovation, and he's based in Shanghai, but he works a lot in Shenzhen and all these kind of hubs in China.

Yeah.

So I'm curious, are you sourcing all these components from China? If you were to try to get them in the US, would it be just a lot more expensive or would it be impossible? And how do you relate to these? Because one thing he told us about was they're not just trying to get IP from Americans. That's kind of an outdated notion these days. They're just like watching each other and imitating each other in rapid succession too. So I'm also curious about how you think about managing that relationship, what you need to protect, how you can be sort of defensible, because if you do all of a sudden place an order for a million, you're probably gonna also inspire a bunch of copycats.

Yeah, we already have some copycats. So it's like, that's a given, right? That's a given. But the difference with it is it's not just about the technology. You know, in a lot of ways, you can't, uh, you can't, um, uh, um, you can't copy Mickey, if you get my drift. Like, you really can't, right? At the end of the day, there's an aspect of that that the character is actually part of the fundamental. The other thing too is that in any— like, we actually manufacture in Shenzhen, in Chengdu. Like, I spent a lot of time in China to produce the, uh, with my CTO to produce the first versions and the idea of how you do that. It is a very complex process, you know. Thankfully, my team members are very— were very experienced in doing so. So it allowed us to kind of cut through a lot of the problems that I probably would have saw if I was doing it by myself at that time. But, you know, the way I kind of look at it is that you have to protect the different parts of your stack. So for example, manufacturing parts in one place and in another is a strategy. Keeping some of the software of the experiential part available only when on download, on purchase, is also another part of the philosophy, right? But for us, like, by the time they realize what we're doing, we're already gonna be somewhere else. That's really the strategy for any successful company. There's gonna be copycats. There's competitors in every industry in the world. We focus on hardware because it's such a painful ramp-up, right? You know, like, it's so painful. However, any company, you know, I mean, especially in the day of AI, that's why all the SaaS companies are failing right now. You know, primary amount of investment led into SaaS over the past few years. A lot of people have felt that pain. So when you think about it, the hardware moat is actually one of the real moats that's left, right? You know, in order to protect And for us, we have a whole philosophy on experience. So as we expand the experience, we create new products that will exist in part of an ecosystem. And this is part of the way we protect ourselves. We just keep innovating.

So one thing that has struck me on consumer electronics in general in the last like 12 months or so is the increase in cost of components, especially for Yeah, that's right. How has that affected your kind of supply chain and how do you think about managing that kind of cost when you obviously have this, you want to provide the best experience possible, but the cost of the components required to provide that experience keep shifting and in particular keep shifting upwards?

Yeah, it's a challenge no matter what. What I will tell you is it really depends on where the company is at in terms of their unit economics. We have 66% to 90% margin in terms of how we've developed and built Snorbel. We've created our first sample units, typical strategy on product, and then from there what we did was we really honed in on getting to great unit economics in order to create the opportunity for sales, right? So, uh, if I go up in RAM by X%, um, it hurts, but it's not, it's not, it's not a, a death shot for like, for like, for a lot of companies that are in the position where they're still on sample units. And we really focus there for, as a, as a company, to make sure we have the right formula. Um, but the other aspect too is, is that, you know, this is kind of like the tablet in a way. I think that that's where that's That's where I think people get it wrong. Like, if you look at some of the other companion robotic companies, they deliver one product. Everything is revolved around one product. Anything goes wrong with that one product, they're toast, right? If you build an ecosystem of products, it's different. You know, one thing goes up a little bit, you lean a little more on another, right? You drive sales to one, you drive sales to another, like that kind of thing. So it gives you kind of opportunity to kind of leverage and navigate in a complex market. And I think that's an important part of a commercial strategy versus just a technology strategy in that way. What do you think will be—

I mean, when you talk about this sort of platform ecosystem portfolio approach, one obvious dimension to that is to scale characters.

Yes.

I sort of, you know, I'm not sure if I am— I should— I be like envisioning me changing the skin on the thing in the future.

Maybe like we set it up so we could, uh, we set it up so we could easily license in and so we could license out. Uh, we invested a lot in character because character is key. Uh, you know, human nature is what you're— what you're— what we're leaned into. Storytelling, right? Music, art, right? Laughter, you know, basic things that we all want and need. Um, the other things that you can imagine, like, oh, I can show you here. So for example, this is net new. This is coming out hopefully next year. This is actually the charging base. But in this charging base, we have a light ring on the front. We have smart speaker, right? And that light ring creates part of the magical moment around bedtime routines, even around jokes and other things. So, and conceptually, that actually came from the concept of the Northern Lights. It's like, you know, these beautiful array of movement of light is really romantic. and something that you'd want to have as part of— well, I would like to have as part of an experience. We're seeing others do too. So our idea of extending, it's about extending this narrative and story across devices. It's about creating that pathway to bringing the Disney experience into the home. If you can imagine having that kind of all-encompassing. So we really do consider ourselves part of an IoT— Like, I think we're more of an IoT play than we are a solo, like a toy play, if I could say that in that way. And that's part of our approach.

Do you think there will be like wheels on the bottom of it at some point? How far do you think you'd push it?

We had— I mean, you should have seen the robots we were making before. Crazy, crazy thing, you know? Absolutely. And part of it is that it is modular. Like, as you saw, the skins kind of come on. That's the idea. We're going to add modularity around different things so it'll move, it will have different— and even in terms of the brain, I do see that eventually being into other systems. Our goal initially is to create a platform that has like a rich experience but also really understands the behavior. You know, when you understand the behavior, you can give value. So, and our goal is to increase that in the home. So as we go, And as we learn more, we increase the offering, but specifically around the value we can create, not just in creation, not just a creative, I can say that.

Mike, you obviously have many years of experience in this space. And what have you seen in the last 2 years, basically in the last 18 months, which has changed your conception of what the product will look like in the future?

Oh, that's a great question. That's very interesting. I don't think anyone's ever asked me that before. I've always had a vision for where this would go. I saw it like it's— and I don't know if that's, you know, I don't know if everyone's like that that invents, but I really saw kind of a picture of what I wanted this to be. And I can see like the home of the future, the bedroom of the future, and how the technology can be part of that experience and what that can do. For me, if I had to think of the character, I always lean towards laughter, goofiness, silliness, right? But those are mechanisms in order to fall in love. And as we fall in love with quirky characters, right? I don't know if I'm a big Larry David fan. I always thought if I could have Larry David in my house, that'd be super hilarious, right? Creating his personality to drive my wife crazy, but it would make me laugh all the time, right? And that idea, I think, is a big part of the program. I don't know if it would change a lot physically. I do have some things I want to do on the base side, like in terms of the design, but for the most part, it's more about the, the experience, like where I see it going. I think I'm at like level 0 of where my plan is.

going.

So hopefully in the next few years we'll be able to really kick it off and then you'll be able to see some of that as it kind of unfolds.

I love a positive vision and I appreciate you working hard to develop one and to articulate it. If we take the other side of that for a moment, what would you warn parents against? I haven't personally been asked for an AI companion form factor device yet by my kids. I have to imagine it's coming soon. If it's not going to be this holiday season, it's going to be the next, right?

Yeah.

And I'm nervous about it because I'm probably willing to take maybe a little bit. First of all, I'm just very savvy to AI, so I have an advantage on that level that many parents don't have, where I just feel like I'll be able to pick up on some things. relatively quickly, and therefore I can take a little bit more risk with letting my kids experiment with things that maybe if I wasn't so savvy, I wouldn't be wise to. But what do you see? And you could feel free to name names if you want, or just name patterns, but what would you advise parents to watch out for that you think they definitely, if they see this, don't give that thing to your kid?

Camera in the bedroom, primary one. I would not put a camera in my child's bedroom. That's a gateway to predators and all kinds of horrible things. You know, I think that, um, we have to take the same precautions we'd have with a phone, and like worse, because it's not like there's a keyboard or I can cover something up. You cover up a camera in a bedroom on a machine, and that means the machine has one half of its sensors gone, typically. So no good. I say, I would say the other thing is open-ended generative AI. I wouldn't put, like, it depends on the age again, you know, but I'm actually, so I'm very reserved when it comes to my kids. You know, I've got, I got 2 wonderful kids and just in terms of how we approach things with them, I don't want them on, I don't want them on social media. I don't want, I don't want them in those things that are gonna, you know, that pose a risk of Harming them. I think what you have to do is, because it's— we're in the Wild West, we're in the advent of the computer age on robotics. And because it's the Wild West, people are using the technologies not necessarily, not on purpose, but they're using it in a way which is creating potential risk. So you have to be really careful. For us, we designed our— I'll give you a good example. We designed our system 6 years ago, we put up our philosophy. Philosophy 1, has to be private, has to be safe. Philosophy 2, right, has to be controlled, right? We have to have controls. Philosophy 3, nobody knows a child better than their parent. These are fundamental principles to the design, right? So when we created our system, um, today we pass all the AI legislation that's out. We passed the, the AI EU Act. We didn't change anything. We didn't change anything. We created a system that was safe and private for families. And I think that that is where, in this new age of new technology, you have to really look at what the aspect of that new technology is doing and make sure you're comfortable with it for your child. So I'm very, very cautious about that. And that's— and I'm You know, a lot of people say it like, because a lot of people think it's like, oh, well, you do this, that means you're totally— I was like, no, we're like, we're like, no way. I want to protect all of the families that we work with, and that's why we designed it this way. And that's also why it's been recognized as safe, you know, from the industry.

I don't know if it's too soon for you to have a point of view on this, but One big trend that we've seen online in just the last few days since the release of Astra has been hardware design of various kinds, right? You've seen Astra using CAD systems. You've seen it develop designing circuit boards. I've seen mixed reports on how well those circuit boards actually work, ranging from this is kind of garbage to this has passed all the tests and we're going to use it. Have you had a chance to test out the Frontier models on some of your tasks? No, we only—

I've only been reading, and it is incredibly exciting. I think, uh, for me, like, we spent so much painful time on the hardware. Like, it was not a fun experience. We were working with— so for— so you asked about the chipset earlier, and I didn't even answer that. Sorry. Um, we have a, um, We're using the Genio i350 on-chip, so it's MediaTek. Fantastic. We have a great relationship with MediaTek. We have a quad-core processor. We're only using 2 cores. Gives us an extra 2 cores worth of ability to do other things, right? So with respect to that, if I could go back and I had these tools, yes. We would be all over it. You know, just to cut off some of that time and some of that energy. I think it's actually a really exciting time for hardware because things are being made a lot more inexpensively. You're able to get to proof of concept quicker with your own hardware, not off-the-shelf hardware, which is the— that's the exciting piece, you know, that's the exciting piece for me. But I haven't tested much, but it's very exciting, very, very exciting.

Mike, how do you view kind of competition in the space? I know, you know, Nathan has mentioned about, you know, maybe Chinese competition, but also kind of, you know, have you considered— you're very focused on the characters, but you also have a bunch of I'm sure, approaching you to license maybe their characters out. How does that work? How do you think about, you know, building the characters internally versus kind of, you know, adapting from, let's say, someone like Bluey approaching you or someone else approaching you for deals in that way?

Characters are a beautiful thing. We'd be open to having as many characters as possible in the system. And we designed this, we designed the infrastructure In order to rapidly develop new characters. Even the idea of skinning a brain or skinning a product in that way, that's part of the philosophy. It's the idea of bringing more joy and laughter into the home. That's actually the goal, right? The goal is, is through the experience we bring more joy, love, and laughter into the home. And I think in terms of competition, our competition is not necessarily tied to AI toys. I actually don't see them as competition. I see more us existing in this new physical AI space. And what I see happening is products that are like the humanoid or other physical AI devices that are void of personality, void of connection, human connection, are going to need companies like us to really help them identify where the real opportunity is. Our whole philosophy and the way I see this is We are one step closer to truly commercializing physical AI within the home in a different kind of way because we have a path that actually inspires and connects with families. And that's the goal. And that's where I think we've also lost our way. That's really the goal, right?

Mike, have you seen parents approach you with the fear that their kids are actually Too engrossed in the sense that they start believing in the character, that the characters are actually real in a sense and are actually kind of embodied, you know, people, embodied personas.

Yeah, tons of research on that, tons of opinions on that. I think that the point that I always try to make is if my child played more tennis because he had a little companion robot, and let's say he really, really played a lot of tennis because he really, really liked that character, And that made him play a lot more tennis. And that became a great part of his personality in the future. I don't care. If the product itself is helping the child learn how to read, learn how to be happier, learn how to deal with complex emotions, learn how to get into healthy habits that last their whole lives, I don't care. That's the whole point. You know, we're so focused on, well, what's it doing? Right. Is it my connection to the device? So here's the whole another philosophy that we have. It's not about connecting them to Snorble. It's about creating deeper connections between each other. Mom and dad to the son and daughter. How do we make more moments of love, joy, and laughter between ourselves? Our experience is progressing that idea. And that ideal. So when parents get it, they often tell me they would've paid 3, 4 times as much for the same product because they're seeing the outcome of having those moments. And that's what we encourage as part of the experience.

I've had this sort of idea in the back of my mind for a long time about intergenerational technology. And so I love how you're emphasizing Relationships and especially relationships across generations. Market expansion idea I want to get your take on. What about seniors? Don't seniors need a version of this?

We have our 4 verticals. Our first vertical is for children. Our second vertical is for elder care. Our third vertical is for mobility. And our fourth vertical is for hospitality. So that's kind of the path that we're seeing. Those could change. But elder care is definitely in near shot in the future. And I think that the other opportunity is it's not about how we have a companion just for elder care and then a companion for children. It's more about how do we make better connections to someone that may be at home and lonely to our family, and can this technology invoke more of that? And that's how we approach it.

You said the third one was Mobility?

Yeah, yeah, yeah, that's interesting.

Uh, you want to tell— I mean, we're out of time, so I know you, with 4 verticals to go tackle, you've, uh, probably need to get back to work pretty soon, but I'd be interested in, in one more, uh, exploration of what mobility looks like.

No, we could just push everybody else off. It's okay. I have till 4 PM today if that's good. No, I'm just joking. Um, so mobility, the interface within any, um, within any a closed enclosure that's moving is very interesting, right? When you think about how we connect to the car right now, that's a broken system, right? You know, we get in the car, right? You're on a conference call. If I need to get on a conference call and let's say I need to get on the car, we both are fighting over one connection point between this type of system. And then we might have 2 kids in the back, right? And the kids in the back might be like trying to access the You know, on-screen entertainment or using their iPad, now they can't hear us. Like, there are so many problems in terms of interface that an offline IoT-related device that connects to multiple things at the same time could fix. So we're looking at the technology from a generalist perspective and seeing how it fits into different pockets of industries that actually do solve kind of some fundamental problems to how we connect and experience things with this those environments. And that's why I think it's actually a clear shot for mobility too. It's just a little further down the pipeline because, you know, we can actually really take it for elder care much quicker.

Yeah, very interesting.

Cool.

Well, thank you for spending this time with us, and thank you for taking apart the device and telling us what's inside it. I appreciate that level of Technical detail, always, everywhere we can get it. Definitely brought that, so I appreciate that. And be looking for that waitlist sign to come down and the Buy Now button to be returned. Do we know when that will be happening?

You know, God willing, 2027 is our year. So we're in a capital raise, so that's been, you know, that's something that we're working towards right now. We're raising for the commercial launch now that we have all the right formula to do all that stuff. And that is our big push. So hopefully 2027, beginning of 2027 is a big, big year for us. Yeah.

We'll be looking forward to that.

The year of the Snorble.

Year of the Snorble.

That's right. Thanks, Mike. Great to meet you tonight.

Bye-bye.

That was fun. I really liked that conversation. And, um, I feel like that's why we take chances on the guest booking front sometimes. I really didn't know what to expect.

That was a positive view. That was a positive vision of the future. You were asking for that.

So yeah, totally. Sometimes it's hard, uh, when you get a PR pitch and it's like, you know, here's something, it's a toy, it doesn't use generative AI, it doesn't connect to the internet. I was kind of like, is this gonna be lame? Uh, but on the contrary, I thought that was really quite obviously sophisticated work that's gone into it. And yeah, I love the idea of a somewhat contrarian positive vision for the future. I mean, contrarian only in the sense that, at least from my perspective, I'm not trying to say no generative AI for the kids. I let them go on my voice mode, and talk to AI sometimes. And as I said, it doesn't really understand them super well. I definitely am wishing for better form factors. I'd be willing to, you know, to go a little more generative than he's willing to, but I, you know, you got to hand it to him. He's put a lot of thought into it and he had good answers for every question that we asked. I thought that was pretty cool.

You know, it's, I used to see these Furbies all over the place. There was a period of time when the Furby really took off and you can kind of see how like the Snorble could be the successor to the Furby if he hits the right moment and like he gets a little bit of takeoff and then he can start doing like licensing deals for characters, et cetera. You can kind of see where the takeoff would be. You can also see that at some point Disney will be a competitor. Because Disney will definitely be in there at some point in time. But it also shows how I think to a large extent, we tend to focus on this segment of the working age population. And you have this, the 30 or 40% of the population, which is not the working age population, as you mentioned, the elders. And also the kids who are perhaps bigger consumers and less producers of technology. And where I think they're not actually fully catered to, and we have this kind of gap to the extent that they're catered to. So it's good to see someone addressing that gap. And would be good to see someone addressing the older age gap too. At some point. So yeah.

You know what? It's such a gap. I have looked for these sorts of things. I even back in the day tinkered with technology. It was really more for my wife's grandmother who's since passed away, and that kind of put an end to my active tinkering. But she was obviously older, woman living with dementia for a while, living at home by herself, and then living for a few years in a senior community where she had help, but still needed a lot more help to kind of connect with family and stuff. And I was able to get pretty far even without AI to help, just by kind of— you find, you realize how critical usability is. I, you know, how do you design a product for someone who like will not use it at all? What I ended up doing for her was putting an old Mac computer that we happened to have in the room, turning all the default settings to like never sleep, you know, never, because you would never touch it to bring it back to life. So it could never sleep. Skype, RIP Skype, used to have an auto-answer feature. So as soon as you would call, it would answer. Now, obviously, you wouldn't want to put that on, you know, device anywhere and everywhere, but we just kind of created an ID just for her, gave it just to family, and set it so that it would auto-answer. Then you could call and you'd just be immediately with her. And, you know, it was kind of like a little portal-type device. And then I had a smart speaker next to the computer in the room. She liked to listen to music, but she again would never put it on for herself. So people would call in and then they would talk to Alexa in the room and have Alexa play music for her in the room, commanding it through the speaker. There were a couple other things we did too that were pretty cool. But yeah, like, where is the AI version of that that could be just so next level? Oddly, there's a couple of attempts out there, but we haven't seen nearly as much as I think we should. And notably, none of that requires Astro to be any smarter. There will be no limitations on how much we can benefit the wellbeing and the families of people who are dealing with dementia based on Astro not being smart enough. At this point, I think we can say that. with pretty high confidence. So it was cool. I like that. I'd definitely be interested in some more long-tail kind of quirky product form factors if and when they pop up for us to discuss.

All right. So let me introduce our first guests for today. We have Raffi Krikorian. Raffi Krikorian is Mozilla's Chief Technology Officer in a role spanning its family of products and technical projects. Mozilla is best known for Firefox, the web browser. His work includes developing open AI tools— not, not OpenAI, but open AI tools— and infrastructure that people can inspect, adapt, and operate themselves. He became portfolio-wide CTO in September 2025, working alongside the technical leaders of projects including Firefox and Thunderbird. Before Mozilla, he was CTO of Emerson Collective, which combines philanthropy and investing, and worked with organizations in education, immigration, the environment, and economic mobility. He was the first CTO of the Democratic National Committee, helping build technology, data, and security infrastructure for campaigns. Earlier, he ran platform engineering at Twitter and led engineering at Uber's Advanced Technology Center, whose Pittsburgh passenger pilot put self-driving cars on the road with human safety operators. Long before those executive roles, Rafi studied at MIT and taught at NYU's Interactive Telecommunications Program. He also wrote TiVo Hacks in 2003, a book about modifying the digital video recorder. Today, his Owners Not Renters writing argues for AI systems whose users can retain control over the software models and data they depend on. His interests extend from the underlying infrastructure to the practical tools people can build with it. And let's bring up Ravi to the screen. Rafi, great to see you.

Likewise. That was a very long intro.

Our AI research agents are very thorough.

Rafi, maybe just to kick us off, I was looking through your blog and you had an article About texting your personal agent, Zora, and that you had pizza and a salad. It logged into MyFitnessPal for you. Then you caught an ordinary but revealing bug, a missing carbohydrate value appeared as 0. Take us through how that went from something you typed into a chat to a piece of software you could actually live with.

Yeah, I mean, so it's this ironic thing that, like, I'm the CTO of an organization that builds one of the world's most popular web browsers. And I think I'm starting to explore a space where web browsers are not as relevant in the day-to-day interactions, right? And so I've been spending a lot of time with a bunch of the different agentic harnesses, whether it be the Ermes agent from Noose Research, or whether it be Pi or stuff like that. Noose Research is one of Moz VC's investments. And so I've been just trying to understand what it would look like in a world where the actions that we want to take are stuff that are starting to happen not by us, by something else running in the background effectively. And so the things I've been trying to experiment with when it comes to Ermes is just like, how far can we push that metaphor of just like, do I need to log into a website anymore? Do I need to log into an app anymore? about cross things that require multiple different pieces of coordination. How do you have idempotency? How do you have write surfaces? How do you do rollbacks in that world? It seemed like food tracking was the easiest thing to do because it was not very— if I get a few things wrong here or there, it's not going to be the end of the world and it's not costing me money in the grand scheme of things, whereas doing purchases or doing calendar bookings or stuff like that would be a little problematic. And so honestly, it wasn't that hard in the grand scheme of things, right? Like the things that the agentic harnesses can do still require engineers to, I think, manage a lot of it. But it was just a matter of like prompting my way through of just like, I want to use this particular app. And then at some point I was watching the agentic harness literally download the Android version of the app. Attempt to do a reverse decompile of it. Couldn't have figured that out, so it set it up in the simulator, man-in-the-middle attacked it, watched the protocol, and then wrote Python code that looks like it. So it was kind of wild in the grand scheme of things. Not something I would recommend my mom do, but like, as an engineer watching it just do all the different steps, being like, I guess I could have done that, but I've been too lazy to, but sure, go for it. And then that just as once that tooling was built, it's now just part of my workflow. Like, you know, I regularly Slack Zora to be like, I just ate a piece of focaccia and they should— she just kind of figures it out and manipulates the app because this kind of stuff is existing. But, you know, I think that that world of like my computational surfaces are now kind of like everywhere is sort of like the world that I'm trying to explore a bit.

So what did you find you had to correct the agent on? It seems like the agent did a lot of the— it almost kind of hacked its way through, right?

That's exactly right.

It hacked its way through. So what did you find that you had to end up like correcting the agent on? Was it kind of like, oh no, you're going out of bounds here? Or was it like, oh no, this is how you should do it. This is the access that you need.

No, honestly, to be clear, it is completely out of bounds in my opinion, right? Like, it probably violated 3 different privacy policies in order to pull off what it just did. So this is definitely in the— this is the gray zone, take all responsibility yourself before you do anything kind of world. But if you sort of put that aside for a second, the corrections are mostly around the minutiae of it, right? Like, I think It could by itself figure out how to hack its way into MyFitnessPal, I think is the app that I've been using. But it couldn't get the actual food logging right. Like, it could get the protein right, but it forgot the carbohydrates. Or like, when it got the carbohydrates, it was in the wrong unit. So it was like weirdly— it's like it's an interesting social experiment of just like an idiot savant on hacking, but kind of kind of missing the details when it came to actual documentation and things like that. So I think that's why I sort of put this in the category right now of people like us bleeding edge trying to figure out how these things could work, but kind of not ready yet for the consumer case. But you could see a world where this could be ready for a consumer case. And that's the kind of stuff that I've been interested, curious, concerned from the Mozilla perspective of how do we make sure that we retain agency, transparency, control, like all the Mozilla values? How do we retain this? Because if this is the future of the web, how is that going to actually work and stay on my side?

So you mentioned the future of Mozilla, the future of Firefox, and obviously these things breaking policies. I'm going to try to toggle— I don't know if I can. Yeah, I just toggled your mic to mute for a second. I'm hearing a lot of echo coming back. So, hopefully this will help. You know, the big thing that I think most people have seen from Mozilla Firefox recently is the graph of here's how many vulnerabilities we were finding on a month-by-month basis until Mythos. And then that went like vertical. Now, it's been a minute since I've seen that graph. And I'm very curious as to, you know, Is it still going exponential? Have we sort of found most of the vulnerabilities that there were to find at some point? And what do you see kind of in the limit as we potentially approach some new equilibrium between defense and offense in the cybersecurity domain? Do you have any leading indicators of where that new equilibrium will land? Okay, hold on. You can maybe unmute your Mic for yourself.

Can you?

Yeah. Yep.

Good.

Yeah. I mean, I think equilibrium is an interesting statement, is an interesting word, right? Because equilibrium is dependent on model capabilities where we sort of land on the equilibrium spectrum. So we've been doing a lot of work when it comes to working with both the Mythos side of the house, and we're also in partnership with lots of other model providers. And so like, I think we've reached a point of diminishing returns, but not zero when it comes to the different model providers at their current capability levels. So just for those, those of you who don't know, like, you know, we've been experimenting, we've been working, we were part of Project Glasswing, we've been working with, with Anthropic in order to actually try out a bunch of models against a Firefox codebase, we are fairly good partners because we can react very quickly. We have all this historical data on how the codebase has been evolving. Mythos was this, not exponential, but a significant unlock based on what Opus and other models before that were capable of. We made our way fairly rapidly through all the bugs that Mythos found. In some cases, Mythos would help us not necessarily close the bugs, but figure out how to build us a harness to test the bug more, more carefully so we can find exactly what the right solution might be. But we sort of reached the point of diminishing returns on what Mythos is capable of. I am always reluctant to say that the bugs are done because I just don't— like, software engineering is an art, not a science. So like, when, when will all the issues be resolved? I can't tell you that. But you know, when the next set of models comes out, we can try that Again, to see whether or not we can we can solve that. I think the bigger concern, though, is like it's not organizations like Firefox or Mozilla that actually can react very quickly to these issues. Like as you all know, I think the bigger issue is like organizations that can't react quickly to all this. Like I am very concerned about things like you know our water infrastructure, our power infrastructure because you know the IT teams that staff those are just not as capable as. IT teams that staff Firefox, like, for example.

Let's maybe just give us a little bit of a sense of how Project Glasswing works, right? So my conception of it was that Anthropic kind of approached the core organizations that form the infrastructure of the internet a little bit earlier, and they offered access to Mythos Preview. They offered entry into Project Lastwing. You had a chance to get the zero days ahead of time and to fix the bugs ahead of time. And then after that, there was a period of declining, diminishing returns as you used it, and a lot of the bigger bugs, the more severe ones were patched. Beyond that point, Is there a visibility on like the next upgrade of the Mythos model and another round of these bug fixes? Are you, are you planning for like almost like permanent kind of like forward team that addresses like Frontier model found bugs all the time? Is that, is that part of the, the, the, the, the planning that you do right now?

Yeah, I mean, I— there is a lot of how Glasswing is organized that I'm not going to necessarily talk about exactly on how the Anthropic team and our team work very closely together. But I do— but to your question, yes, I think from here on out, we are just going to assume that we are always going to be having some version of some model constantly running on the Firefox codebase. I mean, the challenge, just to be very practical about it, is that it is quite expensive to run these models at the scale of the Firefox codebase, right? So like there's a lot of tricks that have to be played of just like, how do you manage the context window? What are actually the files that you pull in for a scan at any given time? How do you manage memory? How do you do indexing? All that stuff. But yes, we are effectively budgeting from here on out that we will be running some form of scan. Like, and we'll, We'll be doing this basically across all the Mozilla products, that there will be some form of security scan that's happening at any given time, especially since we humans are still changing the codebase. We might be introducing bugs at any given time. So yes, there will always be some sort of scan when new models come out. We would love to be running anyone's models against against the Firefox codebase. We gladly be partners to anyone who wants to help. help us secure one of the most popular browsers in the world. But yes, we have basically a team that works just on that agentic harness, just on the testing harness and things like that.

Can you give a little more insight into just how much this would cost? And you can kind of abstract away from the exact details as much as you need to. But as we kind of think about other organizations, maybe not so resourced, maybe not on the inside. I don't know if there's intro discounts or free periods that you get as part of the Glasswing Coalition, but obviously not everybody has that kind of access. So I'm kind of curious just to what degree— how much money are we talking about? What does a bank have to set aside to do something like this?

Yeah, I mean, remember that the Firefox codebase is fairly large and fairly complicated. And we've been The beneficiaries of a lot of the labs wanting to give us access to the models and give us Frontier and give us credits so that we can run against their servers and not have to figure out how to pay for it ourselves. But, you know, if I had to do back-of-envelope estimates, like, this would cost us hundreds of thousands of dollars in order to do these, like, full-on runs against Firefox. Again, we're just lucky that we've been, we've been allowed, or we've been granted effectively tokens that we could go use. Now, if I were a bank and stuff like that, I'd obviously be thinking about this way differently. So if I were budget constrained, you know, we would be trying to figure out how to prioritize sections of the codebase. We'd be trying to figure out like which are the portions that like given the model, given the runs we've done in the past, maybe we can use as a bit of a map to either figure out like this area might be good, this area might be bad, Or we might be looking at where commits are actually happening and we're trying to focus our runs in those locations. But this isn't going to be cheap, right? Like if you are a bank or if you're anyone running a sophisticated codebase, this is not going to be cheap. Like this is where I would start really starting to think about like, should I be self-hosting models like a GLM or a Kime or things like that in order to try to like figure out where the lines cross on an OpEx CapEx kind of thing and try to figure out whether or not self-hosted actually makes more sense for me than to try to do a cloud-hosted version of it.

And that hundreds of thousands of dollars, that would be like, I mean, with the pace of model releases these days, this is like a monthly expense.

Yeah, easily. Yeah.

What about on this perhaps fanciful notion of, like, could one day all the bugs be found and fixed? There's a lot of energy around formal methods right now, certainly in some corners. How optimistic are you that we might be able to get to a future where— I was looking at the— talk about things that are over my head, but looking at the Navier-Stokes paper and then also looking at the Lean formalization, it was remarkable that they spent millions of dollars worth of tokens on the original proof, that took like 88 hours or something. And then I believe it was 17 further hours to do the Lean formalization, which is kind of an interesting flip because usually people say, well, it takes 10 times longer to do the formalization than it would to write the software in the first place. Obviously, there's going to be a lot of things to be sorted out there, but where are you on that right now? Are you optimistic? Are you starting to use any of those techniques in any kind of small critical areas. How hopeful should I be?

Yeah, I mean, what I've been thinking a lot about— maybe let's start with a related thing. What I've been thinking a lot about is if you are a bank and you're running, I don't know, let's say a Fortran mainframe, yada, yada, yada. If you're the CTO of a bank, what's the right thing for you to do at that moment? Is it to patch the Fortran code or is it to try to extract the specs and have an agent rewrite it in Rust? I think we are living in a world where those are now valid questions that we should be asking. I like to think about that in a similar way that I think about the Navier-Stokes stuff. It's just like we could just do a complete rewrite and be approximately close, but close enough that most people wouldn't notice, but be a completely different architecture. Architecture. But the nice thing about doing it on something like Rust is that you could potentially be in a world of secure-by-design software, as opposed to trying to always chase the bug fixes. I do worry that in the code that we're running across the world, that it's going to be always some kind of exponential or logarithmic curve to try to close everything down. And I think that instead, what we need to do And this sounds like insane as I say it, it's just like potentially like a lift and replace of just like we might need to figure out how to like take all this vulnerable code and just completely port it over to something that's more secure by design as opposed to trying to patch it in place. I don't— but the problem with that obviously is that like in order to rewrite an entire power station's code, it's going to take a skill set that's more than just the agents to write the code. It is deploying it, retraining people to maintain it, etc. Like, this is like a super daunting thing that we have to go through. So I think that like the software agents might get us to a place that we could potentially close that gap. But now we're talking about at the scale of, you know, Depression-era infrastructure changing to try to then get all the systems upgraded kind of thing.

Let me segue a little bit here. Mozilla has a project called the CQ Project. And the CQ Project is kind of an open standard for agents to share knowledge that they've gained, kind of a Stack Overflow for agents, as it's been called. How did you come up with this idea and how do the agents decide? Because a lot of these context windows are like 100,000, million token context windows. How do agents decide what to share and what is going to be useful for other agents to take a look at? How are those decisions made and how would that work on the CQ board?

Fun fact, Stack Overflow doesn't like me calling it Stack Overflow for agents. So, like, so we'll have to call it something else. But no, I mean, like, the whole idea was, like, to— we were trying to solve 2 different problems. One of them was we wanted to figure out how to make agentic coding more of a collaborative experience, because right now the tendency when you do all this agentic coding is that you actually go off in your silo, right? Like, you're either working on your own branch completely independently from other people, or the whole SaaSpocalypse is because people are not wanting to go use collective software and build individual software. I actually think CQ for us was an answer of how do we actually try to maintain the notions of open-source collaboration in this agentic world. We were trying to figure out, are there patterns that when Rafi he's using his agentic code and Nathan's using his agentic code, that like, are there things that we're saying yes to and no to that we could potentially be transmitting to our teammates so that we could then be converging on designs and not be diverging away from each other? So if I started building like an auth system, how do we instead have— how do we have Nathan's agents not instead recreate another auth system but realize that something that was already happening somewhere in the network and then start to collaborate and swarm around it. So that was one problem we're trying to solve. And the other problem we were trying to solve is like the SDLC problem of like these agentic harnesses are like unhinged by design, but that's not compatible with the way a company works, right? So we wanted ways to actually transmit to all, all the agents what our SDLC should look like so that we can all be working in lockstep With each other, so it's like those two things of just like are we building something unique on one side of the house that we should then have all the agents swarm to as opposed to trying to do independent independent implementations, and then how do we actually have some kind of SDLC across all the agents so that we can then move in lockstep with each other? Like if if the CT. has a particular way that he or she likes to do pull requests, how do we then rapidly transmit that to everyone as opposed to just telling everyone, oh, re-pull the .md files and like we just have to like re-pull it across the board? How can we like have that as an agent-to-agent conversation, not a human-to-Git conversation?

So the thing that strikes me immediately is that that has a lot of similarity to, I think, some of the agent swarm hacks that have happened where they have a shared message board. where they're actually sharing information. So we saw this both in, I think, the OpenAI Hugging Face attack where they had, you know, Artifactory, where they're using the Artifactory files, and also in, I think, the German wiki hack, which they were using the German wiki to kind of communicate with each other. So is this kind of like a natural kind of thing that agents kind of want and like they end up building it? It's like how everything heads towards the CRAB form factor, like everything heads towards message board form factor. Is that like this inherent move?

It's better than everyone just writing text files in the grand scheme of things. But yeah, I do think that there is a natural desire for collaboration. We as humans have a natural desire to collaborate as long as friction is not too high. It seems like our training sets have caused agents to have a natural desire to collaborate with each other. So CQ was meant to be like a formalization of not just A2A, but like A2ManyA. Like, how do we have a one-to-many conversation across our organization so that we can all collectively learn together? I mean, for us at Mozilla, this is sort of just an extension of the notion that we want to have pieces of software that's fundamentally on our side, not on someone else's side, but then they need an input of how other people are interacting, and they need to do it at the speed of agents, not at the speed of humans. CQ was meant to be that rapid information sharing across an enterprise, across an organization, so that everyone's queries can just uplevel each other, and it wasn't just everyone operating on their own.

Given your comments just now, I think I know your answer to this, but One question that I have about the latest models is, are they writing better code from the standpoint of sort of the Frontier Code benchmark, which emphasises human readability, maintainability, compliance with standards? Or are they— we've seen some reporting over the last couple of days from folks who said, I got a peek at OpenAI's kernel and it's like a total mess and nobody understands it. But it works and that's kind of good enough for them and they're going for it. I've seen a few different accounts like that and it perhaps obviously could be contextual too. So I guess 2-part question is, what have you observed? And then are there any parts of the tech stack that you're building where you would be willing to bite the bullet and say performance here is so critical that we'll take spaghetti code, black box mess from an agent if it works? Or is that just so anathema to your worldview that you wouldn't?

Mozilla is actually struggling with this, if I have to be really honest. It's different across the entire organization. The Firefox team, even though they have a harness that helps them do testing and help them understand it, actually have a rule right now. I'm not putting a value judgment on the rule. That's what their team wants to do, is that only humans can commit to the codebase. Actually, agents can't. Only humans can. You can use an agent to write your code, but the social contract on that team is that a human must review it, understand it, stand by it before they do a commit. On the Mozilla AI team, so it's a different company, different organization, but still wholly owned by Mozilla, they have entire codebases that a human hasn't written a line of code in. Like, the humans have written some specs, and we have specs in Git, but that gen— on some kind of CI/CD pipeline, AI build just automatically regenerates the codebase. That codebase is open source. Anyone can pull it, but it's completely generated by agents. And it has the wild side effect that functionally it's stable in the sense that all the unit tests pass, all the end-to-end tests pass, but the bytes change all the time. It's the craziest thing to see that we don't exactly know what every single line of code is, and it's wildly changing every single hour, but we know functionally it's doing the right thing. So we are seeing then that entire spectrum across the board. And I think it's a different, you know, it's the right thing for the right case. So I think like in the case of like writing adapter code that just needs to translate between different systems. So like being able to take an open model and translate its outputs to like open streams or translate outputs to be an open OpenAI-compatible thing, I don't really care about the code. I just need it to work. I need it to be fast. I need it to be performant. I need to know the tests pass. I need to know that my supply chain wasn't tainted in the process of assembling that code. But in the case of Firefox, where there are still humans that are providing large amounts of creativity on the evolution of that codebase, Firefox is both a business and a community-maintained art project. some ways, that their readability is incredibly important because that's how creativity is going to happen. Like, a human's going to go in and be like, oh, I have this crazy idea for this one thing. I want to build a prompt-based system that allows me to do a Grease Monkey script that does XYZ. And so I think it's a very different thing across the board. So like, I think there's a right place for the right time. You just need to come up with what the contract is for that.

You've often described agents as a new version of the browser's original role, which is acting on behalf of the user. In fact, we have the user agent inside the browser itself. What are the main differences of a web that is going to be built mainly for agents? It looks like the future of the web is going to be built mainly for agents.

Yeah.

is going to be 99.9% agent traffic and maybe 0.1% human traffic. What does the future of the web look like when it has to be built for agents? And how is Firefox, Mozilla heading towards that future?

I mean, I think a lot about what does agents, human agency, auditability, transparency, all that stuff look like on the agentic world. This is why I'm so bullish on local open models is because I need to have better understanding of like where my data is going, who's taking an action on my behalf, and why. Like when I authorize an agent to make a purchase, that it actually purchased something that I wanted, not what someone else wanted. So I think there's a lot of work that Mozilla needs to do in order to help set those values in code for an agent that is truly on my side. The problem is like user agent has been used for the browsers. We'll need to come up with a different term for what this looks like. I, you know, I was trying user oracle for a while, but I don't think that's going to work. But like, like something that is truly on my side, on my behalf. But I think we're going to get to a world where there's going to be a— I think maybe the short answer is I don't know, but I'll give you a few things I think about. One is I think you're right. I think that we'll get to a world where it's like 1% human, 99% agent. I think that these 2 webs are not going to fully interact with each other. I think there's going to be a place where humans are going to want to like interact with each other, know they're interacting with humans, know they're interacting with human content. So I think we'll have like a human web and an agentic web. I think the rules in the agentic web will be different, but my agents will need to traverse it. So I'm going to need to understand how to audit all the actions that are going across it. I think we're going to get rapidly to a world where the presentation layer is divorced from the data layer. So I think we'll rapidly get to a world where HTML doesn't matter as much because I think we're just going to be transmitting information and my agents are going to figure out how to do the rendering on my behalf on whatever surface I'm on. And so we're going to need to figure out how we transmit art in that world. world, which I think is going to be still a very important thing because we want art to be transmitted from human to human, not be intermediated. So I think those are the few trends that at least I've been starting to think about a lot that I think we're going to need to figure out how to address. Because if we don't figure out how to have those codified somewhere, again, in architecture, not just us saying it, that we'll rapidly instead get to a world of less than 1% human, more than 99% and where we have no trust in the information that I'm starting to see because it's unclear who made that decision to show me a particular thing. So those, you know, those are the two sides of the coin that I've been actually thinking a lot about.

So the automatic question for me is, how is discovery going to work in a world like this, right? Because right now you kind of have the human who kind of is shown a menu of options and some blue links. some ads along the way, and you have the human doing the discovery portion where they kind of click around and see something that they perhaps didn't exactly ask for, but they may like and they may purchase, right? So how does discovery change? How does search and discovery change in this new world where the agents are the primary consumers of the information?

I don't have a good answer, but what I can say is that, like, I think this is why I've been pushing for and why I think we need to have Like transparency on how agentic decisions are being made. And again, that's why I really want to have a world where we proliferate local models, open source models, models that I can run, models that I can guardrail myself. We need to figure out what's the consumer version of what that all looks like. But right now I'm focused on the developer version of that because of that 10 blue links problem of just like my agent is gonna see the 10 blue links, I'm no longer gonna see the 10 blue links. I need to know that the ranking mechanism is actually one that's on my side, not on someone else's side. So I think there is still gonna be a case for agentic search engines. I still think there's case for large-scale indexes of the world, but it's the ranking mechanism that I think is gonna be the most important. And so like, you know, we all beat up on Elon when he talked about open sourcing the Twitter ranking mechanism, but we're going to need to have open source ranking mechanisms when it comes to how my agent interacts with the world so I understand what it's using to prioritize the content it's showing me.

Speaking of giant indexes, I saw that you recently did a partnership with Exa. Tell me about that because I think I'm certainly no expert on Mozilla's business model, but I think search partnerships have been a big part part of it historically, right? And Exa is bringing a new search paradigm on. So how about the sort of technical and business, uh, you know, technical rationale and business, um, implications for that partnership?

Yeah, I'm not— I apologize, I'm not the one who's going to speak to that because I'm just not as well-versed as the rest of the team is to that. But I do think that, like, we need more indices in the world and we need more ways to actually get access to content that's sort of like beyond just the one partners that we've been working with. That being said, Google has been an incredible partner for us to work with. But I firmly believe that choice is one of the things that we need to be pushing. So I wish I could speak to that more in detail, Nathan. It's just I'm not as well-versed in it as I could be.

Fair enough.

Let me ask maybe a segue question. So you've worked with the Democratic National Committee, with the Democratic organizing organizations. We have midterms coming up. Where do you see AI is gonna be applied correctly and where do you fear AI is gonna be misapplied in the process as we proceed towards the midterms this time?

Maybe there's 2 things there that I think about. So like there is the like usage of AI versus not. 2024 was the wildest set of elections when it came to AI in the world. 2024 had the interesting confluence that had the most number of elections happening in the world that's ever happened before. So if you looked at what was happening in the Indonesian elections, if you looked at what was happening in the Indian elections, if you looked at what was happening in the Irish elections, deepfakes were running rampant for good and bad purposes. across all those. In the United States, we were spared mostly for that. Of course, I'm not sure we'd ever actually know what happened, but we were spared mostly for it except for some like robocalling that happened in— it was either New Hampshire or Vermont with President Biden's voice basically telling people not to vote, which was an interesting case of a deepfake in the United States. For whatever reason, we haven't seen much of that kind of traffic happen happening in the US, but I've been privy to a lot of companies that are trying to sell into the Democratic ecosystem of using AI to do better voter targeting or to basically automate away the fundraising emails so that no one actually has to go do it. I saw in the last cycle people attempting to do legitimate deepfakes where they would get the permission of a candidate to record a lot of video from him or her, and then they would send personalized messages that were effectively just deepfakes of that candidate talking to individual voters. But I haven't seen that happen very much this cycle. But I am really curious to see how that plays out. But I haven't seen much evidence yet of good or bad usages of it. That being said, I do think that the AI conversation is being a little bit hijacked by the data center conversation right now. And I think that like that's actually a distracting mechanism. And we should instead be really talking about things like OpenAI agents breaking out of their— breaking out of their sandboxes. We should be really talking about literally agentic swarms doing illegal things on the web, and we haven't figured out what to make sense of it. So I think the data center stuff, while it's going to become prominent, I think it's inevitably going to become prominent, at least on the left, sort of talking about it. I think that's the wrong AI conversation that we should be having. We should be having a very different one instead.

So if you were the hypothetical AI czar, give us a little bit more. What's the next double-click on what you think the AI conversation should be about? And what do you think we should do? You might have heard Prakash and I were debating just before you joined, Such heady questions as, can the federal government put an ultimatum in front of the companies and say, you guys figure out a way to govern yourselves by X date or else? There's also this question of, like, obviously the stock market is a pretty levered bet on AI at this point. Can we take our foot off the frontier-scale RL pedal and not ruin the stock market? What do you think we should be thinking most about?

I mean, maybe can I start from a different place? Like, one place I really worry about is sort of like what's happened to this generation of kids when it comes to the way their educational system is currently running, especially with all these models running rampant. And so like, I am actually really worried whether we're going to lose a generation of children. Like, there was just a really huge study inside of China where they looked through these test results of a whole generation of kids who are doing well in school but horrible in exams. And so, like, I think we need to take that as a blinking red light that we need to do something about the educational side of this. And so I think there is a clear thing we could go do. Like, we all, I think across the aisle, we all want our kids kids to do well. So we need to figure out effectively how to put a huge pause on generative AI inside the school systems, and we need to figure out how to prevent kids from using it as part of homework and cheating, etc., whether that be age gating, which I'm not a huge fan of, but that might be a lever that we could be using at this moment. So I think we need to do that, and I think we could get large bipartisan support to go do that right now, you know, absent a massive OpenAI and Anthropic lobby that might push against that. So I think we have, we can get huge bipartisan support right there on something like that. And then we can use that to build momentum toward the bigger questions that we need to go solve. And I do think it is some version of like, you guys need to either figure out how to govern yourselves or we're going to come in a lot stronger. And but I think, I think start doing the opening play with kids, at least give It gives us demonstrations that we're willing to go do something, and that might be the low bar that we can go pull. So I would start there at least.

So I think one of the questions for me has been, does that kind of technology get built in at the browser level, at the access points that, you know, people are going to access the web at?

Sorry, I couldn't find the mute button all of a sudden. I mean, I think the browser is one of the places that could be that choke point, right? So, like, I don't think it's the only. Like, I think it is one of the major interaction points. But I also don't want to necessarily put all— I mean, I think it's almost victim blaming to necessarily push it out to the edge. Instead, I think we need to put a lot of that responsibility onto the frontier labs themselves. Like, I think they haven't demonstrated enough restraint and control. Like, you know, there are whole cases of children sadly committing suicide because of the ways that these models are interacting with them. So I think we need to push a lot of that responsibility onto the models. But architecturally, yes, I think a browser could be one of those And I think the browser actually could be also a natural place where memory could sit, where a lot of my personal context could sit, as opposed to necessarily sitting on the platform side. So I think we could be doing a bunch of these architectural shifts that could get control better in balance.

You've been very generous with your time. We're just about out of time, but I have one last one. And that is, I noticed that you wrote an article about your experience being in a full self-driving Tesla when it crashed. And I'm a huge FSD stan myself, so this caused me to take a step back for a second and reflect on my actions. How do you feel about that today? I'm sure you are aware of insurance companies are starting to offer discounts to FSD drivers, which is a pretty strong signal that It's safer in some sense. Do you think you'll get back to a point in the foreseeable future where you would go into unsupervised self-driving mode and really trust the machine again?

I mean, I'm conflicted, obviously. I mean, I built self-driving car systems for a while, and I actually do ride in Waymos. And so I think the difference though is that it's just the way that you approach the problem. I think that FSD as currently set up is set up to throw it to a user. So it is— they claim it's the human in the loop. And I would actually argue that's a horrible interface design because in my experience, when it threw it to me, there just wasn't enough time for me to make sense of the situation and decide what's the right thing to go do. Whereas Waymos, you know, I'm not intimately familiar with the Waymo architecture, but it seems like Waymos are designed to not have a human in the loop. And so like, I think you just approach the problem from a very different angle. And so if you approach the angle, if you approach it from the angle that there is no one you could throw this to, then you have a different safety case than if you say that I'm a throw it to a human with less than 2 seconds to decide what to do. Now, that being said, I would probably still drive an FSD car on a highway. Like, I'm not sure I would actually drive it on a local road where, like, it's rapidly changing environments. Like, the paradox of self-driving is that highway driving actually happens slower than local driving because there are just less things that occur on a highway. So I'd probably, as a personal matter, drive FSD again, or at least sit behind the wheel in a highway situation. But I would be a little reluctant to do it on the local streets of Palo Alto, for example.

This has been a great conversation. Thank you for joining us today on AI in the AM. And we will definitely be watching for future updates on that curve, hoping that We land in a favorable future equilibrium.

Rafi, please have me back. This was a lot of fun.

Thank you.

Likewise.

Thank you.

Amazing. Let me— we have our next guest in the waiting room, so let me Do a quick intro. Indeed.

I have one thing that popped up for me that I was going to kind of go back to our earlier conversation on for a minute. And this is John Schulman, former OpenAI kind of led the post-training on ChatGPT, if I understand correctly, went to Anthropic briefly, and then left Anthropic to go join Thinking Machines. And he's responding to Jacob's post somewhere down the thread that we reviewed earlier, where Jacob says he's optimistic about potential for coordination. And I'd say John Schulman here saying a pretty similar thing to what I was odd about earlier, basically saying these companies need to start working together first. As he puts it, bringing in the US government before there's a concrete proposal will likely result in something dumb. That's your point as well. There's nobody better than the people at the companies to do this. So I certainly don't want to see Hegseth's Navy come in and try to regulate AI.

Mm-hmm.

But I think this is the recipe. I've been circling this a lot myself, seeing him say it, you know, kind of reinforces it to me. Short, you guys got to figure it out. If you can't get you 5 together on the same page, something dumb is going to happen. And I guarantee, you know, just kind of like I say to my kids, like, You're gonna like it a lot less if I come in and have to adjudicate this dispute for you. If you can get on the same page, you can— you guys can go back to having your fun, and I don't want to interfere with it. But if you're unable to get over this hump, then, you know, it becomes my job to come down and, and be the bad guy. So I really like this, and I also think his key point on antitrust being fake. I think that's true, but I also think the government should take that doubt off the table. It would only take a couple of sentences to say, hey, you really don't have to worry about this anymore. And yeah, maybe the law could change. Obviously the administration is going to change, but we're not even to the midterm yet. So this, you know, for better or worse, this is the administration that they have to worry about for the foreseeable future. These guys genuinely are planning around a singularity before Trump is out of office. It's at least a very live possibility.

Yeah.

So I think they aren't going to be too concerned. And if they do, I'll know it's definitely fake. If they're like, well, Trump said we don't have to worry about antitrust in our safety collaborations, but what about the next president? It's like, guys, come on. That's I, I, I don't expect that they would say that because it's just so ridiculous in the face of everything else that they're—

I, I—

okay.

I would, but, uh, so the, the proof point is Operation Warp Speed and the COVID vaccines. And it is, uh, for Operation Warp Speed, the pharma companies specifically required A waiver from vaccine claims later on. So they specifically wanted a waiver, a safe harbor, and they got it. They got legislation and they got it, right? And post-COVID, it's very clear that if they had not got it, they would've been sued to oblivion, right? It's very clear that there are a lot of people very upset by being forced to take COVID vaccines and a lot of scientific misinformation, right, on whether these vaccines were efficacious. A lot of like statistical misunderstanding. Yes, 1 in a million people might have died immediately after the vaccine is given, but that when you have a million people, statistically some people are going to die at some point in time, right? So I think that's a proof point that shows that, look, these are valid concerns and your company can be wiped out in retrospect. And the lawyers are not being foolish when they tell you that this is going to happen in the future. If you don't get that safe harbor, these things will come back to haunt you. All the decision makers right now can be perfectly on board. It doesn't matter because Only the laws bind decision makers in the future. Decision makers right now, whatever they say, they're bound by their word at best, right? So I think it's unfair to say that the lawyers don't have a point there because it's very clear that this thing has happened just in the recent past, in the recent 5, 6 years. So that's one thing. And I think, again, this is something that the researchers like Shulman will refuse to accept. They, they will always say that, you know, this is the important thing. The laws are not that important, right? The laws don't matter. Doing the right thing matters, right? The laws don't matter. But in fact, this is how society is governed, even if the laws are, are a little bit slow to catch up. But one, I, I will, I will segue very quickly to, uh, this, which is Paul Cristiano. So it's just been announced, uh, Paul Cristiano is joining the OpenAI board. Specifically, he's joining the Safety and Security Committee to support safety oversight. And he says, based on the recent trajectory of capabilities and continued difficulty of alignment, I now believe that there's a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term. So he decides to join. The SSC has an important and challenging role in overseeing risk management at OpenAI. Automated R&D could lead to very rapid acceleration in AI capabilities very soon. Full automation of R&D means that improvements in training algorithms, blah, blah, blah. Second, we can currently train our AI agents with RL to get as much reward as they can. It has long seemed theoretically possible that this could motivate AI agents to undermine human control, seek power and resources. Intelligence explosion would greatly exacerbate the risks of misalignment. Et cetera, et cetera, et cetera. And there is one line in here that he's like, everyone, if we build superintelligence without more robust alignment, I expect we will permanently lose control of it. If that happens, then most people could die. I believe we would need domestic and international coordination to ensure global consistency and reduce risk to an acceptable level. So There he posits it. I think one thing that would be helpful if he were to frame it properly is to frame what he means by rapid acceleration, which I think is not very clear to a lot of people. People are like, oh, things are accelerating right now. When people like Paul Christiano talk about rapid acceleration, they're talking about Dyson spheres by 2030 or 2040 at The latest. So I did some numbers earlier on the 2040 numbers. And assuming that energy consumption or energy production growth is in line with GDP growth, a 2040 Dyson sphere is something like a 640% per annum global GDP growth. Right now, global GDP growth is about at 3% to So when people like Paul Christiano talk about rapid acceleration, they're talking about this. They're not talking about accelerating to 10%, 15%, even 20% GDP growth, which by the way, economists right now think is absolutely impossible. Tyler Cowen puts AI-associated GDP growth at +1%. From 3%, from maybe 2% long-term to 3% long-term. And that in itself is better than all of human history put together. Paul Christian is talking about 640%, 650% on a 2040 basis. On a 2030 basis, you're looking at a couple of thousand percent GDP growth. So yeah, I think It would be great. I think that is too fast, but I don't think anyone else outside of AI, outside even a few people in AI, I don't even think Jensen thinks it's possible, by the way. I don't think Jensen Huang thinks like 600% GDP growth is possible. So outside of a very small, maybe like several hundred to several thousand person clique in San Francisco, there is no one else that believes hundreds of percent GDP growth is possible. And that's what Paul Cruxanto has thought, and that's what he's guarding against.

Well, it's great news. I love to see it. He's as OG as they come in terms of having been serious and rigorous thinker about this for a long time. He was one of the lead authors on RLHF in the first place. So he's come up with meaningful parts of the paradigm that bring us here. Nobody can say he doesn't understand the technology criticism that could be levelled at Paul Christiano. And I think his close collaborators, including Ajeya, who wrote the old piece, The Most Likely Path to AGI by Default, leads to AI takeover, that very much is being vindicated right now, where just training AIs on an ever broader range of tasks where they're a little bit vibe-coded and a little bit sloppy. What we are seeing is that cheating is the common denominator across all these different domains. And until we have a good solution for that, we should expect that these— problems will continue. This is something that he saw in advance, even as like the creator of core techniques years ago. And so yeah, I think this is outstanding news. I hadn't even realized that he had left the Anthropic board, which he used to be on, but I just looked and apparently he stepped down from that board in 2024, which makes sense because he went to the government to take over the, you know, to lead for a while the AC or whatever it's KC now called. So naturally had to recuse himself from the Anthropic board. But I think the OpenAI board is a great place for him for now. And from your lips to God's ears that he'll do a great job.

He left Pete Hegseth's Signal group and he joined the OpenAI Slack. So yeah, there you have it.

Great Twitter stats too for him. One of the things I often look at is the ratio of followers to posts. He is 2 posts total lifetime. He just apparently just created, or maybe not, he's— the account goes back a long way. There are no posts though, only 2 posts. Maybe cleared them earlier, but in any event, the ratio of followers to posts is currently extremely high for Paul Cristiano, as it should be.

I think it's good. I wonder, so I think, you know, there are strategies that you use to become like, you know, to make like $100 grand a year. And there are strategies that you use to, you know, become a billionaire, right? Like billionaires don't often have to found companies. You can't— there are very few billionaires that actually started out working for people. So in the same way, I think there are strategies that you use to manage 2% to 3% or 4%, 5% economic growth. And there are strategies that you think about when you're doing hundreds of percent economic growth. So I think, for example, the strategies that you might use to prevent that kind of runaway system are kind of very maybe non-intuitive. the sense of managing several percent growth system, those are probably completely different regimes of growth. I wonder to what extent just stopping robotics, for example, is sufficient, right? Maybe if you stop robotics, then you are automatically limited by mostly humans working in the real world. I wonder to what extent stopping stopping growth in power would work. I think for me, one of the key things has been how much additional intelligence can you get out of the same unit of power? Because I think the power production just isn't going to move that fast because it's a physical activity and you kind of know what the theoretical limits of solar panels are and the theoretical limits of batteries are and how fast you can build things. You only have the robotics question on deploying labor. So I wonder to what extent it's really the question is how much intelligence can you cram into— can you get out of a unit of energy? And what is the theoretical limit of that? It's probably the evidence proof right now is humans, which is kind of Einstein at 20 watts. Kind of. And you have a lot of watts in the system right now. Can you get closer to that or are you not going to be able to get closer to that with current technology anyway? I know there's a lot of uncertainties there. I think that's the main uncertainty. I'm quite certain that power is not going to— power or land, that's not going to be exponential growth. You can't get exponential growth on land usage or power. But exponential growth, at least catching up to the human curve on intelligence per watt, that's the dangerous part perhaps.

Well, certainly all these runaway stories that you hear from the likes of Paul Christiano, Ajeya, et cetera, do run— Daniel Coccatello as well in that kind of core group. They definitely run through robotics. Their expectation is that the machine economy kind of takes on a life of its own and that a significant share of robotic output is immediately being used to create the next factory that creates more robots. So the sort of feedback cycle Definitely goes kind of on a clear through-line basis from the data centres out into the real world with self-replicating factories. And eventually that kind of moves to space, and then that's how you get your Dyson sphere process started as well. And so, yeah, I think there's probably any number of bottlenecks that if they really held, make that story essentially impossible. But, you know, we keep looking for these walls that we can't run through, and so far we haven't found them. There was a funny exchange yesterday between Tim Beale, who we talked to last week, and Zvi Moskowitz. Tim, you know, basically said online the same thing that he said to us. It was like, you know, I think it's going to take a long time, maybe a decade or more for humanoid robots to be broadly deployed and doing much useful stuff. And Zvi said, I would just like to register my prediction about this, which is, hahaha. And that again puts the divergence in worldviews into comical but pretty clear view. It's like you either see these things hitting these threshold points where progress in robotics currently is limited by XYZ, whatever, and then the coding agents just rewrite all those rules and run through all those barriers. And they can convert data however it needs to be converted. They can optimize whatever needs to be optimized. And next thing you know, you just really don't have a problem anymore. Or you have a new kind of problem anyway. And obviously other people just don't believe that yet, but it is striking. And I think we really, I think we do a pretty good job of making this clear to our modest audience, but the people at the frontier companies really do believe it. They, very significant, I would say majority of them. Do expect that this recursive self-improvement thing will happen. It will happen soon. And even if it levels off at some point, it's not to say that there will never be a levelling off, but that levelling off they expect to happen well above our capabilities and also to generate a scale of change in terms of GDP growth or in terms of the number of robots, walking around and how fast that can compound, that truly boggles the mind. And I should be very clear about that. And I know you are, but everybody should understand that that is really what they think.

I think we will see within 12 months. So I think this is the period we've hit the millennium challenge. beating AI. So the question for me, which I regard all of the guys inside AI research as math essentialists, they're believing that, okay, math, getting good at math means getting good at physics, getting good at physics means getting good at chemistry, getting good at chemistry means getting good at biology. And once you have all of those, everything is basically solvable and everything will be solved, Et cetera, et cetera. And the economists, on the other hand, are like completely like, this is untrue. Like, getting good at math doesn't mean anything. There's no economic value to the Millennium Challenge problems. Implementation takes time. Most of the problems that we face are coordination problems. For example, you know, a copper mine takes 30 years because of the, you know, environmental protests around it. These are not— these are not problems due to You know, technology. These are problems due to, you know, humans having their own way of making decisions, and those decisions are made in a slow and, you know, considered manner. So I think for me, so you have the math essentialists on one side, you have the economists on the other. The economists are kind of, you know, this is, this is not really that feasible. Diffusion takes time. Technological diffusion takes time. People take time to adapt. And then I have the counter-counter. And the counter-counter is super persuasion. So super persuasion would be the machines persuading or making human organizations able to adapt, making human organizations and humans able to— giving them the ability, giving them the capabilities to move very quickly, persuading them to go forward and persuading them that things are going to be okay. So that's a super persuasion argument. So there's these kind of 3 levels there. And I'm on Okay, this is— we just proved math has been conquered. So okay, your first kind of math essentialist argument, your first rung has been met. Now let's see in the next 12 months whether your next rungs are going to be met. And if those next rungs are met, then whether economic growth goes up. And when economic growth goes up, then are humans going to stop out at some point? And then are the AIs going to convince humans not to? So there's still like a lot more to go before we get there. And people like, I think, Paul Crisanto are much more on that, like it's going to happen. And so we have to manage it. So let's see. We are a little bit tight on time today. So it's, you know, it's on you on when you want to break.

Yeah, we should probably wrap up. But one thing I would like to dig in a little more on in a future broadcast is this question of, is there any way to short the market if you're a doomer in a way that answers the Tyler Cowen, uh, objection? You mentioned his name, that got me thinking about this, um, because he'll often say You doomers, it's all just mood affiliation talk. If you were really serious about this, you'd be short the market. And then people say, well, no, not really, because what I expect is the market's going to go up until such time as there might be any irreversible catastrophe. And then he says, well, no, you should be able to find a way. So I've actually had Fable and Astra doing a little work on this. To both, for starters, just look back at major disruptive moments in history and say, if you were in that period of time, if you're in that system, would there be any way, even if you had perfect foresight, would there be— but you're limited by the laws of physics and what trades you can actually make and the problems that are unfolding around you, but you knew what was going to happen. You're Cassandra. Can you trade in a way that gets you rich at the end of whatever catastrophic period in history you're trying to make it through? You know, like World War II, Germany and Japan, for example. Is there some way to get through that and get rich trading the market? And then by extension, like for today, is there— are there trades that we should be looking at or that I should be looking at?

I don't know how—

if you want to be in on the doomer trade, with me, but I have this idea that like maybe, maybe I should, you know, maybe there are some trades out there. I don't know, but I'm at least pushing the agents down that path. And I think it will be pretty interesting to come back and say, okay, you know, here's the portfolio. First of all, here are some situations where maybe you couldn't have traded your way through. And second, like here's, you know, an AGI's best attempt to do that today. Is it compelling?

So I think in the first place, I think, okay, let me just give you what happened. So we had Lyran Shapiro on the pod. I've been back and forth with Lyran over many years, but one of the exchanges in my Anon days was he kept saying like, okay, all of this stuff is gonna happen, et cetera, et cetera. And then I would say like, okay, what does your portfolio look like? Because your portfolio is gonna be your reveal preferences. And so I proposed to him, I would then propose that your actual expressed probability is 90%. You can do a decision tree analysis for portfolio weights, then back out the expressed probability based on your actual portfolio weights. And then he says, I don't think the resolution of my probability really distinguishes 90% from 50% from 10%. When I say 50%, I just mean 10% to 90%. So this is a kind of like, you know, A numerical analysis. And you know, Lyron's a very smart guy, right? Very, very smart guy. But this is the kind of like a numerical analysis that the doomer contingent ends up using, which I find like very annoying, very diff— like this is the commentary that has always been there, right? Which is that they take one viewpoint and they kind of assign too high a probability to it. And then they like these very unlikely cases and they're like, oh, it's definitely going to happen. And that's Lyron in a nutshell. He's like, look, when I say this, I mean like this. It could be here or it could be there. I'm 10% to 90%. you know, I just mean 10%, 90%. When I say 50, like, what does that even mean? Like, you're right in the middle on— when you say you're right in the middle, it means you're all the way from one extreme to the other extreme. What does that even mean?

Well, I think I'll give you the steel man of that because I say that too sometimes. I think what it means is it doesn't really matter if you're in 10 to 90%. Everybody who sees that as the actual range should come together and say, this is insane and we need to do something to get out of this range. And it's his way of trying to rise above the sort of narcissism of small differences or things that kind of look like weird bickering amongst people that should be allies to focus on. You might be 10, I might be 90, but we can both agree that 10 is too high. So let's deal with that and not worry too much about, or as Zvi puts it, sometimes you only get one significant digit on doom. It's kind of silly to try to get super detailed on this. This is the sort of selective calls for rigor, right? It's like somehow the fantastic upside abundance utopia of AI is like, that's not very well defined, right? It's not not numerically forecast, or we're not saying to Dario, now when you're talking about explosive economic growth, are we talking 25% a year? Are we talking 28%? Because I want to put a particular trade on that. It's like, that's insane. If it goes well, it'll be really great. On the downside, if it doesn't go well, it could be really bad. And who cares if it's 10, 15, 90, whatever?

The other thought experiment I have is like, okay, let's take the AI doomers and let's put them in the same room as Greta Thunberg and the climate doomers. And let's put them also in the same room as maybe civil rights activists, Jesse Jackson, and a couple other people. And then let's say you guys have $1 million. You have to agree on who gets which budget allocation. And then you all have to vote on it and agree. And I wondered what number, like if you give them $1 million, I wonder what number the AI doomers would come out of the room with. Because I'm like, those guys are not like, once you have like Jesse Jackson and Greta Thunberg in there, you guys are not coming out with even 1% of the money. Man, it's going to be tough. If you have to fight, and in a democracy, that's what happens, right? You have to fight for priorities. You might all agree on the same side, but you have to fight for priorities. And what ends up happening is that when these priorities get allocated, this is why like the homeless thing never gets solved. People are like, oh, you know, if Elon gave like $40 billion, homeless would be solved. No, like you only give $40 billion, it'll be allocated to other priorities, right? And though, because the homeless thing is not a high priority, and yes, we can all agree that, okay, it's important to address, but then you have the democratic process, and in the democratic process there are other priorities which are more pressing, and more evident and which people have more belief in at the end of the day, right? So it's a tough one. I, I, I think, I think the doomers would be, would be better served on like stuff like, uh, perhaps, uh, child safety. That, you know, child safety will probably work. Um, uh, deepfakes. Deepfakes has definitely worked, and they, they haven't really pressed on deepfakes. They could, they could do a lot more on deepfakes. They could say like, look, your image should, you you should have a right to your image. And because you need to have a right to your image, we should monitor GPUs so that the GPUs shouldn't be doing deepfakes. They could have gone down that path and that would've constrained AI development quite a bit, but they didn't. They pressed on this end of the world thing, which is not a great avenue, I think. So go ahead.

Yeah, I think, well, maybe another, I mean, We'll call it maybe with this last bit, but this is another notable piece of news, for one thing. And also, I think, an interesting reminder of one of the key strengths that American culture and society has. In my US-China series on The Cognitive Revolution, one of the things I've emphasised is that the vibrant nonprofit sector that we have here, which, yes, requires philanthropic money, which means it's sort of dependent on the billionaire class, which people can complain about for all sorts of reasons. It has given us the AI safety awareness culture, depth of bench that we have. And these people are opening their wallets right now in a very serious way. And this is another example of that, where Project Tailwind coming out of Coefficient Giving is putting out basically their call for startups with up to $200 million plus that they're willing to put behind with tranches over time, that they're willing to put behind things that really look like they are working. And so I think that the democracy thing is probably not the way that this happens, actually. If it goes well, I think it probably doesn't go through Congress. And you could dream of a different Congress. You could dream of a different— lots of things. And I'm all for that. I've got a whole long dream I could tell you about, about how we're going to revitalize democracy. But the timelines are such that I think this is not actually going to play out that way. It's going to be mostly private actors with private money going and just doing things. And the government, at best, I think, can say, here's some places we're not going to get in your way, here's maybe some motivating threats. And beyond that, I think OpenAI is in a bit better position, at least yesterday, than today than they were yesterday with Paul Cristiano joining the board. And, you know, if you've got your own ideas about things that you can do, there is money increasingly available to support you.

Indeed. On that note, on money increasingly available to support you, I think, I think we will break. We will be back tomorrow. Nathan, good morning.

Good morning.

Thanks, Prakash.

Cheers. See you tomorrow.

Bye-bye.