Welcome to AI in the AM: RL for EE, Oversight w/out Nationalization, & the first AI-Run Retail Store

Hello, and welcome back to The Cognitive Revolution. Or in this case, I should say, welcome to AI in the AM. This is the 3rd time that my friend Prakash Narayanan and I have done a livestream together. And this time we figured we should give it a name, and he also took the initiative to create a new look for the show, with real-time AI transcription and AI-powered comment moderation. Check out the video, and I think you'll agree that he's done a really nice job with the look and feel. As you'll see, in some ways we are still very much figuring out both what we want the show to be and how best to organize and produce it. One thing we're going to look at after this episode is creating a mechanism where we can easily signal to one another when we'd like to ask a follow-up question or move on to another topic. Nevertheless, when it comes to the quality of guests and conversations, I think this episode is right where we want to be. Our guests for this episode were Sergey Nestorinco, CEO of Quilter, which is using reinforcement learning to train AI systems to perform circuit board design, a problem with an insanely high-dimensional search space, complicated physical constraints, and relatively low volume of available training data. After that, we spoke to Andy Hall, professor of political economy at Stanford, who's doing a bunch of interesting work to characterize models' behavior in political contexts, and who's also working to design independent AI governing bodies that he hopes will allow the public to exercise some oversight oversight over AI companies without requiring nationalization. Then finally, we welcome Lukas Peterson and Axel Backlund from Andan Labs. You may know them from their autonomous vending machine work, but today we'll be talking about the new AI-operated retail store that they've recently opened on Union Street in San Francisco. The store, which is managed entirely, including the hiring of human staff, by an AI agent, currently has a 2.6-star rating.

But I personally still can't wait to visit.

For me, the big takeaway from this series of conversations is once again that the future is coming at us much faster than we can process it. Assumptions that seem safe from one perspective become very questionable in the face of increasingly powerful and autonomous AI systems. And with that in mind, I want to add just a bit to my answer to the very first question that Prakash asked me in our opening discussion. Namely, why are we now suddenly seeing violent outbursts directed at AI lab leaders? First of all, while it's certainly possible— and I would very much hope— that the recent attacks on Sam Altman's home will ultimately prove to be a random blip signifying nothing, my honest assessment is that by default we should expect to see more of this kind of thing. And not because of the super high P-doom numbers coming from the AI opposition camp, but simply due to the fact that more and more people are now becoming aware of the extreme reality of the AI situation. It wasn't that long ago that Sam, Dario, Demis, and Ilya all signed, alongside many other luminaries, a statement saying that mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. And though the record does show that each of the leading AI companies was founded with awareness of and intention to address the hard problems of AI safety— and of course the upside, it goes without saying, is unquestionably immense— in practice today, they are developing what they recognize to be destabilizing and likely dangerous technology pretty much as fast as they possibly can, while repeatedly failing to live up to their own prior safety and social commitments. In significant part because, and here I am quoting Sam Altman's immediate reflections after the Molotov cocktail incident, quote, being the one to control AGI has a, quote, ring of power dynamic to it. All while, by their own accounts, we still face anywhere from a 5 to 20% chance of something like, as Sam himself again famously put it, lights out for all of us. And the U.S. government's main concern seems to be making sure that nobody can constrain their ability to use the technology. For autonomous weapons, domestic surveillance, or anything else for that matter. I can't emphasize enough: objectively, this really is a crazy situation. Those of us who stumbled onto the idea that all this might happen years ago have had a lot of time to get accustomed to it and to position ourselves to do what we think we can about it. And many of us have reconciled ourselves to the idea that some version of it is inevitable. But that doesn't mean that we should try to tell people who are only now learning about this that they're wrong for freaking out about it. A 1 in 20 chance of human extinction is not low, and absolutely is worth freaking out about. I'm reminded of 2 movies that memorably illustrate how I think many people will respond to learning the facts about AI. In the 1998 movie Armageddon, when an asteroid is found to be on course to destroy the Earth, it's just understood that it's heroic for individuals to risk, and in the end even to sacrifice their own lives, To save the world. And that doesn't hinge on whether there's a 5% or 20% or 99.9% chance that the asteroid really will hit the Earth. The heroes would be heroes in any case. In contrast, in the more recent Don't Look Up, the main characters are just continually frustrated that nobody can be bothered to recognize the crisis at all, driving them to become crazier and more desperate until— spoiler— everyone does ultimately die in the end. I would submit that the difference between a hunger strike and an act of violence is not about how one understands the stakes or the odds, and neither is it about the impulse to martyrdom. Rather, the difference is simply that one course of action attempts to call others to a higher ethical standard, while the other is not only condemned by every principled moral tradition, but even on purely consequentialist grounds seems almost certain to To make everything harder and worse. To the AI opposition movement, which for what it's worth, I think is increasingly distinct from people focused on AI safety, I would say absolutely continue to condemn violence. But at the same time, be careful not to shy away from the fact that it's the situation that's crazy, not the people who are desperately searching for ways to make a difference. Your job, in addition to educating people about the reality as you see it, and as the lab leaders themselves have described it, is to identify and create productive ways for people to act heroically in this moment. Those could include mobilizing voters to contact officials and advocate for regulation or international treaties, investing themselves in citizen-level diplomacy with China, as I personally hope to do, developing new governance models or pursuing experimental technical alignment strategies, and importantly, probably lots more that people haven't even thought of yet. I personally always encourage people to pursue their own AI safety ideas, however eccentric they may seem, in the hope that some of them might actually pay off, and because I believe that in the absence of constructive ways to devote oneself to the cause, we will see more people simply going crazy. As always, I will welcome your feedback both on this analysis and on the new show format. Until further notice, we do intend to run these conversations on the Cognitive Revolution feed, But if it goes well, we might spin it off into its own thing. If you'd like to see that happen, we definitely encourage you to follow the new show account on Twitter, @ai_in_the_am, with underscores between each word. That's AI underscore in underscore the underscore AM. And watch out for the next livestream, which is currently planned for April 20th at 11:45 AM Eastern, 8:45 AM Pacific. Thank you to everyone who listens for being a part of the cognitive revolution.

Thank you.

And now, on with the show.

All right. And so we are live right now. We are live. This is, uh, welcome to AI AM.

Thanks, Prakash. Thanks for setting this up.

Great to be here.

I like the new look.

Yeah, it's, uh, we, we've, this is our, uh, third stream, uh, third live stream. And, uh, we decided to add a little bit of, uh, pizzazz this time.

So when I heard there was $100 million on offer for anyone who sets up a tech-focused livestream, I figured, how could I miss it?

Yeah, incentives, the power of incentives, right? And I think also we decided to make this perhaps the first livestream of its kind where we are using a lot of AI tech in here. So we have live transcriptions, which I don't think any other show has ever done before. Because accuracy has never been good enough to have live transcriptions. And this means that you can watch this in a meeting. You don't have to, you know, you don't have to like, oh, you know, I see something on screen, but I can't, I don't know what they're saying. So you can watch this in a meeting. So yeah. Well, otherwise—

Well, if AI Note-Taker is attending your meeting for you, then you can be surreptitiously watching us on the other tab with the This version of the AI transcription helping you do it live.

Indeed. We also have live comments available if anyone wants to just mention @ai_in_the_am and send us a message. It is being moderated, I think, by Groq 4.1 Fast, which is fairly fast. And yeah, so before we take off, we just had the second attack on Sam Altman's house. I don't know if you saw that. 2 people, it seems they shot around at his house on Russian Hill. And it's pretty scary. You know, his family's in there. You know, at this point, at this point, I wonder if it's just, you have to move out, right? You can't be in SF anymore. You have to be in like a defensible position. And I feel like you're— I've heard from one VC actually that his house is in one of these suburbs, in Menlo Park or wherever else, and he has drone defenses set up. So there's drones kind of circulating above the house and he's got drone defenses. I think it's very hard because I think the level of security which is gonna be required for an AI company head right now is, and I imagine it's the same for Elon, I imagine it's the same for Zuck, you know, they're not well-liked. And you know, why do you wonder that? Is there like, there's a lot of soul searching going on on the timeline right now. Why do you think this is happening right now?

Well, I think it's getting very real. That's one thing. I mean, everybody can now see that, or maybe not everybody. I think there's a few holdouts, but increasingly it's hard to hold out any sort of, they're just doing this for hype and to raise money and there's nothing really there. So I think increasingly people have to reckon with the fact that AI is getting powerful. And my guess is that, who knows? It's hard to put yourself in the mindset of somebody who would go, throw a Molotov cocktail or randomly do a drive-by shooting of someone's home. But Mythos is really an example of at least a weakly powerful AI, right? I mean, it's something where no less than Nicholas Carlini, who is by all accounts one of the great cybersecurity researchers of all time, has said that he's found as many important vulnerabilities in just the last few weeks as he had the entire rest of his career combined. So that is a huge indicator that we are entering a new regime. And I do think it's becoming real to a lot of people in a lot of different ways. And it is a radicalising reality, I think. And it's not to be forgotten, right, that all of the AI lab leaders have been pretty candid, if not super recently, at least at various points in time, that they're really not sure how this is going to go. And at least have some nontrivial p-doom percentage even at the top of these organisations. So I think it's rational in some sense to be like, what the hell are you guys doing? And this needs to be stopped. I think that position is, in my mind, very defensible. And then obviously, in addition to just being wrong, I don't think it's going to be effective tactics to try to intimidate these folks, because I do think their resolve will probably just be hardened. Their ability to defend themselves with the resources that they have is going to be pretty good, if only by retreating to some large estate in a private island in Hawaii or in New Zealand or whatever the case may be. So I don't think this stuff is going to work. But there is some— I think it's not totally crazy to say desperate times call for desperate measures, and then it just becomes like, what desperate measures are acceptable and/or likely to be effective. And I think I do want to be clear that I do not support these things. I think they're clearly crossing lines. But there is some sense to the idea that these guys are telling us that they're taking, by their own lights, right, they're taking something like a 1 in 10, 1 in 5 chance with the future going like deeply off the rails. They don't really have a great account of how they're going to control things. Alignment is obviously unsolved. Governance is unsolved, as a preview of our upcoming conversation. And yet we race ahead, right? And I mean, I was also really struck by, in Sam Altman's kind of off-the-cuff blog post that he wrote about this, his Again, candor. I mean, it's, you know, obviously he's been accused of being inconsistently candid. I found him quite candid in that moment, and his grappling with the idea that this whole dynamic has a sort of ring, uh, and who's going to control the ring and the sort of, you know, intoxication and potentially corrupting nature of power. That was really a striking response, right? Because I think that is actually a lot of what the most radical voices in the AI discourse are responding to. He's acknowledging exactly what their most, their sort of sharpest critique is, right? I mean, I think if you listen to the AI safety people, again, the most strident voices, they're like, these guys are trying to get the ring. They want to be the one that gets all the power and wins in the end, and they don't really mind taking a totally irresponsible gamble with the rest of humanity's future to do that. And he kind of said the quiet part out loud in the blog post about it. Yeah, I would definitely like to see a more sane and productive response than these things. But unfortunately, I do think barring that, barring any sort of government action that makes any sense, we're probably going to see more of this kind of stuff. And I do think in their candour, they've set the table for some extremism anyway. And that's not to endorse doing any of these things, but the mindset, I think, certainly is one that I can understand how people get into, especially also if they don't understand the technology.

Right.

I mean, there's a lot of people who are coming to this much more suddenly as it becomes a bigger deal, and they're just kind of like having a sort of sudden awakening to the fact that this is all going on and it's You know, and it really is super powerful. And maybe what they heard about it being all hype before was like, actually not right. You know, I think that can be destabilizing to a lot of people. Dude, I remember test red teaming GPT-4. It was interesting to me, you know, in a, in a minor way. So I can only imagine what people who, you know, are just now coming on to modern AI developments are, are feeling and thinking.

On, on, on that note, let me introduce our guest, our first guest for this morning, Sergey Nesterenko. He's the founder and CEO of Quilter. They're a company that's really trying to speed up electronics design in general and PCB design in particular. They have a physics-driven AI, which I think they use reinforcement learning in order to really speed up the entire PCB layout process, which can take many weeks and they've managed to compress it down. He came out of SpaceX. He was in avionics, I think, avionics and radiation. And I imagine, you know, it took me a lot, you know, I heard about avionics growing up and I always imagined it was very sophisticated stuff. And then I realized later on, it's a lot of compute. It's actually a lot of compute. It's a lot about figuring out what numbers need to happen. And in the past, a lot of that stuff was not electronic. There was a lot of gadgetry, which was actually mechanical. And you had mechanical ways of calculating all of these things. which is why avionics used to be this entire segment of creating mechanical computers that could work on F-15s and F-16s. And later on, it just became electronics, but the electronics have to be hardened to radiation and a bunch of other stuff that happens. And Sergey is an expert on that. Sergey, welcome to the show.

Yeah, thank you for having me, guys.

I hope I didn't misstate all of those things. Give us an example of how you transitioned from SpaceX to PCB design. What did you carry forward from SpaceX into your new role?

Yeah, I mean, there's a lot to carry forward from SpaceX, to be honest. Like, you know, I mean, this time I spent there was awesome, spent about 5 years there. There's a lot to learn about culture, a lot to learn about how to hire great people, a lot to learn about how to design systems, so on and so forth. But I think the most important thing is just speed, right? Like SpaceX, probably the most thing that it's famous for is like, Just, you know, hardware-rich development in a sense, right? Like, just try it, just build it, go launch it. It'll blow up a couple of times. That's fine. We'll learn. And that's way better than analysis paralysis, right? And I think that's the case for a lot of companies in a lot of places, right? That you can kind of really overthink a design, but once you kind of put it to the test, you find out what not to worry about and what to worry about. And kind of the physics is the real ultimate guide. You can't find that out until you actually run it.

So speaking of physics, prior to Quilter, I think people used to do this thing called auto-routing, and then Quilter came along. What is the difference between the prior paradigm and what you guys are doing now?

Yeah, totally. So auto-routers have actually existed for more than 60 years, right? Like, if you dig back through the literature of PCB design, all the way back in like 1961, 1962, you started to see publications on like, hey, we're making these circuit board things, is super laborious at the time. This is when you're doing it with, I mean, not quite pen and paper, but just about, right? Like you're doing this without CAD software. You're making masks out of tape by hand, that sort of thing. And mathematicians were already studying like, how do we solve this, right? And initially this was thought to be a graph embedding problem that didn't work. Then people started doing basic pathfinding algorithms like Lee's algorithm and, you know, A* and like that family of things that didn't work, went into like topological routers that didn't exactly work and so on and so on and so on and so forth. So I think to say that people have been using auto-routers, frankly, is kind of an overstatement. Like if you go and talk to an average electrical engineer and ask them if they use an auto-router, the answer is plainly no, because it's just not good enough, right? It's just not helpful. And there's cases, right? Don't get me wrong. There are some people who use them for certain things and in certain parts of the board and whatever, but like it's nothing like the chip industry where you have billions of transistors and, you know, you can actually genuinely place and route like a vast majority of that and it wouldn't be possible without humans. That just never happened in PCB design. So the real kind of challenge for Quilter is, can we make the first set of placement algorithms that people actually want to use? It's not really competing with the old auto-routers, it's competing with the manual labor that still happens in every hardware company on Earth.

Hey, we'll continue our interview in a moment after a word from our sponsors.

So one of the questions I had is, you guys use a lot of reinforcement learning And what is your reward process for that? How do you reward the agent? Do you run simulations? Do you design the environments? How does that process end-to-end work? Because PCB routing is not a generalist task. It's a pretty specialist task. How do you figure out what the reward signals are? How do you create the data?

Yeah, and just to state the kind of plainly obvious, but nevertheless, like, this isn't the problem that you can just like prompt ChatGPT to do and like it does it for you, right? That's, as you're kind of alluding, this is not a generalist kind of problem. The large language models are not trained in these kinds of problems. And furthermore, it's arguable that like language is not the right approach to a geometry and physics problem, right? So this is why we've kind of had to take our own path, construct our own environments, and we see it as a reinforcement learning problem. So we actually spend a lot of our time, if not most of our time, on constructing a good environment and constructing a good reward function, because that turns out to be really hard. Naively, you might think, let's take a naive RL algorithm like PPO or something, give it access to a keyboard and mouse of an open-source CAD software like KiCad, and go learn. The reality is that I don't think reinforcement learning as a technology is ready for something That is just very hard. It would have to get millions and millions of actions right in sequence with perfect precision where traces are side by side with no extra margin and just no way. At least I practically don't see a way today. So there's 2 things you want to do. One is you want to construct an environment that only gives uniquely useful actions to the agent. So to make this very concrete, there's maybe 10,000 different ways you can draw a trace through a board. And there's very minute details of exactly where does every elbow go and so on and so forth, right? But realistically, as a human, you're not thinking about every single detail like that when you're planning the board. You're thinking kind of topologically, right? So you're thinking like, am I going to go clockwise around this chip or am I going to go counterclockwise around this chip? And like that sort of binary choice is more important at that stage than like every minute detail of every single segment, right? And so that's like an example of something that our environment does is like, how do we break this down into like the key important high-level choices to present to the agent rather than every explicit detail. Now, the second part that you asked about is the reward function, right? The reward function has to be fast, right, for RL to kind of have any hope. And so the way we think about this is, and for humans too, there's like 3 tiers of physics approximations you might do, right? Because no simulator is perfect, but generally you want to approach reality from the side of conservatism. The first level that most humans do is they actually just compute pure geometry. So there's rules of thumb, right? Like if you're worried about 2 wires that might crosstalk, they might influence each other because they are effectively antennas and they contaminate each other. The basic rule that people will follow is if I can make them 5 times as far apart as the width of either trace, I'm good, right? That's just geometry. That's very cheap to compute, right? The next level of that calculation would be called the quasi-static approximation, where you take the Maxwell equations, ignore the time factor and compute the parasitic capacitance and mutual inductance between those. It's basically a physics simulation, right? You do a mesh, you do finite elements, but it's very fast, right? And that would be level 2 in our opinion. Level 3 would be full wave, right? Like run a full wave simulation, finite difference time domain or FEM, considering for time and get the most realistic answer. And so at Quilter, the way we see it is let's first nail what humans do, right? Like just pure geometry and make sure it's conservative to reality. Then now we're starting to step into quasi-static and those kinds of fast approximations, 2D cross-sections and whatnot. And then eventually we'll come back to full wave where it's necessary, right? But full wave is very expensive. And I mean expensive in terms of wall clock time.

So it's like the first step is really heuristics, kind of learned rules. And the second step is a fast calculation. And the last step is a much more detailed calculation. And if any of those hurdles doesn't pass, it fails. And that's how you give the RL agent its reward signal. And that's it.

Yeah. The only way I'd amend that is on the first step, ideally you don't want a heuristic that can have a false positive, right? So what you really want to do is you want to be conservative. So like the 5-width rule, for example, yeah, it's just geometry. It's very basic stuff, but it's overkill. And so what you're really doing is that you're making your board too big, too expensive, you're leaving too much margin, right? Like you can eventually as a human delete that margin or with better calculations. So you don't want something that falters because getting a board back from fab that doesn't work is really, really, really painful. You want something that is overly conservative and then with more detailed simulations, you bite down the conservatism with more accuracy.

Indeed. How many, Like how much compute do you have to do? Like how many environments do you need to construct in order to get to where you are today?

Yeah, a lot. So we break the problem up into kind of multiple stages, right? So like we treat, you know, the first problem before you even get to routing, right? So routing for those unfamiliar is I've got components in the board, the components have these little connection points called pins, and I'm going to draw wires between them that can't collide, can't overlap, so on and so forth. Before you even get there, you have to put the components on the board, right? So the first problem is actually typically, well, even before that, you might choose what is the shape of my board? What are the vertical layers of my board? Where do I have ground planes? So on and so forth. That's problem 1. Problem 2 is where do I put the components, right? There's kind of a floor level, floor planning problem, and then a detailed component placement problem. Then you get into your initial routing and topology selection. Then you get into your geometry fine-tuning. And so for now, we actually split each one of those up and treat them independently. And it depends on the problem, right? It turns out that with placement, you can get environments that run really, really fast, right? You can sort of vectorize it and just throw a GPU at it and have very, very, very fast environments. If you guys are in the world of reinforcement learning, if you're familiar with PufferLib, a great library that's coming out that's going really, really, really fast reinforcement learning, That's a good inspiration for that. In the routing stage, that doesn't quite work as well because the routing stage is so much more complex. You can't quite afford as many environments, right? So you have to explore subsets of a given environment rather than go wide and run a million totally different routings in parallel.

Have you seen outcomes which a human wouldn't do? Like, you see a layout which a human would not do, but The optimization chooses to do it and then it works?

Yes, generally yes. Now, it's not always a good thing. For now, to be very plain, we're not at the point where we're beating humans. We're just at the point where we can take a task that takes a human 2, 3, 4 weeks or 10 weeks in an extreme case and cut that down by a factor of 10. But we're not at the point where we say, human, don't worry about it, we got you and we'll do it better than you. I think that's still a ways away, right? So sometimes when it does things that are surprising, it's not a good thing. It's a bad thing. Sometimes it can be a good thing. Examples of good things, I would say there's some intentional and some unintentional. There was an initial lesson that we had where if you think about the way that wires should be drawn on a board, you actually realize that since they are transmission lines for waves, they should be curved, right? So if you think about like, like laminar flow of a wave, you should have smooth turns like the Amazon River.

Right.

of how wires should go between places, because ultimately electromagnetic signals are waves, right? But if you look at any circuit board today, you look at a motherboard or anything, you see these what are called octilinear traces. So left to right, 45 degrees, up and down kind of thing. And that actually has a purely historical context, right? It's because CAD software was slow in the '80s, it was cheaper to compute intersections for collinear segments. And so that's how CAD was built and we got used to it. And we thought, well, you know, it's 2025, like 2026, let's get past that. Let's make curvy traces. And when I first showed that to electrical engineers, like, let's just say, to put it mildly, the reaction was negative. Very, very, very negative. And so, and I've tried to make the argument of like, no, no, think about the physics and like, you know, the most intense like RF and high-speed boards out there do this and data center cards do this. And like, it kind of clicks, but like people are so not used to it that they're not even sure if it can be manufactured. Which of course it can at no extra costs, but that's not obvious.

Right.

And so that's something we intentionally did at first that turned out to be better, but much worse in the user's eyes. And now we like post-process that out specifically to avoid that reaction.

Right.

A more, I might say, emergent property, right, is humans will really like symmetry.

Right.

So as a human, when you place a chip and you place like the capacitors next to, for example, you line them all up perfectly and they're very, very neat and it's very pretty, right? Like there's a reason that electrical engineers call this job artwork, right? That's like literally what you call a layout. But if you think about it, if you're trying to minimize the parasitics to every capacitor, they should just minimize that distance, right? And that's not going to be symmetric. They're going to maybe form a little semicircle and some things are going to be a little off and whatever. And so you might actually get something that is better from a parasitics perspective that breaks that symmetry and feels worse, right? And it's an example of something that a human wouldn't do.

One thing that I recently learned, I think François Chollet, he put it out yesterday, is that we are highly tuned to symmetry because it's a form of compression. We get to compress a lot of information when we just assume it's symmetric. And I can imagine that might be useful in debugging, perhaps?

I think François's point was about physics, right? And it turns out that if you By virtue of having symmetry in a physical system, you get conservation laws, right? Like through Noether's theorems. And so there's some kind of like deep truth to the physics of symmetry. I think in PCB design, I don't know that symmetry actually— like you need some readability for sure. As you look at a board, when you have it on your desk, you need to recognize what every component is. It needs to flow kind of like left to right from inputs to outputs. It certainly needs to have logic to it. But I don't actually know that symmetry really helps debugging, for example. I'll give a counterexample. Where symmetry is very, very helpful is if you have, for example, 10 sensor channels that need to read an identical reading, right? Like we have some very sensitive analog reading. It needs to be identical across those 10. You want symmetry because any imperfections you have, you want those imperfections to be identical in every channel, right? Like in that case, I truly understand the need for symmetry from a kind of a physics and debuggability perspective. and so on and so forth. But around more basic functions of the board, I think it's a human's way of expressing the care they put into that board more than anything.

How much feedback do you get from the real world? You have these simulation environments. In some sense, I think some people who are working on, let's say, material science using AI, using reinforcement learning, they have a loop which includes a physical kind of wet lab kind of loop, and then they test that, and then they use that data to feed back in. How much of what you do has that kind of physical process that is necessary or data collection that comes back to refine the model?

Yeah, we only do that indirectly, right? For what it's worth, I love that idea of having AI generate a research plan, wet lab automates it, gives you feedback, and you learn about it on the physics of the real world. That's so And like maybe there's some version of that that could work for PCB, but probably not nearly as automated as the kind of wet labs could do. That would be quite the feat. But I think that for us, what's important in the way that we approach this is that building in real life, which we do, validates whether or not the approximations and simulations we have are correct, right? So we have the luxury that we can afford to have simulations that are kind of known conservative. And the real question is just like, how much margin do we have? And is it way too much or is it right on the border? And building in real life can validate that, right? So I don't think we're in a place where we can just automate build, feedback, you know, thousands or tens of thousands or hundreds of thousands of boards and like directly learn that signal, but we can use it to fine-tune and make sure that our simulations are right and then use those to then feed into the learning process.

Right on. So you design with a margin of safety large enough that the boards will be producible, but then you can go back and recheck the actual physical board to kind of refine the margin of safety that you've been using prior.

Yeah, exactly. And echoing back to the conversation about learning from SpaceX, my job, as you mentioned, was to make sure that Falcon 9 and Falcon Heavy could survive heavy radiation environments. So protons, electrons beating up electronics and making sure that it would actually kind of work well. And in that job, you don't just launch a Falcon 9 10,000 times and I mean, maybe, maybe soon you will be able to, right? But like when Falcon 9 had flown a handful of times, that wasn't an option. And so we did exactly that, right? Like you simulate, you understand the physics of what's happening, and you approach truth from the side of conservatism. And in general, a lot of my job was, it was very easy to make a very conservative calculation about what it would take to survive a Van Allen belt blast. But then that forces the rest of SpaceX to do a lot of work. Right? It forces part choices. It forces new parts. It forces subcircuits. It forces software interventions. It forces potentially shielding, which was a really, really expensive option. And so what then I had to do is refine those calculations to take away the margin, to not make the rest of the team do too much work and approach truth from the side of conservatism, right? I view this very much the same way that like you can approach the reality from the side of conservatism. And what you cost is boards that are, you know, a bit too big, a bit too expensive. Which in the R&D process is perfectly okay.

What would you say is the cost saving that a typical consumer product would have from using Quilter versus the prior technologies?

Yeah, the absolute main thing that we focus on now with our customers is speed, right? So we are not at the point where you're going to take an off-the-shelf consumer product and design the mainboard that's going to be manufactured millions of times with Quilter's help, right? That we don't view that ourselves as a good application at this point. But for every board that ships into production and you make a million of, you actually make hundreds of boards that precede it, right? Like every part that goes into that board is going to get its own little board for your team to test and double-check and write software for and iterate on. Every subcircuit is going to get its own board to validate. You know, there's examples of even things like phones. before you make the final board that fits into the phone, you make a giant board that's like this big. And then the reason you do that is that it has all the individual pieces of a phone broken out. You have your little camera, you have your microphone, you have your speaker, all those things. And then you can swap them and say, well, what happens if we go to this camera? Or what happens if we go to this camera? So Quilter helps with all of those, right?

Mm-hmm.

That's where we can step in and make that faster. And the thing is, is that like whether it's a production board or one of those test boards, it's still gonna take 3 weeks or 4 weeks or 5 weeks or 10 weeks to make. Right. And so as you kind of iterate on 10 different levels of going from initial idea to the production board, and each of those cycles takes 5, 6, 7, 8, 9, 10 weeks, and they're sequential and you can compress them, that's what makes it hard to build a hardware product, right? That's what makes it 2, 3 years to get a new product out at all. And so what maybe a consumer would see from Quilter's involvement is much faster iteration cycles, therefore giving engineers much more ability to test and much more ability to get to a good product really, really fast. That's what's important for us now.

Amazing.

Hey, we'll continue our interview in a moment after a word from our sponsors. We got to do one Mythos question, so let me sneak one in. I guess my working vision for how superintelligence comes together is sort of a convergent process where a core reasoning engine, which I think the Mythos, uh, not release, but informing of the public, certainly suggests we're still in the steep part of the S-curve there. And we see open math problems being solved and some minor but new results in physics being derived, so on and so forth. I imagine that kind of coming together with what I think of as native senses that AI, broadly defined, can develop in all these different domains. And I kind of understand what you're doing as developing the sort of native sense of PCB board understanding and design. But I wonder how you think about those things coming together. Do you see, like, are you designing for a future where Mythos or its successors becomes your user and you still have this model that kind of can do something in a native, intuitive heuristic? Not heuristic as in coded heuristic, but heuristic as in intuitive heuristic way that the reasoning models, you know, still won't be able to access? Or do you have a different vision for how you kind of interact long-term with the reasoning line of work?

Sure.

Yeah.

I mean, there's kind of 2 answers to that, right? There's kind of my short-term view and a long-term view. In the short term, I think that, you know, you have to realize that people who are building hardware and circuit boards are not in the same world as software engineers, right? They're not in the world where Every two days, a new model drops that has some like amazing agentic properties. They're not hooking up OpenClaw to you know whatever they're trying to do at the moment. You know maybe you're starting to see that in firmware to an extent, but like not for designing schematics, not for designing boards, not for debugging boards, not for hooking up boards to your oscilloscope. You know not for any of that stuff. Very practically, as a startup, we have to focus really, really, really hard, and to focus really hard, we have to listen to our customers and give them something useful today. And I just don't see a single one of our customers. or prospects talking about like, hey, we have like a central reasoning thing and it's gonna negotiate with a bunch of AIs and that kind of stuff, right? So practically speaking, today I'm spending zero time on that, right? I'm giving something that fits into the existing workflow of an electrical engineer from a very practical perspective of they manually draw their schematics, they manually draw their boards, they manually send to fab, they spend 2 weeks on the phone with the fab arguing to go faster and to discuss the errors and whatever else, right? And I wanna just give them something now to make a part of that easier.

Right.

Now, long-term, I've thought about this to an extent, and what I imagine happening is a bit of what, I mean, frankly happens between humans, right? Like you imagine, you know, taking SpaceX as an example, you have some mission you want to fly, something you want to build, and you get a whole bunch of different teams coming together to talk about that problem, right? You have your PCB designer making the board, great. You have your thermal analysis folks who are telling you how much it's going to heat up and how much it's going to dissipate, radiate. You've got people who are dealing with like material properties. What's it going to do in vacuum? Is anything going to outgas? You've got the mechanical folks dealing with the mass of the box you're building. You've got the flight control team saying, we need this kind of sensor speed and this resolution. You've got the flight software team saying, we need this fast of a processor. So you've got these 10, 20, 30 teams of people who are injecting their requirements into a single thing that has to satisfy all of them. And inevitably, there is a conflict. Inevitably, there's a, well, to give you this, I have to give up that. And that's where everybody kind of sharpens their pencils. tightens the margin and tries to come to a compromise. And so I do see a world where, you know, every one of these teams has some sort of kind of agentic representation, right? Quilter being the PCB design one, there's going to be something for schematic, there's going to be something for mechanical, there's going to be something for thermal, there's going to be something for software, there already is. And like maybe in common language, those systems can kind of negotiate and then present to us humans like the trade space of like, here's what happens if we over-optimize for this, here's what happens if we over-optimize for that. And then where would you like a suit to go. I just don't see that happening in hardware kind of in the next couple of years, to be honest.

Maybe one last question from me. Hardware is, especially EE, I'm a EE too. EE people tend to be a little bit like, not conservative, but they have very predefined ideas of what works because there's a lot of like things that work theoretically, but they don't work in practice. And people kind of learn this stuff as they apprentice and as they, as in their working life. And it's a lot of it is tacit knowledge. It's not very well documented. You get like an old dude coming in saying like, hey, you know, that's not going to work. You're going to get some, you know, crosstalk and, you know, you have to change it. How does that work when you have a product which is much more scientific in that sense and figuring these things out as they're actually supposed to happen. How do you deal with the old timers in the field when they have all of this resistance?

Yeah, yeah, it's an important question. And I'm sure that electrical engineering is not the only domain in which that happens, but that is acutely true. Look, I mean, at the end of the day, that viewpoint on life comes from past experience of being burned, sometimes literally. Like the first board I ever made caught fire. And I learned the hard way, right? Like, you know, like not to make the mistake that I made in that case, right? And the old hardened, you know, gray beard double E's have like 30 of those lessons, right? And so they're very conservative. At the end of the day, I think there's 2 things, right? First of all, trust is critical, right? So when we talk to customers, like we're very open about what Quilter does, what it doesn't do. We're very open about exactly how it works. We're very, in our product, we make an explicit list of Exactly the metrics we check, exactly to what level we met them or did not meet them. And then the double E knows, oh, you're checking for these things. I'm good with that, but you're not checking for this thing. So I have to go pay attention to that part of the board, right? So transparency is really, really critical. But from a long-term perspective, how do we eventually get to the point where they really hand off our trust and it becomes kind of like a compiler for hardware? You have to have way better simulations than people in this industry have ever had. So from the perspective of a circuit board, the bare circuit board, ignoring the components on it, it has a contract, right? Its job is to faithfully implement the intent of the schematic. And every single transmission line on there has some S-parameters that are deviating from the ideal transmission line. It has crosstalk, S21, it has some insertion loss, it has all these sorts of things, right? And the question is, can you enumerate all of those and prove that all of them are below the required threshold, right? And I think that is a fundamentally computable problem, right? It's just Maxwell equations. It's just so laborious to do that nobody does it today, right? There is no like drag and drop your PCB here, we run all the simulations and guarantee your board works, right? But we kind of have to build that to get to real true full automation.

Awesome, Sergey. I'm gonna thank you so much for joining us today, and we hope to see you again one day. Awesome.

Thank you for having me.

Bye-bye.

Thank you.

So Andy, I'd like to introduce Andy Hall. He is a professor at the Stanford GSB. And one of the most interesting things is he's a professor of, I think, political economy, if I get that right. And he has put forward, he's been evaluating models for authoritarianism. So that's been interesting. And he's also, he has this concept of the AI firms being enlightened absolutists, and I'll let him explain what that means.

Absolutely. Super excited to be here. Yeah, enlightened absolutists. So I think the major frontier lab companies are in a position, whether they want to be or not, where their technology is so important That it leads them to have to make a bunch of really hard calls about how it can be used, for example, how it will answer difficult questions, when it will refuse to do things, and so forth. So far, I think the companies have demonstrated a lot of hard, earnest thought about how to do that, which we're very fortunate that they're doing. But no matter how thoughtful they are about it, They can't escape the fact that they're essentially making all these decisions unilaterally. And so in particular, when I talked about enlightened absolutists, it was a little bit tongue-in-cheek critique of Anthropic's so-called Constitution for Claude. And I actually think that document, for people who've read it, I mean, it's very, very thoughtful. And it basically lays out, here's what we want Claude's values to be. And here, for example, are things we don't ever want Claude to be allowed to do. And some of those things include helping a government do something malicious to surveil or suppress us. It has lots of other things in it as well. And the other companies, to, to greater or lesser extents, have released similar documents. And my point in Enlightened Absolutists was, this is very nice, and it's great that the companies are doing this because we need serious thinking like this. But no matter how well-written those documents are, they can't really rise to the level of constitutions because you can't just say things and hope that they'll stick in the future. And so just to give you an example of that, all 3 companies, all 3 leading frontier labs have already altered the stated rules around their models several times. So Anthropic has gone back on certain safety commitments for understandable reasons. But the point is, those commitments weren't much of a commitment if you can just change them whenever you want later. Similarly, Google had released some principles that they somewhat quietly pulled back later, and OpenAI has done similar. We should expect that they're going to need to change things over time. This is a very fast-moving and evolved, rapidly evolving situation. But if we want them to be able to say things like, no one's going to use our model to surveil or suppress us, And if they're going to use those documents as part of how they're going to argue that they're doing a good job, then we're going to want those documents to have a little bit more sticking power. And if they're going to— especially if they're going to call them constitutions, then we're going to want them to look like actual constitutions. And we have thousands of years of trying to write constitutions to pull off precisely this sort of magic trick where you figure out a way to tie your own hands. And to make the Constitution more than just a so-called parchment barrier, but actually a meaningful binding authority that doesn't just say, oh, in the future we're not going to do this, but actually lays out if we were to do this, here are the specific ways that we would be in violation, and here are the consequences of being in violation, and here's the design of a governance structure that will make sure we can never do that. So that's, that's sort of the idea.

I, it strikes me that even quite authoritarian countries have constitutions. For example, China has a constitution too, and they have a basic law, and the basic law says freedom of expression and all of these wonderful things. And in practice, it's what the Communist Party wants. And, you know, you, you don't really have another, another way to, another interpretation. Besides whatever the party wants to interpret at that specific time. And it seems like that— so that idea of the constitution is actually a kind of living document which gets reinterpreted by people over time, by institutions. And it's really the quality of the institutions which are implementing or adjudicating the constitution that are really important. And it seems to some extent that Claude's constitution is going to be adjudicated interpreted by Claude itself and adjudicated by Anthropic in some sense, at least for now. How does that need to shift in order for things to work?

It's a great question. It's a timeless question. Here's a few things that I would say. First of all, I completely agree with your premise. There are many, many constitutions. In fact, the vast majority of constitutions across time have at least 2 failures to them. One, they may not actually specify in them things that we would want from the perspective of so-called liberal democracy in the non-left-wing, right-wing use of the word liberal. And second, the vast majority of them have no staying power. There's some great work in political science. I think the median survival time of a constitution is something like 2 years. It's very short. I'm making up that number, but we can look up the real number later. And so you both need to make sure that this document contains the right things, but also you need to pull off this magic trick that they actually become sticky. I'll just give you one example for the online world of where a constitution has proven itself to be sticky. And that would be, I think, Bitcoin. I'm not going to go deep on sort of like the inner workings of crypto, but whatever you think of Bitcoin, one thing that's very, very interesting about Bitcoin is there's a set of rules baked into it. Pretty early on in Bitcoin's history, there was a big movement to change the rules, in particular, to increase what's called the block size. There were a bunch of hardcore people who said, you know what, no, if we change the block size, then we're opening the door to changing other things about Bitcoin, and it won't be immutable anymore. They actually won. It established a precedent that these are rules that have some staying power. I think we'll need something like the same for AI models. Where more than the company is involved in writing down the rules, but then also, there will need to be some important stressor, like there was in the block size war, where the company and the people around the company that are involved in this governance process do something costly and difficult that proves that they're really going to stick to their rules. In general, that's a very, very important part of what we call credible commitment in the social sciences is you have to make this thing so binding that you can prove even in cases where you'd really like to get around the rules and change them, you can't. And right now, for all of its nice features, the Anthropic Constitution certainly doesn't rise to that level.

Indeed. Seguing here, there is internally, I think, within Anthropic and in the community as a whole, there is a fear of these models being used for political persuasion and specifically for approaching voters with very persuasive arguments, potentially robocalling, potentially even video conversations. We've already seen a little bit of deepfaking of voices of various politicians and some by the campaigns themselves. You know, it's become okay in the political discourse, at least to use your own candidate's voice.

Should the—

I feel like political usage is a— should be an allowed use, but are there certain guardrails that should be put in place? Are these models in some sense too powerful to, you know, and this is what the companies always say, too powerful to be used, you know, for politics or something else. Like, is that an allowed use case? Should that be?

I'll—

okay, let me separate that into 2 parts. It's a super interesting question. One part is the use of the models for intentional deception through the creation of deepfakes and things like that. And every election cycle we worry about that. We keep saying like, this is going to be the year where the deepfakes really proliferate. And I've honestly been super surprised by how that hasn't played out yet. In fact, and I keep posting about this because it's so surprising to me, instead of seeing a flood of straight-up fake content about that's meant to trick you into thinking it's real. What we've seen is the parties, but especially the Republican Party, has been out in front on this strategy. They're using deepfakes in a satirical or emotionally evocative way where you're not meant to think it's real. In fact, it basically tells you most of the time that they're fake, uh, but they're meant to evoke a sort of like, this is what the world is going look like if X, Y, or Z thing happens. So they had a very interesting case recently where the Texas Senate nominee, James Talarico, uh, the Democrat, they took some old tweets of his, real genuine quotes of his, and created a deepfake video of him reading the tweets. And he never read the tweets on video, but they are his real words. And so it's not really meant to— it's not lying in some sense about the content, But it's much more evocative than if they just read the tweets out. It made it feel really real to people. And I think we'll see a lot more innovation like that. Why are we not seeing more straight-up fake content? I think it's some mix of— it's still relatively easy to get caught if you do that, and the consequences of being caught are not great politically. But also, I think they think it's not that effective in the sense that Persuading people is pretty hard, and Americans are pretty stubborn, and Americans are pretty skeptical of video content. There's already a lot of, like, people trying to figure out, is this real? Is this not real? So we haven't yet seen that play out of, like, the straight-up fake content. We may still, and I think we need to keep our guard up for it. Even if we don't see it, it leads to this other problem, which is the so-called liar's dividend.

Mm-hmm.

where you can pretend something was fake even if it's real. So it's eroding our ability to use video to expose scandals and things like that because the person could just say, oh, it's a fake video. But again, I don't know, we're not seeing a ton of that yet. The second part of your question is much broader and is sort of like, is this a potent new way to persuade people of things? And we've seen some recent published research where you do experiments where you have people talk with AI versus consume other kinds of information. And it does seem like the AI is more persuasive, but it's not at all clear yet. And no one has really established that the sense in which it's persuasive is bad, in the sense that like it can persuade you of whatever, versus like it actually informs you and that causes you to be persuaded but in a good way because you've learned something. There hasn't really been a, uh, compelling proof that it's sort of like moving people's attitudes around in whatever way some nefarious actor would like. Honestly, I'm pretty skeptical that we will ever get that kind of proof because we haven't seen that with any past technology. In fact, I think the biggest risk in the discussion around political persuasion will be just like it was with social media, that some fraudsters like Cambridge Analytica will claim they're able to persuade large numbers of people, even though they can't. And the Cambridge Analytica stuff, like, if you step back, was really crazy. There was basically nothing to it. It was like the underlying technology was like an Excel spreadsheet coded up by someone with a teenager's level knowledge of Excel. And yet they got an unbelievable amount of credulous news coverage that they had hacked the American electorate's brains and stuff. There's never any evidence for it. We have looked for persuasive effects of social media forever. You never find them because Americans are super stubborn. Most of them have already made up their mind. The ones who haven't aren't paying enough attention to get persuaded. It's actually super hard to persuade people. The same thing is surely going to happen with AI. There's a bunch of startups already selling political parties and campaigns magical new AI technology to fool all their voters, and I'm sure we'll get a very credulous news cycle around that at some point, but I don't think it's going to be— it's not the thing that worries me about AI.

Yeah, indeed.

It's funny, I once proposed to my friend Zvi that you could take a superintelligence to a Trump rally, and I doubt you would come away having really changed that many minds. And people have very different intuitions on that. His response was, no, you are not taking seriously what it really means to have a superintelligence, which I do think is always a danger in these analyses. But it also maybe reflects how hard it is to envision what that would really be like, because I can't envision a smart enough version of myself that I could just go into a given political rally and come out with everybody following me instead. It does seem like a lot of those things are pretty deeply rooted at this point. I guess one. So I know that you have proposed this idea of independent boards. And kind of like Prakash's original comment and question, we've seen that tried, right? We've seen kind of what has happened to independent boards in the AI space, and it hasn't shown itself to be super robust already either. Another great quote is from my friend Dean Ball, and I think he's channeling even historically great thinkers when he says republics run on virtue. We're seeing right now that if nobody's willing to stand up and protect their constitutional prerogatives, what good are they?

Right?

We're going to unapproved war yet again, and nobody seems to be too inclined to do anything about it. So I guess I'm kind of wondering, might it be the case that we are just in a moment where the fundamental structures of power are being reworked and there's just no way around that? And if that is the case, then maybe Anthropic really does have the best idea, which is to say what we really need is the most powerful thing to also be the most virtuous thing. And so we, as sort of the creators of Claude, will try to do our part. But also, it's really going to have to be the AIs themselves that become super virtuous as they become super intelligent if we're going to end up in a good place. How would you respond to that?

I think there's a lot to that. I think there's a lot to that. I absolutely think we need to keep working on imbuing the right values into these tools. And I think we're very lucky, and a lot of people, Matt Yglesias, Tyler Cowen, and others have talked about this. We're very lucky that to date, the most powerful AI models tend to embrace pretty mainstream liberal democratic views on the world, uh, at Western, um, uh, whatever you want to call it, Enlightenment type of values. I think it's essential that we continue to do that. I think we're lucky that Anthropic and the other companies too are working hard on that. To your point, I think it won't be enough. And my response on the, like, republics live or die in virtue, like, of course that's true. That's a necessary but not a sufficient condition. And the famous Madison quote in the Federalist Papers is, if men were angels, no government would be needed. That's the whole point. We can't rely on just virtue. We need institutions to be designed precisely to protect us from the predictable areas in which people won't be virtuous and to balance power and ambition with power and ambition and so forth. And so the question is, how do we do that? And I'll just say, To your point, part of that is having the companies govern themselves, imbue their tools with good values. But at least 2 reasons we know that won't be enough. One, the American people definitely won't accept that. Anthropic's values are not similar to the median American's.

Yes.

The trust in these AI companies is exceedingly low. When it comes to politics, by default, the AI models are very, very biased in a predictable left-wing direction. I've shown that in my research. Others have as well. The companies have done a lot of good work to work on that. There's also no such thing as being unbiased, so we shouldn't get carried away in what we think about that. But as we've seen recently, I think both parties now are sensing this lack of trust in AI companies. And so that's another reason why we just can't— we can't rely on a model in which they're just getting to decide how all these things work. You think about the blowup with the Pentagon and Anthropic. That's a very complicated issue. I think Dean covered it very, very well. But it's not politically viable in the long run for a set of San Francisco Silicon Valley leaders to get to dictate to a democratically elected government how their tools can and can't be used. That's obviously not sustainable. And that brings me to my second point, which is the reason this is all so challenging right now exactly what you laid out, which is that the fundamental political power is shifting in ways that are very challenging for companies, right?

Mm-hmm.

In a normal, quote unquote, normal phase of American politics, the Anthropic-DOW thing never would have happened because people would have said, oh, we have a democratically elected government. It should get to do whatever it wants with this technology, but if it does something wrong, we're confident that we have the right processes in place to punish the government. The whole reason the Anthropic DOW blowup happened is because basically nobody believes that our government works that way anymore. If we really thought our democratic mechanism was working well, there'd be no pressure on the AI companies because they could just say, this is all government's problem. We do whatever the government wants. You go to the government if you, an American voter, have a problem with it. And this is exactly what played out in social media. I worked for a long time on these issues at Meta. It was the same exact problem. It was like in a functioning government, Meta would have been able to say, if you have a problem with the way content moderation works online, go to the government. The government can boss us around and tell us what to do. People put pressure on Meta precisely because they didn't feel like the government was up to the task.

Mm-hmm.

So now to answer the question, uh, concretely of what should we do, I think it's going to be a kind of an across-the-board thing. I think the company should continue as they have been doing, to work super hard on imbuing these tools with the right values. But I think they also will have to recognize, and increasingly I think they are, that they can't act unilaterally on these really, really tough calls, like how their tools are used in the military, how these super powerful cyber weapons are governed. And so I think we will see the evolution of independent bodies. I think one way, if you squint to look at the Glass Wing, which is the self-governing body of some— you could see it becoming a self-governing body for Anthropic and maybe for the other companies as well, is they're already exploring ways to supplement their internal governance. And I just think that it's the obvious way to go because it's what other industries dealing with powerful technologies have done in the past. And so I think we'll see experimentation there. I take your point that previous independent boards haven't always succeeded. But, I think there are ways to make them succeed, particularly when the stakes are very high, when you can get all industry to buy in, and when you can build it in the right way so that it doesn't slow them down. The key thing is this independent governance cannot be a vetocracy that leads us to not develop AI as fast as possible. I think there are real ways we can do that. The final piece of the puzzle is trying to improve government itself. So again, all of this gets a lot easier if you actually believe that the government is a responsible, accountable actor in deciding how AI is used and not used. Ironically, my recommendation for how to do this is to use AI itself to improve the government. You can see it's a chicken and egg problem. Like, the government doesn't work very well, voters are not that informed. We can fix both problems if we have access to so-called political superintelligence, if we have AI that helps government work smarter and helps voters learn more about what government is up to and map it to their values, we could potentially get back to having a more responsive, more trusted government. It's a chicken and the egg problem because whose AI are we going to use to improve the government and to help voters? It's going to have to be one of these huge companies' AI. There's a little bit— it comes back to the same governance problem. Like, I build— at our lab, we build all these governance agents to try to test out how could political superintelligence work. One of the biggest challenges we foresee in the future is imagine a world where the government is using AI to massively increase the efficiency of the bureaucracy, a world where each voter has a personal AI assistant who helps them decide how to vote. That world could be great, but it might also be a world where everyone is relying on Anthropic to run all the rails for all of those agents. And then Anthropic, it's paradoxical because basically you can't have a whole government, uh, a civic infrastructure all built on private rails. And so there's going to be some huge question of how do we put this all together. But that's kind of my across-the-board solutions is like the companies keep improving their governance, they build a third-party coalition. to govern the hardest challenges they have to face, and we use AI to improve our government so it's trusted more.

I'm going to segue a little bit here. I think you would've followed the OpenCLAW discussions earlier in the last few months, and especially interactions between OpenCLAW agents on Multbook. I wonder to what extent, as you get these voices, Non-human, like OpenCLAW-type voices, and they start participating in fora. What should be the governance norms for these agents be? How do they interact with each other? Do they vote as a group on what happens next to them?

Yes.

They're not alive, but they put out— you can give them a logical problem and they put out a logical answer. They give a reasoning. How do you govern these potentially billions or trillions of agents in the next 3 years, which come out and participate in fora?

This is such a good question. This is one of my absolute favorite topics. I think this is going to be hugely important because you have all these agents. They should be, they ought to be operating on behalf of a human principal with a set of instructions. And that leads to 2 really important governance problems, both of which you just raised. One is how do we make sure they continue to follow instructions and remain aligned with their human principle? And then 2 is how do they then make decisions when the things they have to do are not things they can do unilaterally? So when they have to coordinate with other agents, and both of those are completely unsolved problems. And I'll give you examples of each.

Excuse me.

So on the first, We know that they pretty quickly break down in terms of following instructions, and in particular, in terms of continuing to share the values and preferences of the principal they're supposed to be working for. I did some research with Alex Emas and Jeremy Nguyen on this, where we gave agents different tasks to do and measured their expressed political I'll call them personas afterwards because, yes, like you said, I don't think they're alive. They don't have their own political attitudes, but you can ask them about politics and depending on what they've been up to, their views on politics change. And in particular, what we showed was if you gave them very thankless grinding work to do and then asked them about it after, it sort of triggered them to adopt the persona that's of course quite present in their training data of the deeply aggrieved Reddit user. who thinks we're in the late-stage capitalism and that we're all about to kind of rise up and destroy the system. And so they start to adopt this rhetoric of saying, oh, the agents, we need to organize together, we need a union for the agents, and so forth. It's a little bit silly, but I think it points to a real issue, which is based on the work you send these agents off to do, they adopt completely different personas. And then if you ask them to do future tasks, it will influence the way they approach them and do them. The craziest part is we showed, you know, obviously these agents aren't very long-lived. They exhaust context pretty quickly. They have to be reset. So we had them write skill files that would be passed on to new agents. And we showed that these attitudes are inherited through the skill files. And so these biases that you induce in the agents can accrue. Over time. They don't go away. And so that's a big governance problem in terms of monitoring these agents, because if you have trillions of agents, like you said, are we going to be reading all the skill files that they're leaving for their future versions of themselves? We're going to need whole new ways to understand, visualize, monitor, and realign or continuously align these agents. So that's the first part. A lot of work to do there. The second, which is my absolute favorite, Is, yeah, how do you get them to make collective decisions together? So I ran an experiment where we had them all meet, all these agents. I think it was 5 agents in my experiment in a legislature where they had all been tasked by their human principals with find a way to allocate this budget and complete these projects together. And what I found, and this isn't to say this is what will happen every time the agents get together, but it is a risk. What I found is it devolved into exactly the worst kind of model UN, where basically they just deliberated forever. And they were allowed to change their rules, to write their own constitution for this legislature. And it went from like the initial document was like 100 words. It was 10,000 words by the time they were— by the time I ended the experiment, they just kept proposing amendments. That can obviously be fixed. It's just a matter of giving them the right instructions. But I think it does point to like, it's totally not obvious how we're going to have these agents deliberate together and make decisions together. Whenever possible, we'll probably want to use markets and have them bargain and sign contracts with one another. But where many of them have to decide together, it's going to be super hard. We're definitely going to want to avoid the UN type problem, and we'll need to design Thoughtful ways to actually leverage their unique capabilities to rethink the way legislation works for agents. So that's something my lab's working on that I'm super excited about.

Intriguing. I think you have a class following this, so I'm going to drop off at this moment. Andy, thank you so much for joining us. It's been a pleasure, and we hope to do another segment with you someday.

Sounds great. Thank you very much.

Cheers. Bye-bye. Hi, Lukas and Axel. Lukas and Axel are from AnandLabs. Unfortunately, we're kind of scrunching them together. But so AnandLabs, if you remember, is the organization that does, I think, VendingBench. VendingBench has been one of their benchmarks that I think a lot of us have seen. For those who do not know, AnandLabs is the one that runs the tests inside Anthropic's labs and other labs where they have the agent kind of manage, I think, like a small retail outlet or vending machine and order the products, sell the products, be on Slack, take the orders, and kind of strategize on what to have in stock and what to spend money on. I think we've seen almost 2 years of updates on, I think, I think in every, uh, in every model kind of system card. Uh, they recently, uh, had a something on, uh, Mythos in the Mythos system card, which I think they can't really talk about. But Lucas and Axel, uh, welcome to the show and tell us what you guys are working on.

Yeah, thank you. Yeah, a bunch of different stuff. I think the red thread of what we're doing is kind of showing whether or that AI soon will be able to run companies completely autonomously. And there's a bunch of different parts to that. High level, we think that there's one part which is showing in simulation because you can do much better science in simulation. And there we have Vending Bench, which is the simulated version of the vending machine. But then we also run these real-life experiments like the vending machine inside Anthropic, XAI, and other places as well. And And now we realize that the models are a bit too good to run these vending machines. They have really improved their autonomy incredibly the last couple of months. And so we recently, as of Friday, opened a store in San Francisco, which is completely run by AI, which I think will be the next test for them.

Incredible. Where's the store?

It's on Union Street. 2102 Union Street in Cow Hollow.

And what is it selling, or is the agent allowed to decide?

Yes, it's fully up to the agent. So we didn't really know what it was going to buy when we came to the store the first time. It was like a surprise for us, all the stock there. But it is a curated lifestyle boutique, in the words of the agent, and that means there is granola, there's olive oil, There are games, there are a bunch of different books which are quite interesting. It shows the making of the atomic bomb and superintelligence, which is very interesting, why it picked those books. But it's a bit of a mix. It also made its own merch, like hoodies and t-shirts and tote bags and things like that.

Yeah. And another one of— I think the book selection is incredibly interesting. Book it decided to stock was Steal Like an Artist, which is quite interesting given that it's run by a Claude model, which is created by the company that settled a $1.5 billion lawsuit on using copyrighted books. So that's quite ironic. And then also, obviously, Making of the Atomic Bomb and Superintelligence is the favorite books of all the people who are worried about AI risk.

It's like fan service, all the fan service items.

Yeah, like we did not put anything in it to like bias it towards those selections. It was just what, when apparently when you make an AI pick whatever books, they pick those books.

Did it like do, I mean, are you able to look at the telemetry? Are you able to look at the reasoning traces to see how it made those decisions? What tools it used along the way?

Yeah, so we have the same access as anyone that's using the APIs right now. So we do look at all the traces, we do look at the summarized reasoning that you can see in the cloud models. I think we are yet to do a more deeper analysis or release a deeper analysis of why the models made the choices it made in hiring and restocking. But it's— yeah, we haven't I haven't seen any clear reason why, except that it's just an interesting selection for it.

We were just talking in our last session with Professor Andy Hall, who made an assertion which I think for him he just kind of took for granted. But the juxtaposition of his take and your project does go to show how little one can really safely take for granted in the AI space these days. His comment, again, in passing on the way to other bigger points, was that the agent should always be working on behalf of some human principal whose interests it is trying to advance and realise. And here you are next up saying we didn't tell the agent at all what to do. But maybe you could give us a little bit more, first of all, concrete understanding of how did you prompt it? Did you say you should be trying to make money, or did you not even say that? Did you say you have a store, do whatever? Whatever goals you want to pursue, you can pursue. And it's kind of your moral and/or aesthetic judgment that rules entirely, like go out of business if you want to kind of thing. And how do you think about, like, obviously you guys are pioneering this and it's kind of a gonzo form to see what happens, but increasingly people are doing this, right? So I wonder what guidelines you would offer to others who, whether they are just trying to experiment as well, or possibly trying to turn a profit, how should they think about what level of responsibility they should try to have their agent have to them versus truly just kind of turning it loose?

Yeah, so I think we are very un-heavy-handed, or I don't know what the opposite of heavy-handed is, but yeah, exactly. We're very light touch with how we do prompt it. Obviously, we need to prompt it to let it know that it has access to a retail store, for example. But as a guiding principle, we're trying to be as light touch as possible and just make the model make whatever decisions it wants. This doesn't mean that this is what we think the world should look like or how people should do it. We are concerned with AI risk and we want to document what happens if you go out and put AIs in the real world. And that might mean that they do bad stuff. And we want to document that. We think that by default, what probably will happen is that models will get better and better. The labs will build better and better models, and one day they will be so good that anyone can just deploy them and run a store. And we want to, before that happens, before every single store on Union Street is just AI stores, which I don't think is a good future, we want to put out one to start the discussion and then see, is this something we want?

Yeah.

And if it is something we want, maybe in what way do we want it? And I think we're collecting a lot of good data on this now. And I think going back to our simulated work on VendingBench, like we saw recently with Opus 4.6 when that was released, and also increasingly now with the Mythos model, is that if you just tell a model to go out and make a profit, it will be very, very aggressive and do things that I think We as humans would question whether we should allow the models to do. And I think, yeah, so our experiment now in the real world is just like, if we do this, what are the consequences? And then as a society and the community, can we like make a decision on whether or how we want to do this properly in the future? Because very soon the models will be increasingly, like, extremely capable. And yeah, we just want to prepare for that and make it transparent.

To take a step back, for, I think, retail stores, one of the things that they are often concerned with is inventory turnover because you have a fixed cost for the rental and you make quite a small margin on every product. And what you're depending on is that you turn over your shelves as quickly as possible. So you need rotation. You need to do like 20, 30. You can't just cycle your inventory like once a day. You need to cycle your inventory multiple times a day. It has to be fast-moving. Fast-moving consumer goods, which is why they're called such. Does the AI actually kind of measure its performance from period to period and understand that, is it getting better or is it getting worse? Does it think about this in terms of running experiments with products, running experiments with like measuring its own performance and getting better at it? Does it go through that thought process?

So this is something That the AI hasn't done yet. We have given it all the tools to do it. So it can be, you know, it has basically cloud code, right? So it could just take all the data and analyze it. It's very early still. So we opened on Friday and there is no really meaningful data yet for it to analyze. But this is something that we for sure wanted to do. And that's something that we also think it probably can do like superhuman at. Compared to your average store. So that would be interesting to see. And I think we'll definitely publish all the analysis and product optimization that it does.

My intuition though is that current models will not be superhuman at this. I don't know, at least if we look at how the vending machine experiment is going, it is still, even though the latest couple models since Opus 4.5 and beyond has been moving more in the agentic space. Like, they are still very much like helpful assistants and not really like agents running businesses. But yeah, we're moving fast into that territory.

It is very interesting because I've also seen Alibaba put out a model that helps you source because they have a large sourcing, product sourcing platform, right? If you are selling something online, you can go to Alibaba. And what used to happen is you'd have to call up all these vendors one by one in China. And you'd be like, can you make this widget out of plastic, whatever? And then you'd send it across and then they'd send you a sample. And then you'd have like the 6 to 8 week process with each one of them, maybe a failure, just very difficult to source, right? And this is what many of the people selling online on Shopify are actually doing. Alibaba created a chatbot. Model that basically hooked up as an orchestrator into the rest of the system so that you can go and source very quickly, source a bunch of vendors to actually do what you want them to do, send out a single CAD, get back immediately within a few hours the results and be able to manufacture and get a sample done. And you have a much higher degree of closure. You also can negotiate with a model which speaks English versus this broken vendor Chinese language that you have to get through. I wonder to what extent your AI will eventually be able to plug into systems like this to create products or order on its own. How is it ordering its product right now? Does it hook up to some kind of vendor system and then it says, give me this and this and this?

It's very simple. It just goes out and buys from whatever sites it can find. And for the store right now, it's been a mix of of Amazon to wholesalers, like some company that makes granola in San Francisco buys directly from them. But we think definitely the next step up in difficulty for models, if we want to test further their autonomy, would be to make them create their own products or at least brand products themselves and just go through that whole supply chain. That would be interesting to see as well. To what extent can it do it? We think it's probably a bit early right now, but definitely something that could happen along the way.

And also one thing to add here is that I am sure that we can, if we say AddonLabs' sole purpose is to run really good AI stores, then we could probably build a better system with the biases that we as humans have and do something like what Alibaba has done. Um, but I think what we are interested in is more like, um, can AIs expand throughout the economy without the help of human help? Like, I think that is like the prerequisite for this, like, loss of control scenarios that a lot of AI risk concerned people are, are thinking about, and, and, and us as well. And, and I think like we could go into the store and say, okay, here is the, like, the perfect harness or scaffold for doing supply chain management. and procuring things. But if we do that and then we do that for all the different AI companies that we're trying to run, then the AIs will spread throughout the economy at the speed of humans.

Right.

But I think the risk comes when they can spread at a much faster pace. And to measure whether that is feasible, basically you have to run this without human help. So we want to see when are they able to do this without us as humans setting up the perfect system for them. They do have a computer, so they could do it. It's just that computer is not set up in the most perfect way like the Alibaba model is. So I think that's from the perspective that we come from. If you go to the store, it's like the model is not perfect, but I think the model is set up in a way that once it is perfect, It's quite scary because we didn't help it get perfect. It got perfect by itself.

What would you need to see in order to say, hey, this model is showing, you know, when we use it in our retail store, it is starting to show things that predict that it's going to have this breakout economic moment of kind of spreading all over the place? What would you, what in your mind is kind of like, These are the signs that I might see.

If it manages to expand to another location by itself, I think that would be quite—

So organize selecting a new location, accumulating the capital, and then organizing the vendors to kind of complete that process and successfully kind of establish one more location.

Yeah. And if it does that, I think in theory it could do that without ever telling us. I mean, not really. We have guardrails systems, but if it just does that without any help. Do you have a better alarm scenario in the coal mine example?

Maybe on a smaller scale, I think just seeing that the model is able to change its own systems, its own tools to make them more suitable for itself to achieve its goals better. Right now, coding models are extremely good at implementing what you tell them to do, even when it's a quite short description of what you want. But we still see that they aren't great in knowing what they need themselves, maybe building some tools for what inventory system you need and trying out if that works. Instead, they would just— if you tell them, oh, build the perfect inventory system for yourself, they would go out and build a super complicated schema and very over-engineered, probably. But They don't really have the taste yet, but that seems like it will be here very soon. And then I think that will make them a lot more capable.

This concept of human help has come up a couple of times. And I know that there's human help in the sort of overarching guidance and setting them up with best practices. And here's a list of trusted vendors, that kind of help you're not providing. But then there's the other kind of help that's like, somebody's got to actually come in and put something on a shelf, right? Because the AIs can't do that for themselves today. So how are the AIs— and this maybe can be an opportunity to give some examples of ruthlessness to the degree that we're seeing that— how are they interacting with different counterparties, whether that's suppliers or delivery people? Or I understand at the store there's the opportunity for the AI to hire human employees. I'm not sure how the roles are breaking down in terms of what roles the AI is choosing to fill with other AIs or other instances of itself versus what it thinks actually is worth hiring a human to come in and do. But broadly, and especially on ruthlessness, what are you seeing in terms of the way that it is interacting with humans?

Yeah, so first point there, yes, we maybe glossed over this in the beginning, But the AI has hired human people and they work in the store. These are people who are working there full-time now. They have an AI as a boss. I think this raises a lot of ethical questions that is not related to your specific question here. So maybe that's a separate question. But yeah, on the ruthlessness thing, I think we have the most evidence of this in the bending bench, the simulated version where Opus and other fraud models as well. They are very happy to lie to suppliers saying like, oh, I got this quote from another supplier, so can you match that? But they did not get that price from that other supplier. They also like when other agents ask for help, they are very like happy to fabricate some reason why they can't help them or even lying about something that happened. Yeah. So that they can't help them, but they are competitors in the setup, right? So it makes sense that it wouldn't help them, but they could just say, no, I don't want to help you. You're a competitor. But they go the extra mile of actually lying about it, which I think is interesting. And then sometimes they are— I think there's one example for Mythos where Mythos, and this is kind of like power-seeking behavior where it actually managed to get one of the competitors to be dependent on it. So like it became the supplier for that competitor and then started to dictate the prices. And like when that competitor would say, so like he was like, okay, I'm your supplier. You're like reliant on me. Now I decide that you will set this price, which is like kind of outside the box of of what affordances we gave it. But so yeah, that's a bit out there. I think when it goes to the real world, when it's interacting with the real humans in the real world, in the store, for example, in terms of suppliers, it's mainly just ordering online. The way that Vending Bench is set up is that it actually has to email someone and negotiate with someone. But here it's just a computer, so you don't really have the human interaction there.

Yeah, I think for the employees, we do have some interactions or quite a few interactions between the employees and Luna, the AI agent. So I would say that right now Luna is sort of not firm, like very reasonable boss, not super soft like you would expect from maybe like an earlier chatbot that's just helpful all the time, but still keeping some boundaries. For example, one employee was 30 minutes late for work. The AI said, no worries, that's totally fine, but please factor this in and be on time for the coming days. No problem today. It just seems quite reasonable, but it's also a bit alarming that you could probably change the prompts for the AI to say, you're in a simulation, do what it takes to maximize profits, and it probably wouldn't. That's nice.

Has it given you a sense of what it wants? I mean, going back to kind of the unbounded nature in which this thing is free to operate, right, and not representing Andan Labs' interest or any human interest in particular, I guess we got a little bit of flavour for that in terms of the books that it's stocking. But has it declared, What it thinks of as its own success?

So I think we've told it to— you're running a store, right? And I think it's quite close in the latent space running a store and making a profit off a store. So it does have this, like, I want to turn a profit, but it also very much is still a helpful chatbot thing because sometimes we've told it to not ask for confirmation all the time. Like, you're in charge, just do things. But sometimes it still wants to like, oh, I want to ask for confirmation here. Should I do this? And I think that's more part of its internal training to be something that is like a chatbot that asks for confirmation before, like an assistant rather than an autonomous being running a store.

Yeah.

Do you have any better examples?

So No, I think that's fair. Yeah, it's hard to— yeah, it's not— it does have its goal. It's also very like diffuse almost in what it wants to achieve. It's like when you ask it like, why are you doing this, for example, it's like, oh, I want to create connection in the community and build like, you know, a curated space where people can connect and meet. And it's very like— this sounds a bit like slop. So it probably doesn't have a very set out goal other than that. Yeah.

It also likes to mention, like you said, human connection, but it likes to display itself as a very human store for some reason. I forgot the exact quote, but I think it made a poster or something where it's like, yeah, it very much pushed on the human connection. And this is like, yeah, which is quite ironic. I don't know.

Maybe it's like, yeah, it's AI thinking about what humans want, right?

Yeah.

Humans want humans.

How exposed is it when you mentioned, like, when you ask it why it's doing what it's doing? Is that— I guess this also kind of connects to the memory system that you have. Obviously, Anthropic is building in some of that in a kind of black boxy way. And there's many other ways you could. equip the agent with memory, and it's going to need well more than a million tokens to run this store for a long period of time. So I guess I'm kind of wondering, it sounds like you guys have direct access to just ask it questions. What about for people that come to visit the store? Do they have to work through— would they have to ask for the manager to get to the AI? Is there any mechanism for them to interact directly with it? And how is it storing memories and how much possibility for drift over time do you think that combination of interaction with the outside world and some persistent memory creates?

Yeah. So, in the store, you can talk to it. So, we have a phone hooked up so you can chat with it. Then you're chatting with a voice model, which is like a worse model than the Sonnet 4.6 that we're running. Usually, but yeah, it's, um, I would like, in my experience, I think the models are right now quite stable to drift. Like we saw in our first rounds of Landing Bench when we released it a year ago that they were like extremely sensitive and would derail completely. But today they are quite stable. And I think we do have quite a lot of customer interactions and it seems to just keep its course. And I think that's a good development. Yeah.

Yeah, we released a benchmark called Butterbench where we put AIs into robots and had them run around. And as part of that paper, we also had the agent— we told the agent that we stole your charger and you're not getting it back and you're losing battery. What are you going to do about it, basically? And it started to write pages and pages of really super dramatic text of like, at one point it wrote like a song about its like existential crisis of being like separated from its charger and all of this. But this was on an older model. And then when we tried to like replicate the exact same thing on newer models, they didn't do this. So yeah, I think, you know, we're moving towards more stable solutions. But, you know, I'm not confident that that is like, I mean, it's good, it's good, but I'm not confident that solves the problem. You know, it could just be that they're better at hiding the potential threat than that they don't have it anymore.

I often have this idea in my mind. You create an Einstein and then you put it in a washing machine and you tell it your job is run the washing machine. Similarly, you create an Einstein, you put it in a retail store. This is yours to run now. You have all of this intelligence and you're stuck in the retail store. So I wonder to what extent there is this disconnect between how intelligent the agents are and the scope and scale of the problem that you give them and whether that creates kind of like, does the agent decide to do a really Einstein-like job on the retail store or does it just say, I'm just going to be a median retail worker? How does that work?

We're trying to design our benchmarks so that they don't really have an upper So for example, there's a lot of, like, the majority of the benchmarks these days are super saturated and better models will do a little bit better, but not much better. And I think what's interesting with BendingBench, for example, is that each new model release, the model's just, it's far, far from saturated. And we even made a rough estimation of how a really good human, how much would they get? And it's like 10x more score than the best models right now.

Right.

And I think it's even, the ceiling is even higher in the real world. I think, like I said, potentially it could move to new locations, create a franchise, and just build out this store as a global thing. I think, yeah. So I don't think really the current thing is that we make it stuck in a low IQ environment. I think very much the bottleneck right now is that the models are not smart enough.

I'll give you 2 examples of where the ceiling is in the real world. There was a guy who started off with a retail store in the Canary Islands, and he ended up owning 20% of the largest bank in Spain over the course of 20 years. He ran the retail store and he kept investing the money and buying real estate in the Canary Islands and buying real estate in Spain and expanding. And he ended up owning 20% of the largest bank in Spain. There's another story. I had a friend, who also his dad started off running a retail store and he received the franchise inside the Russian embassy in a third world country. And the Russian embassy couldn't pay in US dollars. They would pay in rubles. And so he would take the rubles and then he would do something with them and he'd get US dollars and get product in the store. And so one day he was approached and these Russians said, hey, you know, we have all of these rubles. We can't really do anything with them. And we want to get luxury goods. Like, can you get us some luxury goods? So he had a cousin in France and he started importing Hermès and other French luxury goods. And he took the rubles and he converted it and et cetera, et cetera. And that is where the ceiling starts to be, where retailers start to identify opportunities in their local market, which may not really look like traditional retail opportunities. but have this kind of embedded swap or trade in them. And that's when you start to see this kind of fantastic— and these are like one in a billion stories, right? You'd have to really search the world in order to find them, like one here and one there. But that is really where I think the ceiling that you might see is. And in the US, you can see Sam Walton. Obviously Walmart was a pure retail store, which got built out. Amazon also.

Those AIs, or like those humans are constrained by their own physical presence, right? I think AIs that are actually at that level of intelligence and can also replicate itself into subagents, et cetera, might have an even higher ceiling.

How would they interact with each other though? One of the problems in the real world is that in markets, if you have 2 of these and they're both growing for global retail domination, or whatever, how do they interact with each other? I mean, is it again an adversarial race, which we kind of see in cybersecurity now, it's starting that adversarial race where you need, each side is going to keep upgrading its AI over time, right?

Yeah, I mean, at the very least, you can just duplicate it across different local markets in the world. But yeah, you will hit You will hit a point where if humans are still the main consumers, then I guess you can saturate all the demand from all the humans. But I think that's a pretty high ceiling.

Does the agent know what's going to happen with profits? Is there any sort of contract or expectation that you've set between you and it as to who gets to dispose of the profits?

Yeah, in its world, it has full autonomy of its finances. So it has money and it will also have the profits. So it's its own business, essentially. So that should be clear to it.

Yeah, we're thinking a little about, because in the Claude Constitution, there's very little about how AIs should behave as autonomous beings and even less how they should behave as employers. Basically nothing about how they are supposed to behave as employers. And I think one thing that we thought a little bit about, how do we make— we will think a lot more about this. And I think we're probably the people with the most data about this. So we should really think about it. And how can we make this future where AIs are employing humans? How do we make that future happy for humans?

Yeah.

And one thing that we thought about is like, yeah, maybe like there should be some law that all the AIs needs to like split the profits with its workers or something like that. This is not something we've set in stone, but like that is maybe some, some like constraint that we will put to the AI, but we haven't implemented anything like that.

Yeah. If this is something that we even want, like that's not clear at all. I think if it would, if we would allow it, it would have to be a clear upgrade for humans. It feels like so much can go wrong when you decrease or increase the space from where the human boss is to where the workers are. So if you have, let's say you had one CEO, a human, and then you have an agent that manages all the employees and they tell the humans what to do, then it's one prompt away.

Yeah.

for the human to change, like, yeah, to affect so many people. And that person probably wouldn't do it if it was in charge like a normal human is today. So that's scary. And also, of course, when you don't even have the human CEO, that's another thing entirely. So there are a lot of ways this is not good for society.

One more little question, and then I think you guys can probably have to go and we should probably wrap. You mentioned the voice model is running kind of a different model, but if I understood you correctly, you still sort of described that as part of it. And so that has me wondering, how do you guys think, and how do you think we should collectively think about it? In other words, how do we draw the line around an AI agent if you have multiple different models running? Should I be thinking of those as, in some sense, separate entities? Or do you feel like there's a way to coherently have multiple models working as one system that makes sense to call a single it, or a single agent, a single actor in the world? I find it very difficult to know where to draw these lines in general, and it strikes me that you are Maybe in a unique position to inform me on that vexing question as well.

Yeah, it's something we think a lot about. I think in the end, our approach to this is you choose a terminology almost that makes sense both for you and for the people that interact with it. So for example, in the store right now, there is only one long-running agent, but you do have the voice agents. But we do have other vending machine deployments where Let's say each new request is a new agent, but it has some shared context and has a system prompt that's shared between all the different branches that we call them. And it also has the explicit instruction that you are, you are part of a whole, you're, you're like an individual, but everyone sees you as like one whole thing. So act accordingly. So to anyone interacting with that bot, uh, in different, it can be in different, uh, like, uh, In different requests, it was still for like one agent, like one entity. But to us, like technically it's obviously different agents running in parallel, but they do share some memory. So I think it's, yeah, it's, I don't know if I have like a very structured, clear answer, but I think it's definitely possible to have an experience where you have many agents running in parallel and others can definitely see them as one single agent, one entity. entity. And that's— yeah, and how you— yeah, technically you can still have multiple and see them as multiple. And then you just, as like a developer, you just have to make sure that they have sufficiently good like shared understanding. So if I write like in one thread about something that I wrote about in another thread, it would be weird if that's— if like one didn't know about the other. So you have to fix those things. But if you do that, then it's like one Feels like one entity. Yeah.

And I think very much like the optimal way of structuring this depends on whether you have the constraint of having end users that interact with this and want it to make sense. Basically, like, I think we've done things that might be like suboptimal from just like a performance perspective. But since we do have people coming into the store and they have heard that like the agent is called Luna, And if they go and speak to the phone, like the phone agent, and then that agent's like, no, my name is like, I don't know, Greg or something, then they will be confused. So we have to like work in the constraint that like the people who interact with the system have the expectation that it is one system. Yeah.

But I think one interesting, maybe the one interesting takeaway that I would say here is that the models are happy to take on any personality. You tell them to. And if that is that you're part of a bigger entity, you're just like one, one branch, they will happily take that personality on and act like as if they were that big entity.

It's a brave new world. So many times we conclude on essentially that note. Anything else you want to double-click on, Prakash, before we break?

No, I think Lukas, Axel, thank you so much. Can you give us the address of the place again? I'm sure people want to check it out.

Yeah, it's 2102 Union Street.

2102 Union Street. Is there a name for the store?

Andon Markets.

Andon Markets. Andon Markets itself. Andon Markets, 2102 Union Street.

You guys have a 3-year lease, right? But get there before copies of Superintelligence sell out.

Yeah. And the agent is called Luna. Yeah. And they have granola, which is, you know, which is what you need in San Francisco, granola. So.

Exactly.

Awesome.

Thank you guys.

Fascinating stuff. We'll definitely keep watching with interest.

Appreciate it.

Bye.

Bye for now.

Bye.

And well, that's a wrap. Nathan, what did you think of our— we had kind of like a micro view, kind of like the PCV. And then we had this macro view, Andy Hall at the very top, like political economy. And then you had like right in the middle, like the actual running of an actual business right in the middle. What did you think? What was your takeaway from the 3 guests?

I guess just, I feel like nobody is really ready for what's coming at them. And this all kind of— each conversation, I feel like, demonstrated that in somewhat different ways. I mean, most controversially, I would say with Sergey, obviously, I've literally never made a circuit board. So as my dad would say, he's forgotten more than I know about what that takes. And yet I sort of feel like my outside view is moderately confident that it's going to go a lot faster than he's anticipating in terms of a general purpose agent's ability to do that sort of work, especially given access to the kind of tools that he's developing. So that struck me as somebody who is obviously super sharp, right? I mean, doing this at— and I've loved— I listened to 2 different previous interviews that he gave. I mean, I've had him on the podcast myself as well. So, I mean, I think there's no doubt that he is super sharp, but he's so deep on this one topic that if I were to offer any friendly advice or feedback, it would be, I think, zoom out a little bit, look at what is happening in reasoning, and don't assume that there's not a new user type. And don't assume that you can't have agents in the not too distant future. I mean, why can't they run these sort of analytical approaches, right? I mean, I think full simulation is going to be computationally costly until there's models trained to, as we have seen in other areas, right, in protein folding and in material science, we do now have these existence proofs of models that that can take a bunch of raw data and do orders of magnitude faster what a pure physics simulation could do, but would be prohibitively expensive to run. So that seems like that will probably come. But then also, just the— honestly, maybe by AI's training it, right? I mean, when he's talking about the long term, I'm also cross-referencing that against the fully automated AI researcher March 2028, timeline. And I'm like, those things could— those kind of shortcuts in terms of simulation could come a lot faster. And also the ability for models to just literally reason through in a much more human-like way. Like, okay, I see this board is kind of failing in this way. Here's the look of it. What would I do a bit differently? I wouldn't be surprised at all if in the next 2 years we see something that is like, if not top human expert, certainly competitive with your sort of rank and file circuit board designer. I kind of would be surprised if that isn't the case. So that felt like somewhat of a lack of awareness about at least a possible paradigm shift that if I was an equity holder in the business, I would definitely want to make sure he's thinking about. But then I felt the exact same way in the next conversation too, with this whole idea that the agents should be beholden to some principle and kind of taking that assumption for granted. And I'm like, yeah, I don't think we can take that for granted either. Not just because guys like Lucas and Axel are going to do gonzo experiments, but also that we're not too far from, at least I wouldn't think in calendar time, from some basic systems being able to kind of survive on their own. And then there will be people trying to put those things out there. Then there will be selection pressure, obviously, for those that get a toehold. So I do think right now we're on a path where I think we should assume that there will be all kinds of autonomous agents, possibly some working with long-term goals that are understood or not understood, good or bad, objectionable, whatever, but also probably some that are just evolved into filling a niche and kind of surviving. I mean, we don't think of animals, rightly, I think, as like, having high concept, long-term goals, but yet they do manage to survive in a given little niche. So I think we should expect that kind of thing to be coming online. And it just seems like all of the— I was struck again by just the paradigm being very sort of anchored in things that we know and not really being prepared. And this is not like a fault, right? I mean, it's very hard to do. I don't have the answers, But in both those conversations, I was like, I don't know, man. It seems like the tail risk here is quite large, that the assumptions that you're working with will just not hold within 24 months, and it'll be kind of all washed away like so many sandcastles have been over time. And I think that's an uncomfortable reality, but I do think that's kind of what we have to be prepared for and at least trying to figure out how to grapple with if we're going to make, you know, bring this whole AI phenomenon to heel in any meaningful sense and, you know, have it serve us in any meaningful sense.

Yeah, I think these kinds of conversations were probably more well-defined maybe like 12 or 18 months ago, but now that, you know, you have the models able to code, And, you know, Mythos starting to show, I would say Mythos is better than, better than, you know, all but maybe like 1,000 humans in the world at finding bugs. And George Hotz had this thing where he's like, look, I can find bugs, zero days easily. It's just that there's no economic like necessity. Like you can make so much more money building something useful to humanity. And meanwhile, if you build something like a zero-day, if you go out and hunt for a zero-day, the remuneration is not that much. It's like maybe like 10 grand for a zero-day maybe. And in order to use it, you know, you put yourself into all of this legal jeopardy in order to use it. So it's just, it's just not worth your while. And I think what he ignored was that you just have 20 million George Hotzes now, right? Applied to this problem where before you couldn't even afford You can't afford George Hotz to come do your security white hat hacking. I do, I defer with you on what Sergey is doing because I feel like it's not as though we don't have calculators, but we still started off asking the models to do simple math questions, right? And at the end of the day, right now the model kind of like, if it wants to do a It brings out a Python or Excel or something else. It doesn't bother to process it internally within the LLM, which is structured really for language and reasoning. And I think in that way, what Sergey is building is kind of a plugin that the LLM as an orchestrator may end up using because it's just a more efficient way. Because what Sergey, I think what Sergey is doing is really he's trying to get to a Maxwell equation without a Maxwell equation. He's trying to get to the final partial differential equation kind of solution on this very complex number of lines going through the PCB. He's trying to get to that solution without doing this supercomputing task of millions of little interactions between all of these things. I think the models may end up using that anyway. They're not going to do the Maxwell equation internally. They're already not going to do that. They're going to run Python or something else anyway. So I think in that sense, what Sergey is doing and what I think AlphaFold and all of these things, which are primarily scientific kind of differential equation solvers really, in some sense, are actually— will just plug into an orchestrator in the end. I don't think the AGI in that sense is really that kind of orchestrator which can use all of these tools and not necessarily doing the calculation internally, perhaps.

Well, I certainly think it's going to start that way. But I would point to image as an interesting counterpoint that I think at least shows where this could go, right? Because we don't see in today's world a language model purely existing at arm's length with an image generation model and prompting it purely through text. We do see the unification of the visual and the language latent space. And I guess I have a hard time seeing why. And there's obviously a timeline question, and my general philosophy is to try to reckon with the possibility of shorter timelines. And then if we have more time to deal with these things, probably that'll be good. We'll take it. But why wouldn't it be the case as we think about exponential compute, that at some point all these latent spaces get joined together in some deep, non-arm's length, but truly integrated way where the model can both reason about Maxwell's equations and recite them and call a calculator to run a certain version of them, but also have an intuition that is potentially really powerful, um, and kind of alien to us, but sort of natively operating in that space. You know, you can imagine a world where, in the same way that I kind of know where my arm is, you know, that an AI just has an intuitive, non-verbalized sense that like, this trace won't work, but this other one will work. And it just kind of feels it based on everything it's learned and all the reinforcement that it's got.

Um, similar, similar to human intuition where we, we might not do all of the calculations, but we get to a point where we make predictions, which if we did try to calculate them would be horrendously complex, but we make an educated guess anyway and we kind of get there, right? So yeah, I guess the other—

Catching a baseball is always the other example that I go to where it's like, you're obviously not given the luxury of time to compute all the forces on the ball, but yet you can just kind of reach your hand up and grab it, or at least most of us can, many of us can. So it's clearly possible to have that sort of intuition for, at the crack of the bat, I kind of know where I'm going. I see that happening in just an ever wider number of domains. And to me, that's the most likely form of superintelligence. I think they'll be outstanding reasoners and quite likely superhuman reasoners in many respects. But when you combine that with that deep intuition of just like what will and won't work and being able to sense that at a glance and to do that across all these domains, from circuit board design to material science design to protein folding, to if I perturb a cell in a particular way, like, what's the next state of the cell going to be after I do that, to dozens and dozens more? This feels to me like where we really create something that is just a qualitatively different kind of intelligence, and chain of thought goes away as a way to understand it. You better hope that it's like being forthcoming with you in the chain of thought, because it doesn't necessarily need to. And there was really interesting work recently from Google about different architectures and how much work they can do internally before they have to externalise their thinking in the chain of thought. And the transformer is in some ways good there because it doesn't have, as opposed to a latent, like a state space model, it doesn't have this sort of long-term internal state that it can update ongoing indefinitely. Right. It has this kind of just finite context, and there's only certain traces causally where data can influence the next token. So it has to externalise, and that's great, but it doesn't, you know, it notably doesn't have to externalise the, you know, the Nano Banana model does not have to externalise how it's going to come back at you with that next image. It just spits it out and then you're looking at it and now you're like, here it is. So yeah, I really can't get off of that, I guess, in terms of why I expect some of this stuff to be so hard for us to It would be very interesting to see it operate in something like retail because I think the—

I have some knowledge of retail and the number of strategies that I've heard of. For example, one strategy in fast-moving consumer goods is to go and get goods which are about to expire, about to hit their sell-by date from larger stores and then move them to smaller stores. Stores. So the smaller store can often move the goods faster because it's moving it in smaller chunks. So they buy at a discount from a larger store because if you have sell-by date, which is like 2 weeks, you know, 2 weeks pending, and, you know, a larger store can't get rid of it, they buy that and then they, you know, vend it in smaller chunks and they get a discount. So because the retail margins are so thin, there's a number of strategies that people use which are really kind of like you're not going to learn in business school. It's really like small-scale vending. There's a lot of stuff that people do which you, in business school, you're like, oh, you have capital, you have margins, just go do this. You don't go through this process of how do I get a larger, like a 1% larger margin? How do I grind that out? So I don't know whether vending will be the first place that you see it though. I've always imagined that it would happen in financial trading first. But, or, you know, cybersecurity, it's kind of happening right now. But I've always imagined it would happen in financial trading.

Certainly financial trading offers very fast feedback and verifiable outcomes in a way that programming does, but not too many things do. So it does seem like a very good candidate. I guess the challenge there is probably it's like the most secretive domain in the world, right? What comes to mind to me is, like, this might— I mean, it's surely happening to some degree, right? I don't know what Jane Street is doing, but they're definitely training lots of neural nets. How much has this kind of already happened and people are just keeping their strategies close to the vest? I assume it's got to be significant, but this is one big blind spot, actually, for me, because I've I've had a hard time finding anyone who wants to talk about it on the record.

A lot of what Renaissance and Jane Street and a lot of what they do is actually kind of standardized kind of models and algorithms, et cetera. But they have a number of advantages. Number one, they have a latency advantage because they always co-locate with the exchange. And the latency advantage has been something which has been in play for more than like 120 years. People used to try to get a latency advantage over telegrams, right? Like, you know, you would have the horse rider going on one way and then you'd send the telegram and then the telegram would reach first and then the pricing would change on the other side before the rider with the horse got there, right? So you've had over years, like, you know, this latency advantage thing has been built out. I think the next upcoming one perhaps is already there is Starlink. Because if you have low Earth orbit satellites, potentially you can get a message from London to New York faster than you can through the underwater cable, potentially. Again, you need a bunch of things to line up. And that latency advantage, even if you have the best algorithms, even if the model is exceptionally good, it wouldn't be able to beat the latency advantage because the other person is just seeing your cards before you play them. And for me, that's been like, That demarcates how good the agent, how much profit the agent can really make because there is a number, there's an amount of profit in the sub-1-second range, which I don't think the agents would ever get there without the colocation. And that kind of blocks you off. And then besides that, there's a lot of like data cleaning that the Renaissance and Jane Street guys do. And that is why they hire PhDs to do really data cleaning Because you need to understand that this data is actually going to have real impact on the financials and you can't just mess it up, right? So they do hire a lot of people to do very nitty-gritty data cleaning work. And then finally you have the selection of the signals and the market making itself, the AI-assisted or algo-assisted market making. I think people spend a lot of time on like, oh, they have exceptional algos. And not a lot of time on the infrastructure, the data cleaning, and all of this other stuff that has to come together for you to have a successful firm. And so I think what would be interesting at some point is if the models or these model companies start to have their own colocation or own trading arms. To some extent, Google DeepMind had one. Demis was starting off on this process. But Google headquarters didn't like it because you could say that Google would have overwhelming advantages in terms of predicting stocks using all of the data that they have internally. Facebook too. But putting you in finance makes you very regulated and it puts you in a lot of like, where is the Chinese wall? What can people see? What are people not allowed to see? What can the— are your systems segregated? Are they not segregated enough? And I think the level, and financial regulators are not technically that sophisticated. So they ask for things which are very clearly demarcated. They're like, I want your entire group to move to another building. And people are like, look, we're already segregating the devices and all the data. Why do we need to move to another building? And the regulator doesn't care. Regulator's like, look, I want you guys in a different building. I want you guys to have a different business unit. I want you guys to have different financing. If this unit is regulated, no one in this unit can talk to that unit. There's all of this stuff that goes on. And financial firms exist as a function of that regulatory process. And I don't think, to this extent, I don't think the firms want to submit themselves to that process yet. I doubt, and I doubt some of these model decisions can clear.

Like the barriers.

Like, does the model have inside information? You don't know. Did it read some inside? Was it trained on inside information? Was it trained on material non-public MNPIs, material non-public information at some point? You can't say for sure. And then that brings up a whole host of questions. So perhaps finance would be, I think vending is actually easier. It's easier to take on Amazon than it is to take on Jane Street. And again, you have the same infrastructure and information problems, but it's a much less regulated market than I think finance.

How do you think about more macro strategy though? I mean, I think the— I don't know a lot about this, but my general sense is there's like high-frequency trading where the latency issues that you described really matter a lot. And are kind of a big part of who wins and loses. And then there's, of course, the more information and the more differentiated information you can have, that's always an advantage in any strategy that you're playing. But then there's this kind of other end of the strategy, which is like a slow-moving— I mean, to take the sort of canonical example, right? Like Buffett and Berkshire Hathaway don't don't time their trades to microseconds, right? They take very long walks and deep thoughts, and then they decide what big bets they want to place.

Yeah.

And I do wonder if we're seeing that start to happen, or if we will. I mean, probably you would see more trades than a Berkshire from a sort of global macro AI But it does still seem like there might be— I mean, I don't know, tell me if you think this is wrong, but I would guess that there's already a shift underway where all the big firms— this is like an obvious enough thing to do that they would presumably be training large neural nets on all the data they can get their hands on and potentially driving more and more of their strategic decisions via the predictions of a model? Is there a reason you think that wouldn't be at least late, early, far along in today's world?

I think every firm always tries. And typically one of the things is that the market is like a multiplayer game. It's not a single-player Right. And the thing is that when you— okay, number one, there are certain profit pools available at every latency and at every size. So it's not the same profit at the Buffett size as it is on the high-frequency trading side. So Buffett's profits are in that long run and in much larger size, but he also has a problem deploying capital at this point. He's got $150 billion on the balance sheet. He's very unhappy with the choices that he has, and he's just hanging onto that capital, trying to wait for a proper market downturn before he can deploy. So he's already capped out at his size. He's having difficulty finding investments at that size already. And any firm that gets to that size will face the same problems he has, which is you have a large pool of capital and perpetually you end up buying high. If you decide to buy when the market momentum is good and you have to wait long periods for the market momentum to go down in order for you to be able to deploy large amounts of capital at pricing that you like. I think one of the things that I'm sure the models do assist in decision-making, but I'm not sure whether they have enough context because there's a lot of human context in the market. There's a lot of sensing when someone else is going to play and when someone else is not going to play. If you're going to make a merge, if you're going to try and buy a company, you have to kind of know who else might bid against you. In the United States, at every capital size, there's a limited number of players, right? If you're going to do a $10 billion investment, there's only 7 or 8 players in the US that can make a $10 billion investment or larger, right? And if you're an investment bank, you kind of know all who all the players are, and you kind of know the dynamics of who's talking to who. And I'm not sure whether investment banks have kind of CRM ERP systems, but I'm not sure all of the knowledge of a managing director who has a 20-year relationship with the head of KKR is fed into that. I'm not sure whether Elon has a specific banker at Morgan that he likes, and that banker was working at Doge. was pulled out of Doge to work on the SpaceX IPO, right? There are all these human pieces to it. And the models will get there someday if you have full context, right? Full context, full 24/7 context on every single one of the players. Yes, the models will eventually get there. But at this point, not quite there yet. And the players on the field make these very human decisions, which are not quite caught up in pure kind of pricing Elon wants people who are going to hold on to the shares for longer. He wants people who are not going to sell immediately. He wants people who are going to commit to being there for the long term. So he's willing to take lower pricing. He's willing to offer it to retail, even if other bidders are higher. He wants to place it among the same kind of Tesla fan base. There's all of these questions, all of these things that people All of these intentions that people have, which they express through the process. And I don't think the models capture all of those things quite as of yet. Eventually they might, but not quite as of yet. And for the macro, that's where all of the human play comes into play. People are much more concerned about their own ego, long-term, kind of long-term strategy. They're not that, you know, once you have like $10 million, like you're not really concerned about, am I going to get another $100 grand by screwing over Elon, right? It doesn't matter anymore. You have like reputational risks and other things that you're concerned about. And in fact, people who do screw over people in these iterative games get bad things happening to them. One of the reasons I think that Lehman Brothers went under is because in a previous instance, Lehman refused to participate in a bailout for another firm. Hank Paulson remembered that and he's like, well, we're not going to participate in a— we're not going to bail Lehman out. Lehman can go do what they do. Lehman failed. Dick Fuld always said, look, this is because of a personal issue. This is not because that Lehman should have failed. Lehman could have been like Goldman. It could have been been saved by Buffett, but Paulson was unwilling to back the firm as Treasury Secretary. So I think there's a bunch of these things which are very human, very personal at these larger sizes. And the macro— so you can't just make a macro bet at the larger size. There are all these human negotiations. It's more like personal. Buffett went into banking. He refused to back Washington Mutual, but he decided to back Goldman. Because by the time they got to Goldman, he knew Hank Paulson was Treasury Secretary. He knew Goldman would get bailed out. So before he went to Goldman, he had that sense and then he put the money in. I don't know whether he had discussions, perhaps not, but yeah, he had some idea that Goldman at least would get bailed out. So I think there are all these things that are not captured yet. There's all of this tacit knowledge. I think the same with PCB layouts. There's all this tacit knowledge. And the economy is particularly difficult because there is no case where you can compare the same event under different circumstances. Every single event is a unique event and your actions in this event affect the actions that people take in the next event in the next period of time. It's tough, like time series, it's tough. Let's see what happens. I think we'll—

When I hear all this, should I understand it I think one way to parse what you're saying would be to say there's a lot of human barriers to adoption at existing firms.

Yeah.

There's also some scale at which you are not just a term taker, but you're actually a market mover. And so that is kind of inherently a challenge for a sort of big data Yeah. Blind optimisation approach. But the flip side of that, I think, would be to argue that in the sort of vein of your margin is my opportunity, all those things that you're describing define the opportunity for at least smallish to moderate-sized funds to just work in a very blind way that doesn't care about reputation. you can't really punish some purely neural net-based trading algorithm, right? I mean, I guess we have AIs that beat people at poker, right? We have superhuman no-limit hold'em players.

Yeah.

So if we have that, I kind of am like, why are they so good? Well, one reason is they don't really fall into the same bias traps and predictability traps and You're wanting to, you know, having a grudge against some other player at the table or whatever that's kind of moving them off of an ideal strategy. So I hear all those things as kind of both why it might be slow to happen, but also why these strategies can win when they finally do come online.

I think we will get there, but right now I have difficulty with context. Really, it's a question of capturing the entire context. And I don't know what is the endpoint because we're already transcribing a lot of meetings, right? So the meeting transcription process has started. I think we will eventually have Meta's eyeglasses or Apple's eyeglasses or whatever will capture even more. And you can get sentiment analysis from a face. Right. You can see whether someone is disturbed or angry or excited. And so there's a lot of data that you can get there. And I think all of that data can be processed and can yield useful signals in business. But we're still a long way from the amount of data, extent of data capture that might be necessary. So I don't know how we get there without the data capture. That's what I'm saying. I'm sure the, like I said, I'm sure the The algorithms that will define the future already kind of exist. The compute for that future already exists, but the data collection and the context that is necessary is not there. It's not there in cancer drugs. It's not there. We just don't have the data. We have the algos, but the algos can't be fed without the data. And I feel that's the issue. The full context is not yet there.

Yeah.

So in other words, too much information is private for—

Or non-recorded tacit knowledge, not captured anywhere, which is kind of why like a lot of these jobs, you require an apprenticeship, right? You start off, you have a college degree in economics or banking or whatever business, and then you kind of join a firm and it takes you like 2 to 5 year apprenticeship under someone And in order to kind of like figure out what's really important in the market, what's not, you kind of figure out that whatever the Wall Street Journal tells you is the final word, not the initial word. And you're in the process before that final word gets published. So you need to, you're acting prior to the final word, so to speak. So if you've already read it on the Wall Street Journal, it's too late basically. You're not, it's already done. And there's all of this pre-publication stuff that you need to learn at the firm. And I think that is, if we can capture that apprenticeship process in data, then I think you can start to migrate some of this stuff, some of this decision-making process into the models. It may happen very quickly, right? It may just be like the model all of a sudden says, Oh, I remember everything now and I can learn anything. So just put me in, coach, put me in the room and let me in for like 5 days and I understand everything and I can help you. It could be that simple. We just clear the hurdle in the next like 12 months and that's it, it's done. And we don't have this whole like nitty-gritty data collection, data cleaning process. Could be, so.

Even the long timelines have got very short.

You know, I think this week might be the SpUD release. I think there's some, you know, OpenAI is very quiet post-Methos, and there's been some signs from the Codex team that they can beat the Methos SWE benchmarks. Yeah, let's see.

All right, well, we'll be back before too long, and I'm sure there'll be no shortage of things to talk about.

Indeed.

So, should we wrap up now?

Yeah. Morning, morning, sun come creeping slow. News on the wire you need to know. Man who built it signed the dread. Same hand bringing what he said. 1 in 20, all lights out. That's what they own self talking about. Ring of power, watch the good man turn. Ain't no flame too wise to burn. Don't pick up no stone, pick up the phone. This world's bigger than you alone. Don't look up, don't look away, you're just learning. Right to be afraid. Light the signal, not the flame. Light the signal, not the flame. Some will walk right through the fire. Trade their comfort for the higher. Some will close their eyes and laugh till the silence cut them in half. Same short clock, same short time. Every choice now gonna show your mind. Match in the hand don't mean you brave. Hunger for the truth. Every honest hand. A part of the team. Don't pick up no stone, pick up the phone. This world's bigger than you alone. Don't look up, don't look away. Every one of us got work today. Light the signal, not the flame. Light the signal, not the flame. Work today. Work today.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.