AI:AM — AI Engineers, Workflows, and Agents · June 22, 2026

It seems like it's pretty credibly claimed that GLM-5.2 has some significant amount of Claude in it in the distillation sense. It's been described as thinking that it is Claude, like, what model are you? It sometimes says it's Claude. It supposedly has the distinctive Claude voice. If there's no Fable, then it definitely stands to reason that their timeline to achieve Fable-like performance could also be significantly pushed back by that.

I'm worried about the Illuminati all collectively deciding that this is the time to exit. For years, we're like, nope, private is good. Like, I don't have to answer to SEC. I don't have to do this quarterly reports and all this thing. And suddenly it's all changing. It's very suspicious, right? The bear case is everyone with insider information is selling. The retail is buying. Like, This is it as far as AI progress is concerned for the next 5 years.

Why do you think they're so unlikely to do it?

I just feel like they don't have enough time. You know, realistically, summer is dead. Like, no one— the market doesn't respond until August. There's this whole, like, sell in May and go away kind of meme. So it's really like you get August, September, October, and Thanksgiving, you're done. So November 26th-ish onwards, markets closed. So you realistically have September, October, and November. You got 3 months. And in order to get all your docs in order, your docs have to be basically ready basically in June. So I feel like the timeline is— they already missed the timeline for this year. And I think the earliest timeline is going to be Q1 of next year. Good morning. It is Monday, June 22nd, 9:02 AM. Nathan, good morning and how are you?

Good morning, Prakash. I'm pretty good. You know, I had a lovely little Father's Day yesterday with my family and my dad and feeling refreshed and ready for a 5-day sprint this week.

How about you?

I'm good. And surprisingly, we didn't have that much new news over the weekend. You know, I was— I saw Peter Wilderford. He had a comment. It's been 8 days without any new AI news.

Yeah, well, we're all kind of waiting with bated breath, I guess, to see what's going to happen with this whole Fable situation. I wouldn't say we got no news. You know, we did get a GLM 5.2 out of the deal, which is something that will have a chance to talk about a little bit. But I think one thing I kind of wanted to start off with is I've been thinking a little bit, and for any of the listeners that tuned in before we came on and heard the song, been thinking a bit over the weekend about Jud Rosenblatt's comments from last Thursday, in which he called out the AI safety community broadly for a lack of cognitive empathy for the Trump administration. And I think this tendency sometimes is all too common a little bit in, you could say, AI safety community. You could say the sort of effective altruism spaces. There's a tendency at times to have a sort of circular firing squad dynamic and kind of hold allies to sometimes unreasonably high standards. This happened a little bit with the Pope too, when the encyclical was dropped and people were like, well, there's a lot good here, but he's not taking the possibility of AI consciousness seriously enough. And it was like, hey, count your blessings. You've got a Pope that's engaged. That's not to be taken for granted. And I think I probably have to say on reflection, I consider myself a bit guilty of the offense that Judd was calling out. I am a little still kind of not sure exactly where I land on it, but I made this song for our weekly highlights episode that tries to channel the perspective of the Trump administration or even Trump himself. It's kind of sung in the first person. From the perspective of Trump. And I do think, boy, yes, for all the concern that people have shown and for how kind of a weird mix of positions Anthropic has taken, it's certainly weird for anyone who's not been in the AI safety discourse for a long time and really understood the nuances of why they think this is so dangerous, but they need to be the ones to do it. I think, yeah, a little more cognitive empathy for the administration right now and just feeling like, hey, wait a second, if you're not that deep in this stuff and all of a sudden it's happening very quickly and you've got a ton of other things on your plate and this one company keeps saying that it's super dangerous and you need to do something, and then they put out this model and it's too dangerous to release, and then they release it. I don't think that they've shown tremendous sophistication in terms of how well they've understood this or how well they've handled it, but I do feel like maybe I was a bit harsh and maybe I should be giving more credit where it's due, even though the tactics used and the, you know, the sort of sledgehammer of export controls doesn't really seem like it's necessarily the right way to go about this. They're engaged, they're doing something, they're taking it seriously. And maybe that is a clumsy but ultimately praiseworthy first step. I think that's where, after a weekend of thinking about it, I come out. And certainly A lot of questions still remain unanswered. I do not really want to see them take this deeper and deeper into the national security apparatus and make everything secret. I do think diffusion is going to be important. That was probably one of the biggest takeaways from my extended conversation with Dean Ball last week that went out as a Cognitive Revolution episode. We can get into that a little bit as well. So I think definitely want to watch closely what they're doing going forward and don't want to act like just because they're doing something that it's necessarily good. But we're in such early innings of the government really engaging and taking this seriously at all that I think I have to say I was probably a little too harsh. And we should be using, to the degree that the AI safety community is serious about, hey, this thing is kind of taking on a life of its own and the whole phenomenon risks getting out of control, we should probably be a little more constructive and a little more inclined to use this as an opportunity to engage and educate and not ridicule because the level of understanding doesn't rise to match our own.

I am often extremely shocked at how inside the bubble we are. Because even when I speak to other people in Silicon Valley who are software engineers, the distance between being fully inside the bubble and outside the bubble is vast, absolutely vast. And it starts to get even bigger as you go further away from the bubble. Right? At some point you get to people who believe all of tech in general has been a scam for like 20 years. I've definitely come across people who are like, you know, all of tech has been a scam since 1999. I've come across many of those people. So, and you see them often. For example, I will give you one clear and evident example. My favorite, Warren Buffett. Warren Buffett did not invest in tech until he went into Google in— went into Apple in like 2014. And then Charlie Munger came— comes out like 3 or 4 years later saying that he regretted that they didn't invest in Google because GEICO was paying Google like $1 billion a year In ad fees. And they just didn't catch it. They just didn't realize it was basically the same as a billboard business, except Google owned all the billboards. But because they didn't realize that, they thought it was a technology company. And they thought technology companies can be disrupted by other technology. What they didn't realize was that Google was basically a billboard company and they owned all the billboards. So it's very surprising also that very recently Warren Buffett has gone into Google. So he has made a big investment, $10 billion investment in Google in their latest $80 billion equity raising. And it looks like he's going to back them to the hilt going forward. But it just shows you how much of a distance there is because it took him— Google 1998 and 1997 founded. It took him basically almost 30 years, right? Almost 30 years. So in 2026, they've been founded for 29 years, and then he makes an investment. It's taken him 29 years to get over like, this is not really a technology company. This is a billboard company. And all of the things that we have in the real world, in the physical world, have now transitioned to the digital world. And people are going to be paying the same amounts for them that they were in the physical world. And I think that just shows you what kind of a distance there is, how much time it might take for the rest of society to kind of acclimatize to this idea that everything is going to be— everything that moved from physical to digital is now going to move from heuristic-based software coded by engineers to AI running things without the engineers doing much more than making up a few rules. It's very, very difficult, I think, for most people to grasp that and accept it as a way that the world will be run. It's always nice to have this idea that there's someone responsible making these rules rather than the AI just kind of put things together and like, whoa, that's your world. And I think that that's part of the story of human disempowerment that I think people, and especially people in the AI safety field, express. We had a good guest last week, Liron Shapiro. Liron was, as you pointed out, he was on the side That whatever the Trump administration did, he was happy. Even though they might have might have been a bit ham-handed, he was happy that they'd come in and made a position because he wanted a pause, and he saw that as a legitimate way for a pause to happen. So how did you feel post the Lyron? You know segment. Was this was this was this was your opinion changed by Lyron?

For some reason, Judd's comments hit me harder, but I, I mean, I do listen to LeRon quite a bit as well, and he's definitely had the influence on me in the past of just making sure that I don't put my big picture worries too far in the background. For whatever reason, I didn't maybe find it as— I guess I've just had so many experiences with the Trump administration over time where I feel like They may be, you know, people often say things like they're— they've often got the wrong answer to the right question. And I do feel like that has happened enough times that I'm kind of burned out on it. So the simple argument that, hey, this may be a clumsy move, but at least it's some move, that didn't quite resonate with me as much. But then the cognitive empathy argument resonated more because it was like, okay, well, yeah, maybe it is kind of on us to meet them where they're at and really try to be constructive as opposed to just being critical. Even if there is still something that relative to, as Biden used to say, don't compare me to the Almighty, compare me to the alternative. We can imagine better alternatives here, But nevertheless, you know, we're in this moment that we're in and making the best of it. I think Judd kind of usefully chastised me for not being, uh, yeah, just not being as, as constructive initially as I, as I might have been.

Um, let me segue here to a friend of the pod, Dean Ball, who has actually been very constructive in his both, uh, criticism and commentary Of the administration and of the Frontier Labs. So you had a chance to sit down with Dean Ball, who basically was part of the Trump administration for some period of time as an advisor. And then he left. He went back to his Substacking. And then I would say about 3 months later, last week, announced that he was joining OpenAI. to head a team that was going to focus on strategic futures. So you had a chance to sit down with Dean. What is his perception? What is he about to do? What is his focus going to be?

Well, he's really stepping into one of the more pressurized environments, I think, in the world today. And it was striking to hear him say that he thinks that this role will be more important, more impactful, higher stakes than his work in the Trump administration. I think probably everybody knows this, but he was the lead author, essentially, even though it's not signed in his name, right? But he was sort of the lead staffer on America's AI Action Plan last year and really somehow is like walking on water. I mean, how many people can you name that went into the Trump administration, left the Trump administration and somehow have generally higher esteem after being part of it than they went into it with. That is not very common from what I have observed over the last however many years. But he managed to pull that off and now is actually not really going to be in a day-to-day government relations role, which you might naively think, okay, this guy came out of the Trump administration, now he's going in industry, he's going to be sort of a lobbyist. He said, no, it's not really like that at all. They have a government affairs team. That team is the one that is responding to all kinds of state bills and all the many, many things that are coming at them from governments both in the US and around the world on an ongoing basis. And they're the ones that are really kind of lobbying for and against specific policies. His new team, Dean's team, which is getting started now, is really going to be more focused on frontier AI policy, which only sort of exists. It's been striking that at the state level, there have been some of the moves that Dean and others have approved of, including these transparency bills, things like the RAISE Act, which Alex Boris championed in New York and has now obviously become a big focal point in his congressional race, but there's not too much there. And there's especially not a lot when it comes to governing recursive self-improvement and governing internal deployments of the latest and greatest models. One interesting story that came out over the weekend, which is, I think, unverified, and my guess is it's kind of overstated, was that Anthropic has finished another model, that there's now Mythos 2 or whatever, right? This kind of popped up. I don't know the person whose post I originally saw on it. It's not somebody I'm super familiar with. I saw another commenter say that this person is super credible. You can believe this is true, certainly just on timelines alone, right? If they had an early Mythos early this year, then 6 months later, they probably should have a meaningful V2. So timeline-wise, that makes sense. From what I've heard from people at Anthropic, they still don't have Mythos or Fable access internally. Like, it's still off for all of them, or nearly all of them inside the company as of now.

Mm-hmm.

So This now raises this interesting question, and I haven't heard anything. Now, they wouldn't necessarily tell me. What I have been told in clear terms is we have no mythos or fable even internally right now. What has not been— I haven't asked directly and I haven't been told, is there another non-public model that you do have that you can lean on? But this is where it starts to really get super interesting and fraught, right? Dean, on the one hand, said, I don't want to see this stuff go all super secret. I don't want to see the government try to monopolise AI. I want to see broad diffusion. I don't want to see nationalisation. We're going to need diffusion to counter the state's impulse to try to control this technology, because only if it's diffused will there be a broad enough constituency to say, hey, we don't want this to be nationalised, we need it. So that's kind of one big force. But then this other force now is like, the government is going to have a hard time regulating a model that they don't necessarily know if it exists or what the details of it are, and it's not being put out publicly. So there's certainly some— and again, I have no idea if this is even true, although it stands to reason, but I have no idea what's going on internally at Anthropic. Are they deploying Would they— are they, or would they even deploy a Mythos 2 internally while having Mythos 1 and Fable off for their own internal users? That would be a bit of a sleight of hand, you know, it would definitely be a little weird, but I do think we're headed into weird times. And so yeah, Dean's gonna be at the center of all this trying to figure out like, what should we do?

How do—

how does society How does OpenAI make good decisions in a regime where we don't know how fast this recursive self-improvement thing is going to go? We don't know how far it will go. We don't know how the government's going to react if we bring forward more powerful models. If we're told to take them down, we sort of feel like we have to take them down internally. Do we want to risk that with our new models that are even better, that we really want to use because we're hitting some sort of 8x coding output, escape velocity. It is getting a little bit uncomfortably AI 2027-like in some of these ways. And he's going to learn a lot more. I think we covered a lot of ground in that conversation. It's like more than 2.5 hours, a lot of little interesting details. He did comment a bit about OpenAI's relationship with Alex Borras. He had some interesting thoughts on the corrigibility versus character debate. Which, despite the fact that he's going to OpenAI, he intuitively said he comes down on the side of character as the more promising path to getting durably good, robustly good behaviour from AIs going forward. But he also said one big thing is he hasn't started in the job yet. And so a big part of the reason, and this has been a huge theme obviously for us the last few weeks as well, big part of the reason he feels like he needs to go in and take this job, is unless you have access to the inside information on just how is recursive self-improvement really going, where it's being pushed the hardest, pretty hard to make good policy recommendations around that from the outside if you're just kind of reading the tea leaves or reading between the lines of what is publicly disclosed. So he kind of, on many issues, you know, I think gave a pretty sort of mature thought or kind of a thought that I don't necessarily expect to change too much from him in the immediate term. But on some of these questions around how does one govern RSI, a big part of his perspective was, well, I got to get closer to RSI as it's actually happening at the research level. I'm going to have to be in close contact with researchers to really know what the shape of this thing is. And only with that kind of information Will I be able to do a good job in making policy recommendations? So again, it's kind of another one of these things where everything is sort of collapsing into not that large a number of entities, and watching them closely is one thing, but being inside in his mind is really what it takes to be informed enough to really be confident that you can hopefully move the needle with good policy analysis. I thought that was pretty in keeping with everything we've been saying over the last couple weeks, but definitely again, a kind of stark reminder that, yeah, this is getting to be a kind of small game board in some ways.

So let me segue a little bit before coming back to that topic. So First thing is we have GLM 5.2. It is an open weights model from a company in China, and it was released roughly about a week ago. And over the weekend, what happened was people finally started to get access to it and they finally start to use it in deployment. And the first thing that happened was we have a VP at— a former VP at Meta and Google DeepMind, Matt Veloso. He goes all day using GLM 5.2, didn't miss much. First open model that passes the bar as a daily driver. Things are not going to be the same. Damn. Now I want to buy some serious hardware. So this goes out on Friday and I remark on it roughly Saturday morning-ish. I remark on it and he has maybe, I don't know, 50 views or something at that point. The post goes to 800,000 views on my side, about 1.5 million on his. The next thing that happens is This is Knowledge Atlas Technology, which is the creator of the JLM 5.2 model, which has— and this is their stock price over the last 5 days in Hong Kong. And as you can see, this is our June 18th. This is where we started making remarks and this is the spike. So you can see the spike going up there. We don't— I don't trade on Hong Kong stock exchanges, but it's just a note on how quickly things move. The next thing that happens is that Elon says within 3 to 4 months, the Chinese models will catch up. Knowledge Atlas's co-founder chimes in, tells Elon it's not going to take that long. So we're at the stage now where we are probably going to see a Chinese model by the end of the year. Which is not that long, 6 months, 5 months, 6 months more, that they are going to get to the Mythos Fable level. So you have roughly February, Mythos preview is announced. December, you're going to get a Chinese model out. So the administration realistically only has about this kind of 10 months from announcement on the US side to when not only the government, but corporations in the US to be fully secured. Because again, a lot of the government in the US runs on Amazon and Google, on these massive hyperscaler clouds. So you can't afford to say like, look, the NSA is just gonna secure their operations and leave the rest of GovCloud and AWS unsecured and have those entities get hacked. And so that's where we are right now. Going back to who announced the next version of Mythos, it's a poster by the name of Andrew Curran. Andrew covers AI developments quite closely. 99% of what he announces is stuff in the current press that people are not actually paying attention to. This is one part of the 1% where he has some inside information. And he just— he does announce that he doesn't have a source that he can reveal, but on the whole, it is not something which is out of the bounds of what we expect. I had remarked also that the administration banning Fable probably accelerated Anthropic because All of the GPUs that would've been devoted to doing inference for Fable, and we're talking about millions and millions of GPUs here, would then have been freed up to do the next version of the models themselves. And the final point is that, does it matter that the teams internally don't have access to Mythos and Fable when they probably have access to The next version, Dream or whatever other version or Sonnet, like some other version which is more advanced than Mythos and Fable. Does it matter if they have access to those things? If they can, for example, create synthetic data that will train the next version of the model? What really is the administration going to be able to do when you have Last week, a Nobel Prize winner, the co-creator of AlphaFold with Demis Hassabis. And really while Demis was doing more of the core kind of ML and organizational work, John Jumper was doing more of the biological work. And John Jumper jumped to Anthropic last week. How long do you think a US administration Can hold out in the public's opinion against a bunch of Nobel Prize winners. It's very difficult if a Nobel Prize winner says, look, I want to do this research. And you're going to say like, I'm banning you. I think the public opinion, the public's opinion, the Supreme Court's opinion, it's just not going to fly. So I think the administration is actually fairly limited in what it can do. Because especially in the time of social media, I think public opinion matters a lot. And this administration in particular is very keyed to social media.

They—

what happens on the internet is real life for them. You can influence their decisions and policymaking by posting on Twitter, right? So I think we're in that stage where there are some people in administration who would like to slow things down in order to give the government more time to adjust. But there just isn't that much time in this kind of 10-month window that they have. And if they don't act quickly, events will outpace them. And I expect events to outpace them as they are already.

Yeah. I mean, I think events will definitely be likely to outpace them. I agree with that. I think 2 little qualifications I'll offer on your statements there. And then I don't see Swyx in the green room, but maybe you do. There he is. Okay, here we go. We'll have Swyx in just a second. The first is, don't forget that AI is super unpopular.

Yeah.

So I am not so sure that they will feel like they can't override a Nobel Prize winner. I mean, Demis is a Nobel Prize winner. And I don't know, I think they're not afraid to bully some scientists. I think we've seen that over time, right? So especially if the public is like, yeah, this AI stuff freaks us out, we don't really like it anyway. And now these companies are all saying it's super dangerous. I'm not so sure that they won't be able to hold the line for longer. And the other thing I think to keep in mind too, and this is analysis, not confirmed fact, but it seems like it's pretty credibly claimed that GLM-52 has some significant amount of Claude in it in the distillation sense. It's been described as thinking that it is Claude, like, what model are you? It sometimes says it's Claude. It supposedly has the distinctive Claude voice. I haven't used it enough to have a real point of view on that. But I think it's not entirely obvious that the Chinese models will surpass the American models if the American models are meant to freeze on that same timeline. I mean, given enough time, they will, but to the degree that they are sort of doing a lot of the post-training and refinement of behaviour that is making them really useful as a daily knowledge work driver, if that's coming from Claude, then if there's no Fable, then it definitely stands to reason that their timeline to achieve Fable-like performance could also be significantly pushed back by that. So a lot of fog of war there, obviously. But I do think those 2 things both kind of strengthen the durability perhaps of the administration's current position.

We will, we will certainly see. Without, without further ado, let me, uh, bring Swyx up on stage. And hey guys, hi Sean. Hello, hello, hello, hello. Let me, let me do a quick intro. Um, I have with us today, uh, Swyx, or Sean Wang, universally known as Swyx on the internet across the developer ecosystem. He's a cornerstone figure in the transition from traditional software development to what he effectively named and defined as AI engineering. Before organizing the global community around artificial intelligence, he was a quantitative finance analyst who taught himself to code, eventually leading developer experience and infrastructure design at major technology companies including AWS, Netlify, and Temporal. Today, he is the founder and primary organizer of the AI Engineers World Fair, an event that kicks off next week at the Moscone Center in San Francisco, bringing together over 6,000 developers, researchers, and enterprise leaders. Furthermore, Swyx is the co-host of Latent Space, an immensely popular technical podcast that serves as the definitive audio record for how frontier AI is actually being built in the trenches today. His front row seat is to the most critical debate in the technology sector, how the industry stops AI from generating unmaintainable, low-quality code, what he calls slop, and starts building reliable autonomous engineering teams. Operating as an advisor to Cognition, the company behind the autonomous coding agent Devin, his technical stance focuses strictly on the craft of implementation. He advocates that the raw intelligence of an AI model is insufficient on its own. What matters now is harness engineering, the complex infrastructure, memory systems, and security guardrails required to turn raw AI outputs into robust, production-ready software systems. Swyx, welcome to the show.

First of all, what an intro. I hope that was quite Claude, or I don't know if it's more insulting than if it's like AI written or not. But no, it was very well done, very well researched. You even talked about the finance side.

Sean, one thing that I have always found while following you on Twitter is that you are really at the forefront of the latest developments in the field in AI engineering. So what are you looking out at right now? What are you concerned about?

I think concern is a very strong word. I think it's, you know, that invokes like safety and what have you. I am mostly an optimist around the potential of AI and like, you know, how much we have left to go. I have a few themes that I'm curating. So basically, once a year, I was describing to a friend, I get to be dictator of AI engineering. Because I run my World's Fair and it's a very, very big show, right? As you read out, we have 30 tracks this year and basically all those selections are mine. I used to be a speaker, I used to do my talks, but now I don't only choose single tracks. My sort of main highest order bit is choosing what tracks to feature because that effectively sets the agenda. So things like auto research, I think very, very popular. We used to have a coding agent track. Now that has been killed for software factories. We killed the RAG track last year, and now we're on search and retrieval. We have split the GPUs into pre-training, mid-training, post-training, and inference, and those different cycles of that. We're also focusing a lot on data quality. as part of our focus on pre-training, which is basically the idea that your model is only as good as the stuff that you train on. And actually you can train a lot more efficiently with better datasets. And people actually want to buy and sell better datasets, including RL environments. So there's a lot of really interesting themes there. And then maybe the last one I'll leave you with is memory and continual learning, Which also obviously a very popular topic. I think a lot of what I care about, I think, is the— what is like, okay, one is like the near term, which is exactly what you read in your intro, which is like the system stuff, like a way to put together the harness. I think most recently, like even Greg Brockman was like, yeah, the model is no longer just the product or like Something like that. Like it's model and harness that is the product. And I think that is something that, you know, we on the system side have been saying for a while. But it used to be cope and now it's less cope. It's still a little bit cope. But I do think like there's all these interesting directions, right? Like I haven't even talked about like voice and real-time AI and all those other things, but those are smaller categories compared to the major stuff. So I'll pause there. But those are like the rough categories that I've drawn that are the main topics of conversation. I will say there are things I am worried about that we can talk about, which is regards to the IPOs of all the labs, and also what happens next year once everyone is like exited and we have all shipped Fable-class models in every lab. And then what? And I think that's something that's very interesting to dive into. Cool.

Let's do it one by one. And I don't know how much time you have, but we can run a little long if you can. So what are you seeing on the continual learning front right now that you think is kind of most interesting? That is something I've been watching for a long time and kind of waiting for something to tip. And I thought nested learning might suggest that Google was going to take the baton for a minute there. And obviously that hasn't quite quite happened in the way some of us might have speculated. So what would you, as you scout out the frontier of continual learning, what do you see right now?

Okay, an interesting, very fun meta point that I actually double-check every time I talk to a Frontier Lab person is what gets published and what doesn't get published. And for the consistent last 2 to 3 years, every time people have talked about Google, The overwhelming consensus is that if a paper is good, or if an idea is good, it does not get published. So you should have that in mind when you read anything that's published from Google, right? So, you know, okay, so with regards to continual learning and memory, so there, this is a choice, like I almost did not do this track, because there's a big split between the models guys and the systems guys. The big split is, do you update your model weights or do you not? Is this a glorified RAG in another format where you store things in a database and you look them up? Is that memory? Is that continual learning? Yeah, actually, when I do a thing and I write a skill and then I do a thing again and it calls the skill, it does learn, but it's not machine learning. It's zero gradient if you want to be really fancy, but it is not machine learning. It is in-context learning for sure. Anyway, so the point being, okay, well, what are we talking about when we have a community come together and do continual learning? And the more machine learning side of the spectrum would be like, well, we will update model weights, and the less machine learning side would be Yeah, I think this all comes down to how controllable and interpretable you want your memory to be, right? You're going to recall bad facts, you're going to not be— you're going to want to forget things. And can you control that? Obviously, the maximum control is you don't update model weights, and you just control what gets into the system so you can delete and monitor and debug. But for full internalization in the model of the things that were learned, you probably do have to train on it. And so, well, that is a whole other discipline that Trajectory AI, Ngram, all these speakers that are— and Adaption Labs, these are all speakers at my conference. The first half is the people that update model weights, and the second half is the people, the more systems people. And I'm not choosing a side here. I'm just making the observation that these guys don't like each other. Basically, the model people don't view the systems people as legit. And systems people are like, well, the model people, have fun training your model, but you're never ever going to have a memory system that you understand because you're just updating weights. And so it's just continued pre-training or Whatever. And I think that's fair. That's a fair discussion. Yeah.

What do you think the enterprise wants? I think one of the things that is really interesting about your position is you're both on the leading edge of the technology itself, but then you also have this kind of Fortune 500 executive surfaced your engagement where I think you would probably have a much more informed perspective. I'm very drawn to the weights side because it's just more interesting. I feel like there's some eureka moment there that I'm gonna get to unlock and grok and it's gonna be fascinating. And it seems like if it really works, then it could be more transformative. But I could imagine that you might hear from VPs of engineering that like, yeah, that all sounds great. But I do actually value being able to open the file and see what's going on inside. What attitudes do you think the customers have toward this stuff?

Quite simply, it just takes one security incident where you leak information that you weren't supposed to leak because you trained on customer data. Or my information was somehow exposed to my teammates' information, even though if we work in the same team, I'm like, wait, hold on. I am not giving any of this to the model. I think enterprises want cheap and perfect and private, let's call it, right? Those are the 3 things. And so unfortunately, that skews towards the system side today. But that's not to block companies like Ngram and Trajectory from doing very good POCs with some of the large enterprises. I think it's still at the POC stage, but even at this level, like POC is like a few million dollars, which is great. I think the question is, can they figure it out before the money runs out or patience runs out? I think that's what all startups and what R&D is for, I guess. It's figuring this out on production traffic. I mean, the beauty of this is that you can just run your traditional harness system that is fully old school implemented with Basically RAG on some memories. And then you can run a shadow system that has the online thing and you can just compare and A/B test. So I think all those problems are solvable. They are open research questions. We are unlikely to get any papers about this because it is so valuable as a problem for all startups. And they may all get steamrolled by the next architecture. Right. If like, we're only doing this because context length is the slowest Moore's Law in the industry. Like, we've, we've basically gone from like 1,000 token context length order of magnitude to 1 million in 3 years, which is actually kind of slow as far as like everything else is concerned. And I don't think we're going to see like 100 million, 100 trillion It doesn't scale like the other stuff scales. And so I do think if something comes along, state-space models become a lot more effective or whatever. By the way, most people don't think they will. That's a whole other topic. I would say that then you need to update weights. You can't rely on infinite context because we don't have infinite context because we don't have infinite memory. And so I do think like we do need to figure out the systems there. In the meantime, people are just stitching systems together as they should.

So one of the questions that I had for you is, to what extent do you see this kind of balance between the hunt for capability versus the hunt for cost. Because I think in the last, let's say, 3 to 6 months, especially as we've had these more capable and much more expensive models start to be published, there's been this question of, you know, are we paying too much for tokens? Are the tokens being used effectively? Are enterprises actually getting value or ROI out of these tokens? Does this even make sense to deploy? One of the questions that I often see CEOs ask is, I'm already paying my developers, why do I have to buy them the tokens? Right. So to what extent do you think like startups are shifting from, oh, you know, I got to deliver more capability versus I got to be more cost effective on what I deliver?

Yeah, I think startups should max capability, whereas enterprises probably should max cost effectiveness. And I think that feeling has definitely shifted in the last year or so, mostly because the publicity on token maxing issues from like Meta to Uber and all those things. So I do think that is a problem, but there's always going to be a core group of influencers and experimenters and R&D people that will get a lot of the headline by capability maxing, as they should. I think that's something that continues to lead the industry regardless of the focus on efficiency. Efficiency will always be a topic that we want to better understand and deploy. But we also want to understand the best capabilities that we can get. That's why even if Fable was very slow, very expensive, we would still try to explore as much as we can on it because that is probably a preview of things to come. 2 to 3 years down the line when it is more affordable. So I think that makes a lot of sense. And I like that we are able to try all those things from labs because there is a darker future where all these things are tied to their products, right? Where Anthropic doesn't release Fable in API and you can only access it through Claude code. And so To the extent that all the model labs are still committed to an open ecosystem, I think that's great. And so because, because like ultimately, they're like, we should all, we should apply, we should be applying AI to more than just coding and B2B SaaS. Like we should probably try to apply it to more creative and unusual things. And that will only be done via the API by like the very long tail of people trying different things at the capability frontier.

So you guys just put out, and I know you were significantly involved in the development of this Frontier Code Benchmark, and I feel like this ties a bunch of different threads together in an interesting way. One is it's the latest and greatest yardstick for where we are on the automation of software engineering. I think the results as I understand them, at least strongly suggest that there will be a high willingness to pay for a Fable-class model, just based on what I understand in terms of success rate per dollar spent. But I definitely want to get your take on that. And then there's, you kind of alluded to the RL environment cottage industry, and in some ways, a benchmark and an RL environment are squint at them, they're more similar than different, right? In terms of what role they play in the ecosystem. So maybe just first of all, I think people have probably seen the headline, but give us a little bit more on what were you really trying to do and get at and measure with Frontier Code? And then based on the results, you can kind of comment on those different angles.

Yeah. Frontier Code is my baby at Cognition. Basically, so there's been 3 generations of evals at Cog. When the first Devin launch happened a while ago, there was Cognition Golden and then it was Junior Dev. And then when I joined, I became a lot more of an advocate, like basically day one of like, we should publish this instead of keeping it internal. And obviously when I say it's my baby, like there was Most of the work was done by the research team and the contractors that published this. We worked with Macquarie but also did a lot in-house as well. I named it Frontier Code mostly because it's inspired by frontier math. I realized that Epoch wasn't going to do code, they were just going to do math. I was like, okay, well, we'll do the code one. We tried to make this the last eval. I think that the interesting thing is that actually You observe about this RL environment and benchmarks collapsing. That actually started with CBench. It was the first agentic benchmark as far as I'm concerned, where you have an open-ended task and you can use whatever tools you want. But these are the golden problems that you need to solve. Frontier Code basically adds rubrics and updates the distribution of the task towards something that's more production. not just Django issues, but like, you know, C++ and Python and Java and all those things. Okay, so that's that. The reason that we were so excited about Frontier Code is, you know, you stop being able to articulate the differences in model quality with more saturated benchmarks like SuiteBench, because they're all like, at most you'll get like a 1 to 2% bump. And they're like, cool, but like, how much of that is memorization or what have you? Frontier Code is all out of sample. They're not in the training set. They're all graded and heavily rubriced. Like, we basically found like DeepSWE, Frontier Code— sorry, DeepSWE, SWE-Bench, all these things, they actually allow a lot of false positives in a way that models can cheat in the same way that, you know, during training, they basically have reward hacks, same thing. And we have an internal catalog of like 20 different ways that models cheat. And so basically, we just translated that to rubrics and, you know, ship that as frontier code. And I think like that is like how we want to judge models going forward, like the not just that whether they can pass the test, but can they write code that we would merge, right? Meter had this very, very interesting blog post where they were like about 50% of Sui bench code that passes the Sui bench test is completely unmergeable. Like it's just, it's like so low quality. Like, yeah, like technically you'll pass, but like, you know, like just on really stupid benchmarks. Like, did you, did you like modify a whole bunch of files you weren't supposed to touch? Or like, did you cheat on the test? Or did you adhere to like code style? what have you, just completely unmergeable. We want to guide the evolution of models towards maintainable code and against slop. A lot of models are incentivized for lines of code, and so we want to provide a KL divergence score against that basically. I do think the results show that the significant uplift that Fable and other upcoming models have on the current Opus class, which I think is very exciting to me. The fact that we have now a benchmark that can clearly articulate, okay, it's passing on these things and not passing on the other things. And there are entire classes of problems that are now being solved. I don't think we released them yet, but we will. And I'm excited for Cognition to do it. I'm just advising on Frontier Code now, but we have the roadmap planned out for the next 2 years for Frontier Code to continue to be a leading benchmark.

So when do you think Frontier Code gets saturated? What's your—

Never. So there's 2 parts of the strategy. Frontier Code 2026 will be saturated by the end of this year. We, you know, my estimate that it will probably hit like 80% by the end of this year. That's as designed. That's expected. It is based on open source repos, which will eventually get trained on. So they just leak. So like, you're screwed if you think that if you want one benchmark that will never get saturated, especially if you're based on open source. So the answer is very simple, just do annual cadences. So then we'll have Frontend Code 2027, 2028, all these things. And I think every year we'll just move the benchmark, move the goalposts from like, okay, this year it's rubrics for code quality. That's like the easiest possible thing. Next year, what is it? Right? My candidate right now is security. We want people to write secure code. But every year we can have a defining theme and that will be the focus of the year, very similar to what I do for AIE. But here we set the agenda through benchmarks. And then I think the other thing that I'm very keen on, which we haven't talked that much about, but I've talked about it on Hacker News, is the private held-out sets, the private evals. Cognition has Goldman Sachs and Citi and JPM and all these large banks and also the rest of the Fortune 500, not only banks, but banks are very, very big. Creating evals that reflect the problems that they have but are not solved, that are all private, is good for them, good for business, good for the industry. So that's what Cognition is doing. So Frontier Code Private or Frontier Code Finance, Frontier Code Retail, Frontier Code Telecom, Frontier Code Government, all these are like the work that remains to be done for building out all these private evals that we can work with the model labs on. But it is the way to communicate or translate industry problems through an agent lab like Cognition into problems that we can guide the model labs on improving the models, which I think is the function of an agent lab.

Is there any worry as you— and I guess let me check myself first, right? This notion of slop code that passes the tests but wouldn't be merged. I sometimes wonder about myself first and foremost, when am I being too precious to not just run with this AI output and use it versus when am I exercising my unique human taste? And I get that there's some of that going on here. Some of it's probably a little more objective than that. But I do wonder if in sort of emphasising the taste, the preferences, the readability, maintainability from the perspective of the human repo maintainers, do we potentially take ourselves away a little bit from the sort of Move 37 upside of AI? Because I feel like you wouldn't do this in math, right? You wouldn't say, like, oh, you solved this previously unsolved math problem. But maybe you do with elegance, right? That people are trying to do, like, an elegance factor in their proofs. But there's something about the math proof still where you're like, if you proved it, even if it's 100 pages of kind of incomprehensible whatever, it's a win. And here there's like a different sort of vibe. And I wonder if there is any concern around kind of putting a lid on the absolute ceiling of performance because we're kind of anchoring to human preferences.

Yeah, absolutely. I think this is a very good subtle point. I will say that when you talk to mathematicians about the math solving that and the Lean theorem generation that some of these models have been creating, they are like too detailed. Some like, like no human would actually like write that. And it's, but it's like systematically provable. It just doesn't contribute to knowledge. It's like Well, okay, you brute force this thing or something. I don't think brute force is the right term, but let's just run with it. Yes, I do think that at some point you should just actually step away from looking at lines of code and just say, does this work as I expected? That's great. Except to the point where other agents also need to read your code that is generated by other agents and they need to work together on it. And except to the point where you maybe get into debunkable places where you have very critical code that you're liable to the SEC for, or you're liable to whatever the healthcare authorities are. And you're like, I'm sorry, I've hardcoded this thing. I don't know what's going on in there. Is that an acceptable excuse? And probably not for the next 30 years. After 30 years, who knows? But I think it's a very astute point. I do think that for most people, you can get by with like, here's a black box, do whatever you want in the black box, but the inputs of this, the outputs of this, I expect this to match this quality. And I expect this to adhere to some coding standards so that my other agents can maintain this and parallelize this. I have a skill called Kakuna where I only harden code. Because that's the only skill I know. And it also tries to make code parallelizable because most models, if you let them run today, they will make monster files of like 6,000 lines of code and it's completely unmanageable. So I do think that you want to have some governance, but maybe that will ease off over time. And I think that's right. I think people maybe anchor too much on Move 37 as a dramatic changing point. Really, the water temperature is just slowly boiling around us. And I think that's probably a better description. It's not as fun and movie cinematic as Move 37. But I do think there has been obviously a sharp increase in agentic coding. We've seen it at Coggy, you've seen it everywhere else. And I think that should be managed well and you should not go YOLO straight into the maximum, like, I no longer care about the code quality anymore.

On this cost side, tell me if I'm reading the graphs right. My basic takeaway from a not super deep dive into the internals of Frontier Code was Fable gets something like 25% correct, approved, you know, would merge basically, right? Whereas Opus is kind of high single digits, a little lower than 10%. And that comes at an increased cost, but it's basically like a 2x, maybe a little less than a 2x increase of cost. And I look at that as a simple person and I'm just like, well, I'm getting more than twice the success rate. I'm probably also on a lot of other things that maybe didn't quite pass. I'm probably a lot closer to passing and I'm paying less than 2x for it. So sounds great. Sign me up. I think I should move all my tokens to Fable, obviously pending the availability of the model to do that. Is it that simple in your mind? And all this kind of, it's too expensive talk is just—

Yeah. So efficiency, like model and trajectory efficiency is very underrated and Anthropic and the other labs are doing very good work here to improve them. Right. Like one other thing that we're showing is the previous generation models are not actually giving you that much oomph for like the extra reasoning efforts that you're getting, right. You're spending a lot more tokens, but actually you're not getting that much better. So just keep them. on the lower side. But here with like, you know, Fable High or Fable Max, you're actually getting a lot more uplift, which is fantastic. I would say yes, on the hardest problems, you want to do that. But on most people, for most people, you are actually working on commodity problems. And on commodity problems, you do not need Fable.

What do you think of this new— well, not necessarily new, but I think Several people on the timeline are pushing the idea of the advisor model, the advisor structure, where you have a more competent model or less competent model, which identifies when it has a problem that it probably can't take care of. And then it calls up an advisor model, which is more competent. I think Alex Atala at OpenRouter, several of these firms, which will benefit from this kind of structure, are pushing this kind of idea. What do you think of that kind of harnessing?

Yeah. So OpenRouter, Fusion, and then Sakana Fugu, I think recently as well. But this problem extends back to any problem of model routing. You're effectively just doing model routing. You're just giving it a different name. So even like NotDiamond or what are the other model routing companies? out there, Martian. And then like actually the sort of SG Land group at Berkeley also came up with some similar ideas before. The other things I also mentioned is that actually Walden is one of the co-founders of Cognition, had this early on and they called it Smart Friends. They released it in Windsurf. Not that many people use Windsurf, so they don't talk about it. I mean, it's still like over 100,000 people, you know, But I think the interesting thing about the advisor strategy is that it is probably the right call as the next level upgrade from using the base cheap model. But it will still not have the level of intelligence. You can still benchmark a lot. And I think it can be very efficient as far as using the base model, the base cheap model as a model router. But basically, the dumb model doesn't know what the smart model can do. It just knows the rough shape of what the smart model can do. So like, you actually need the smart model to be able to answer what the smart model can do. And so like, when you are asked a question that is more complex than you have the intelligence to answer, you don't even know. Like, you're just going to straightforward answer it without knowing that like, oh, I'm supposed to call out to the smart model. And like, that's just like the theoretical, like, limitation. But in practice, I would— so, you know, I was, I was thinking about this 3 years ago. And I didn't do this because of this argument that like, actually, you need smart model first, and then you can delegate to the dumb model. But in practice, like, for cost reasons, for efficiency reasons, you actually want to do it the other way around. You want to, you want to start with dumb model, and then you go smart as a tool call. It just isn't as satisfying. It doesn't solve anything. Is everyone using it in production?

No.

Will it be a trend? Maybe for 3 months. Then you get the next Claude with more adaptive routing and you're good. Have one model trained end-to-end for adaptive thinking. You're probably better judged there than a systems-level thing. I am not toeing the party line here. I would say that Cognition wants there to be a multimodal future where we can be the model router rather than a single lab. But the model labs are very incentivized to do this all in-house. OpenAI acquired StatSig for precisely this reason. Um, and, um, uh, we're— yeah, I mean, like, your, your hands are tied if you're using any closed model APIs on how much you can affect the, the model choice.

Precisely because they, they are incentivized to bring it all inside the— their own boundary.

I mean, absolutely. You just, like, you just want the god model and, like, just, just, like, just figure it out, right? Like, who else wants anything other than that, right? Like, do you, do you want to, like, have these, like, I mean, only tinkerers will be like, oh, I'm using like the Franken model with like the Opus, but then like, you know, like the Gemini plugged in here at precisely this point. Like, like, I'm so smart. I know this, you guys don't know this and I'm better than you. Like, sure. But you know, that'll never be an industry thing.

Yeah.

So Andrew Lee from Tasklet said something to me that has echoed in my head since then.

By the way, what a pivot.

Yeah. What a—

like, I know you covered Shortwave and I was like, I, he came, he came to me for that, for that podcast. I was like, you got next thing, dude. And he got the next thing.

I still use Shortwave actually. Although less, I mean, it has, they've taken their focus off of it. And so increasingly these days I'm just having Claude code, you know, go into my inbox and make sense of it for me that way. But I mean, even that is sort of motivation for this next question, which is like, so he said basically everybody is trying to build the same thing. And he also said there's only kind of 3 sorts of software survive. You've got the model companies, of course, which are pushing the frontier of capability. And then he sort of sees himself as playing, and I think you could make the case based on what you just said, that Cognition is kind of also playing at this level of some sort of neutral layer that tries to help you not get locked into one model provider's ecosystem so that you hopefully have the best of all worlds and also some recourse, right? If something gets taken offline or the price suddenly gets 10x'd on you overnight or what have you.

Yes, that is true.

So do you think, how much How far do you see this convergence going? Is everybody kind of building truly the same thing, or do you see meaningful divergence still between these kind of neutral platform layers? And then one more I'll give you to chew on is, how much of a problem is the price discrimination? And would you advocate for something— I've floated this a couple of times, nobody's really taken the bait on it too strongly yet, but I could see a government action being like, you can't price discriminate beyond maybe a certain point. Right now we seem to be getting greater than 10x per token cost price discrimination for your first-party subscription versus what you pay on the API. And that seems like it's a huge problem for app layer companies. So you could imagine a sort of price neutrality rule potentially really helping to push the ecosystem toward more diversity and more, you know, less lock-in, more option value for everybody?

Okay, I'll quickly respond on the second one, because I think, no, it's never going to happen. Not in— not especially not in this government. But also, like, it's a very— like, at that point, just go ahead and nationalize the labs. Like, you know, like, don't half-ass it with, like, price controls. Like, just nationalize the labs. It's okay. But, like, short of that, which, like, probably not going to happen, like, people will revolt. Like, you will have, like, Atlas Shrugged. Like, literally, people are just going to leave the US and, like, Anthropic just move to Singapore. And like, you know, like any other country will take them. Like, so I don't see that happening. I think the first thing, can you sort of restate the problem for me to queue up to respond?

Yeah. How much are people, you know, in the competitive landscape that Cognition is in, is everybody kind of building the same thing and competing to out-execute on essentially the same vision? Or are the visions themselves different enough that you see diversity in kind of the winner's circle?

Yeah. Okay. I think as far as an early market is concerned, yes, it becomes very, very commodity and very, very competitive. So like, for a while everyone was building their VS Code fork and then now everyone's building their background agent and trying to be Devin. And I think that's as it should be because we haven't really figured out the market segments yet. But look, luxury handbags, there's like 5 of them and they're all great and they all have differences that all 3 of us on this call probably don't even know, but the women know. And they target different market segments. I went to business school and my worst class is Marketing 101 because they were like, oh yeah, Chanel's clearly here and Louis Vuitton's clearly there, Kate Spade's down there. And I'm like, they're all bags, dude. But that's what a mature market looks like is they do have market segments that eventually spread out. Right now we're all making bags and we all want the coding agent bag, we want the background agent bag and that's okay. And now everyone's building the Slack bag, the internal Slackbot bag, like Ramp has one, Coinbase has one, whatever, Shopify as one. But, and like, everyone loves it. Everyone's like, this is the most transformative thing we've ever done. And they all sound the same. And that's exactly how it is, like, because it's so immature. But I think eventually we reach a point of market maturity where like, they're like, oh, like, there's this one's like for the, for like the smaller organizations for this kind of compliance needs, like, this is what you do. Everyone in the industry does this one. And then for the mass market, you do this one. Probably there will be a couple winner-take-all, winner-take-most companies that fit most people. And I do think those will be effectively the new enterprise tech giants. And so Cognitions obviously trying to be one of them. Tasklet would try to be there. But everyone's trying to tackle this from a different perspective. And I don't know if there will ever be— like a steady state for the next few years because it's so early, right? Probably took like 10 years for the landscape of social media to shake out to Facebook, Twitter, and what have you. And similarly, I think, similarly for enterprise software and similarly for AI. I think what's interesting is always there's a tendency towards that. One company is the Android and one company is the Apple. Who is the vertically integrated one and who is the open ecosystem? We will do some core thing and that's about it. I think Microsoft is skewing towards open ecosystem. I had my chat with Satya at Microsoft Build and then he wrote the article which went super viral. My God, to have a CEO posting like that on X, I think it's like, Something I never thought could happen. But that guy's still with it. He's still got the energy, which is really amazing. But I think as far as OpenAI and Anthropic are concerned, they want to be the Apple. They're vertically integrated. You will live within our ecosystem and be happy with it. And then Microsoft wants to be the more open one. And I do think that makes sense. That's how things do shake out. And then the smaller startups that come in and try to be the sort of disruptive incumbents will have to take on a different tactic because the big players are going to do and exercise their brand advantage and their moats to do that, right? So the Tasklists of the world are going to have to do something very different. Otherwise they will just be acquired by them eventually.

So looking at startups versus large firms, We see, I think, Cursor has this Cursor Origin, which is, I would say, a Git replacement.

Yeah, yeah.

Replacement for agents, basically. And the reason being that GitHub itself is not able to manage the number of commits and the exponential increase in really activity. How do you see this shaking out? Because it seems like some of our Software infrastructure has not been built for agents, so to speak, for the sheer amount of volumes that these agents are generating in terms of hits. So where does this go in terms of all of these bottlenecks that we're seeing in Cloudflare, on the edge, in Git? How does this get resolved as agents start to become 99.9% of the volume on the internet?

So first of all, I did have the CEO of GitHub on the pod recently. And I point blank asked him, how come you're not handling all this volume? And he gave a reasonable answer, which is like, it's the largest scaling that they've ever done in every dimension ever. It's like, just imagine like a 10 to 15x in every dimension of scaling. I do think there's like a broader point about how all cloud infra basically needs to be rebuilt for agents. And I think that's true because we have this fundamental assumption of identity as a human, of API tokens being obtained through a UI. All that needs to go away. And then that's not even talking about the scaling side. The scaling side means that there needs to be a lot more instantly forkable sandboxes and file systems for agents to operate in. And I think that's okay. But also, I think it also means a lot more fraud and spam that comes in as well. I think that all that's fine. My worry is actually that people start to wall off parts of the internet. So you get not only a dead internet, but also closed gardens, closed walls internet, walled garden internets, where in China you have like, I live in the Baidu universe. And then your other friend goes like, I live in the Tencent universe. The BATs just carve up China. And I don't think that's what we want. We want a fully open and interoperable web. And so Cloudflare going like, we'll just We'll just ban some agents that are not part of Cloudflare, but then if you're within Cloudflare, you're fine. And I'm just like, well, I don't know if that's the ethos of the open internet that we want. And then to say, by the way, I'm not picking on Cloudflare. The same will happen to OpenAI and Vercel bans and what have you. So I do think we do need to scale. I do think that what Graphite and Chris Origin are doing is fantastic, needed to be done. And I would just observe there's a structural mismatch here. It's just unfair to GitHub that anyone good at GitHub who would be able to solve the problem can solve it at Microsoft. And you can get a really good level, I don't know, 50-whatever pay of, I don't know, $300K. Or you can go to Cursor and effectively get $5 million a year doing the same thing. But you're the cool, sexy one. And it's very interesting, right? When does that switch from like, oh, a startup can never beat an incumbent, to the startup is actually preferred to the incumbent? And GitHub has somehow gotten itself there.

Yeah, that's really interesting. A lot to chew on.

It's very unfair. I think maybe I'll also mention something like this. I have other stats that have come up in my podcast. The degree of scaling, right? We talk about GitHub is 14x in terms of number of commits, but also 10x in every other dimension of parallelism or CI/CD and all those things, right? It just induces so much infrastructure demand that is not human, is probably wasteful, that we are not really set up for, but we are just paying for it anyway, which is good because at least the economics have worked out. But in terms of number of CPUs in the world that we— in our cloud that we need, We're running into real shortages of everything, not just GPUs, but also CPU and memory. And the sandbox companies, like the E2Bs and Daytonas of the world, they're growing at least 50% month on month every month for the last year or 18 months or so. And it's stupid amount of just slop that's just spewing out from agents. And I get so much— I run a large newsletter. And every time I send it out, I get so much replies from Claws reading the mails and trying to reply to me. And I know there's no human on the other side, but I have to manually go and block these Claws from replying because this is just clogging up my inbox. So then I need an agent on my inbox to read their agent's emails. And so this is just all a huge, giant recursive bullshit. And I stopped reading emails. It's terrible. I don't know what to do about it. I just think like, you know, when you talk about scaling infrastructure like that, that's a very human story that is really happening. And we haven't given these agents money yet. Imagine what happens when they all have wallets, they all have stablecoins, and they're all buying and selling, and we don't know what the hell is going on.

Mine does have a Mercury credit card with a modest limit on it. What has it done to go do whatever they want with? Not that much really yet. You know, set up some things like the Zero XYZ, which is kind of a, you know, Swiss Army knife into lots of other tools. And I've got 'em buying groceries for me and stuff like that, but it's not, it hasn't taken on any—

Hook it up. Yeah. Hook it up to Rent-A-Human, see what it does. I think there's one, a lot of people are talking about it, but like, I do think like if you give humans as tools, tools to an agent, like it can do a lot more unhinged things and probably you have to pay for it obviously. And you have to give the human a budget as well, right? Like there's just all amounts of delegation going on multiple levels from agent to human to agent to human. And so like I do think like that should be doable. I haven't done it, but like if I was not running a conference, I would go explore that for sure.

Give an agent a human as a tool. That's inverting the normal.

Yeah. I mean, we've done this forever, right? The first time I called a human via API was with Amazon Mechanical Turk. And as a developer, you're trained to think that API calls are cheap. Until you've run hundreds of thousands of dollars in lines of code, you haven't really felt the pain of Well, if I don't debug this or dry run this before I run it, I'm going to waste $100K. So yeah, Mechanical Turk, CrowdFlower, Rent-A-Human, there's all this series, even Scale AI at some point was like that.

So number of things I still, I don't know how much time you have left, but a few things I would love to get your take on. One is like, maybe I'll give you a kind of menu to choose from. You can decide what you want to comment on. One is like, to what degree are we seeing this rebuilding of systems of record in-house? Like for my agent setup, where I have my main one on the laptop that sort of has access to all my stuff, and then I've got the Mac Mini, which has the more proactive Constitution agents that are supposed to go out and you know, wake up regularly and solve problems even if I don't give them new directions. You know, they have lower access, but they have their own computer, their own accounts to go do stuff. Point being, when it came time to have a messaging system between me and these several agents, I ended up writing my own. Or, you know, Claude wrote it for me, of course. But it's— I didn't go to Slack. I didn't go to, you know, Telegram, all these other things. I was just like, ah, I have— let's just do this my own way, solve my own problem. And sure, there's like tons of features that Slack has that I don't have. I also don't really need 'em. So I wonder, you know, to what degree we're seeing people kind of saying, yeah, we don't really need 90% of, you know, Git is out there, right? It's open source. GitHub is adding all these zillion things that maybe we don't even use that much. To what degree can we sort of in-house some of these functions that we used to pay for? I'll just ask that first as a question. Is that a trend that you're seeing? There's been a lot of talk of it.

Yeah.

There's been pushback. What's the status right now?

Not only that, okay, I have my own GitHub clone that I'm using. I have my own Slack clone that I'm using. And I just, like, just before this call, I just got off a call with my team where we have 3 people who've all vibecoded their own sources of truth for my speaker reconciliation. And so we were just like sitting there, like, check, like, I'm checking my clanker, they're checking their clanker. And we're all just agreeing on what's going on. And it sounds wasteful, right? What do you mean instead of one system of record, we have 3? But actually we all caught different things because we all have different points of view. And then we're all negotiating, here's what our collective view of the world is, but from our different points of view. And between the 3 of us, we caught more than what individually any one of us would've caught, which sounds like the right thing. It's wasteful in the small, but actually more precise in the large. And I think that's very interesting. But yeah, I mean, absolutely. I think people are replacing their SaaS with their internal vibe-coded versions. You're going to have dramatic data losses as a result and dramatic outages and what have you. But it's okay because it's your one versus it's your special baby. I do think that people do need to be more careful around keeping data and auditing data and all those things. Most people haven't been trained to write auditable code. What is an audit log? When do you need to roll back? How do you recover when you implement your authorization and all those things? But it's fine. We'll speedrun that along the way. I think the models will guide you quite a bit. What's interesting to me is a few people, a few like the top founders I talked to have talked to me a lot about how you should, the company should have a sovereign system of record, not the individual SaaSes. Because ultimately a lot of the SaaS economy is built on, we will sit on this, we will be a system of record for this kind of data. your meeting notes, your calendar, your email, whatever. And then now with the AI age, they just slap a chatbot on top of it, right? Like, we'll do the little sidebar thing that answers all your questions. Like, what would you like to do today? And yeah, Mercury just shipped a thing on Mercury stuff, right? But that's not integrated with anything else I do. That has no memory of anything else that I prefer, that I have to set all my skills again. I probably can't even set up my skills because there's no way to import that. So all that is bullshit. What OpenCLAW has is like, well, this is my personal agent and all the data is synced to me and then I'll decide what to do with it. Thank you very much. So how many systems of record should there be versus how many $20 a month subscriptions are you going to need to pay? And I think that is a big reckoning for a lot of people. What's really interesting or innovative is that you would expect the largest companies built on being the entire system of record to be most protective of their data. But Salesforce and Marc Benioff is out there saying everything will be available by API. You can just take it. I think that's very forward-thinking. It does mean that he's going to have to change his business model somewhat, but it's like either he does that or The Salesforce killer does that, right? And so he's just heading off the Salesforce issue. And I think more people should probably think about what the business model looks like beyond just sitting on your data and being— and then slapping a chatbot on it.

So one of the questions I had is, is there a— it seems like that system of record is valuable. Is there an opportunity for a system of record vendor to actually move into the business? So for example, if you have Veeva, let's say, right? Veeva is providing a CRM for pharmaceutical sales, and a lot of pharmaceutical companies basically use them. Is there an opportunity for Veeva as an expert on systems of record for pharmaceutical sales and workflows and processes to actually move directly into pharmaceutical sales? Going the other direction.

Yeah. Yeah. I mean, what's the?

Because you have a chatbot now, which interfaces with your you know everyone.

Look, it's gonna be so messy, right? Because like, if I'm a maybe like a pharma and like IT administrator, they're they're just gonna try to upsell me into this new thing, and I'd be like, no, like just just do it. Just do a little thing like. Stay in your lane. Everyone's going to try to be the operating system of blah. And there's going to be like 1,000 operating systems. And everyone wants to be that top layer. And then most of them will lose. And that's just how it is. So they will try. And to the extent that they're good at products or good at coding or whatever, they will get there. I think whoever owns auth, tends to win. So whoever owns like the thing that cannot be moved, or like, you know, you're authoritative anyway, you just have like an undefeatable prime position as opposed to the other people. So like, I think when you play this like sort of Game of Thrones, like you— where you start really matters because that dictates where you can move next. And whoever starts with off does tend to win there. So for example, like, like Rippling, right? Like, their start was famously compound startup. They want to, they want to do everything. They want to be the operating system or anything, but they picked the right starting point, which is we will own your org chart. And then you can layer on like IT admin on top of that. You can layer on like payroll on top of that, whatever. But like, we will first own your org chart because once we know everyone in your company, It is so easy to add anyone and do like SSO and what have you. And so I think that, I mean, I'm very much betraying my current biases because I do think there's been times where I'm like, oh, you want to own developers, you want to own frontend, but actually auth is better.

Yeah, that's really interesting. I want to go back because I definitely want to get your take on this IPO thing. You said earlier, you know, there's going to be obviously a lot of liquidity. What are you watching or concerned might happen?

Yeah, I'm not worried about the liquidity. I'm worried about the Illuminati all collectively deciding that this is the time to exit, right? Like we have, like for years we're like, nope, private is good. Like I don't have to answer to SEC. I don't have to do this quarterly reports and all this thing. And suddenly it's all changing. It's very suspicious, right? And it's like, okay, there's a group chat somewhere. And by the way, this has happened like twice in my career already, like where like suddenly all the good companies are IPOing and you're like, why? And then, and then like 2 years later that the market is in like a deep slide and you're like, oh, like, so, you know, nobody knows the future, but the powers that be have all decided that now's the time to IPO. And probably like, yes, all this money will go in and it will be great. And then it really remains to be seen, like, okay, do you have anything follow up that will be like the next like 5x in AI? Or is this it? Are we at a permanently high plateau? That's the bull case, man. The bear case is everyone with insider information is selling. The retail is buying, this is it as far as AI progress is concerned for the next 5 years. That is the bear case, right? That is like the, actually we've scaled LLMs to the max. Fable is about it. We now have a pseudo ban on all Fable-class models. So what do you have to look forward to apart from deploying the current gen into enterprise? Which is already priced in, by the way. So yeah, right. So then we're back to sort of the Ilya Sutskever type, like we're age of research, we need new model architectures apart from traditional transformers. And then it's an open-ended thing. So of course you would IPO there if this is an accurate reflection of what the Illuminati is seeing, which I have no insight into. But just like, Like, you know, you're playing poker, you read their hand actions, right? Their tell is they're all IPOing at the same time. And this has happened before in like the internet era and like the sort of Web 2.0 era. Like, what other tells do you need?

What odds would you actually put on that? I mean, it seems to me like—

So I didn't, you know, I haven't voiced this publicly. What odds do I put on that? I don't know. I think it's like somewhat like conspiracy theory. Like this is like the most tinfoil hat that I get because I think there are savvy players on the other side, but they also like truly fundamentally believe they're going to revolutionize the industry. And they did, you know, I'm sure this IPO conversation is very, very complicated. So I will, maybe I'll give you the upside, right? The upside is you and I and even the the people running the Frontier Labs and the Illuminati just seriously underappreciate the earning power of agents deployed to the whole world.

Right.

To the extent that your life is being— you're experimenting with your Claw and paying for things and buying little grocery things. That's cool. But what if it ran your whole life? What if it ran your mother's life? What if it ran your neighbor's life? Like, you're not seeing that yet. And that's probably a 100x from here. And so that's like, so if Anthropic's earning like $50 billion right now, 100x is what, $5T? $5T in revenue? Then the stocks keep going.

Yeah, a lot of headroom, I think.

But okay, then I temper it back again with like, okay, like you just, okay, what? You just casually forecasted a 100x in compute consumption. Where's that compute coming from? Excuse me? Didn't you just hear that Taiwan sold out for the next 2 years? Right. So then you need 10 years to spin up semiconductor industry in America, and that's not happening anytime soon. So there's all these multifactor things that go into the timing of an IPO. But like, yeah, on balance, I think like, I think you should be at least a bit worried that the big run-up has happened and you have, you're now either permanently wealthy or in the permanent underclass.

So let's say the IPOs happen and people leave, some of them retire permanently. For the ones who don't retire, What do you think are the ideas that are feasible for them to look at with large amounts of capital and pretty much no restriction on what they can, you know, do or research on? I mean, give— I mean, I presume they're not going to go into coding agents anymore, given that there are so many other people in coding agents. So what would a Frontier Lab, you know, Deeparty with millions of dollars, invest or research going forward?

Yeah, I mean, I think the categories are kind of obvious. Like, the first immediate category is robotics, right? Like, you solve the software world, cool, but what about the physical world? You know, what's the TAM on that? Don't even bother, just do it. Just don't even run the calculation. Just like, the more question is, can you do it? You know, are you a fraud? Beyond that, medical. I was at the Midjourney medical launch event. That was eye-opening. Maybe it doesn't work out, but as far as worthwhile, interesting things to work on after your cat picture generator going into, we will do CT scans quickly and fun and make it sexy. That's great. More people should solve health. travel to the stars and do all the grand quests apart from coding a B2B SaaS. Unfortunately, that's what we will solve first. But yeah, I think the new rich should probably dedicate their resources to the new grand quests, which we all know. We all know we want to solve aging, we want to cure cancer. We all know what the open problems are. We just don't know how to do it. And like, yeah, when you have infinite money and funding, you can actually go and solve that if you're so smart. And you have all this intelligence now to deploy on figuring it out. And to me, like what David Holtz did with his Midjourney money is existence proof that you can be rich and still try to do meaningful worthwhile things. I mean, and same for like Elon and Jeff Bezos and all these guys per se. doing ambitious things. Same for Mark Zuckerberg and Priscilla Chan, like they're funding Biohub with like the largest charitable donation in history apart from the OpenAI Foundation, which God knows what they're doing. But like Biohub will do like something to the order of like a million times larger donation to science than the Human Genome Project. And, you know, like in, in, in, in 1,000 years, no one's going to remember Facebook. But we will know so much more about the human body because of Biohub. So like, Facebook would just have been a funding mechanism for Biohub.

Yeah, we get— let's get some diseases cured. I love it.

Diseases is rare as one, but like, just like fully understand our human body. It's like so weird, right? And like, even, even like at my micro scale, like, the fact that I don't know how to operate my own body, but I know how to like run a server. Like, that's so bullshit. Like, we, we should be more curious about how we work. And like, you know, like, uh, our health advice is like as simple as like drink water. Like, we can't even do that. So like, what, what business do we have? Like, you know, uh, doing anything else we can't fix ourselves, right? So, so I do think like, um, we will have so much more intelligence and resources. Uh, to me, that's what maybe what abundance means. is that we start to work on Star Trek problems, not Star Wars problems. And I do think that is a really good ambition and goal for people. And I feel like a lot of people, at least in San Francisco, are there actually, that they're working on these things. The room at the Midjourney launch was like, here's a SpaceX investor who's a billionaire just from investing in SpaceX. And Like, yep, he's all about healthcare now. Like, he doesn't care. Like, uh, I mean, classic Brian Johnson story. He built like some Stripe competitor that no one cares about, but like now he's like, you know, Mr. Health. And, uh, honestly, like, not a bad pivot.

Yeah, more power to him. I, I don't necessarily want to live his life, but if I can benefit from it—

he's loving life, dude. He was so happy at the Midjourney launch. He was like, you're telling me I can measure myself every day with a these guys, cheap and free and just water. My hot take was like, you know, so his Kate was there, the one from the documentary. And I was like, he is looking with more love at David Holtz than he has for Kate. I'm like, your girlfriend's right there.

Oh my gosh. We're gonna cut that bit out of the pod. Like, that's not going public. So up to you.

No, no, you know, he's, he's, he's, he laughed it off. He's very, very funny. I told him.

He's a very pure guy. I love, I love Bryan Johnson. Every interaction I've had has been, has been superlative.

Okay.

I've got one more and then last thing will be let's pitch the conference, which I've attended and it was excellent and I would love to come again this year and I gotta talk to my wife about it.

What's up? What's up? I love— so, Tyler, I love that your question is just Google what's up, man. I mean, like, the departures of like John Jumper and Noam Shazir, very, very like, man, it's just bad timing, you know, like that they all come so closely together. I do think like, okay, I do think that hardware and data are destiny. And on that, they're still winning by a huge margin. And I think that that's one of those things where you don't want to count out— they were basically first to invest in TPUs and obviously LLM research and all those things. I do think there's an org issue that they do have to navigate. I think that hiring people like Logan Kilpatrick, running AI Studio and it's basically just being the product face. They just don't have that good of a product gene. They have early hits like NotebookLM and then you can see the moment someone has a hit, then they leave. I think that's not healthy for the Google org as a whole. But then hiring people like Varun and the whole Windsurf research team, and having Sergey come back and work on it. I think that's like very— like bodes very well for their R&D. I think it needs to play out in their products, obviously. But like, as far as like them keeping up in terms of the model quality, I'm actually pretty happy with like 3.5 Flash and, you know, Pro eventually presumably coming out soon. I think they're just not like leading leading. But other than that, like, what's the issue, right? They're still Frontier Lab. I do think like they are going to be the most prime positioned to be the personal operating agent that everyone wants. So Scout, I think is the name. Everyone has a different name, maybe Scout or Spark, maybe Spark. Is there like, you know, I interviewed Jeff Dean and he was like, yeah, I want like a personal agent that has access to all my emails and all my calendar and all these things. Who has that? It's Google. Everyone else is just syncing from Google, but who has that natively?

It's Google.

Apple's not going to do it. You should probably ask what's up with Apple more than what's up with Google. Then more on top of that, OpenAI has shut down whatever, the Sora. I think basically all side projects are dead. OpenAI. They're just doing coding in enterprise. But Google still has the funds and the appetite and the resources to do a Veo, to do a Nano Banana, to do Omni, to do text diffusion, which is like a huge success at AIE Europe, by the way. They are still funding and doing fundamental research that will set them up for the next 10, 20 years. And all the startups labs are pursuing the near-term IPO thing, but only Google has the scale and resources and talent. I mean, what is their weight class? Their weight class is Facebook. Their weight class is Microsoft. That's their weight class. At that scale, who else? So I don't know. I can make the— I made a strong case. That said, I do talk to Google insiders and they want their models to be better and they want their products to be better. They know it. They're working through what it is. And, you know, like, I don't know any better to say, like, you should do this differently. I'm glad to have what we have from them.

One last question from me. So this year, Stanford computer science graduates down 42%, Berkeley down 61%. Shocking. Some shocking, shocking numbers.

Has anyone analyzed the breakouts of like, are they just dropping out or are people just not signing up or what's going on?

Well, I know Stanford a little bit more than perhaps Berkeley. In Stanford, at one point, 2/3 to 75% of graduates had a computer science minor or major. And what basically ended up happening is everyone took CS as a safety degree, with another degree. So a lot of people were doing this kind of dual co-term degrees. And what ended up— what has ended up happening is a lot of them do not do the co-term anymore, and they've dropped basically the secondary degree, uh, CS as a secondary degree. They have some knowledge of, you know, computer— they say they take some CS classes, but they don't bother to get the degree as well. And I think that's really what's driven the drop-off at Stanford. So core CS people are still in core CS. It's really the peripheral CS people who have dropped it off. And that's what's happened at Stanford. What kind of advice would you give to someone going in at this point? Because you're right at this intersection of seeing all of the trends that are gonna be happening for the next, you know, 4 or 5 years. You can, you have, you have a roadmap in your mind, kind of, of what's gonna, what's about to happen. What kind of advice would you give to graduate students?

Yeah, I think this is a bit of a non sequitur versus the 45% down with the secondary thing. I think the market for junior engineers is very challenging. Not that many people are hiring junior engineers now. So you should— you are right to be somewhat concerned. That said, all the cool kids, all the very cracked, like, highly compensated, whatever members of technical staff at OpenAI and Cognition and all these, they're all under 21. Okay, I'm the oldest person in the room, right? Like I'm twice their age. And so if you're, if you're in computer science, you have a lot of potential. You have like, people do believe in you, people do want to hire you, but you have to demonstrate that you are cracked. Do interesting work, demonstrate taste, research taste, project taste, execution ability to ship an interesting product. That people, that people will say, oh, what's that? Like, or like, oh, I want to use that. Um, because, um, you know, it's not the coding ability or the LeetCode solving ability that's in short supply. It is the execution and taste that is in short supply. Um, and people do have that, and people still are hiring for that. I see that the— I see the demand is super high. People still doing on-campus recruiting and all those things. There are increasingly this genre of blog posts of people doing their job search and then like how I got into Frontier Lab and they write it up and I would go read those things obviously. And then that's a really good guide. I would attend leading industry conferences and see what people are talking about and working on and what problems they have.

Is the leading industry conference in the room with us right now?

I don't know. This might be made up and imaginary. But so that all said, and I think there's something that I have been known for since before this AI thing of what I call learning in public, right? Where you want to basically periodically collect everything that you're thinking about and processing and then just put it out there mostly as notes to yourself at first, but then eventually you start being seen as an authority. A lot of people are worried about looking too junior, looking— being wrong in public. But, you know, there will be no better lesson or a teacher than like being wrong in public. And like, you will— people want to see the slope. People don't actually care about the level that you debut at, even though some people debut at really good levels. But if you can see the slope and see the work, people want to coach you because then you're giving them reward on them coaching you. And people do want to coach juniors, right? Like, it's like, we just don't see that positive feedback from, from the young kids a lot. And so when I do see someone who does respond to my coaching, then I give them more. And so like, I, you know, I have a whole blog post on this. Go search my name, go search Learn in Public. There's like a series of 7 blog posts where I've written about how you start your career and how you get mentors without asking them for for a cup of tea, to buy them coffee, to pick their mind. That's all outdated. You get mentors by working on interesting things and responding to them and picking up what they put down. So that's all in there in my blog stuff that if you want, that's my best career advice so far. It has helped a lot of people.

Awesome.

Well, you've been super generous with your time and it's always good to get your takes on all these things. You're definitely one of the—

Good to catch up in general.

Yeah, I appreciate it. And you're definitely one of the most plugged-in insiders who has opportunity to interact with people across all these different kinds of organizations and perspectives. So I think the synthesis that you provide is really valuable. Let's close on the event. Tell us again, when is it? Where is it? I believe we have a code that we want to share with the audience.

Actually, I don't know what code I gave you guys.

Let me pull it up. I'll tell you.

Okay. I gave you some code. It is next week. It starts literally a week from today in San Francisco. It is the AI Engineer World's Fair. It's the largest version of the AI Engineer Conference that I do. Basically 12 simultaneous conferences running side by side. It's a buffet where you can set a pick and choose and, and all that. Big party of, um, everyone online that you've seen. Um, we have poster sessions as well for researchers for the first time, but we also have poster sessions, poster with an A, uh, where people who post hot takes can just print out their tweets and stand in front of them and defend them against all comers. Uh, it'll be quite fun. Um, we— and, um, I think, uh, for me personally, you know, I, I— this is the first World Cup with the World Fair with the World attached to it. And so we're— there's a lot of side events. There's about 40-something side events. All the companies, they're just taking over San Francisco. And where I think— I think it's really interesting to me that I get to sort of curate this group where you can see like a material difference in the quality of discussion because I put all the, you know, speakers and the curation and the marketing and obviously me leading it means that you get a very different kind of discussion with the attendees than you would with a normal conference. Like, anyway, I don't want to name competitors. But I do think like, the people like actively working on things in that you've— on projects that you've all seen will be here. And that's what I really curate for and somehow managed to get people to show up. And so that's why I'm very grateful for this role that I get to play in the industry. Like when I started this whole thing, it's going to be— it's also going to be the 3-year anniversary of the Rise of the AI Engineer article. So when I like started this, I only had like an inkling that it would become something like this someday. But to get here in 3 years to me is a miracle. Like New Relic has been around for 40 years. And the fact that we're basically at that level for our sponsors and our speakers is a blessing.

Amazing. You've ridden the exponential thus far just about as well as anyone. And again, I have had the chance to attend in the past. It really is an event where the conversation really does feel up to the minute. And I think that is not something to be taken for granted in general when one gets on a plane to, or in my case, I'd be getting on a plane, others will be local, But, you know, it's, it's, I, I'm always asking myself that question. If I go to this thing, am I gonna have sort of, you know, 3+ months ago kind of warmed over talks, or is it really gonna be people that are living on the, you know, the very edge of what's possible now? And that is a great strength of the AI Engineer event series. And, and that comes from you being so plugged in and on the edge yourself.

The code is—

It's really crazy. Yeah, you go for it first. I'll end with the code.

You need to have a benevolent dictator, which is me, of just like, hey, we're going to throw this out and we're going to put this in just because it's cool. I just yeeted our MCP track and put in auto research because that's the thing that we all want to talk about right now. It doesn't mean MCP is dead, but it is part of the context engineering track rather than You know, like a headline feature. But a year ago, it was absolutely the headline. Like, we were the first to have a focus on MCP in New York. And now we're like the first to have a full auto research track. And I think that means that we get to stay ahead compared to everyone else.

The code is THANKSAIAM if you want to register for the conference. THANKSAIAM. Thank you, Swyx. This has been a lot of fun. Really enjoyed the conversation and some great takes as always.

Yeah, thanks guys.

Hope to see you next time. And yeah, you weren't able to attend the one that I actually attended for reasons that maybe we'll leave to history for today. But it'd be great to actually connect in person finally after all this time online. So hope to make that happen. And either way, keep up the great work and check Check out the AI Engineer World's Fair, everybody.

Yeah, thank you.

That was great. I really enjoy talking to him. And he's genuinely as plugged in and connected as they come. And of course, the Art Space Podcast is perhaps the most—

one of the most plugged in people that we've had on the show. Like up to the minute, literally up to the minute.

How are you feeling on energy? We went longer with Swyx than we planned. We could do— I've got a little fun activity for us, which we could do now, or we could save it for another gap in the schedule later in the week. What's your feeling?

I'm up for it. If you'd like to do it, I'm up for it.

All right, let's try it. This is going to be definitely an experiment, to put it mildly. But let's give it a go. All right, can we see? I'm trying to share the screen, but you may need to promote it or whatever.

Put it in. All right, there you go.

Nice.

Okay, cool. Maybe I'll make it a little wider, see if that fits a bit better. Did that change? Yeah, I don't know if that really made a difference or not. Okay, so here's the game. I had Claude, Opus of course, one silent tear because I had to use Opus for this. It could have been better, but I had Opus go out and look across the different prediction forecasting market sites— Polymarket, Kalshi, Manifold I think makes an appearance. We got to work on our Metaculus MCP or whatever to make sure we're getting access to that in the future. And basically just had it go find some interesting questions and then create this little quiz site to present to us. And I haven't looked at the questions, haven't looked at the answers, so we can have a real-time friendly competition to see who can guess the current values in the markets for these key AI-related questions. This is kind of a shameless, uh, gimmick steal from Bill Simmons, uh, and the guy known as Cousin Sal, that they've for many years done NFL Guess the Lines. And so this is sort of an AI, you know, guess the, the current parking poly market or Calci or manifold value. We'll see how well calibrated we are, and we'll see if we're inspired by the strength of our own analysis to maybe actually trade in some of these markets. Does that make sense? Any questions, or do you want to dive in and see how we do?

Where do I enter the numbers? How do we do this?

This is actually just hosted right now on my local computer, so I'll have to enter numbers for you, and then we'll actually see if this thing works in terms of scoring the competition effectively. We're going to have an interesting discussion either way, but I'm not sure how well it's going to work.

All right.

Okay, so we've got 12 questions. Question number 1, will Anthropic hold the number 1 ranked AI model at the end of 2026. And our job is to give a probability guess that Anthropic will still hold the number 1 spot at the end of the year as determined by LM Arena's Chatbot Arena.

Okay. All right. I have a number in mind.

Okay.

I'm gonna say, I'm my, Brain is kind of like, yes, I think they're the most likely to have it, but what are the odds? I'm gonna say 60%. What's your number?

55.

55, just, just sneaking in under. All right, let's see what we've got. We reveal the market. It is 63.5%. Interesting scoring. It's giving us total error in terms of points. Something tells me, uh, we can come up with a better scoring system than that, but This is why we're iterating in public here. Okay, well, but we're both pretty close to the market there. I don't think that screams that either of us should dive in and start trading.

Yeah.

Number 2, which company will have the number 1 math model at the end of— oh, this is just the end of June 2026. Okay, that's very soon. I'll note that we should probably have a little longer time horizons, although this also begs the question of Do we even know today who has the top math model? So I guess this question is basically reading who has the top math model right now with some little room for improvement. I mean, Anthropic had really, with Fable of course, had really, really strong results on Frontier Math Tier 4. We covered that a little bit in our Fable discussions. I had predicted at the beginning of the year low 60s as the Frontier Math tier 4 success rate for the best model, regardless of provider, at the end of the year. And already we have Fable in the high 80s. OpenAI, I think, also is in the high 80s with, I think, 5.5 Pro, if I recall correctly. And my gut is like, I still feel like we've seen a little bit more overall impressive stuff from OpenAI. So I would say OpenAI probably has the best model today, although some of this stuff isn't public. And so then there's, how does that get scored on these resolution criteria is sometimes a bit of a challenge. But I guess I'll say I think it is like, Maybe 50% OpenAI, 30% Anthropic right now, and I'll give 10% to Google and 10% to the field.

So, so similar. Oh my gosh. So I had 50, 30, 15, and 5. So 50 there, 30, 15.

15 for Google?

Yeah, 15 and 5.

Okay. That's, yep, indeed very similar.

Yeah.

Not giving— neither of us giving too much weight to Harmonic or Axiom, though I've been really impressed by those folks for sure. And I do think they've got—

I'm gonna guess that the Anthropic numbers come out higher on the markets. Let's see.

Here we go. Google, 73%. Okay. Now let me just open the actual market itself. Can you see the— okay, hold on, let me, let me redo my share for a second here. Yeah, um, we're gonna do Google, share this whole window, because now we can look at—

Startling how, how different the insider, like closer to the bubble, inside of the bubble versus the outside of the bubble evaluates.

Yeah, this is really kind of bizarre under the— okay, so again, this is an LM Arena evaluation. So let's go pull up this and see. It's presumably Google's got to be up there really high.

Well, this is text arena math. I see. Yeah. So maybe we're just not aware. We were just not aware of like the LM arena kind of like specifics of how they rank.

Yeah, it's a little— maybe we should do a follow-up on that and kind of see like what exactly is that experience because It does strike me as a little— let's see, can we go into math? It's been a minute since I've used— I don't know if I have to pick math mode or if I can just automatically detect. Well, we might have to come back to this. As much as we're doing this in public, watching me figure out this UI is probably not that exciting. But I am quite surprised by that. And it does sort of suggest, honestly, speed. might be one big reason that Flash would be rated so highly.

Yeah.

I don't think we're seeing— I think we maybe took the definition of best model for math quite differently than the actual resolution criteria. I don't think we've seen 3.5 Flash solve any open problems, to my knowledge, although I wouldn't be shocked if it had done some. But if it's the best at answering people's math questions, um, you know, potentially the speed factor alone could be a big reason that it's, you know, winning so many head-to-head bake-offs. Um, yeah, Fable down here is, you know, basically right in line with the others, but yeah, something a little strange about that. Okay, well, you know, we'll note that feedback for the next round of, uh, Guess the Markets. Let's keep going. Number 3, will any company achieve AGI before January 1st, 2028? This is kind of another classic. Well, what do we mean by that? The company apparently has to announce it.

I mean, they would just announce it anyway, right? Because if you have a fundraising round and you can't raise, you're just gonna announce it anyway. They have to be, unless there are legal repercussions, what stops like any small company from announcing?

Right.

Any company, any company, public or private.

Yeah. Now that's an interesting question on what is any company. If I would think of this as any credible company, I sort of think it's not going to happen because I don't think they want that smoke really. I think they're kind of, I don't think we're ever going to see a moment from Ever strong, but my guess is they will be lagging indicators in terms of their own recognition of having achieved AGI. We've got Tyler Cowen called it at '03 and Dean called it at Opus 4 point whatever exactly. And my guess is that they will continue to say, ah, we've still got more work to do. And they won't say— Yeah, this counts as AGI, if for they we are talking about the actual trillion-dollar companies that have a credible claim. Now, if Nathan's LLC comes forward and says, we've got AGI, that makes this a little bit hard to resolve. But I will say, with my own understanding, that there's some relatively narrow set of companies that would be taken seriously, that we're considering, I'll say 10%, because I think otherwise they're gonna not want to own up to having achieved it, even if society at large kind of feels like they have.

I'm gonna rules lawyer this, and I'm gonna go with 80%. And I'm gonna go, you know, I'm gonna defer, and I'm gonna like say that this is gonna be rules lawyered. And so I'll go with 80.

Here we go again. 45. It is exactly in the split the difference space between us. And this is often the problem with these, these things. It's like, it is really hard to write a good question sometimes, or to write good resolution criteria.

Yeah.

Um, is there anything here that they're saying about Okay, so it'd be in these different things. Yeah, it does say any company. Okay, that to qualify, the announcement must explicitly state that the company has achieved, attained, reached, or developed any publicly traded or privately whose existence verifiable. Okay, whose financing slash valuation or corporate profile is covered by at least 2 of the listed major business news sources. This to me feels like we're kind of talking about what will 10 to 20 NeoLab kind of companies do.

Yeah.

I feel like there's enough companies that have raised this kind of $100 million seed round to go chase recursive self-improvement or whatever, that they got 2 articles in these sources. And now this will ultimately kind of boil down to, do they come out and say that they've achieved something in that timeframe? Obviously, if any of the really big companies does, then that's a clear resolution. But my guess is most of this weight comes on that second tier of company that would have more of an incentive to try to fuel a new cycle with that kind of announcement.

Thinking Machines or SSI stays alive, by just doing coding agents. They're going to have to announce something more than just coding agents in order to stay alive or they won't, or they'll have to get bought out or absorbed. So I think that there will be sufficient market pressure that someone will announce. So yeah, but yeah, it's really a rules lawyer thing. Exactly in the middle. Unbelievable. Yeah.

To be continued. Okay. Number 4, will the ARC AGI grand prize be claimed before January 2027? It pays out when an open-source solution scores greater than 85% on the private ARC AGI evaluation set within the contest's strict compute and efficiency limits rules.

Is this 1, 2, or 3? It's 3, right?

Uh, I believe it's 1 because I think it would be— yeah, that 85% and the strict rules. I mean, we can— I can't open the market without spoiling the answer, but I'm, I'm gonna operate under the assumption that it is 1. And Frontier models have hit at this level. Yeah, I think they actually have surpassed 85%. If not, they're definitely in ballpark. But the compute limits are actually quite low. It's like basically a sort of 10 cents of compute. If I mean, it's been a while since I really studied the details of their prize criteria, but it was something like 10 cents of compute per puzzle total. And the cost of the big models coming in and solving these things is You know, is much higher than that. They also want it— ideally, they want it to run in a little container in their environment as well.

And we only have 6 months left. So, like 5, 5 and a half. I'm gonna go with like 40%.

You might pull me up a little bit. I was initially thinking 10. Just because I do feel like it's— I haven't heard much recently about the low end? It seems like we've seen these, you know, Arc AGI 1, 2, 3, and I feel like there's been much more energy on big models are making good progress on these even as they get harder. But the really small models that actually hit their performance criteria, I haven't heard as much that those are getting close. Although, of course, there's— I'm sure there's progress continuing. So Since you went first and I can adjust myself, I'll go up 5 points and say 15%. I think you might be winning this one, but this might be one where I might actually auto-trade. It's almost again exactly right in the middle.

Oh, man.

Yeah. Okay. So it is AGI-1. We do have frontier models in that range, but the efficiency, the compute caps are are still very much binding. Um, let's see if we can get over real quick to an actual Arc AGI leaderboard. And how much has this moved around? Actually not that much. And how liquid is it? It's not— well, trading volume, $19,000 Manifold bucks. It's not super liquid. 84 traders. It's, it's moved, but it hasn't moved like that much over the last year, which is kind of interesting. Um, let's see if we can real quick get the ArcAGI 1 leaderboard. Somehow we got to 3. I wanted to be on 1. Let me try that again. Yeah, I don't— whoops, I don't see a leaderboard link here. Okay, I'm not trying to sign up. Well, I might have to go— okay, wait a second. ArcAGI 1. Yeah, but again, we have all these things that are just like massive models. They really have done— yeah, it's funny. I mean, you can just see the, the trend. Like, the graph is just showing so much of their, uh, you know, how much of the energy has come from these big models. Where are the— like, yeah, just not seeing much in terms of small models. I want to get down to the cut where it's like These are the open weights, ones that actually hit our compute thresholds. Yeah, I mean, because even the open weight ones are still broadly way big compared to what they are asking for.

I feel like someone's going to auto-research a small model solver that just solves it.

Yeah, I mean, that would be—

And I don't know to what extent you can, you can say that's really like the small compute which is solving it. It's really the auto research compute which is doing a pre-training step on the model itself. And then that's being used. So—

Yeah, there's always a little bit of hide the compute possibility going on there. Okay. Well, we'll see if we can maybe find that compute sleight of hand, sleight of hand on the compute step. Okay, so we're through 4 questions. We are in an incredibly tight race. Let's go on to number 5. Will OpenAI complete an IPO by December 31st, 2026?

15%.

Wow, really interesting. Okay, I was gonna go significantly higher because, I don't know, it seems like they're talking about it and As we've discussed even in the context of Google, right? They're starting to issue, sell new stock to fund their build out. Just like how much longer can they keep raising these like $100 billion private capital rounds? It seems like at some point they will run out of people who want to, or maybe not who want to, but who are able to write those kinds of checks.

I—

my first impulse was 50. Since you beat me to it, I'll drop down to 40. But yeah, interesting. Okay, let's see. Let's see, 54%. All right, you should trade on this one. That's not enough for me to trade, but it might be enough for you to trade. Why do you think they're so unlikely to do it?

I just feel like they don't have enough time. Realistically, summer is dead. The market doesn't respond until August. There's this whole sell in May and go away meme. So it's really like you get August, September, October, and Thanksgiving, you're done. So November 26th-ish onwards, market's closed. So you realistically have like September, October, and November. You got 3 months. And in order to get all your docs in order, your docs have to be basically ready basically in June, like May, June. And then you spend like all your time like marketing for the next like 3 or 4 months. And then you kind of close by end of October, early November. So I feel like the timeline is they already, they already missed the timeline for this year. And I think the earliest timeline is going to be Q1 of next year. So that's, that's my sense of the timing. Like, they just— and Anthropic too, like, they just don't have the time. The timing is already skipped. So that's my sense.

Decent volume on this. You might want to get in there and make a trade. Okay, on to number 6. On December 31st, 2026, will Anthropic carry a higher valuation than OpenAI?

Tough ones. Oh my gosh. Tough one. Whoa. I would say that's a yes, but like a 55% yes, not like a strong yes, maybe like a 55% yes.

I'm also a yes. I'm a little stronger. I'm gonna say 75%. Yeah, uh, the growth rate has been higher. They have the best publicly known model. And it was interesting, we didn't even touch on it, but Swyx did do a little, you know, and other upcoming models. Uh, clearly, you know, there's going to be an OpenAI answer, but—

This week, maybe this week. I'm hearing this week, 5.6 launching this week. So let's see.

Yeah, I may— you know, one reason I may ought to temper this a bit is, uh, the government is, um, meddling in the market in a way that doesn't necessarily seem fully even-handed, and that could definitely move the market to, um, to a different place than I might naively expect. Still, I'll say 75%. Let's see what happens. 82. All right, another trading opportunity for Wow. Interesting. That seems a little— after our brief discussion, I felt like I was a little high. It makes me feel like the market is a little high. I just think, man, the, the market probably shouldn't be that confident about this, you know. Even if directionally it's right, that just feels a little strong. Let's see how liquid this one is. $35,000 in volume. That's not Super liquid.

I feel like, I feel like the, the market is not really judging the future. It's really judging the present, the present view of the future, perhaps.

Yeah.

The other thing to keep in mind too is like OpenAI has invested so heavily in compute that, and I think this is like, you know, obviously Google's kind of ultimate backstop as well. When you have 25% of global compute, you can be a little lax from time to time on shipping the next model, and you're not exactly in an existential crisis. Whereas I don't know what percent of compute Anthropic has, but it's obviously much smaller. And OpenAI is, they're making a run at getting to a significant enough share. that they could sort of have some resilience to any number of different wrenches that might be thrown their way. So that feels high in the end, and I think my number still actually feels a little high as well. Number 7, will Nvidia be the world's largest company by market cap on December 31st, 2026? For reference, I believe they are today. Last I checked, Google was number 2. I don't see Google being on such a tear that they are about to overtake Nvidia. So my guess is probably yes.

The spanner in the works is, I think, SpaceX. SpaceX could have a ridiculous— like, all they need to do is get that Starship, you know, in the sky on a regular basis, actually delivering cargo. Um, I'm gonna go with like 70% Nvidia?

Yeah, I'm a little higher. I'm gonna say 80, but I mean, never underestimate an Elon stock's ability to pop, but they're at like 2, right? So that would have to be a— they'd have to add like more than $3 trillion in market cap just to get to where Nvidia is today, which if that happens, my guess is Nvidia is also jamming because it's probably just like there's a broad run-up there. So Does SpaceX get to like 6+ and overtake Nvidia? It seems if anybody's gonna do it, I would probably bet on Google. You know, it would be like a, um, Gemini 4 or, uh, you know, or a robotics, um, moment where all of a sudden they're like, oh, these guys even have more cards up their sleeves than we knew. Yeah, but it seems unlikely.

Yeah, I don't know. It's hard for Google because it's a big company and they have so much cash flow. And it's hard for Nvidia also because they're also going to be judged on cash flow. SpaceX, they're going to be judged on vibes, right? The market can run on vibes. You don't need cash. I don't know. Let's see.

Let's find out. 73.5. You win that one by 3 points.

Yeah, but it's small.

Yeah, we, I'd say we're generally speaking, showing pretty good calibration to the market so far.

They have like low values of SpaceX. You know, the vibes-based company is going to run much stronger than the cash flow-based company.

So yeah, only 3 candidates.

If Elon doubles his revenue at SpaceX, he's going to be at like still like 1/10 the revenue of NVIDIA and he's going to be valued higher. So there's a lot of SpaceX fans in the house. There's a lot.

Yeah.

Well, maybe that's why I won't be pouring my assets into SpaceX. As much as I do believe in them, it feels hard to justify another doubling of the Valuation. Interesting. Okay, that one was pretty liquid too, by the way. $3 million, $3.6 million in volume.

Um, yeah, because you can bet against the market, you can hedge it off the market on that one. So you can buy call option— call options on, uh, the market directly, so you can hedge the volume off the market.

So, all right.

Okay, now this, this one's going to be tough because I honestly Don't even know the Bitcoin valuation right now. I take a certain amount of pride in kind of knowing these basic stats for fluidity of analysis, but I don't know Bitcoin's market cap. So question is, will Anthropic's valuation surpass Bitcoin's market cap at any point before 2027?

I would say—

Now, if we really wanted to be super forecasters, we would be allowed to Google simple facts, Like, what is Bitcoin market cap? Do you want to know Bitcoin market cap right now?

It's like roughly $60 grand a coin, 21 million coins, which puts you at like $1.4, $1.3 trillion at $60 grand. I guess it's like $61 or $62, somewhere low 60s. So that's where we are.

So will Anthropics surpass that at any point in the remainder of this year? Yeah, and again, the resolution criteria here is always a little tricky, but what we've seen already is that their evaluation is at a trillion.

Yeah, $950 billion.

Uh, and all reports—

I'd go with a yes, and I'd go with, um, you know, 60% yes. Let's see.

Yeah, I think I'm basically with you. I was going to go maybe slightly higher. But I think my main— it's funny. Is Anthropic more valuable than Bitcoin? Yes, I think I can say that definitively in terms of many aspects of actually making money, actually having use cases, impact on the world, all these sorts of things. Why wouldn't it happen?

Well, because the Chinese can invest in Bitcoin and they can't invest in Anthropic.

Yeah, but I mean, the read on the most recent funding round was like, it was super cheap. People were begging to get into it, et cetera, et cetera. That still feels right to me. I just don't know if we'll— why wouldn't it happen? I think the biggest reason would probably be we just don't see a public repricing in the timeframe that would actually allow it to happen. But I think I'm even going to go a little higher. I'm going to say 75%. You want to stay at 60 or you want to make a last-second change?

I, I think it's higher, but then I don't think the market will judge it as higher. So let's see, 66.5.

My move just— I was going to be slightly closer and then I moved to slightly farther. But hey, we're bracketing this stuff pretty well. Um, not super liquid, $150,000 there, and it has ticked up, but this isn't, uh, this is not a very old market, so this has only been trading for 6 weeks. Okay, um, number 9: will any AI model reach a 1550+ overall Chatbot Arena score in 2026?

Hmm, I'm gonna go with 80%.

Yeah, do you want to look at what the current leaderboard looks like?

I thought Fable was already up there. Like, I thought Fable was— yeah, 1,500+ already.

So if we open the leaderboard, we very well— I wouldn't be shocked if we see a Fable has already had that number and it's happened.

No, we should have intellectual integrity and like not open the leaderboard, and we should just guess and see. You should, uh, make sense of it without, without checking.

Yeah, I don't know. Oh, it's a pretty low confidence guess given that I haven't looked at the leaderboard in a minute, but on the, on the general theory of market mind, you know, if I'm just thinking like, who made this market, when did they likely make it, and, you know, what were they thinking was a reasonable sort of over-under at the time, I think these things have kind of moved faster and have exceeded people's over-under estimates from 6 months ago in general. And of course we still have 20, you know, we're in 2026. So I'm gonna say 80% based on just kind of the general background sense that things like this are like tending to happen rather than not happen. The questions people are posing with any runway are being answered in the affirmative. What do you wanna say?

I'd say 80% also. Yeah.

All right, let's see. I'm gonna— now that I say that, it feels slightly low, but let's find out.

18.

Holy moly. Okay, okay, now we need to dig into what's—

now we are really miscalibrated.

Uh, where is the actual leaderboard? Here we go. Where is the—

it says 1,650. What is the ELO for Fable? Yeah, Agent Arena. What was it?

Why am I not even seeing Oh, this is agent. Should we be doing just overall text? I think that's maybe the one we're supposed to be looking at. Okay. Yeah, so this is—

they're saying that the, uh, the improvements are marginal, like 4 or 5 points. Yeah, every, every 4 months is like, uh, 3 to 4 months is a 4-point, 5-point improvement. So 1550 from a 1508 would be 30 months of improvements. So 2 to 3 years of improvements before we get there. That's the sense that— so yeah, if you're only improving 4 or 5 points every 3 months. Yeah.

So yeah, interesting. This feels like another one of these things where we're maybe hitting the limit of it. It just seems like there's something here that's like, People are not asking the right questions, right? They're not— I think people are not asking questions that effectively resolve the differences. Is there a hard version? I thought there was like a hard version here somewhere. Expert. Yeah, let's look at just expert. So here they're higher. Yeah, I mean, you can sort of see a little bit of my argument there and just the fact that there's a higher score at the top of the expert versus overall. It does feel like maybe this sort of thing should be recalibrated to an expert subset because, you know, if Fable's a little slower or whatever and you're just asking very basic questions, you maybe don't see a big difference. Maybe you don't even— maybe you like it less because it's a little slower.

Mm-hmm.

And it, it doesn't feel like it's maybe capturing the signal as well as I might hope something would. I certainly think like looking at the Frontier code, Fable versus Opus 4.8, tells me much more of what I really want to know than looking at the arena. Um, you know, Fable beats the recent Opuses by a couple points. It seems like something is, something is a little lost in that whole process.

Yep.

But maybe that's just sour grapes because I had a bad answer. Okay. Which bank will— this one is for— this one's for you. This one's not for me. Which bank will be lead left underwriter on OpenAI's IPO?

I think it'd be Goldman, but I'd say the market is going to give— I think JP could be in there as well because Elon didn't like Jamie Dimon, but I think Sam is fine with Jamie Dimon. I would say Goldman 60%, Morgan 20%, rest of the field 20%. So 60-20-20.

Okay.

I have no idea. So just since these are big banks that I know of, I will say 30-30-40. You're probably going to be more right, but I'll be honest in expressing my ignorance.

77.

It's much higher to Gorman, but it's all taken from the field. It's really apparently just those 2 that—

yeah.

Okay, um, interesting. Is there any more analysis of that that's worth, uh, spending time on, or— well, who cares? Like, in the end of— at the end of the day, why is this something that people would even be interested in betting on? Is it just pure degenerate gambling or is anything really at stake?

Well, it's, if you're in the industry, it's something that people want to talk about. It's about who's gonna make the money that has to be made. Investment banking in the US for IPOs is a little bit like a studio system. You're gonna have to pick an underwriter and the underwriter is gonna make money. and they're going to make a lot of money. So, um, it's really a question of, you know, who gets to make that money. Is it gonna be like a bunch of like very smart people at Goldman, or a bunch of like very smart and also, you know, people with like very good relationships at Morgan? Or is it going to be JP Morgan with like a large balance sheet that they can support the IPO with? Uh, it's very hard for any other bank to get in there because you need all of this like financing relationships that need to have been there for like a long time that you can draw upon. But really, you know, it's really more important on the East Coast where people are making that money and it's not that important on the West Coast. Like, you know, tech people don't really care that much anyway. So whatever.

What do you think the odds are? And this doesn't feel like a Sam Altman thing to do, 'cause I think he's generally more of a political operator than somebody who would do what I'm about to propose would need to be, I think. But it's been a long time since the Google IPO. I know they kind of did some things differently. What would be the barrier for an OpenAI to just do their own process or do it differently, say, hey, we're just not going to spend all this on this. Why do we need a banker? Everybody knows who we are. We have agents that can go make presentations to institutional investors or answer their questions, what have you. Is there any risk of a— I guess there's 2 questions, practically and sort of on the level of execution. Could they do something like that on their own? And then the second question would be like, would they even if they could, or maybe it would just be better to kind of pay this toll to keep everybody happy. But do you think that, is there something that the banks are doing that is so hard to replace that they couldn't get their next model to do the job?

So it's a complex question, which I think people have debated for a very long time. People like Bill Gurley will tell you that you should just kind of do a listing, which is basically like no one sells any shares and you just go out and you say like, okay, you can trade the shares in the market. Anyone who wants to sell shares can sell shares, whatever you want, blah, blah, blah. And Spotify had some success doing that. So Spotify did that. The problem becomes that most of the time, and especially in this case for OpenAI and Anthropic, you want to raise money. The company wants to raise money on the market and not just allow people to sell shares. And when it wants to raise money on the market, even if it does the listing upfront, it's going to have to go out into the market and say, I want some money, guys. I need to do an offering. And the problem is when you want to do that offering, you might as well get that offering out of the way immediately. And when you get it out of the way immediately, you have a chance to pick your price, and the price is a negotiated price. If you just put those shares in the market, you don't get to pick your price. After that, the market— people trade on the market and readjust. And then after that, you have to do an offering, so the market chooses the price for you. So if you want some power in choosing your own price and choosing how many shares you've got to sell, you're going to want to do the offering. And when you do the offering, you get to lock up your employee shares. So your employees don't get to sell the shares. And so you get to create the scarcity in the market for your shares and you get to control the price. And so all of this process is really about, do we get to control the price that we sell our shares at on day 1? And do we get to manage the price for the next 6 months or so, so that we can also manage our share price and we can do follow-on offerings and do a bunch of strategic things that we need to do? And that has value for the company. So you get to basically be semi-private and control your share price for another 6 months while still raising money from the market. And you get to control your narrative. After the 6 months, the 180-day period disappears and everyone can sell shares. And then the market readjusts the price again. So if you have employees who are disgruntled or who wanted to sell the shares and didn't get a chance to sell, they get to sell at the 180-day So when a company wants to raise money and they want to control the share price and control the narrative, they're going to want to do the IPO. And they're not going to want to do a direct listing, have the market choose a price, and then be forced— if the market chooses a lower price, be forced into doing an offering at the lower price. So you get to take away that getting forced to do the offering. Off the table. You're not forced. You get to do your offering and set your own terms. And you get to go out and ask the market like, hey, what's the price? And that's what the banks are doing. The banks are basically rounding up these investors and they come and they sit in a room together and the company makes a presentation and they come and all these MBAs basically gossip with each other on the Slack channels and all the WhatsApps. And they come They're kind of like, hey, what are you thinking about doing? What am I thinking about doing? And they kind of collude with each other and they're kind of like, okay, this is what we're going to send our price in. I think they're going to put in for like 10 billion shares at this price. Those guys are going to put in. If we want to get some shares, we've got to put it in somewhere around this number. And they kind of go out and the banks kind of get all these bids. Then the banks come out and they say like, look, we can only give you like this amount of shares, but if you go up like A little bit more, we can give you a little bit more. So what are you thinking of doing? Then there's this 2-way negotiation, like blah, blah, blah, blah, blah, blah, blah, blah. Finally, everyone settles on a number, goes back to the company, and then the company decides, okay, this is how the shares are going to be split. This is what we're going to do. This is the price we're going to go in at. Put in your checks in. Money is locked up. And then we go on the first day, the shares are transferred, money is transferred. And everyone gets to do whatever they like after that point onwards. There's a little bit of tango. And this tango is around that transition period. And that tango is best managed by the banks because the company wants to choose the price. And that's really what's going on. And you end up— company management has usually found it more advantageous, even Elon with his unlimited capital raising ability, has found it more advantageous to basically have this price control in order that you can have a control over the next 6 months or so. So that's the ballgame.

Interesting. I also feel like something tells me that just in the political economy of it all, even if you abstract away all the value, there is somehow an argument for just paying the piper on this one and kind of Somehow karmically believing that that'll pay off down the line in ways that I couldn't even articulate. But it just feels like you don't screw all the big banks out of a big payday they were looking forward to without it somehow at least creating risk for you downstream if you're ever in a vulnerable spot.

Yeah. In accelerationist theory, it's also about techno-capital. It's about Giving capital owners the reason to root for you. So if people have bought in your shares and they're up 20%, they're rooting for you and they're going to— and even politicians, right? One of the reasons I'm very supportive of congressional representatives owning stocks is that if they own the stock, they're going to be interested in the welfare of the company and that the company does well. And I think to a large extent, policymakers need to have that enthusiasm for for their constituent companies that employ their constituent citizens. So, you know, I'm very for it.

So, well, we can maybe bracket whether or not an index fund would be enough for them to, you know, have that exposure and enthusiasm for another— 2 more, 2 bonuses. I'm not sure why Opus decided that the first 10 would be numbered and the last 2 would be bonuses, but that's just what Opus felt like doing. Bonus number 1, will a Chinese company hold the number 1 AI model at any point in 2026 according to the LM Arena overall arena score, uh, with style control off? Interestingly, um, I'm gonna give that— I am gonna say that's real low. Yeah, I'll say 1%.

I'd say 20.

20?

20.

That's really high in my opinion. Let's find out what the market has to say.

9.5.

9.5.

I am— this feels like, you know, I've at times been, uh, an enthusiastic supporter of some dark horse political candidates. One, Andrew Yang, for example, comes to mind. And people have made pretty easy money betting against the Andrew Yangs of the world on these sort of marketplaces. The theory being that the super enthused people that are buying the lottery ticket of Andrew Yang's nomination are just dumb money basically, and betting with their hearts rather than their heads. This feels to me like that. I mean, this says I can get a 10% return. Obviously I have the risk of losing my principal. Yeah, but I can get a 10% return as long as there's no Chinese model at the top of this leaderboard at any point in 2026. I would, I would be shocked if that were to happen.

Um, I, I feel like, I feel like the, uh, LM arena itself can be hacked, um, in terms of reward hacked. And I feel like the major frontier labs have no incentive at this point.

Yeah, we did see Meta do that specifically. So that is a good point.

So I feel like we're not— it's— I'm not saying that it's going to be the best model. But I'm saying that the frontier labs have like, why would a frontier lab ever care about LM Arena at this point if they can have more revenue off coding, right? Like, and for the Chinese models, it matters because they're still fundraising off these benchmarks. So they're still benchmark hacking. So I don't know.

Yeah, let's look. Do we still have that up? Let's look and see where's the first Chinese model today. Do we have a GLM-5.2, for example? No, we don't yet. GLM-5.1 is the latest GLM, and it is basically 30 points from the top. So GLM 5.2, you know, will probably close that gap substantially. Yeah, I would guess it would come in, you know, maybe halfway, maybe up here around the kind of—

Say probably like 4, around Opus 4.6, thinking not, not below Fable 5, above Muse Spark, I'd say. So probably like number, number 4, number 5, probably, I'm guessing.

Yeah, that's interesting. It's also interesting that Opus 4.8 is lower down. Um, okay. I, you know, I'm not that degenerate of a gambler, so I, I don't think I'll be placing this trade, but I hear you. The, the gameability of it is, um, is definitely— feels to me like for sure the biggest risk. Okay, and finally, will OpenAI publicly release GPT-6?

Oh man, I'd say that it's a yes, but maybe 50%, not that high. 50% maybe.

Yeah, it's— I feel like this was sort of a, sort of a tricky one for all kinds of reasons, honestly, right now. I mean, there's the inherent uncertainty of what they're— how fast the progress will go. There's the fact that Last time a Frontier company did a full number increment, the model got yanked off the market. So it's— there's no doubt that there's like a bunch of weird incentives that might make them say, well, maybe we should just call this 5.8 versus 6 for the time being when they get there. I, I kind of feel like they did that with 5.5, right? Or maybe it was 5.4, but there was sort of this moment where All of a sudden the GPT-5 series seemed to have a step change, and there was a bunch of commentary that was like, this was kind of the real GPT-5, but they put one out as GPT-5 sooner than maybe they really should have if they were being fully honest about how strong it was. But then they got there a few point releases later. I don't know, I'm stalling for time. I feel like I'm kind of with you that it's It feels like a toss-up. I think I'm gonna go— what am I gonna do? I think I'm gonna say yeah. I think I'm gonna go basically right with you. I'll do— I'll go, I'll go just slightly higher just to have some difference, but honestly my gut says toss-up as well. Let's see, 71%.

Market thinks it's gonna happen.

Yeah, I mean, you know, they got IPO, right? So, uh, gotta have something to, uh, drive the narrative. Uh, pretty good, I'd say, overall by us. Just a quick scan back. Uh, we were pretty close on that one.

This one was a weird question. The biggest question is that 18% versus the 80%, the, uh, yeah.

Which one are you thinking of?

Uh, I think that was the Oh, this one.

Okay. Yeah, that one, the 1550 on Chatbot Arena.

And I think we're just badly calibrated on that one.

Yeah. Okay, cool. Well, I don't know, it's a pretty fun exercise. I thought if, uh, anybody has any feedback, I'd love to hear it. It could be an interesting little app to put on the— maybe I'll ask Opus to maybe put this quiz onto the website and then people can go try it for themselves and Obviously they can cheat, but they can compare their performance to each of ours as they go through it. Anything you would do differently next time or any feedback you want to put into the transcript for Claude to self-improve on?

I don't know if the error term is— should we be using squared error?

Yeah, there's almost for sure a better scoring system.

Yeah. Maybe squared error so that to basically penalize more dispersion, basically.

Yeah, there's a super forecasting thing as well that sort of, I forget exactly how that's done, but there's some sort of logarithmic scoring options too where You're, you know, if you say 0 and it's not 0, you get like infinite penalty, which could be a little, um, a little weird and different. But yeah, I wouldn't put too much— I think as of now I wouldn't put too much, uh, weight on the score as, as calculated by Opus on this first run. Definitely some room for improvement there.

Indeed.

Cool. Well, hey, That was, you know, that's an experiment. That's what we're here to do. I enjoyed it. Anything else you want to talk about before we break for the day and come back tomorrow?

No, it was— that session with Spix was a great session. So yeah, absolutely.

Cool. Well, once again, the code for— and I don't even know if we ever said what the code was for, but it's a discounted registration. So it's basically Free money if you want to go to the AI Engineer World's Fair. ThanksAIAM is your code. And I did have a great time the one time I was able to make it in person. So do encourage anybody who's interested to check it out.

Indeed.

See you tomorrow, Prakash.

Bye-bye.

Bye for now.