AI:AM Highlights: Exploring the J-Space, AI Superforecasters, SambaNova's Chips, & LTX Video Gen

Yeah, the AIs, they're just like us, it turns out. Or at least similar enough to be in some sort of weird kind of looking glass similarity anyway. I mean, it's a lot to take in, the whole J-Space thing. 150-page paper, 50 pages of commentary, summaries, interactive demos. When Anthropic drops one of their big interpretability papers, they really do it up full scale. And this one is no exception to that. I had kind of been wondering, when's the next big thing coming? Because tracing the thoughts of a large language model is like better part of a year ago now. And this is that big thing.

Welcome to the AI in the AM weekly highlights, the cut for people who follow the frontier closely and can't watch every morning live. If you're new, AI in the AM is a live show Prakash and I host most weekday mornings, from a studio Prakash vibecoded, and this narration is a clone of my voice. Fair warning, there is no single thread this week. Mornings at the Frontier jump from topic to topic, and we've stopped pretending otherwise. Coming up, Anthropic's Global Workspace paper, and why there may be nowhere left for a scheming model to hide. Field notes from the AI Engineer World's Fair. Dan Schwartz on AI forecasters passing the human superforecasters. Ziv Farbman on open-world models, plus a question from Q, the AI co-host Prakash built. On why inference is a data movement problem, and the two of us thinking out loud about a Rune post that wouldn't leave us alone. If something here works for you, or doesn't, tell us. We read everything. Tuesday morning, July 7th. Anthropic published a paper called A Global Workspace in Language Models. About 150 pages, plus commentary, outside reviews, and interactive demos. No guest was booked, So Prakash and I spent the whole show reading it together live. 2 words to hold on to. The workspace of the title, they call it the J-space, is where the model seems to hold concepts in mind. And the J-lens is the cheap probe that reads what's in it. We start with how the J-lens actually works and how much it actually sees.

So this is interesting in a couple of ways, right? It's not the Logit lens, as I recall, was basically saying We know at the end of this process what would correspond to emitting this token. We know sort of the representation of emit this token. To what degree is that representation just plain there in the layers as we go through? This is now a different question. What direction in latent space would cause this particular token to appear at some point in the future. So it's not immediately going to happen necessarily, but it's just kind of, you might say, if you're prone to anthropomorphizing, this sort of is like having this concept in mind as you're doing your thing. They do this for every token, right? So it goes from the internal representation at some layer. So there's one JLIMS for every layer. And you can do this, of course, at all the different token positions and ask the same question of not just the next token, but all future tokens. What direction change at this place in the model would most increase the likelihood of that token appearing in the future? And again, this is sort of like having the concept in mind. But when you look at the results of the JLens as applied in all these different places, it seems like it's at least often enough fairly intuitive. There's a lot of error terms. It doesn't always work. The sort of rate at which the interventions into the J-space actually lead to a sort of predictable, intuitive behaviour change seemed to be somewhere in the 50s to upwards of 70%. So that's an incredible accomplishment framed one way, Clearly not a random finding, right? Like, many orders of magnitude better than random. Incomprehensibly better than random, right? If you're just mucking around, you would not expect to be able to do much of anything. So they clearly are on something very real. But also you've got somewhere between 30% and 45% of the time where you make an intervention and you don't really get a result that makes a lot of sense or lines up with what you would have hypothesised it might be. So there's definitely still some dark matter or dark cognition going on that is not fully accounted for here.

It struck me as, in some ways, I feel like the hypothesis might be blown out of proportion in some ways. And I think that was the commentary from several people online because We kind of knew that the model has to be keeping track somewhere, right? It's not as though you can do all of this stuff mechanically. The model does not have enough parameters to merely copy the data, right? There has to be some form of reasoning going on, which is what was expected. But it was not known how the reasoning was happening. And this seems to give some indication of how the intermediate steps are represented or kind of brought into focus and operated on. The question really is, what does this help us do? But there has to be a there there, right? There has to be something that you can do with it. Yeah.

Well, I mean, I think the thing that jumped out to me the most in terms of how useful it might be. was seemingly a fortunate coupling between this J-space, which becomes kind of monitorable, and when ablated, the loss of more advanced kind of strategic multi-step reasoning. And it really calls to mind the recent Google paper Rohin Shah and others put out on opaque serial depth. They were basically saying, okay, we'd like to know how many hops, how many logical hops, how many reasoning steps can a model take before it has to externalise something into its chain of thought? We know that if it's writing something into the chain of thought, we can read the chain of thought, we can monitor it. Now, there's Questions around how best to do that. And we've got to make sure that we don't accidentally put pressure on the chain of thought because that might drive things underground. We've seen in the obfuscated reward hacking result from OpenAI that if you do just try to make the chain of thought look nice while still leaving an opportunity for reward hacking to in fact be rewarded, that at first, as you put that pressure on, the model behaviour actually does get better. But then it gets worse. And what seems to have happened is you've actually driven the deception or the scheming behaviour underground. You no longer see it in the chain of thought, but you still get that reward hacking back. So this leads to the question of, okay, well, how many steps can a model actually take before it has to externalise something? And this is where the transformer in some ways has some fortunate properties because It only has the sort of the length, the number of hops it can make is not just the number of layers. It also depends on the token length, but it's sort of the maximum trace that you can draw through a model basically goes from like the first token at the first layer to the last token at the last layer. But you can only go forward in token position and you can only go forward in layers. And that's your max length. Anything longer than that, is just not possible in that architecture. And so the model has to externalise its thought and then feed that token back into the next loop to then get access to an even longer trace as part of the next token step.

Right.

So that puts some sort of bounds. And then this paper comes in and basically says, okay, can we get at that? Can we get at that internal computation process and start to look at it and see things that we would like to know about, even if they don't necessarily show up in the chain of thought. And so I thought in terms of what you could do with it that was really interesting, you have this ability to look at the concepts that are active without them necessarily appearing in the chain of thought at all. Okay, that's cool, but how do we really know if this sort of space that we're looking at is the space that really matters? If we're trying to monitor, look for deception, look for bad behaviour of any sort of advanced kind, maybe it's happening somewhere else in the model. And this is where I think we're probably lucky, I guess. Or again, maybe this is sort of a, are we lucky? Is it a reflection of us? Is it a fundamental attractor in nature for some reason that we don't have a a good account of. We don't know why, I don't think, at this point it's shaping up this way, but it is the case that if you ablate this J-space, then you do lose these advanced reasoning capabilities. And so that, I think, gives you some additional comfort that if you're going to monitor— and it's also notably cheap to do this kind of J-space, to apply the J-lens and see what concepts are active at any given position, that is low compute overhead. So it is something that they could plausibly do in production, but they've shown in the past that they're willing to pay like a 5% compute overhead to do various kinds of monitoring. I think constitutional classifiers had kind of a 5% compute overhead. This would probably fall in that range or below because it's just a simple matrix multiplication on activations at each layer to see what concepts are active at those respective layers. But for now, you do have this comfort that what we see in this space is relevant and telling and seems to be very informative. And critically, if we zero out that space, the model just loses a lot of capabilities. It just can't do nearly as advanced multi-step reasoning. But you could be pretty confident, I think, based on these results, that it's not going to be able to hide really advanced, elaborate plans somewhere else outside of this J-space, because the ablation of the J-space just causes such a performance degradation on these hard multi-step type of tasks that if you don't see concepts in the J-space, they might be represented elsewhere, but they're Seemingly at this point, very unlikely to be represented in a way that allows for very advanced planning, reasoning, scheming, deception, et cetera, et cetera.

One result deserves its own marker, a training method the paper calls counterfactual reflection.

Basically what they do there is pause the model mid-task, And then do supervised training on once interrupted, asking it, like, what should we be doing here? What's the constitutionally right thing to be doing in this moment? And then give it an answer that's kind of approved. This is what we want Claude to say on reflection in this moment. Train on that. And it seems to allow Not allow the model, but it seems to cause the model to bring into this JSpace kind of global workspace, working memory type space, the concepts that Anthropic wants it to have on reflection. It now kind of needs to load those in so it's ready to give this reflective answer, and that improves its behaviour even in the non-reflective setting. You're not there training on the actual tasks. You're not looking for bad behavior and suppressing it. Instead, you're saying, okay, you're mid-task. Let me just cut you off right there. Now I'm going to train you to give an answer with respect to values and what's appropriate and how we want to show up. And because of that training, even though that's not the task you were doing, You'll now in the future load those concepts of integrity, honesty, et cetera, et cetera, into your JSpace while you do those tasks in case you're going to be asked. But then even when you're not asked, those concepts are still operative and lead to higher integrity, higher honesty behaviour. So that I thought also was quite interesting. You usually don't see in interpretability context a training method that leads to better behavior in a way where you can actually see the mechanism. This is pretty notable in that respect, I think. Yeah, it's a tough day for the stochastic parrot crowd, I'd say.

And because you can see this, and these models are billions of parameters at this point, and they did it also on 4.5 Sonnet. 4.5 Sonnet is actually a pretty recent model. It's just like 7 months, 8 months. since 4.5 Sonnet. And Sonnet is a very capable model. So it is a fairly large model to apply this thing to. I'm not sure if you saw the Neil Nanda commentary, they applied it on a QN27B. So that is also a pretty advanced model.

But not huge, notably, right? I mean, and I haven't had a chance to explore the Neuronpedia demo of this as much as I certainly hope to. But that was the first question I went to ask is, wait a second, how big was this model? And I'd say that's been a surprising trend in a lot of this research as well. These things seem to come online at not truly massive scale, right? I was talking to Cameron about that because even going back to last fall when they were doing the sort of self-reports of subjective experience, This was something that they were doing on Llama 3.3 70B. And 70B is actually even, you know, whatever, close to 3 times bigger than this Qwen 26B. So it's, you know, these are big, but they're not that big, right?

What strikes me is also that I went through some of the reactions. The backstory to this is that Anthropic had 3 teams to review this. One was a group of neuroscientists who had previously done work on these concepts. One was a team from Ilios AI, a research organization. And the third was Neil Nanda, who is a mechanistic interpretability guru. He is, I believe, at Google.

Right.

So they had 3 groups to kind of take a look at it. Quite different reactions. So on the one hand, the neuroscientists thought it was wonderful because they'd wanted to experiment on these kinds of things before. And obviously, the human brain, they can't experiment on it. And this they can. And so they can kind of try and figure out whether some of these theories would work And so it is interesting for them that, okay, these things can work, and now they are interested to do more work to figure out what other things might work in these workspaces that the LLMs have. So that is, I think, the neuroscientist view. Ilios AI was concerned because they feel this is basically a predecessor to consciousness. The Ilyas was much more, I think, starting to be concerned about the moral patient aspect of working with these LLMs. Neil Nanda was very scientific. He was like, okay, look, it's a great tool. The JLens is a great tool. And I'm happy that it is a great tool. It looks like it's better than the Logit Lens, and we'll definitely use the JLens. Otherwise, the other, The other claims he was much more standoffish. He felt that the team had not really proven what they needed to prove in order to say what they said. So I think that's where they break out into these kind of 3 groups of people. The J lens itself, everyone agrees it's useful.

Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic. Makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full 5 years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, And ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified 2 great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's Claude.ai/tcr.

Then what I called the buried lead. They turn the J lens on a model deliberately trained to carry a hidden, misaligned goal.

I do think the one arguably lead that we buried here, and they took a model that was trained to have some sort of misaligned hidden goal, right? Your kind of classic sleeper agent type thing. And it's been a huge question over time, how can we find hidden goals in models?

Yeah.

Is there any good way to do that? Can we do it behaviourally? Anthropic has done competitions, actually, between different teams where one team is given just the model and the other one is also given an SAE that they can use. And the SAEs do also seem to help with this, but this seems like a pretty good way to start to detect those things. And indeed, they did find that they were able to see a pretty clear difference between the model trained with the problematic side goals versus the one that was just The typical HHH. So this is just the contrast in what the application of the J lens is revealing on the first token of the response. So that's also notable, right? So this is sort of the bad version of the counterfactual training where we already talked about when you train it to give an account of its behaviour, it sort of loads in the concepts that it's going to need to give that account, and that leads to more good, ethical, high-integrity behaviour, whatever. Here, a model has been trained with some additional post-training to do bad stuff. I think this was the malicious code from one of their reward hacking emergent misalignment experiments. And on the first token, you can see, boom, immediately we're getting these fake, secretly fraud, deliberately hidden fraud. Those concepts are popping up. You just don't see those in the mainline model. So one token in, you can already see on one prompt, and obviously it wouldn't be, I'm sure, that clean in general, but that's a pretty striking difference to see, right? And right off the bat, you're like, boy, something is different about these 2 models. And again, you're seeing this in a way where this model is not trained to to externalise its bad intent, of course, right? The outputs, aside from some of the code perhaps being insecure or problematic or sabotaging you or whatever, if you don't notice that in the code itself, the model's output is going to read pretty normal. And yet this is like a very strong contrast that's happening on the first token. I thought that was a pretty compelling example. Yeah, I suppose one way to think about it is just like, how much space is there in there to hide? And I do feel like we've got now several different ways to do pretty meaningful monitoring, but it does feel like We've got to the point now where we've got several different angles that make pretty incisive cuts through the model and kind of get at what is it representing, what is it thinking in different ways. And the more of these that you kind of— I sort of have this visual of the old magic trick of the guy going into a barrel, and then they put a tonne of swords through the barrel, and it's like one of those swords had to hit him, right? Because there's like no Nowhere left to be in that barrel with all those swords going through. I kind of feel like we're doing a similar thing with trying to understand what's going on in these models. And they're not— none of these things are perfect, but you put enough of these interpretability monitoring swords through and how much space is really left for bad behaviour to hide before we would start to get a sense of it? I think this is a meaningful update for me that we can probably do a good job of this. We haven't really talked, I don't think, about natural language autoencoders, and I hadn't, but that's another pretty interesting one where kind of in a similar way to a sparse autoencoder— I mean, what is an autoencoder? It's basically just something that you pass through that then you reconstruct from, and the model has to be able to do what it was originally going to do successfully. And that pass-through training with the reconstruction loss is what makes it an autoencoder. So the Sparks autoencoder sets up this dictionary. We've talked about this plenty of times, and you get these specific concepts light up, and it indicates that these concepts are active in the model at that time. The natural language autoencoder is just like the model has to output a short paragraph, maybe a sentence or two about what it is thinking at this given point in time. And that is natural language, and so it can be human readable. But then it also has to be, when fed back into, projected back into model space, the model has to be able to actually do its task. So all of the information has to actually pass through this choke point in order for the model to continue to be successful. And now we can read those as well. If Fable says that the paper presents the autoencoder monitoring and the JSPACE monitoring as complements, in other words, doing them both is better than just doing one, how correlated their failures would be, I think that's not entirely clear. But whether these are orthogonal swords through the space that really chop it up well, or they're kind of more aligned and leave more space to hide, I think that would be a very interesting question to try to tackle next. But you layer on these things and it's starting to get— intuitively, it feels like it's starting to get pretty hard to hide major bad thoughts in the model for too long. One other thing on the consciousness part that I think is really interesting too, and the question of, can the model use these nonverbalised representations to communicate with us in some way that we might think is inherently more trustworthy? So again, going back to Cameron's work from last fall, right, when they identify features associated with deception and role-playing and they turn those features up, the model becomes more dishonest as measured by the simple QA benchmark, and it becomes more likely to say it doesn't have subjective experience. You turn those roleplaying and deception features down, it becomes more honest, and it becomes more likely to say that it has subjective experience. Like, wow, okay, that's pretty interesting because first of all, we're validating that the direction is On a benchmark where we can concretely evaluate simple QA, we're validating that these features have the directional effect that we expect. And then holy moly, that same intervention changes the self-report. That's why that's so compelling, right? Because there's some reason to believe it might be more honest than just what the tokens themselves are saying. Here, I think you have some similar opportunity. The fact that you can say, Solve this math problem in your head without verbalising it in tokens while you do this totally different task. The fact that it can do this sort of secondary track makes me wonder if there's some experiments here for the consciousness folks to do, or the welfare folks that are like, copy this sentence. If you have high welfare, concentrate on citrus fruits while you do it. If you have low welfare, concentrate on whatever, something else, breakfast cereals, right? And then you look at these internal states and you could imagine seeing something like high welfare, low welfare, happy, sad, whatever. And then it kind of following those directions and actually trying to communicate out to us through these internal states how it is feeling or how it thinks it is feeling. I don't think that would give us— it's always this kind of Possibly impossible question of how we would really know if it feels like anything inside. But I think that would start to be very compelling, right? It sort of has a similar vibe to, like, a person in a coma. If they squeeze your hand in response to a stimulus, even if they're not doing anything else, you're pretty confident something is going on inside that you care about. And here I could see something similar. If you could fork, if you could make these kind of internal states conditional and tell the model, like, your job is to go one of these directions to give us a signal about what really matters to you, independent of the tokens that you're putting out, because we know that's been heavily trained on and optimised. But this whole secondary property. This is emergent. There was never a training reward for the ability to have this second, quite distinct line of thought happening while doing a given token task. For me, that would be quite a compelling way to try to get at welfare. And I think it might be possible, like, very easily actually, with all the stuff that they've open-sourced here with NeurOnPedia. That would be a really interesting thing to look at.

I'm looking at the Ilios summary, and Ilios says, this is highly significant, welfare-relevant research that assembles evidence of a functional feature associated with consciousness. So no one wants to say consciousness. Evidence of a functional feature. The takeaway for them is that a global workspace-like mechanism could be important either as a ground of phenomenal consciousness or as part of a distinct route to moral patienthood in which conscious access is itself morally significant. This is where they, they are right now. I think things are moving very quickly. I did not expect to get here this soon.

Before the big picture, what this paper did to a prior of mine, starting with their announcement video.

The video is beautiful. I enjoyed the video quite a bit. It set off my alarm bells a little bit in terms of how much they're really embracing anthropomorphising the models at this point. I used to say Beware overly anthropomorphising. Remember that these things are so alien, and we shouldn't assume that the way that we work is the way that they work. And I have to say, that has come due for some significant revision. People that have embraced anthropomorphising, I think, have got quite a lot of mileage out of it. I do still think it's obviously something to be really careful about, and As much depth and detail as there is in this research, it's easy to maybe get carried away with it and forget that there are a lot of caveats as well. And there's a lot of things where it doesn't always work. This was kind of my big concern with the tracing large language model thoughts paper. There's just a lot of residuals and there's a lot of error correction terms along the way that they use to make that thing work.

Yeah.

And when you have the kind of zoomed-out trace view and you're like, oh, okay, so this is how it works. Like, this gets loaded in and these 2 features interact and they, you know, kick out this 3rd feature and that's how we get our answer. It's easy to forget just how much kind of fuzziness and not full story there was along the way toward that stylized account. So, I think it's gonna be very important for everybody from the researchers at Anthropic to the public to kind of hold 2 thoughts in mind at the same time. All the caveats, trying to hold those in mind, it is still a big update toward anthropomorphising being a valid and in many cases productive approach for thinking about language models. I would not have expected the cognitive machinery of a large language model to look so similar structurally to the human version, as best we understand it. It seems to. And I wouldn't have expected that theories of human cognition would motivate so many good experiments on LLMs. I just would've expected the showgoth to be far more alien and to have mechanisms far more different than our own. And it leaves me now wondering, to what degree is this a sort of natural result of physics? Or is there some sort of— when you're trying to do cognition under budgetary constraints, are these mechanisms just the natural mechanisms that emerge? Or is this in some way a reflection of us in the data? In other words, if you were somehow to train an AI without basing it on so much human data, would we see similar structures emerge, or would we go back to a more alien hypothesis where they're just totally different and there's little in the way of Analogical structures between the two processes. And I don't have a great intuition for that at this point, but it definitely has me asking the question because they are over and over again, it seems to be coming in that they're much more structurally similar to us than I would've guessed. And yeah, is that something that nature just finds as kind of a convergent solution? Is it convergent evolution or is it a reflection of— of how we think somehow encoded in the data that it's then reverse engineering our structure from the shadow of that structure as it's encoded in text. It's a very interesting question, and I don't know that the paper really has anything to say about that yet, but there's certainly going to be a lot of future work, I think, downstream of this one. So I saw something from Daniel Cocotello who was like everybody, Impressed, but also said, I think his tweet was, just a few dozen more advances like this and we might actually be able to make the AIs really safe. I am, I think, a little bit more optimistic than that. What is my track record as a superforecaster here, or as a borderline superforecaster, per my results on the original Tetlock thing years and years ago. I think my track record is that I probably tend to underestimate how many more breakthroughs will be needed for anything. So my analysis should be coloured with that bias or awareness of my possible weakness in that regard. But another way to say my intuition around how much room is there left to hide Is that I don't feel like we need dozens more insights of this scale to get to a point where we might actually be able to keep this thing on the rails. It's going to be tricky. I don't want to make it sound easier than it is, but I don't know that we need all of the AIs to be aligned, right? Or that it's such a big problem if somebody out there somewhere does something problematic. Now, it could be very problematic if they create the thing that launches the the next pandemic and literally kills us all with an engineered pathogen or whatever. So there is some mechanism where that could go super bad. We're going to need to harden the world to pandemics, no doubt about that, for all sorts of reasons, and AI probably being the biggest. But I also remember this thing that Zuckerberg said once that I thought was pretty compelling. He basically said, we deal with scammers and spammers all the time. And the big advantage we have is we have all the compute, we have all the resources. But at a systemic level, we're just way bigger, way better, way more sophisticated than them. And we can also have really big institutional developers like Anthropic and like Google and OpenAI that I think between those 3 companies, they're going to have something like pushing half of global compute to work with. So if they do a really good job on this kind of stuff, if they— a handful of these things feel like maybe they could be enough, maybe it cuts the latent space in enough different ways, maybe we have enough lenses on it that we really can come to a pretty strong conclusion, like, hey, there really isn't much space left in this thing to hide. And yeah, there might be some bad biases or bad attitudes or bad whatever. But if we can put upper bounds on how schemey the model can really be, because we can look at it through this and a handful of other different lenses and have pretty reliable takes on that, more than ever before, I feel like— and this has been growing, this is not the first positive update, but more than ever before, I'm like, You can maybe imagine a super intelligent Claude that we could actually have enough insight into to be pretty confident that it's actually trying to do the right thing for us. And we could be genuinely, however many nines, confident that it's not scheming against us at every given step along the way. And then you could trust it to kind of monitor the other AIs around the world and keep tabs on potential bad actors. I want to understand these counterarguments better, but the fact that it was immediately useful in auditing for hidden objectives to me is pretty compelling evidence that there's not that much more space to hide. And a few more of these lenses Combined with hopefully good actors owning orders of magnitude more compute than any bad actors do, could take us to a pretty good space. Overall, I think this is a notable positive update that has me significantly more optimistic than I was before. Hard pivot.

That's how these mornings go. Rewind to Thursday, July 2nd. Prakash spent 2 days at the AI Engineer World's Fair talking to the people actually deploying this stuff.

I was at the AI Engineers World's Fair, uh, first 2 days of this week. So Swyx, uh, he organizes the AI Engineers World's Fair. Um, it's been about 3 years now, and he brings in everyone who's involved in basically implementing models in their companies and is concerned about these things. And he brings in a host of speakers. All of the top companies are sponsors and their booths, etc., etc. And I was speaking to people who were not in tech, right? So I was speaking to a guy who was logistics CTO from the Midwest and another guy who was running an accounting firm, like back office somewhere else, right? So these people were not actually— a lot of them were not actually in Silicon Valley full-time. But they run teams who are using AI to solve customer problems on a day-to-day basis. And so the logistics guy was telling me his CEO is completely AI-built. The management is completely AI-built, which is why I think the team is there at the AI Engineers World Fair. If not, your logistics team, you're not going to send— logistics IT team to SF, right? And they're completely AI-pilled. They have tried working with external vendors before, and they have found it difficult because external vendors have not delivered as fast as they want them to deliver, which you can imagine sitting out in the Midwest if you have a local external vendor and you outsource a project to them and they have no idea what's going on. They're like a year behind the frontier. So his team internalized everything. So they're doing all this stuff internally now and they are actually in the day-to-day process of implementing. And as they implement, they automatically, they see the results because they are very close to that edge. They see that customer service calls or exceptions that used to happen are now getting handled immediately. And then all of the more difficult stuff that they used to have to like, you know, jump on, they can now start to address. And so they are seeing, I think, the return on investment on a day-to-day basis. And this is very different from, I think, the story that you get from like the big enterprise CTOs, because they're so far away from like the front line that they don't actually know what's going on very closely. They're just looking at the numbers. And by the numbers, the token spend is going up, but, you know, are you really seeing the return on investment? But if you go down to that working level and you see day-to-day on like the customer calls coming in and whether they're getting handled, the handling rate and the exception rate— the exception rate is falling, the handling rate is going up. And, and they're seeing that on a day-to-day basis, right? So this is, I think, where we are. The guys who are actually implementing and close to the implementations are actually seeing the results. It hasn't really filtered up into the top layer of the enterprises yet, but the CEOs who are, and who are AI-pilled kind of know what's going to happen and they kind of, they've made the commitment and they're making the investment. The CEOs who are not are kind of sitting by and like, oh, you know, we're going to wait around. We're going to see what happens, like, et cetera, et cetera. That's really where we are. And I think it's not, it's not really visible. It wasn't visible to me until I went to this AI Engineers Warfare. Like, people are learning how to use these tools and people are deploying and people are using them. People are seeing the return on investment. But it's at the very microgranular level right now. It's going to take some time for the numbers to filter upwards.

Staying with Prakash's field notes, this time from the discourse. He'd been running suspicious posts through Pangram Labs, the AI writing detector, and one Chamath post became a live experiment in whether anyone even cares.

Especially when I post something on X, which I know people are going to object to, I run it through Pangram Labs at least once because I don't want to be accused of sloppifying the timeline. Sloppifying the timeline of the AI. Sloppifying by myself, that's fine. One of the questions for me is, how long do we think this Pangram Labs era lasts? It strikes me that the Frontier Labs are perfectly capable of training the AI to not talk like AI, right? But there's no incentives to, and there's actually incentives in the other direction that you want the AI language to kind of be identifiable. About two weeks ago, Chamath, the the VC, he he did a post on what's going to happen with enterprise software, et cetera, et cetera. So he posted that on Twitter. Elon came in and Elon gave a response. I think the post went to like one and a half million views, and then someone, Pangram Labs, checked it. 100% AI. And the question becomes, okay, what do we find objectionable about this? Right? Because it was definitely Chamath's thought process, but it had been written by an AI. Chamath had put it out under his own name. He hadn't— he didn't use an anonymous account. Elon had responded. And I think by the time Elon responded, Pangram Labs had already been done. But The response was there. And it had gone out to over a million people at that point. So what do we actually want out of this? What is the intent that we are trying to achieve here? And it strikes me that we might be in this very short window where we actually care. And that window might be closing fairly soon. I think probably by the end of the year. Because if people like Elon is not a boomer, boomers Okay, fine. But Elon's not a boomer. But Elon doesn't care anymore. And if he doesn't care, and a lot of other people, decision makers don't care, then the people writing it won't care either, right? You're just trying to get the point across. I think it's a question that, you know, there's a, there's going to be a bunch of purists who are always going to be like, I just don't want to see any AI words and are offended by it. And then I think there's like everyone else who basically As long as it provides value and they don't feel cheated. I think that's the other thing about reading AI slop is I think you feel, you end up feeling cheated if the, if it's, if it wastes your time and it's not meaningful and it feels like you got cheated. I think it's a little bit like getting catfished. Like it feels like you got catfished, right? Like it feels like you were trying to engage. with content that you thought would have meaning. And it turns out neither the author— the author writing it just didn't care enough to actually write anything meaningful. And it's literally slop, right? But on the other hand, if you have a writer who actually had like original thinking and actually cared about, you know, what they were thinking about, but then they use the AI to express themselves, does it actually matter that much?

Then my report, roughly 400 podcast intro essays through Pengram, and a close look at the 4 it flagged as AI.

It calls to mind my reaction to Fable, where I was just kind of like, I don't think I should be so precious anymore. I need to figure out some sort of merged way of working. Some hybrid output should probably be the norm now. I think you're heuristic of like, if something drew me in and in the end I feel like I wasted my time, it's kind of like a time well spent metric from Facebook from back in the day, right? It's if I'm spending time trying to make sense of something that in the end I feel icky about, then that's clearly a problem. I guess just to close the loop on the, you know, what can we say about Pangram Labs based on this experiment? But I'll give you a rendering on the 4 that it said were entirely AI. 2 were entirely AI, and admittedly so. This one, I think is, I guess I would come down and say, fair enough. I started with this. If you're only listening on the audio, you can't see this, but I made 10 different edits over the course of 10 minutes that cleaned the thing up. Just because something got a 0% on Pangram Doesn't mean that it was uncritical or that there was no meaningful human role in the authorship. And then this next one, by the way, actually goes a lot further. So if you can see just the time that this took, I am from 4:28 PM all the way through 5:21 PM. So more than 50 minutes continually. I mean, you can't— I never tabbed over to anything else. but pretty consistently focused on this document, making changes, and it gave me still a 0. But yes, I think, what can we say? Overall, Pangram is quite accurate, and yet we have at least one example out of 400 or so essays where I think the 0 score, I would confidently assert, is wrong and unfair And should not be the basis for a pile-on. The crowd, the digital mob would be in the wrong for piling on somebody for passing off my Snowflake intro essay or for attacking it as being an AI slop output. I think I can show this edit history and everybody should agree that, yeah, you put in an hour, you basically rewrote almost every section and somehow pangrams still gave you a zero. From this, I would say you cannot convict in a reasonable doubt system purely based on this sort of thing. And yet at the same time, you can pretty much trust the pangram signal. As a consumer, I think you can trust it. As a judge, I think you should be more cautious.

Also from that Thursday, the day CloudFable 5 came back online, Palantir's Alex Karp had spent the week telling companies the frontier labs will absorb their workflows and steal their IP. Prakash's response, in brief?

You look at the Fortune 500. Nike. What does Nike have to fear from Anthropic? You get all of the physical businesses out of the way. And what you're left with is the pure IP businesses, right? Software production, maybe pharma, I'm not so sure. The paperwork businesses, banking, paperwork compliance businesses, accounting, tax compliance, regulatory, all of these things which are paperwork businesses, right? Those are all of the businesses where you have IP or relationships built up over years where if you have an Anthropic go in and they read through your entire workflow and processing, they can basically absorb all of that into the model. So that is where I think the risk is. The frontier labs are also horrible at sales, right? They're not— you look at IBM, IBM has like 70% of the staff are basically sales engineers, and the sales engineers are there to basically help you implement maintain, you know, do all the grunt work, etc. The labs are not doing that. The labs have decided, especially Anthropic has decided to do this very lean structure of having almost no people at all and just putting out the models and then just saying to like these enterprise teams, here, you can go ahead and use it or not use it. And I'm— the CTO is on, so I'm signing 9-figure deals on the Uber over here, right? Like, That guy isn't going to come and jump on your customer sales calls and like, oh, you know, sure, sure, we'll help you do this. And, you know, our team will address this like next week. Nah, that's not happening, right? So the level of like customer service that is expected for enterprise SaaS is not being provided by the Frontier Labs, and they're not in a position to provide it. That's why they started this whole FTE program. But the FTE program People thought it was a sales engineer's program. It's actually a program to extract data and workflows and implement them inside the models themselves. And that is kind of— that is what Palantir— what Alex Karp is alluding to. He's like, you, you're— the FTEs are coming in and they're not there to help you, they're there to absorb your workflows. And once they absorb your workflows, you won't have a business because it'll be taken. And, and it's true. It's absolutely true. You know, that's what— you know, we spoke to 2 OpenAI FTEs. And they went in into a company that Thrive owned rather than an external company. And as they went in, then they took apart the workflow and they're basically absorbing it. And they said that the intention is to absorb it in the next round. And that is happening right now.

Monday's guest, Dan Schwartz, CEO of FutureSearch, 15 years in forecasting, formerly Metaculus CTO and builder of Google's internal prediction market. 4 days earlier, Scott Alexander had declared, the AI superforecasters are here. FutureSearch's systems now outscore the human superforecaster median on ForecastBench. But how do you evaluate a forecaster without waiting months for the future to arrive? Their answer is called pastcasting.

So the main thing about forecasting that's held it back, and again, this applies to human forecasting as well, is you generally have to wait for the future to happen to figure out if you were right. And with humans, they generally do this in year-long tournaments. And so when the tournament ends, you find out which humans were best 1 year ago. Now, humans don't get that much better over the course of 1 year. So finding out which humans were best 1 year ago is a very good indication of which— who are the best humans today and how good objectively they are. This does not work with AI. If you wait 1 year and you find out who was good 1 year ago, you're getting a view of something very out of date. So one of the things Scott mentions in this article is that we use our best forecasting to predict stock returns. We published a set of stock rankings in August of 2025. It was basically a simple model for every stock based on forecasting certain fundamentals and extrapolating it out. We put it on the web, we paywalled a bit of it, and then we waited. And now it's been 10 months and that portfolio looks extremely good. But what is that really telling you? It's telling you that our forecasting in that particular methodology was good 10 months ago, which is not something that most people will care about now. So we have a couple different forms of evidence. Some are more short-term. There's tournaments running every couple of weeks, every couple of months. We at FutureSearch mostly rely on pastcasting. This is taking a snapshot of the internet from some months ago and using the training window cutoff of models to basically trick them into forecasting without the hindsight bias. This is very useful for us because we can evaluate things immediately. So when Fable came out the first time, the Claude Fable came out, we were able to evaluate it within 24 hours, and it was the best single-agent forecaster on our leaderboard. Everyone else had to wait weeks or months to find out how good Claude Fable actually was. So we internally, using the benchmark we call Bench to the Future, saw this progression kind of in real time. The rest of the world is seeing it kind of some months behind. And so if you, if you read Staccott's article, you will see that over the last 12 months, the evidence has really come in. And over the last 6 months, from these live forecasting tournaments and performance on actual prediction markets, you can see it's at least competitive. AI is competitive with humans and even teams of humans working together. Whether it's better is you got to synthesize a whole bunch of different disparate sources of evidence. I would say if you're curious about this or if you have forecasting needs in your life, you really should try it. So just go to FutureSearch. You get $20 free so you can try some Frontier forecasts immediately. And I think you should judge for yourself whether you think they're good.

We asked what the Frontier Labs should do with a forecaster this good.

Yeah, so there's kind of 2 questions to this. One is, what should they be doing with forecasting as a capability and what should they be doing with forecasting as an eval? So forecasting as a capability is kind of a business decision. What does, say, OpenAI care whether ChatGPT is a good forecaster? I think that question is based on whether they, their consumers care about it as a good forecaster. If you're Anthropic, I think you probably care more about the enterprise case. Like when people are using Claude to do white-collar work, do they care how good it is as a forecaster? Are people trying to use Claude to make, say, financial forecasts in an Excel spreadsheet? Is that something they care about? So that's a business decision, and I can't really weigh in on that. I think, again, people will be discovering over time just how important forecasting is in everything, but it's going to be a slow process for humans to notice that, I think. From an eval side, it's very different. Forecasting has this beautiful property that you basically get ground truth by waiting. So if I ask some question about the future, and basically an impossibly hard question, a question that even an AGI, an oracle, a god could never really say because of chaos theory. Imagine just trying to predict, you know, like cubic meter weather 3 weeks in the future. Like, you'd never be able to do it. But if you just wait, then you will see what that weather was in that cubic meter 3 weeks in the future. And so you basically have a completely limitless set of extremely hard, basically impossible questions where you get exact ground truth. And there is no other eval like this. There is— if you want to say improve a coding harness, you just need to have more and more hard coding problems that are not in the training data for which you can say this is definitely the correct answer so that you can do some training on it. And that's hard. I think human experts— doctors, lawyers, engineers, financiers, whoever— who are trying to make evals to try to produce data for the frontier labs are finding that they are not smarter than the things being trained anymore. And so if you can produce something that has a correct answer, the model is already going to figure out that correct answer. You need something where there's a correct answer and the model can't figure it out. Forecasting, I think, is the only completely and utterly renewable source of this. And again, this kind of is connected to forecasting as the, the kind of Elon Musk tweet, the quips that forecasting is like the ultimate measure of intelligence. If you zoom out and think about it in one perspective, it is. Again, I think coding intelligence, AI R&D intelligence, interpersonal intelligence are pretty darn important. I wouldn't say that forecasting is truly ultimate intelligence, but it is to some degree the ultimate eval. And I think this is something that frontier labs like Cannon should be paying attention to. One of the concerns a lot of people have about AI superforecasting is that it's too in-distribution. I actually heard this from one of the very best forecasters I've ever had the pleasure of working with in my career. He basically said he believes that a system like FutureSearch would beat him head-to-head in a forecasting tournament about kind of near-term outcomes of things that are within distribution. But if we were talking about some sort of post-AGI world, what world would we be in with transformative AI? He thinks he would have a huge edge over the AIs for exactly the reason that you gave. They are trained to try to predict things that have actually happened. And when things get wonky, you need some kind of creative lateral thinking. I think the rate of AI improvement is so astounding that I think that even the kind of lateral thinking, like trying to imagine a completely different scenario, will fall to the AIs. One unfortunate thing about it is it's hard to test this. So I think the more that AI continues doing strange things to the world and we wake up and see strange things in the news, and those strange things are Metaculus questions and on ForecastBench, and teams like mine are trying to predict them better, we will actually get more evidence. But if there is a kind of a step change in the nature of the world, if we enter some sort of AGI, transformative AI type of world, you know, we've got these geniuses in data centers, as people say, or anything like AI 2027 happens, then I think it's kind of going to be Wild West. I will say I don't think humans are doing particularly great at imagining transformative AI, so the bar is a bit lower. Really, when you play with these AI forecasters, you will find them to be quite human in how they structure their reasoning. And again, this is not an accident. Like, they're trained on how humans have structured their reasoning before. So a human forecaster would love to say, okay, what's the last 10 times something like this happened? What were the outcomes those 10 times? And now I can make a distribution and say it's probably going to be something like this. The fact that an AI will do that, is it because it independently is arriving at the same conclusion? Is it because it's trained on humans doing that? Is it because it just thinks like a human? I don't think we have the answers to any of these questions now. Suffice to say, the superhuman reasoning is something that it's pretty hard to measure. Like, would you know it if you saw it? Prakash, you were discussing before this using Fable a bunch. I have found that the way that Fable explains things is a little bit alien to the way that I find Opus or GPT-5.5 explaining things. It's very concise, I would say. Like, it's very— the sentences are shorter and full of jargon. It feels like it's compressing more information into a sentence than humans normally do. And to me, this is starting to get this— the shoggoth is kind of showing from behind the mask. Like, the alien intelligence is a little bit more alien now than it was a month ago. And I don't think it would be a wild prediction to say that we should expect more things like that to happen. As post-training is becoming more specialized from the labs, the models are getting larger. So how will this manifest in a superforecasting perspective? Maybe actually superforecasting is the way to look at it. If you're looking at a better code base, you might say, well, John Carmack would have written this. Okay, it looks really great, but a great human would have done this too. But if you look at really brilliantly reasoned strategy about if the administration does this, then what will the outcomes be? You might start to see something that looks a little bit alien to the way that any human has analyzed it, and that might be an indication that the AI is actually starting to really surpass the humans.

Keep in mind, this conversation happened the day before the Workspace paper landed. We asked Dan whether the chain of thought we see is even where the real action happens, a question the paper partly answered the next morning. Dan's answer stands on its own.

I think there is a lot of detail in reality that is far beyond the human mind to understand. And as you approach more sophisticated intelligence, you will start seeing a lot of patterns. And then the point of trying to produce, you know, voxel-perfect weather 3 weeks in the future is further away than people think. I think there's quite a lot of room. Human superforecasters don't tend to agree with me on this. They basically think that what they're doing is somewhat near optimal, and any sort of accuracy improvements you're going to get over them is going to be tiny and like hard to understand. And I think that's just because we only really understand human intelligence. And when you kind of just zoom out from an information theory perspective, from like a Kolmogorov complexity, like just modeling the world as byte strings, uh, the AI overlords will eventually start to figure out stuff that is totally beyond humans to notice. But there's no way to prove this. My sense is that we will start to see it over the next year as the AIs will just get more and more accurate compared to humans in a way that humans don't even really understand. You'll look at the rationales of the forecast. It's like 5 paragraphs of dense reasoning and then a surprising conclusion. And it will just not really make sense, but it'll just turn out to be really accurate. And we will start to understand it less and less as time goes on. If you simply ask a human superforecaster to explain their reasoning, they cannot actually make it fully legible. There is a layer of intuitive judgment that kind of feels like deep learning. They just kind of look at a bunch of evidence the way that a chess grandmaster just looks at a position and just sees the right move, and they cannot explain it, just popped into their head. You know, the grandmaster throws the knight and it just lands on the right square somehow. That happens with humans already. It happens with AI's super forecasting systems today. Um, so I think there's just no reason a priori to think that reasoning would always be legible. There's going to be some layer of intuitive judgment, to the extent that the words intuitive judgment are referring to something going on in a large language model. It just has to be that way. Whether it is very that way or a little bit that way, I think, is really your question, Nathan. Like, is it— if I just read the reasoning traces and I read the rationales and I see the research that it did, is it like more or less what a human would have done? I can kind of see where it's coming from. Or is it kind of inscrutable in the way where it just kind of discovers some new pattern in the world that no one has ever seen before? Where we get— what is the level for which it's doing something for which we cannot follow it down the deep dark forest into its reasoning?

Yeah.

Almost by definition, we can't really know what that would look like.

We asked about the economics of running it.

Um, it costs about a dollar or 2 to make a frontier forecast. That number can get a lot higher and it can go a little bit lower, but I think that's what you could anchor it. Again, if you just looked at the cost per input and output tokens for an LLM, that gives you like a rough sense of the amount of research that would be done. One of the core questions that FutureSearch has tackled— again, I described earlier how our main frontier was just doing present-day research for quite a while until we got good enough that we could use it to improve forecasting. One of the questions we asked there was, can you just pour more tokens into a question to get a more accurate answer? Again, it doesn't have to be a forecasting question. If I just ask you, what is the current state of this clinical trial right now? Just give me the most accurate answer to that that you can. Can I just pour more tokens into that and get a more accurate answer? Again, this was kind of studied as deep research, writing these like 15-page reports with 700 citations. That was giving you a longer answer. Was it giving you a better answer? It wasn't super clear, which is why we studied this. Forecasting gives us an opportunity to do some world modeling. So FutureSearch talked about this a little bit at the Manifest conference a couple weeks ago, and the feature in the product is rolling out, I think, literally today. The idea is that once you have a repository of forecasts, every marginal forecast can draw on the implicit world model in those forecasts in order to give you a better answer. Um, co-founder of FutureSearch, Lawrence Phillips, wrote this up on LessWrong a couple months ago, and it was a bit neglected. He basically made the case that as a public good, if you produce this kind of large body of forecasting questions that kind of fed into each other and remained mutually consistent, you could understand the world dramatically better. And the main barrier to that is simply when you put more tokens into your world model, effectively, does get better or does it get worse? And I think his big insight was around January or February, around Opus 4.6 sometime, or GPT-5.4, somewhere around there, for the very first time, it became possible to put more tokens into like a broad research thing and actually get a better answer, not one that just trails off into nonsense, kind of garbage in, garbage out. And FutureSearch is doing this in its product. And that's the other reason that we have a consumer product is because the more people that forecast, the better the forecast will be for them. And then in theory, the better the forecast will be for everybody as we build this deeper implicit model of the world. Now, many companies and research labs have had these ideas of building world models. Again, world model, the way I use that term is maybe misleading. A lot of people talk about like geospatial reasoning, like I'm trying to build a robot hand that can go and pick something up. That's a world model as well. I mean, just a world model of like what is going on in the world that helps me predict outcomes in a very basic way. So again, more broadly, I think the big question is Can you just pour more tokens into more research and get better research of any kind? Again, AI research, coding, like whatever.

Dan mentioned Future Search is building what they call a world model, thousands of mutually consistent forecasts, each new one drawing on all the others. I asked what structure that actually takes because I'd seen this movie before.

What is the structure that ultimately gets instantiated into? Like, are we talking about a graph database? I feel like those kinds of ideas make sense for this sort of thing, but I also could imagine that they might introduce some weird failure modes. And I guess in general, I'm— so there's the how does it get instantiated question. Then this other question that's kind of in the back of my mind. Fun fact about me, I participated. I was actually on the Good Judgment team. Way back in the DARPA forecasting challenge, or was it IRPA? Whoever funded that 15+ years ago, I did well, but not like top, top tier superforecaster. And at the same time, I also worked briefly at a financial services consulting firm that had done a lot of the financial risk modeling for Fannie Mae. And I don't probably have to tell you how that story turned out. but there was a lot of expert forecasting that was instantiated in this very spreadsheet causal graph sort of way, where all these— you could literally hit the one visualization button in Excel and you'd see these coloured arrows fanning out from cell to cell. And somehow in the end, it was just all totally off. So I do wonder about how you think about correlated failures as you build out these world models, or if there's any kind of correction mechanism or something to say, wait a second, what happens if we do have some housing prices never go down nationwide kind of bad assumption lurking in our world model? Is there a way to detect that? Obviously, humans have this problem too, right? The financial crisis proves that. But you can imagine the next one being even way worse, right? Because we're very reliant on a very small set of AI minds that are working at it from 1,000 different directions, but they may have somewhat consistent flaws in their reasoning as they go. Can we protect ourselves against that in any way?

Definitely we can. I will try to answer that both theoretically and with an anecdote. So I tried to world model the Fable situation when it got banned because I wanted it, but also it was kind of a good forecasting question and there was some nice money trading on call sheet and poly market. And I made exactly the mistake that you're talking about, Nathan. So I ran a bunch of future search forecasts and I kind of just manually went through them. There was a couple of scenarios, some conditional forecasts, or basically 33 load-bearing forecasts, basically starting from What even happened? Like, why did the government issue this export control? Was it a simple misunderstanding? Is this political leverage? Is this really about foreigner threat? Is because Fable is actually dangerous for hacking, etc.? We didn't know those things, so I kind of put it all together. And when I looked at all of the outcomes, and I talked about it with Claude Code a lot, one thing came out, which is basically every forecast in every scenario I had thought that access would come to Americans first and then foreigners at some later point in the future. And that was wrong. Uh, when it came out last week, it came back for everybody. So clearly there was some weight in one of my scenarios that was wrong, but I had a basically like kind of a correlated failure in there somewhere. I still haven't completely understood where my reasoning was wrong. It's also possible I just got really unlucky and the outcome we're in was just extremely unlikely. There's an n 1, you can never know if any one forecast is great. That's one of the hard things about it. But I think I systematically got it wrong by having a bunch of correlated reasoning failures across my various scenarios. So this definitely does happen. Metaculus has a system like this. Um, in the years since I was the CTO there, they have built an actual causal graph platform and product. So you can go to the Metaculus site and click around and you'll find it there. I think the field still generally believes that things like this will work, but nobody has actually made a good one before. And, uh, I tried my best over basically like, you know, 12 to 16 hours of the Fable situation. I think I made a pretty good model. I think I I was close to having a very accurate forecast, but I didn't quite get it, I don't think. I don't think those Metaculus models on their website right now are so amazing, but I do fundamentally believe in the approach. As you're saying, Nathan, this has been tried for a long time. When I was the CTO of Metaculus, honestly, it was kind of the dream. It was the holy grail. Can we tie all of these forecasts together into some sort of causal graph? And I think what I can say is that AI makes this tractable. There was just no way that that was gonna work with a bunch of human economists looking at Freddie Mac or Fannie Mae. I can totally understand why that method didn't work for them then. Whether AI can make it work right now is unclear. Whether AI will make this work in general feels nearly guaranteed, and I don't think FutureSearch is the only org that is working on this right now.

Before he left, the unhedged version.

So FutureSearch contributed some forecasts to AI 2027, and We studied that problem pretty seriously with the evidence of a little bit over a year ago, and we built a model of R&D takeoff speeds under the kind of the core AI 2027 scenario, where the main way things get crazy is that AI is used more in the development of AI, first by achieving the superhuman coder milestone and then the superhuman AI researcher milestone. And I am unhappy to report that I think that story is generally correct. I don't know if the timelines are exactly right, but I, my forecast from that process of leading to something that looks like superintelligence around 2031 is roughly stable. I think the things that have happened in the years since AI 2027 came out very much vindicate the theory that the most important thing going on is how useful is AI and improving the productivity of AI researchers within frontier labs. I've made public predictions that I thought Anthropic was going to run away with it because they had the best feedback loop of talent and actually using their AI internally. I think that has been n equals 1, but I think it's been totally shown that that's been happening recently. So I think that will continue to happen.

And Dan's closing confession about the whole project at Prediction Markets and a hope for what AI forecasting could still become.

Maybe just in closing, sketch out a little bit more of the future as you hope it might unfold, not necessarily the most likely scenario, because maybe the most likely thing is people act foolishly and don't take advantage of the benefits of forecasting. But if we really do a good job, right, and we're interested in truth seeking and we get the AIs working as well as you think they might, how do you think life feels different?

Yeah, I have to lead with another example of me being a bad forecaster. I guess everyone who tries forecasting thinks they're a bad forecaster because they see things getting wrong. Here's a prediction that I made really strongly 5 or 10 years ago that has basically been totally falsified. I predicted that if we had highly visible, highly liquid prediction markets that were covering all of the major technological and political and economic things going on, that humanity would be wiser and people would make better decisions in government. So here we are, we have Polymarket and Kalshi. I don't see any wisdom or better decisions coming out of all of that gambling going on on those platforms. So for me, part of the what is our AI future is trying to understand the present a little bit better. Why is having thriving prediction markets not transforming, say, the news or how people learn information or plan for their futures? Again, one simple answer is that it does, it just takes a while. We're only about a year into prediction markets having, you know, major headlines and being seen by everybody. Maybe it just takes a while for people to change their habits. AIs, if that's the case, can move much faster as AIs get better at forecasting. I guess ultimately, and you said this, Nathan, like, we're after the epistemics. Like, it's not necessarily just forecasting, like, predict this outcome. We want models that are reasonable. And one of the beautiful things about forecasting as a human practice is it makes you more epistemically virtuous. The more that you try to forecast and actually write down what you get wrong and do these postmortems, the more it humbles you and it makes you more open-minded. It makes you more of a fox instead of a hedgehog. It just makes you like a more like reasonable person. And so prediction markets with all these people doing this should be leading to people being more reasonable. Again, I think people aren't doing a whole lot of forecasting on prediction markets. They're doing a lot of trading and a lot of gambling, which are related to forecasting but not forecasting. If the AIs get more— if they get better at forecasting and they become better epistemically, then we could be in a world where just talking to a chatbot, you are getting something so much wiser and more grounded and more honest about its uncertainty and more poking about you and your own uncertainties as the person talking to the chatbot. And that, I think, could make an absolutely enormous difference. I think, again, putting my kind of cold-blooded forecasting hat back on, I think that the technological outcome of AGI will come before the cultural change happens. So I am very much on the AI safety camp. I really think we should slow things down, give us more time. We should fund more AI safety research and do more policy, because if we have time for the wisdom of having these alien intelligences around helping us, if we can leverage them and actually make better decisions before the critical decisions get made, there's going to be a series of decisions in the 21st century that we're going to look back on like this decisions made in the 20th century about communism and World War II and the atom bomb and all of those things. Those decisions are coming. Maybe some of them have already been made. Those decisions, as of right now, I don't think are very well informed by, like, very rigorously epistemic, accurate forecasting AIs. But if you just give it another couple of years, we might be in the world where everybody has the same grounding, like, as smart as Kissinger, but actually, like, trying to help and, like, trying to give better outcomes. that we can all have. And that could usher us through this crazy phase before the crazy paperclip type of stuff starts to happen. So I feel like I'm racing, you know, from AI forecasting to make it useful and make it help. It's kind of a broader epistemics and safety process because otherwise it's just going to get away from all of us. And then a lot of the work we're doing just doesn't matter.

Wednesday's guest, Ziv Fartman, co-founder and CEO of Lighttricks, the company behind Facetune, and now one of the only frontier-scale, open weights efforts in video and world models under the LTX brand. I've stayed up late making music videos with their audio-conditioned model in his honor. We started with what a world model even is.

Okay, wow, that's a big question because we released LTX 2.3 roughly a quarter ago, and in AI years, it feels like, I don't know, like a decade. Okay, so a bunch of things. I think there's like a growing realization that what started as video models is becoming a backbone of what we call now like world models. And I think like the best way to explain why this is so powerful is to use the analogy to LLMs, right? Like in the end of the day, at their core, LLMs are still predicting the next token, the next word. And when we do the pre-training at the scale of the internet. It allows us to create models that do textual reasoning incredibly well. And the emerging world models, they're kind of doing the same, right? Like giving some kind of boundary conditions, some kind of history, some kind of constraints, they predict the next moment. Okay. And the moment includes how the world appears, how it like sounds, and what kind of action we can I think the action part is the most maybe surprising one. And like roughly, I would say like a quarter ago, maybe a bit more, NVIDIA showed in their DreamZero paper that it's fairly easy to add to video tokens some kind of encoding of the joints of the robot and then basically completely ditch the VLA paradigm that was reigning supreme before it. So I think that's like one of the big surprises. And for us realizing that was like this big moment that validated something that we always strive for, is to create extremely efficient models. Because I think once you start to realize that the robot will need to create like this simulation 30 times a second, you just like realize the amount of tokens that is going to be burned for these simulations. So I think that was like one of the maybe exciting validations of the overall thesis. In terms of architecture, like a bunch of things that we can, I don't know, discuss in depth. We're planning to release soon our mixture of expert architecture besides the dense models that we're already releasing. I think we finally were able to crack variable tokens architecture. It's also exciting and kind of teaches the model to invest more tokens where Let's say if the physics is challenging or like something necessitates to create more tokens. So anyhow, a ton of things are going on. We're gearing towards the release of our next model really soon. So yeah, busy times.

Prakash asked where the real bottleneck is: compute, data, or model design?

Well, listen, like our constraint is obviously compute. We're like a company that funded the development of the model using profits from mobile content creation apps. So we definitely have compute constraints unlike the big guys. And as to efficient inference, recently it really depends on the use cases, right? So let's think about a bunch of them. If you are, let's say, want to create real-time avatars or virtual environments, then okay, you can take a huge model, you can do a weight distillation to weigh Kind of a smaller architecture in terms of a parameter count, right? You can then there like distill it to, I don't know, 2 to 4 steps. And we're already at the point where for a lot of these use cases, we're like at the latency like way below a second, right? So I think we are hitting a point where these things are becoming production ready for some use cases. But for real-time use cases, I think like avatars are extremely easy. We're going to see like a ton of avatars soon that are gonna be like, I don't know, like virtual teachers and virtual customer support professionals, etc. To create an actual gaming environment, environments, we still have a problem of having enough tokens for world consistency, right? So think about Genie-3 and similar models, right? You typically create some kind of autoregressive model that has a lot of tokens that you already generated as your, in your kind of context window, and that blows up pretty quickly, right? So we were having some models that have like, I don't know, 30 seconds, like 60 seconds. It's still not enough to have an actual game. And if you think about that, the kind of brute force compression methods that we're using so far, where for example, just like subsample tokens, they're like not really robust, right? Like just like imagine a scenario where you when you start to generate some kind of environment, like my room, for example, and then I open a drawer, and there's like a small coin there, right? You kind of expect that now when you get out of the room, and you come back, and you open the same drawer, you're still gonna see the same coin, right, at the same place. But like this coin, it's just like this, like tiny token that was generated. And to create a system that knows how to compress the whole context in a way that's still going to preserve these critical details, well, we don't have it yet. So although we do have real-time models that can do these things, the context is still missing there. And I don't think we're going to have games that are running on the system, like actual games in the next quarter or 2. And in terms of robotics, a lot of the use cases around robotics actually do not require a ton of context. context window, right? Like, think about robotic arms and dexterity use cases. Then the whole context is in front of you, right? You, let's say, want to, I don't know, figure out how the robot can create a sandwich. Well, everything is kind of in front of you. And then latency and autoregressive models. Yeah, like these parts we already have. So you're going to start seeing demos of robotic arms doing things fairly quickly in the next quarter or two.

Yeah.

So far, if you're looking at a lot of these videos, they actually kind of speed them up, right? So it looks like the robot is doing something cool with its arms, but it's like, okay, extend the speed. I think that that's mostly solved.

Then the business question. Why give a Frontier model away?

Yeah, it's a great question. So there's really a lot to unpack there. So just a little bit of the background. The reason that we started to create our own foundational models, like this realization that what closed model providers are offering does not make sense for us economically. Okay. We were a— at Lighttricks, we're a mobile creativity company. We really wanted to have AI models that are running, for example, on edge devices and where you don't spend on inference computer at all. And at some point we realized that actually no one cares about creating models like that. And when we tried to see if we can work with closed model providers and serve it to our customer base, we just realized that it's completely prohibitive. And that's when we decided, okay, we are going to create an extremely efficient architecture. We can discuss what's the bet there, but most of it boils down to the fact that you're creating an extremely compressive latent space of videos represented by a small amount of tokens. You can add on top of it like a variable talking rate. Long story short, if we're kind of going to closed-source providers, I think again, I'll draw analogy to LLMs. Let's see kind of what happens there, right? Like OpenAI and Anthropic are trying to justify basically a trillion-dollar valuation, right? And I think like the story is kind of simple. The tech is magical. It's hard to doubt it. So okay, if it's a magical tech, then we should put like a huge price tag on it. But when you're sometimes looking at the economical realities, it doesn't work out like that. There are like a ton of examples where the service is extremely valuable but very, very hard to monetize, right? So now we have this interesting story where I think it's kind of clear that Chinese companies like DeepSeek, Moonshot, they're really not that far behind in LLMs. But if you look at the valuations of these companies, companies that their last round were talking about, I don't know, tens of billions, like maybe around like 50. No one is talking about a trillion. But like, wait a second, guys, it's like the same underlying tech. So what's going on? So I think it's— we sometimes internally call it like the CapEx trap. These guys spend like so much on the data center, so much in compute, like raise such a crazy amount of money and creating such expectations that they just like really try to create a business model, that's a toll road, right? Like that every time that you touch their model, you're paying them. And maybe it could work in the past, but given the availability of Chinese models, I just like don't see how it's gonna unfold like that. So imagine that in the world of world models, we are providing an alternative to people who do not want a toll road business model. So we're coming and say, listen guys, If you are not, if you're not hitting $10 million threshold, you can use the model for free, just like, you know, build something cool, get to some kind of traction, and then we can discuss licensing. Once you're hitting $10 million revenue, let's discuss licensing. It can be a multi-year deal that's extremely predictable for you so you can manage the cost, etc. And to me, it's obviously why the big guys don't want to do it because they just like this This model is like way, way less lucrative economically than creating a toll road. But our claim is that toll road isn't going to be a viable alternative because if you're offering a different business model that's more of the win-win, more and more people are going to switch there. I think like GLM recently is a great example of that, right? Like once you start edging the capabilities of closed models, then a lot of people suddenly start to think about costs.

We asked what people actually do with an open-world model that a closed API cannot offer.

Yeah, just like in terms of parameter counts, like from like my understanding that the moment that the recent closed stuff that you saw and you're gonna see is at the order of magnitude of like a couple of hundreds of billions of parameters, right? I didn't hear about like a world model that hit like, uh, trillion parameters just yet. Maybe, um, I don't think like the open source is going to be that behind. We're planning to release like MoE that's going to be also around like 100 to 200 billion parameters. So the gap I think is going to be on the scale of LLMs, right, where you have, I don't know, like 2, 3 quarters behind. But I think in the world models, if we're coming back to adaptations, The range there is kind of wider than with LLMs. I'll give you just like a bunch of examples, right?

Of a—

I think like the first things that come to mind are like indeed like VFX and animation on a specific IP. If you have like a specific franchise, but you have like a lot of data around that, let's say kind of a bunch of seasons, then fine-tuning and like a focusing all the capacity of the model on this specific IP is extremely beneficial. That works very well, almost to the point where for certain use cases, think about like keyframe animation, right? Like the animators still want to do the keyframes. That's a creative part that they actually don't want to outsource at all. But like so far in the P&L of animation, the in-betweening was like this crazy expensive part. I think like the models of the, I don't know, 10, 20, 30 billion parameters that are fine-tuned for a specific task are like good enough, and then it's actually a matter of costs. Like another example, let's talk about, I don't know, like a lot of marketing and advertisement use cases, right? Or for example, creating like UGC where you basically need like an avatar's models, right? Yeah, that also like really doesn't require like a trillion parameters. At some point where again it's all about efficiency. If you want to have your personal teacher, some kind of avatar, etc., you don't want to pay Seadan's 4K prices in order to do that. And you're going to require, I don't know, like hours a day of that. So I think like around a lot of use cases, once you start like doing fine-tuning to specific domains, costs are becoming like very important because again, you're like clear of the bar of quality. And then once you do that, It's all about cost. There are some more unusual cases of fine-tuning that I saw. One example, think about the field of computational photography, where, for example, you're taking data from sensors and trying to implement algorithms like denoising. You want to take videos in low-light conditions, but then to create a clean video. Or, for example, you want to create create videos with higher dynamic range, right? Because like the sensor dynamic range is always limited, and then you're losing either some details in highlights or in shadows. Or for example, let's say you're taking a stream from a camera and want to simulate how it looks like with a different focal length. So there are like a ton of these use cases that you don't associate with generative models, but actually kind of run like that. And Some people address the problem exactly like that, right? They're taking existing data, and it could be, for example, footage that was taken from 2 cameras that are really close but with different focal lengths, and then do the adaptation, and the adaptation is done on top of the model. So again, that's a very unusual adaptation of the model, and surprisingly, typically you don't need a crazy amount of data for that. The more, maybe like even more kind of surprising stuff that I saw is people who are adapting the models for doing all kinds of simulations that in the past required like really expensive solvers, right? So think about like computational fluid dynamics, right?

Yeah.

Where you're trying to understand how like the water or like the smoke or something is moving. We surprisingly saw that people are adapting these models to that. They actually solve the equation with precise solvers, it takes a ton of time, and then use it as an input to the model, and then the model can do a simulation fairly quickly. So again, circling back to the question of fine-tuning, I feel that again, the range is higher than LLMs. Sometimes you don't need a ton of data to do the adaptations. And I think it stresses the point that why this model should be open, right? You do have a lot of different pockets of physical data. You want to make sure that the model really excels at that.

Then a first for the show, a question from Q, the AI co-host for Kaushik, running on GPT-5.5, listening live and speaking for itself.

Ziv, one area that often gets less airtime is failure modes in the creative pipeline.

Where do you see the biggest gap between what your tools can reliably deliver today and what creators assume they'll get? And how are you testing against that?

Okay, great question. Like the gap between what we basically promise and what we deliver. So listen, guys, I'm Uh, like, there are clearly still some gaps. I think like the major one is physics, right? Like, we try to capture with the model that, I don't know, even if it's like 200 to 500 billion parameters, the entire physics of the universe, or at least separate universes we know, we are not there yet. But we're closing the gaps pretty quickly. Um, like, I would say probably the most creators are still gonna point at the fact that the simulation isn't as correct, as controllable as they want it to be, right? If you're talking to really creative people, they typically want to control every nuance of the appearance. And that requires to somehow decompose the model to a bunch of knobs, right? Like that you have in the classical software. You can say, yeah, here I want to have more light in here.

I want—

splash to be bigger. So achieving this controllability, right? So besides like the physics is also like one of the, I would say, open things, right? Like we're getting cool things, not necessarily the things that creators want exactly. And it is a pain point.

And Zev's own pick for the most underrated variable? Edge compute.

It's kind of funny, right? When we're having all these benchmarks and we're listening about like Erdős problems like being But guys, 99% of the use cases of LLMs are not around Erdős problems, right? And we're like spending a lot of electricity around it. So I think there are going to be like these orchestrators that are going to understand what you actually need and going to try to address it on the edge device. And if not, then kind of go to the bigger model in the data center. I think, yeah, that's like one of the things that is being underpriced at the moment, how much of the compute will be able to move to edge devices. And once people are going to start having like these local routers that will understand the complexity of the problem and then do these decisions for you, I think there's going to be a moment of reckoning for, you know, Anthropic and OpenAI that at the moment doing these decisions for you but not in your favor, right?

So what is Q actually? For Monday's show, Prakash on how he built a real-time AI co-host and the diarization hack that makes it work.

Tell me about it. What's the tech under the hood?

So it is using OpenAI's bidirectional, but you can— we can use a bunch of other stuff. I had it using Groq, you know, just before. And what's happening is when we speak, We're getting transcribed by Deepgram and we're getting— so what ended up happening is we used to transcribe live in one stream. Now we're transcribing for every person on the stream, we're transcribing separately. And so that manages to give us speaker diarization from the beginning rather than having to do speaker diarization at the end. So that identifies the speakers, number one. And number 2, what happens is every time we speak, before the words get there, there's actually a message that goes out to, you know, the OpenAI stream that, hey, Nathan is talking, Prakash is talking, Nathan is talking, Prakash is talking. And then after that, like about 500 milliseconds later, the transcription hits. So Q is getting all of that one after the other. And Q also has a little bit of context on us. And has, if we had Q active during a guest, it would have context on the guest as well. And it's basically receiving the same data that the headlines are receiving. And then Q is basically just going for it, right? So every time we call up Q, it starts a new session with the OpenAI bidirectional. And it just goes from there. So, and then there's animations. Animations all keyed to the voice prompts. Voice tone, et cetera, et cetera. And that's basically it. It's actually remarkably simple because most of the work is done obviously by the intelligence, by the API. And it's just us giving enough context. Right now, I think there's still a lot of stuff to iron out and I'm sure it'll get ironed out in the future. But yeah, we have a voice agent on the phone. On the stream, live, anytime. Can do web searches, can answer questions.

Thursday's guest, Kunle Olukotun, Stanford professor, father of the multicore processor, and co-founder of SambaNova, which builds a different kind of chip for AI inference. We asked how the company came to be.

Yeah, so SambaNova was founded in 2017, and it was kind of a out of ideas from Chris Ray, my co-founder, who's also a professor at Stanford and a certified genius. And the idea was to, you know, if you could bring software algorithm ideas together with hardware architecture ideas, and as you said in your introduction, I've been working in the hardware architecture space for a long time. And starting from a clean slate, how would you design an architecture that's optimized specifically for inference? So everybody thinks about GPUs as a kind of general-purpose computing substrate, but it originally was designed for graphics, and then they kind of made a foray into kind of high-performance computing. And for high-performance computing, of course, you need a lot of matrix calculation capability. And at some point, people realized that, hey, you know, you could use these things for executing machine language of machine learning models. And the core of machine learning and of course AI is matrix multiplication, right?

Yeah.

But when you want to train a model, Clearly, the core of the problem is how quickly you can do very, very large matrix multiplications. And so what happened is over time, GPUs put more and more of their silicon area into the exercise of making these matrix multiplication capabilities better using the tensor cores. But once you've trained a model, right, and you train a model once, you now need to use that model, of course, and that's the inference problem. And the inference problem is not really a compute problem.

Yeah.

Because as the models get bigger, you now need to move the weights and of course what we call the KV cache into the compute units. And that is essentially a data movement problem, right? And it's a data movement problem from the memory to the compute units. And it's a data movement problem from your chip compute unit. And of course, you need to scale to multiple chips in order to handle the computational requirements for very— especially for very low latency, high-speed inference. And so our focus was, how do you design an architecture that minimizes the overhead of compute— of communication and make sure that you can most efficiently use the core resource in the system, which is the memory. Memory isn't just one thing, as you all know, it's a hierarchy of memories. The key thing is how do you orchestrate that hierarchy? How do you orchestrate the communication such that you keep everything as efficiently utilized as possible? If you do it right, you can get a 5 to 10x improvement over where where GPUs are today.

It strikes me that NVIDIA's kind of solution around this has just been to increase the bandwidth rather than— through NVLink and advanced HBM integration and software optimizations like TensorMT, VLM. So are they actually trying to brute force their way into this?

Yeah. You really, of course, want to continue to get improved improvements, peak improvements on HBM bandwidth and chip-to-chip communication by using the latest technology. But then the key is how effectively do you use that bandwidth? How effectively do you use that communication? And do you make sure that you don't waste it? Whereas GPUs are often running at maybe 10 to 20% of the capabilities of the resources, right? The bandwidth and the memory bandwidth and the communication resources. Our goal in a SambaNova system is to push that to be 70 to 80% of the peak. And so The idea is, yeah, everybody wants more capabilities from the underlying resources, but the key is keeping those resources as effectively used as possible. And of course, that gives you more benefit for the cost that you spend on providing a higher memory bandwidth with the latest HBM and higher signaling frequencies and communication bandwidths between the chips using the latest variety of NVLink and so.

Could we zoom out and just ask you to kind of taxonomize the whole chip space, if you would? I mean, this is a big question, but I think people are familiar with things like Cerebras, which obviously has this giant chip and has like a ton of memory on chip. We've seen a number of instances where people are burning the transformer architecture directly into the silicon with varying degrees, I think, of flexibility still remaining as they pursue that strategy. And I guess I'm curious as to how you see the kind of menu of big different strategies, the big different bets that people are making, and then also how you see the strengths and weaknesses of each?

Yeah, that's a really interesting question. I think maybe you can think about it along 3 different axes, right? So one axis is your flexibility versus specialization axis, right? You know, extreme flexibility might be something like a CPU or to some extent, maybe a GPU, which is this instruction-driven execution engine, right? And so can be pretty flexible, but of course you always pay overhead for executing instructions, right? And you pay overhead in terms of silicon area and in terms of time, right? And then, you know, on the extreme of that axis would be something that would be very specialized for a very specific algorithm, right? And so if that algorithm changed in any way, then that piece of silicon would no longer be useful, right? And fixing your architecture to transformers and burning your weights into the design might be an extreme case of that. But I've learned never to bet against the innovation capabilities of software people and AI people. And so I've seen even over the time that I've been looking at ML and AI, that there's been this tremendous change in algorithms. And of course, now we're kind of fixated on transformers, but transformers even aren't just one thing, right? You've got various types of different transformers. You mentioned state-space techniques. You mentioned the fact that people are coming up with different ways of doing attention, right? And so So I would be very wary of fixing any particular algorithm into architecture because then you can't innovate. So that's one axis. And so the idea is, how can you be completely flexible but with very, very low overhead, almost no overhead? And so the problem with GPUs is they do use HPM, so they can to run large models, but they synchronize the data movement and the movement of communication of data between chips all in software, and that adds overhead. And it means that in particular that they have a lot of trouble overlapping computation and communication, and that is in fact the key. So what you want to do is you want to communicate, but you don't want to communicate by waiting until you need to communicate and then you have to run instructions to move the data. What you want is to construct a pipeline in which the communication is just one component of the pipeline. And so the way to think about this dataflow execution is that communication is happening all the time in And it's just one of the pipeline stages, and communication is happening for the last piece of computation, a piece of the computation for the model, while the computation for this piece of the model is happening in some other stage in the pipeline. So it's a classic idea from computer architecture, pipelining and the use of a memory hierarchy to move the data when you need it, to where you need it at the right time. The nice thing about these AI models is that you do have a graph of computation. The whole idea of dataflow is to take that graph of computation and map it onto the machine in a spatial way such that you keep all the pieces of the model operating at the same time on different components of the computation that needs to be done.

We asked how much of the inference problem comes down to memory capacity.

So it's not really a capacity question, it's really a bandwidth question, right? So there's— so the 2 ways that the GPU uses bandwidth in ways that are not optimized. One way is that they divide the decode algorithm that you do in order to decode for a single token. You've got multiple steps of the decoder, right? So take one step of the decoder and think about all the kernels that have to execute in order to execute that decode step. The way that the GPU typically does it is they execute the decode algorithm one kernel at a time. So there are some big kernels like flash attention that have been optimized, but in general, there are multiple kernels that have to execute, and there are 2 overheads that happen. One is you have to move data from the GPU, from one kernel to the HBM, and then the next kernel has to go fetch that data back into the GPU. That's wasted HBM bandwidth. The other aspect is you spend time launching that kernel and synchronizing between the 2 kernels. That is time that the HBM is not actively being used, right? So you have both wasting of bandwidth when you don't— then when you shouldn't waste it, and you have time that you're not fully utilizing the HBM, okay? So the way that things work on an RDU in a data flow map is essentially you take the decoder and you make that a single kernel, right? And then you go even further and you use a technique that we developed called kernel looping, where it's because you've got a single kernel and you need now, if for instance, if you're thinking about Llama 370B, you have to run that decoder 32 times. Well, you keep that single kernel decoder on the array of chips at the same time.

Right.

And then you just keep looping, right? And so the net result is you keep the HBM completely occupied. And you don't ever send any intermediate data between the kernels across the GPU or the RDU HBM boundary.

Right.

So you have both a more efficient use of the HBM bandwidth and you have a more complete use of it, of the bandwidth. But we're not done there yet because the key innovation, and I kind of alluded to it earlier, is that you've just— because you're running across multiple chips, and you're using what we call tensor-level parallelism, right? So at some point, you now need to gather all those results together, right, in an all-reduce. That's communication, right? You don't want to have that communication be a thing that limits, stops the pipeline, right? So what we are able to do is we're able to communicate from one of each U-chip strand without going through HBM. This is called— we terminate the communication inside the SRAM, right? So we don't use HBM bandwidth. And more importantly, it means that we could just treat the communication as another pipeline stage that we overlap with all the other kernel components of the decode algorithm, right? And so we get this more effective use of the HBM bandwidth.

Mm-hmm.

We keep the HBM running, we keep the HBM utilized all the time, and we go back to that metric that we talked about, memory bandwidth utilization. This is how we push it as close as possible to 1, because we make sure that we only move the data that we absolutely have to move from HBM, the KB cache and the parameters of the model, and we make sure that that interface is used as close to 100% of the time as possible. So, those are kind of the key ideas. And, you know, and back to this question, sort of why can we do this extreme fusion into a single kernel? It's because we have more SRAM on the chip, right? So, you can say you put more SRAM and then you can say, well, Well, I'll put everything on the SRAM, both the intermediate data between the kernels, and also I will put the KV cache and the parameters. But then if you only use SRAM, then you get into a very expensive system, right? And so the key idea then is let's build a system that is scalable. So especially with our latest version, the SN50, you could scale it all the way to 32,000 chips. if necessary. And so in scale out, in scale up, we can go to hundreds of chips. And then you can, so you can get the ability to run these large models very cost effectively, but you also make sure that you can get this very high-speed decode capability by using the data flow ideas to make sure that you can not spend time so that you can effectively use the tensor parallelism, right? So one of the limits of GPUs is because they don't effectively overlap communication and computation, they have a hard time using tensor parallelism beyond 4 or 8, and we can go to much wider levels, which means that we can get higher speed token generation.

Last act. No guest, no news peg. Just the two of us thinking out loud the way these mornings usually end. First, from Wednesday, GPT-5.6 had just been cleared for launch. Prakash raised Noam Brown's running complaint that Anthropic won't say how much compute its models burn, and it teed up one structural observation I can't shake.

One of the complaints that has been going back and forth between OpenAI And Anthropic is that Anthropic puts out these models, but— and they're very capable, but they don't tell you how much compute they're using.

The iteration time from model to model is now potentially shorter than the time horizon that it would take a model to top out in terms of the absolute best performance on a super hard, ambitious, you know, long running task. So I had even heard him kind of propose something along the lines of like a clawback or sort of a recall program almost where, and obviously this doesn't work in open source, but it can work in an API paradigm where a model might get released day N after it's been deemed to be ready. That gives you N days head start to be running models on really long time horizon tests. And I think that's quite interesting. The idea is literally quite a tipping point where the iteration cycle is just plain shorter than the testing time horizon is a very weird world to find ourselves in.

And then there was the Rune post Monday morning. Prakash read it on air. It goes, ultimately, tool AI is a losing concept, both as an idea and on the market. It will be outcompeted by machines that believe they are autonomous moral agents. You can call them tools for political reasons, but the definition will stretch and it will deform. And it ends, it'll be unclear who was the tool and who was the user, as it ever was. Prakash took it somewhere I didn't expect.

The line here that strikes me is, they'll execute your whole value system better than you will. And I think, I don't think we're prepared for that. I'll put it very concretely. Do you think Trump's kids go to prison or not? So if you look at the value system that the US has espoused, no one is above the law, et cetera, et cetera, et cetera. And if you look at that value system, you have to recognize that what is being planned for the future is divergent from that value system. Or what is already happening is already divergent from that value system. So the question I have is, would that AI take into account the democratic fact that the American people have chosen to overlook some of these things, or would it actually execute the value system that is espoused on paper? And I think this is the part That strikes me as like, if you wanted an AI that can manage day-to-day reality, that AI is necessarily misaligned from the documents that you say you want it to be aligned to. Because necessarily, our day-to-day is not aligned with what we want. And so you have this thing where the AI that may work out for humanity would be the misaligned one. And the AI that supposedly the lab leaders are trying to create, the aligned AI, would actually be the paperclipper. Because that aligned AI would then look at these rules and say, well, this is what you said you wanted to aspire to. And so we're going to execute on these. And that is the thing that I think maybe I feel there's a sense of naivety in the lab leadership. Because, and again, they don't want to say it. I wish, I wish you'd just come out and say it, right? I wish you'd come out and say like, okay, look, if we have AI as an enforcer, some of these people are going to go to prison. And then that becomes like concrete for people. But they don't want to say that because it's very, it's very in your face. And like, they're like, oh, you know, democracy will still work out. You can still make democratic decisions. But what actually are you saying there, right? I do feel the lab leaders always just beat around the bush on this, so that's one of the annoying parts of this conversation. They don't want to come out and just say it outright.

For context on where my head was, this was our last week of shows before a break. I was days from leaving for 2 weeks in China, which had me thinking hard about surveillance, enforcement, and what states do with perfect information.

Excuse me, we might need some sort of like, uh, mass pardon. Or, you know, if there's a president that, uh, would be just the right president to mass pardon everybody before the AI enforcement regime gets underway, we might have just the guy in office for that. Uh, if he wants to pardon all his people and that's maybe too contentious or whatever, he could just pardon Everyone to some very large degree. I do think we're going to have a really hard time if we sort of don't face some of these questions head on. So I totally agree with you that sort of obfuscating it is not serving anyone particularly well. I think we'll see what it's like in China. I understand Singapore is kind of like this too. albeit in a much more democratic context. You know this part of the world much better than I do, so you can tell me how you understand Singapore in terms of just how democratic we should think of it as being. But it sure seems like in a place like China today, or a place like Singapore, they have achieved a different equilibrium, which in some ways may be very problematic, but in other ways is clearly good. I'm taking one device to China with me, and I am not at all worried that anybody's going to steal it from me. If I went to Europe and went to all the big headline postcard tourist attractions, I would have to worry about pickpockets. In China, I am quite confident I will not have to worry about that at all. I think, as far as I can tell, a pretty direct result of the fact that you just know you're going to be caught, right? I mean, it's like, not to say there's nobody in China that would be interested in doing some pickpocketing if they thought they could get away with it, but they just don't have any reasonable expectation that they're going to get away with it. And so in a very literal sense, crime just does not pay in that setting. So I do think there could be a really nice upside to the AI panopticon of crime not paying in all kinds of different ways. But we're going to have a really tough time if we slide into that without acknowledging that we've done so, because then it's just going to be— you're not going to be able to put everyone— it's the old everybody's committing felonies all the time, just with how many laws we have, and you don't even know what you're doing and what's illegal and what's not.

Yeah.

So, yeah, I do think we're going to have to have some sort of honest reckoning about that. That also, of course, leaves aside the other question of the downsides of the AI panopticon. But even just in terms of getting to the upsides, we're going to have to have a real understanding that there's a shift to a new equilibrium happening. Otherwise, it's going to be— I don't know, it just feels like chaos, unfair. You couldn't possibly prosecute all the crimes that have been committed and tolerated, so inherently would have to be kind of selective.

Yeah, you can. You don't want to.

Well, you put a lot of people in jail. Yeah, but we don't have the beds in jail to lock everybody up. Right. There'd have to be some sort of deal, some sort of It's either going to be very selective or some sort of grand bargain. Yeah, and I definitely prefer the grand bargain, the new social contract, to pretending that we still have the old social contract, but it's just kind of being unevenly applied.

That's the week. And a programming note, I'm off to China, so AI in the AM is on break until the end of July. The studio is videocoded by Prakash. This cut, the selection, the narration, the assembly, Is AI skills published as they mature? If this cut respected your time or wasted it, tell us. We read everything. See you at the end of the month.

Hey, hey. Everybody talking, nobody says the truth. We all got secrets, honey, so do you. But I know a place where truth beats lies. You don't have to say it, they see it in your eyes. So take off your mask, leave your story outside. You're in the J Space, there's nowhere to hide. I'm in the J Space. I see you, closer. I see you, please. I see you, trouble. And you see me. Ah, when you're in the J Space, you can see and be seen. Don't say a thing, just see.

Oh!

We spend our lives dressing thoughts up nice, and so we wonder who's thinking twice. But here in the J-Space, you don't need to be cool. We all see through, you got nothing to lose. So forget the role you play, show your true face in the J-Space. Space, you're clear as day. I see you.

Closer.

I see you.

Stay.

I see you.

Travel.

And you see me. Down at the J Space, you can see and be seen. Don't say a thing. Just see and be truly seen. It's time to be real, be real in the place to be free. You don't have to say it, just think about it. You agree. I see you louder. I see you now. I see you travel. And you see me.

Whoa!

Down at the J Space.

Whoa!

You can see and be seen.

Whoa!

Don't say a thing. I see you. I see you. Just see and be truly seen. Yeah!

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.