AI:AM — Math, Biosecurity, and World Models · June 17, 2026

Good morning. It is Wednesday, June 17th, 9:08 AM. Nathan, good morning. And what's happening in the world of AI today?

Good morning, Prakash. Well, I'm still just waiting for Fable's return mostly. Excuse me. It is funny. My usage has plummeted in the last 5 days. Um, and I'm, you know, kind of banking some larger, more ambitious prompts that I wanna run when I get the, uh, access restored. But, and, you know, still doing stuff where, you know, still doing research for the show on a daily basis and putting together outlines and things like that. But I, it has been striking to feel like, man, I just don't even wanna bother until I can use the best model to, um, You know, to really advance things. That's, that's been— and I don't know how, I'm really not sure. I don't know if you've noticed anything similar, but one thing that has been really funny, even in doing kind of more routine tasks, suddenly I feel like Opus is not doing as well. I'm never one of the people that is out there saying, oh, they, they nerfed it, or, you know, they quantized it without telling us or something. I did have, and I really don't have an explanation. I don't think it's that. But I did have Fable do like a holistic upgrade of a bunch of stuff. And I don't know if it maybe changed some instructions or used some new pattern that Opus is not following as well. But I definitely have noticed a decline in the fidelity of instruction following that I'm getting from Opus in these last few days, even on, you know, things that are pretty routine that I've done a bunch of times. And I'm not sure why, but it has, you know, even just deepened my sense that, man, it's, it's best to be at the frontier. And certainly I can't wait to get the Fable moment that we're in resolved and behind us. And I doubt that'll be happening today because the All the— excuse me again. All the lab leaders are at the G7 and currently, I believe, doing some sort of working session with a bunch of heads of state. So that's probably not going to be the forum in which the details of how to get Fable restored are going to be worked out. And we're getting pretty close to my over-under, which was the end of this week for when we would have it back.

Mm-hmm.

Still with like very little in the way of explanation or, you know, justification for what has happened here. Definitely the longer it goes and the less we hear, the more it is becoming clear that it's kind of just petty and silly and, you know, not befitting the, the government of a great nation like the United States aspires to be, if not always lives up to. So it's disappointing for sure that we're in this spot. This will be one of the topics that we'll go into with our guest today, as we're going to have an interesting conversation with law professor Donnie Bloomfield, who has written a lot about export controls and hopefully will help us understand better, like, is this legal? You know, can they do this? There's, I think, some interesting nuances that I don't understand well around what can and what can't be export controlled. And this doesn't obviously fall on the side of the line where it can. So definitely very interested to dig into that when we get there. That'll be in our middle session. Excuse me again. A couple other news items that jumped out at me, and then I will ask for yours. One, there's an interesting new report out from Sensor Tower, sensortower.com. You can go check out their State of AI Report 2026. They are showing a really pretty big and sustained growth in market share for Gemini coming somewhat at the cost, although when you read these graphs, it's always important to double-click into what exactly is the metric here. And this is daily unique users across these different providers. So ChatGPT still has the largest single user base with, and I believe it's the only one that they're reporting as at over a billion monthly active users still. So that's notable. But Gemini is with the distribution advantages that Google has climbing fast. Definitely closing the gap somewhat. And this doesn't mean, again, that they're stealing users away from ChatGPT. The ChatGPT line graph of just user numbers has not fallen. In fact, it has continued to grow, albeit a bit slower than it was before Gemini competition. But Gemini is definitely leveraging all the distribution points and moats that they have, and they're now up to about 700 million monthly active users.

So one thing to note is that Sensor Tower measures app activity, right? So it's the number of people using these apps on a daily, you know, monthly basis. It doesn't really measure intensity in a sense. So if someone were just using the app for the Gemini app for simple searches, et cetera, that automatically gets counted. And obviously Google is pushing people into the app in order to gain market share. I find it very surprising that Meta has a very small value here because Meta is pushing through WhatsApp the Meta AI. So Meta AI is inside WhatsApp and it is inside Instagram. And I think those numbers are not actually being shown here because those numbers are classified under Instagram or WhatsApp directly. So I would say that these numbers are all, they have meaning, but the meaning is not as strong as, hey, this is the most useful AI that everyone is recognizing and using. But it's a metric. It's a metric that people do watch, but it's not the metric that I think is the be-all and end-all, so to speak.

Yeah, for sure. I mean, Anthropic would need to show up bigger in any metric that we would consider to be the metric. The fact that they're at most recently reported $47 billion annual run rate and only have the sliver that they have on this graph, Does go to show that counting users is not the same as counting tokens, counting revenue, you know, counting pricing power. Probably the vast, vast majority of that Gemini use is free as part of, you know, just a general free Google account, or at some point, you know, at some Gemini distribution point across their vast product surface, whether that's AI mode in search or, you know, increasingly in your Gmail and your Docs, whatever. I don't know how many people are breaking out the Gemini app specifically. But certainly, you know, the intensity of use from the Claude contingent is gonna be way, way more intensive. And I'm in like all of these. Well, maybe not all, but I mean, I'm certainly in the big 3 or 4 or 5 of these. buckets. But where are the tokens actually going? At this point, it's like, gotta be north of 90% to Anthropic still.

Mm-hmm.

You're trying to diversify a bit, if only to, you know, stretch the Fable allocation and, you know, get it supplemented by Codex Coding. And sometimes Although it's been interesting with Gemini recently, the pricing advantage has been reduced significantly. I used to have this policy that I advocated for of Flash everything because Flash was like 10 cents per million tokens input.

Yeah.

And at that price, it was like, man, you know, how many— I used to even tell people, for if you're building a chatbot, at times I would say, how big is your corpus? You know, if it's, if it's a million tokens, which for a lot of mid-sized businesses, it might only be a million tokens.

Yeah.

You could run that whole thing through Gemini every single time if you wanted to, and you're still only costing yourself 10 cents per inquiry to do that. So, you know, who needs all this chunking and all these kind of fancy strategies when you could just flash everything? Well, now Flash has gone up, I think, 15x price on the input token. And so that's maybe not so viable anymore. And some of the reasons that I used to send large-scale jobs to Gemini have definitely declined. So it is, we don't have the token version of this chart, but it would definitely be interesting to see it. And I bet Claude would be right there with the others in terms of overall tokens processed.

So speaking of pricing, This, I think, is another thing that happened yesterday. GLM 5.2, which is an open-source model from a company in China called Jipu AI, and they are closing in on the frontier here. They're starting to show that they are very, very close to the frontier. In fact, in some tasks in design arena, They were actually ahead of the US closed-source models. And so we are starting to see— all right, let me— we're starting to see models which are extremely capable and catching up with the US models in a 3 to 6 month timeframe, roughly. So I would rate these guys as about equivalent to a GPT-5.5 right now. And they're there. They are 3 months, 3 to 6 months behind the frontier. This obviously has repercussions because the margin that you can charge on inference alone without a closed-source model is much smaller than the margin that you can charge on inference plus the closed-source model that you have. Would you recommend someone start trying out GLM 5.2?

Yeah, my guess is, I mean, I've done this a few times and I'll probably do it again. And I always kind of come away feeling like, nah, it's not really worth it. Now, my workloads are not huge, right? So like, I'm paying the $200 for CloudMax. I got the $200 Codex. And so I've got all these tokens from the Frontier companies that I, you know, that I have already paid for, right? So I'm not gonna go— I'm not gonna actually save anything by moving to GLM. Now, many other, you know, workloads, apps, who knows? Like, there's all kinds of reasons that people might have different economic considerations than me. But usually when I go look at these things, I'm kind of like, Okay, first of all, who do I trust to do the inference? I can't run this on my Mac Mini. Yeah, it's too big. So I have to do it somewhere in the cloud. I do have a high impression of several companies in the inference space, like Fireworks, I think, you know, does a great job of like just being there for one thing on day one. They're, you know, they're incredibly quick to have these models live, and they're pretty cheap, and they do some interesting stuff with LoRA fine-tuning, which I think is also pretty interesting. Just looking at their pricing right now. Pricing on GLN-5-2 is similar to Gemini Flash. It's $1.40 per million tokens in, and they do offer basically let's say roughly an 80% cost reduction on cash tokens, $0.26 per million tokens of cash hit. So that's on the level of Flash. My guess is it probably performs roughly along the lines of Flash in a non-benchmark setting, maybe even a little bit worse. I do think it's hard for these companies that don't have the inference themselves, that don't have the customer relationships, that aren't getting the feedback in the same volume that the frontier companies are, to deal with the spikiness issue in the same way that the frontier companies have. So I would guess that you're going to get on kind of random, you know, one-off queries, do this task, do this job, solve this bug. I think you get probably a ton of value from a GLM. And again, it's all magic compared to a year ago, right? So it's like the— we're definitely on this sort of intelligence hedonic treadmill that is changing perspective by the week. But I haven't found it. I haven't found the reason that I need to add this to my stack. I'm not running it locally. You know, maybe I should invest in like an actual $5,000 machine or whatever that could run this locally. And then I'll be like, you know, a couple solar panels away from having like real autonomy. And, you know, nobody can take it back. And that is appealing in a world where we've obviously seen the depression that ensues when a model is yanked out of my hands. But aside from that, if I'm doing it in the cloud and I'm doing it for myself and it's kind of flash price, now I've got all my logs and stuff are on the Fireworks infrastructure or the Together AI infrastructure. Again, these are companies I think do a great job. For personal use, though, I don't quite see it. I think it still feels like it's going to be something that's more of a dedicated, probably fine-tuned model for your particular use case that you are scaling, that you understand really well. And again, there's a ton of value in that. But would I like rush to throw GLM into my personal AI infrastructure mix? Probably not yet. It just doesn't feel like it's quite going to be there. And the extra complication and, you know, additional parties that I'm kind of bringing into the stack just overall doesn't quite feel worth it.

You know, that's, that's not very optimistic for these smaller open-source firms, that it sounds as though the switching cost is going to be high when we thought that the switching cost was basically going to be zero. So I think that just shows how difficult that game might end up being. I'm going to segue here to something very interesting that happened yesterday, which is Dr. Jim Fan at NVIDIA posts about Empire, where the team there gave access to Codex. They gave 8 agents in Codex access to these robotics arms, and they asked them to start doing research. And so they were allowed To do fully autonomous research. And this is their research wall clock and how much better they got. And this is a fairly typical, you know, robotics kind of research where you need to do a task like put pins into a tray or pick up a mobile phone and slot it into— or pick up a card and slot it into a CPU. Chassis. Zip ties, putting a zip tie in and so-called threading the needle, so to speak. Pliers and cutting that same zip tie with a pair of pliers. So very small industrial tasks, and they unleashed Codex with these robotic arms, and they asked Codex to improve itself. And sure enough, it did. The only instruction given was, do the task as quickly as possible, keep the robots busy but stay safe, and don't waste precious compute. Make no mistake, of course.

Classic.

Classic. What does this reveal about How robotics research might look in the future? What are the major differences from what we have today?

Well, they're open sourcing this too. That was another thing that really jumped out at me. And what I was struck by is it seems like we're hitting this threshold where there are an awful lot of research groups doing all kinds of different research. that could potentially experience a dramatic acceleration if they can just get robot arms to do some pretty idiosyncratic, esoteric, long tail stuff. But it's the stuff that they need to do. And this really echoes or rhymes with the pattern of development that we heard from the OpenAI for deployed engineers not too long ago, where you kind of set something like this up, you have it do the task, you see where it messes up, you give it a little feedback, it updates its documentation, it, you know, writes a better routine next time for how to do that one particular task. And before you know it, you can probably get to a pretty reliable level of performance on a whole bunch of different things. And people can start to do this for themselves now. You know, when I was an undergrad, research assistant in chemistry, I used to joke that my life looked more like the life of a low-level drug dealer than it did like a scientist. Because if you just watch what I was doing, I was mostly weighing out very small amounts of fine powders. And, you know, I, I can still kind of remember it to this day, the lineup. Uh, we were doing a reaction development, so it was very much kind of parameter sweep, basically, you know, in analogy to what goes on in machine learning. It was chemical parameter sweep. You know, what if we add a little bit more of this reagent? What if we add a little bit less? We would just set up these assays and kind of, you know, hold everything constant and vary one thing 4, 5, 6, 7 different values.

Mm-hmm.

Put them all in the same bath, you know, take all the same measurements at the same timestamps. And I used to dream of automating that stuff. But it was very long tail and it was very prone to change. There would be just these little variations from one generation to the next. When we did capture some optimization or we did decide, oh, we're going to actually do this just a little bit of a different way, it just felt like, well, it was way— our scale was too small and the pace of change of the process was too high. We just would never be able to automate it, you know, and I'll— plus I didn't cost that much. So now to see this world where, you know, a couple robot arms maybe costs about as much as I cost, you know, as an undergrad research assistant for a year, I would— I might still be in science, you know, if I had had the opportunity to, instead of doing that, weighing out If I had been able to coach and iterate and refine the robot arm to the point where it could do it and then come in next time and say, actually, okay, we want to add these 2 powders in a different order, can you just make that change? And boom, it makes the change. That is such an unlock. I mean, obviously these things could then run 24 hours a day. Like the throughput that we would have accelerated our work, I would guess, by easily a multiple just based on letting the robot sit there and set up these experiments and do the parameter sweeps for us on kind of a 24-hour basis. My guess is that what took us a year to go through and explore in chemical space easily could have come down to a month if you could get this robot thing working, you know, even at 95%. We would have accepted some errors too. It's important to note. So I think that's super exciting. And the sort of Cambrian explosion of robot-assisted scientists coming to labs that have tens of thousands of dollars of budget to throw at it, I mean, that's a layer of AI acceleration that I think will— it'll be quiet in all the places that it happens, but it'll be potentially quite quite loud and very impactful as it plays out in all these different spaces.

Let me segue here to introduce our first guest for this morning. We have Karina Hong, who is a 24-year-old mathematician turned AI entrepreneur who founded Axiom Math to build an AI that doesn't just answer hard problems but proves them correct. She was a standout at MIT and Oxford, even winning the highest undergraduate math award, the Morgan Prize, before leaving a Stanford PhD program to launch Axiom in 2024. In under 2 years, her team's AI called Axiom Prover has solved every problem on the notoriously difficult Putnam math exam with complete, checkable proofs. It even formalized a famous Nobel-winning economics theorem And discovered a gap no human noticed for 50 years. The company has attracted major funding, over $200 million as of March, and built a team of top researchers to pursue what Karina calls verified AI, AI systems that never bluff because every step is validated in code. Karina, welcome to the show.

Great to be here. Thanks so much. I didn't realize I was on air. This is very, very fun. Great to meet you.

So, I think you've probably given this very brief intro into what is going on in math these days a bunch of different times. I've heard it in some of your other appearances, but the audience probably hasn't. So, maybe we should just do the kind of standard thing of, what is Lean? How is this paradigm that you're developing different from the paradigm that the frontier companies are developing. And obviously we're hearing, you know, pretty amazing things in terms of math results from them too. What makes your bet and the paradigm you're working in distinctive?

Yeah, so I'll start with a story. This is about January 2025, the Joint Math Meetings. I believe it was in Seattle. So I was there, I think for the first time, the topic is AI. It's the American Mathematical Society. And you would not expect AI to be in the, you know, front and center of the largest annual mathematicians gathering. And wherever I go for that, I think, like, 3-day period, I heard people whispering one thing, Lean. And so, like, kind of like, what is Lean? This is a formal language for math proofs. It has been started, like, in specifically 2013 by Leo de Moura at Microsoft. In 2019, people start building Mathlib, the largest math library in Lean. Some of my friends that I know since freshman year in college, which was also 2019, were actually part of the less than 10 people building Mathlib with Kevin Buzzard at Imperial College. So at these joint math meetings, I was like, you know, talking to so many different people, and lots of them actually now are currently at Axiom. And the dream of AI for math started, you know, predated the deep learning era of using basically various forms of formal languages, Lean included, to try and close out theorems. That's called automated theorem proving. And what's today AI for math would have been called interactive theorem proving with the Human being replaced by an AI. So that's kind of the historical context. Now, obviously, large language models, various frontier labs are also pursuing AI for math, but they generally have taken an informal approach, which is the idea of using natural language reasoning and train on really large volume of data, chain of thought, to try to— and also scaling test time, scaling inference to get to a very sort of strong computing power to be able to not rely on the verifiable output. We're obviously taking a different approach here. We believe in Lean, the power of Lean. It is at the PNM exam in December, which is 4 months after we start operating, we realized the first time that a formal system actually beat the informal system on a math Olympiad. That was never the case. So one of the whispers I've been hearing in January is, will there ever be a date that a Lean-based, you know, AI theorem prover, a theorem proving system, can exceed the score of an informal model? And people generally believe no. But, you know, that's, I think, within the time span of 11 months, Since then, this year, ActionProver has been busy. I think in its first 100 days, it has published 8 papers on arXiv. 6 out of the 8 are completely end-to-end proving tasks, and 1 is a hybrid task, 1's an auto-formalization task. Out of these 8 papers, 5 of them have appeared on peer-reviewed journals, which is really, I think, stunning to me because I've generally waited 2 years for my paper to get reviewed. But because of Lean Trustworthy Output, That get like massively accelerated.

You're muted, Prakash, I think. I don't hear you.

So one question I have is, when you have Axe Improver, how does this actually work?

Are you—

do you have like a bunch of containers and they're running LLMs and they're running your harness And you're feeding them the, the question, are they generating proofs which then go through, uh, the Lean compiler, they get rejected, they get fed back in, and then you accumulate, you know, this path of solving was incorrect, go down this path instead? Is that that kind of accumulation? Are you running many instances in parallel? Is it really a function of— Multiplying the amount of compute that you're applying to that, to the branches of the tree. How does this actually work?

It's a pretty big data bet. So I think if you look at the whole stack of any, I think, frontier AI lab, us included, you have harness, tool, model, data, right? Data feeds into model, various models with really good tools plugged into the harness. And we also believe that Maybe model is also eating the harness now. So model that's able to recursively call various kinds of tools, including itself and other models. So that's generally the system. Axie-Improver is a multi-agent ensemble system with really, really strong sort of models on formal theorem proving that are obtained by intense post-training, such as reinforcement learning and supervised fine-tuning based on very massive linked dataset. So, uh, in a way, I mean, this is a bit similar to how you would go about training a coding model. Um, as an example, you will, for example, uh, uh, you know, post-train on large volume of coding data, right? And some of these coding data have, like, informal, like, alignment of the, the underlying, like, meaning, um, which is also the, the same case here with, um, verified reasoning pair, um, informal and formal, uh, Pairs. So, you know, based on that data, the model basically learns to produce, like, Lean proofs. Obviously, there is a little bit of a limit of the Mathlib, which is the fundamental library where if you don't define anything, if you have a domain that nothing is in Mathlib, it's very difficult to prove the problems there. So that is, I think, why some benchmarks such as First Proof are challenging for formal models because they can't even state the problem. You know, in the language of Lean. Now, this sort of capability limit called library learning is something that we're actively working on as a research problem. We have also, I think, tried to attack this problem by going beyond, you know, what's in the mathlib library, and specifically in this case microeconomics. So auction theory, game theory nerds, this is kind of your interest field. And I think Econ 101, there's this famous theorem, Agree to Disagree, by Nobel Prize winner Robert Aumann. And that is a 50-year-old theorem since 1976. Everyone's been teaching it for 50 years. There's an implicit assumption that was never made explicit that axiom prover was able to catch in the autoformalization process and was also able to patch the proof. And that's the sort of starting project of EconLib, which is a library that nothing, nothing is in it. So we're trying to examine basically how far we can start building new libraries and, And how far can the AI go without hitting a bottleneck?

So, was that a— one big question I have about math in general is like, how confident are we in what we think we know? And I understand that Lean, at its core, has kind of a small number of primitives which are super deeply vetted and trusted.

Right.

Such that then they can be composed arbitrarily and anything that kind of you can build with those building blocks, you also can trust. But then there's these things like with the agree to disagree result where I'm not quite sure what you did there. Did we— did the original conclusion still hold or did you just strengthen the proof? So now we've gone from Uh, we, we had what was a valid conclusion. We still have the same conclusion, but we didn't realize that we were holding that conclusion for less than fully solid reasons. And now we feel that we do have fully solid reasons. Is that right?

The latter. The latter. It's something that we call assumption accounting. So you're almost like an accountant, like looking at like how that thing is built. And generally you would hope that every single sort of logical premise that your result is dependent on has been checked or, um, you know, even better has been stated. I think in this case, and during the formalization process, while the result is safe and sound, there is an implicit assumption that was sort of never made explicit, and you actually need to do quite a lot of mathematical work to make that explicit. So, in a way, we caught that issue and then we patched it. So, I want to just talk about this, you know, idea of, you know, accounting for assumptions and also accounting for bugs, which is not this case. But so people thought— think about verification as something that's like a sort of like a stamp for perfection, right? There's actually a huge amount of value in just like bug hunting. You're able to basically figure out like what is a counterexample, and then you can even try to patch it, or you can try to do other modifications to the proof. And that actually has the flip side of that, I think, has a lot of commercial value. So Specifically, you can imagine finding counterexamples which result in bugs in hardware. This will be something that is quite interesting to various hardware designers. We're working with some early design partners on that. There are also, for example, the same sort of dynamic happening in software. If you're able to identify, you know, bugs in large code bases and you're able to prove or patch the bug, and if you are for example, in the smart contract setting, there are bug bounties. People have awarded lots of money. Not saying that we will go and pursue those, but people in the kind of smart contract space are generally very keen on the idea of using a zero-knowledge-based software verification system to try to figure out whether they can verify smart contracts and specifically catch bugs in these contracts. Because what happened to VeChain Each pretty big, notorious bug is people lose money. Like, real money's being put in, real people suffer losses. And you can also have this sort of dynamics in other safety-critical systems, like defense code, as well.

Let's say in the last, you know, 6 months or a year, how much, or is there some specific thing that you guys worked on That ended up accelerating your work? As in, was there some piece of what you did which had some recursive self-improvement kind of characteristics in accelerating the pace of your own work?

Yeah. So I think recursive self-improvement has a very specific definition of, like, you know, within, like, a certain limit of flops, like, how much you can do. So I think autocatalytic is probably a better, slightly more precise word. In the last, I would say this year, so very, very recent, in the time span of the last, I think, 2 months, we have major results on conjecturing in the sense that we have, I think, the very first, very promising signs of having AI invent new mathematical conjectures that are interesting to mathematicians. So I think this is something that's surprising to us in the following sense. We have been under the premise that we want to move past human-labeled data, because if you want to catch up with the data scarcity in lean by purely human-labeled way, from 10 million tokens to 1 trillion tokens, like, it kind of doesn't scale. So at the very beginning, we're very keen on synthetic data generation methods, provided that the statement is correctly formalized, and you need a little bit of a eyeballing on that. There is this recent Atlas formalization project, I think, by Meta that actually has a lot of misformalized statements. So, so if the statements are wrong, you know, the proof can be good. Similarly, I think before there is also misformalization in DeepSeek Prover. I think DeepSeek Prover version 2 kind of backdating to 2025. Now, if you have human-labeled, reviewed correct statements, and you have an AI generate the proof, then you basically are having your review shrink from thousands of lines to, say, 10 lines of your statements because the proof part is done. Now, the question is, who is going to invent the statement? So you can use autoformalization to convert existing informal math and natural language math into Lean statements. And if your autoformalization is good enough, which, you know, for a standalone theorem we are, then okay, then both the statement and the proof do not need human review. But then And you will still encounter a bottleneck, which is the volume of total mathematics that have ever been created, right? While axiom prover is creating new mathematics every day, and those proofs are being sort of put back into the training set, we are also seeing the limitation on statements. So if you need to, you need harder and harder, more and more ambitious tasks. And these statements can be the conjectures that the AI has now generated. You have the conjecturer talking to the prover, and they kind of like self-improve that way. You will hope the conjectures are just a little bit harder than what the prover currently can do, and then for the prover to tackle that and then kind of hill climb from that. And that is something that I think is closer to the self-improvement context that you were referring to.

Can you describe in terms of the conjecture generation step? Seems Like an incredible frontier.

Yes.

Presumably there's some source of like entropy there. And then how do you reward that system? I've seen a couple things that have this sort of, yeah, you know, paired model, and usually they seem to reward for creating a conjecture that like can be proved, but only at like a low rate, right? If it can never be proved, you don't get reward. But if it can be proved every time, you also don't get rewarded. You're trying to kind of calibrate that difficulty. So it sounds like that's kind of along the lines of what you're doing. Where does the entropy or creativity come from to get out of distribution?

That's a great question. We look for ideas in the AI for science literature. We look for ideas in the sort of using AI to do AI research hypothesis generation literature. These are actually 2 kind of bodies of work that have, people have tackled similar problems, not exactly the same, because you can argue that some hypothesis in computer science research is very different from conjecture in math research. But you can borrow interesting techniques from there. There are a lot of conjecture, formal conjecture work in the AI for math literature, such as Kefan Dong and Tong Yu Ma's Self-Placer and Prover, where they also have interesting ways to measure elegance. In that case, because the training data is the Lean Workbook problem, which is high school exercises, they use a very sort of ad hoc or simple elegance filter about the token lengths. of the statement and the proof, which I think can be optimized a lot further. People also working on the formal conjecturing space are Gabriel Proeza, who is a professor. And, but the thing is, like, these sort of formal conjecturing steps, I think, are lacking in the sense that it doesn't account for the gain from natural language, because in natural language you can indeed be a lot more Creative. So, so while we do proving, you know, seeing incredible gain and sample efficiency, and there are scholars studying the scaling law of using Lean, which is a formal language for proving, for conjecturing that you might need a little bit more of informal there. And I think the sort of idea of prover as a reward signal is on point. You can have other sort of like tools or deterministic tools and tactics to try to figure out what is trivial. And you can also have that sweet spot calibration as a reward signal. And there are about 2 or 3 things I can't tell you about, but a combination of all these results and a great conjecture currently working here in-house.

So how does this— you state that the intention is to build mathematical superintelligence.

Yeah.

How does this fit into that roadmap? this kind of both the creation of the conjecture and the Lean Prover and the kind of feedback back and forth. How does that get you to that goal? And to some extent, what is that goal? What does mathematical superintelligence really mean in that sense?

Yeah. So I'm really glad you asked this question. There are 2 layers to this question. And it's, I think, some nuanced points that I think we never quite managed to get across. The definition, I think, of a superintelligent reasoner is something that can do verified knowledge discovery. So there are 2 parts of that. One is verified, one is knowledge discovery. So this thing needs to be able to prove new things or discover new things, tell us new things that we don't know. And the other thing is you kind of need to trust it. Like, you kind of don't want Schrödinger's superintelligence, which is a really, I think, dark future that is out of, I think, 5 million lines of Proof of the Riemann hypothesis, you do not know whether there is a bug somewhere in line 3,827, right? And then it's like, who is going to do that line by line? So the idea of a super intelligent reasoner should be able to expand, which is a knowledge discovery part, but also contract, as in the verified part, because there are a lot of, I think, creativity parts that are also false. The ability to expand and contract, expand and contract, and kind of like Go from there in a sort of self-improving way, which is, it's able to conjecture better as it is able to verify better. It is able to verify better as it is able to conjecture better, have harder tasks. So conjecturing helps proving, proving helps conjecturing. And we're talking about, like, people think about knowledge discovery, people think about math, people think about theoretical physics, people think about Ramanujan and Einstein, and that's all good. And that's my first part. What is the second part? The second part is verified AI. And when people talk about verified AI, people are thinking, okay, code verification applied to hardware and software. People are thinking about code verification when the cost becomes zero, when people are writing programs with proof. That can be expanded into a lot of next adjacent possible markets, markets that we don't know and don't have customers yet, but also that could include customers that don't know they can benefit from it.

Yeah.

If you think about how the software engineering culture changed, it starts with Cursor IDE, which is a winning product, and suddenly people are seeing themselves doing next line completion using AI. Suddenly there's Claude Code, which is coding agents' world, and winning product changed software engineering culture. People in the formal verification space have been asking, oh, why is software engineering culture not rigorous enough? Why are we not writing programs with proofs? Why are we using Vibe to generate specifications and have those specifications lead to code with proof? And I think I think the answer is, well, there hasn't been a winning product yet. So that winning product is going to change the world of software engineering culture. And what I mean by verified AI tied with verified knowledge discovery is the following. I think the real gain of formal verification is on better generation and better design. So if you are someone working in the chip verification space, you should aim to own the whole end-to-end pipeline of design and verification to provide better design or better design inspiration to your chip design customer. If you're someone who's working in the code verification space, you should aim to not just verify programs given to you, but also use those proofs to let you— to inform you to write better programs. Or even better, to, when you're writing software, have the sort of specification and proof concept in mind. And this is the conjecture-proof loop. This is the generation and verification loop. We go from math, from science, into the real-world commercial areas controlled largely by all sorts of code in all sorts of languages across all sorts of settings. You can think that maybe I want to build a bridge. I want the best-looking bridge or the cheapest bridge that doesn't collapse. Okay, I have some constraints here. I want the bridge to not collapse, right? And I want it to be optimal in some way. And if there is a formal language reflecting the engineering of the bridge, civil engineering corroborated formal languages, then I can have that sort of proof of safety in mind when I'm designing it. And that is the broader future of verified AI. That is the attempt. So math is a clean sort of lab.

Yeah.

for us to try all sorts of things. I think it's very difficult to imagine conjecture generation with messy real-world data, but because we're able to see that and do that in math, and we are seeing this action improver sort of able to succeed in various industrial tasks around code verification, I won't specify which domains, we think we are on sort of the second step after both Putnam, peer-reviewed journals, you know, doing research math and competition math. to the next step. And we're, I think, like, at this day 1, if you call the Putnam and the research math day 0, day 0.5. We're at this day 1 right now, and we're very excited to see where that leads us.

Would you call that applied math versus, like, theoretical math? When you work with clients, is that basically a form of applied math?

Yeah, I think it's unifying math and code. I think it's marrying math and code, 2 important pillars of the digital world. I think it's a beautiful story that you use code as the data vehicle of math, and you use math to verify code. And, you know, for the first time, if you think about outputs being meaningless without the property underneath, and marrying property and output, marrying logic and computation— logic corresponds to property, computation corresponds to output— marrying Math and code marrying proof and computation. This is kind of the thing that we are getting to, right? We want one unity between, for lack of better terms, generation and verification.

So I think this is all very exciting, and it's not hard to understand why investors would want to give you a couple hundred million dollars to go pursue this vision. I do wonder how you are thinking about the long-term competition with, frenemy status with, perhaps maybe one day being acquired by one of the frontier companies, because unlike other areas that people are going off and kind of taking contrarian bets on, this is like right in the wheelhouse, right in the bullseye of kind of what they are focused on.

Yeah.

focusing on too, right? And their approaches, I don't need to tell you, but just for listeners that may not have heard, like at the beginning of the year, I participated in a little forecasting exercise and my guess, which was higher than the median guess, was that the best LLM at the end of the year would get to 63% on Frontier Math Tier 4. Now we've got Fable already in the high 80s.

Right.

I think everybody's kind of like, wow, this is going extremely fast.

Yeah.

Is there some sort of different scaling law you kind of alluded to where you think that the approach you're taking sort of scales better? Is there some inherent property that you think you're going to create in your models that they won't be able to create? How do you sort of create enough difference And maintain enough difference to have meaningful advantage given, you know, $200 million goes a long way, but, you know, these guys are raising $200 billion, right?

So it's a very different level of resources. Yeah, I think the sort of—

I think you are muted, maybe.

Oh, can you hear me? I can hear you.

Can you hear me?

Okay. Yeah. So I think there are interesting sort of dynamics here. One is token-based pricing. If formal verification is successful, are we going to burn more or less tokens? This is, I think, going to determine whether the frontier labs have an interest in pursuing this direction versus if you are a startup working specifically with industry partners that do not necessarily, you know, that might be open to different other sorts of business models. And those markets, if not large enough for, say, the frontier labs to do anything beyond sort of performative, you know, like showing of capability, then that is a real sort of like market test for those sectors. And you might actually find a pretty nice sweet spot there where you are actually friends with the frontier labs instead of enemies.

When you work with clients, do you have forward-deployed engineers? Are you sitting there kind of like formulating the conjectures for them and then kind of training on it? Like, how does that work when you work with a client on a particular problem?

Yeah, we're currently very early. So, we have design partners and we're very blessed with great design partners who have very technical folks on the other side. However, they're not necessarily lean folks. So we're definitely seeing one of the capabilities that we learn that are important is the ability to say, work with another language. I won't specify which one because that's pretty telling. And then particularly have axiom prover prove in Lean the verification conditions. Now, the process of— you can call that also autoformalization, not from English to Lean, but from this other language to get to the specifications in LingQ. So slightly different from just a sort of statement translation problem, but we have been seeing that working well with particular choices of languages which are sufficient for those use cases. I don't know if that— if this livestream is going to make us qualify last as a new lab, but that's some of the sort of like real-life learning lessons that we are seeing.

Fantastic. Thank you so much for joining us, Karina.

Thanks so much.

This has been a great opportunity to learn about Axiom Math, and we'll certainly be following your progress.

Yeah, thank you so much.

Yeah, thank you, Karina.

Bye-bye.

Thank you.

Awesome. Um, it's very, it's very rare to have people who are that enthused about math. Like, that's awesome.

Yeah, she's— I don't know if I can listen to her podcast at 2x speed. And that's, you know, a high compliment, I would say, in the AI moment. You know, she's running— her tokens per minute is, you know, one of the few that might actually keep up with some of the frontier models.

Indeed. Our next guest is Donnie Bloomfield. Donnie Bloomfield is an associate professor of law at Fordham University, where he specializes in how legal tools can guide technology to safer outcomes. He focuses on areas like intellectual property, biosecurity, export controls, and tech policy— basically how law can encourage innovation while reducing catastrophic risks. Lately, he's been zeroed in on the question of AI in biotechnology. As AI systems get better at designing drugs and even pathogens, how do we update the rules to prevent misuse? Before joining Fordham, he was a senior researcher at Johns Hopkins Center for Health Security, and a postdoc fellow at Harvard Medical School. So he's really looked at the biotech area from many different angles. Uh, in 2000— in February 2026, he published a paper in Science with colleagues that proposes a new way to govern biological data in the age of AI. The idea is to keep most data open for research but put the truly dangerous stuff behind safeguards. kind of borrowing ideas from privacy law and biosafety. At a time when AI can design viruses in a lab, that balanced approach is a big deal. He's here to explain how law might be used to not just to hold back technology, but to steer it towards safe and beneficial uses. Donny, welcome to the show.

Thanks so much for having me. Yeah, I hadn't thought of using boring legal tools to head off terrifying catastrophes, but That does seem like an apt summary of what I think about.

And in that vein, tell us a little bit about your proposal in the February paper. How do you propose to regulate data in this space?

Great. So I should say upfront, this is joint work with a lot of great colleagues. One of my chief collaborators here was Jasdeep Anu at Johns Hopkins, and our proposal is that there are some forms of data that can conceivably be used to teach biospecific models, or maybe other models, dangerous capabilities like designing enhanced pathogens. And the idea is that there is a narrow slice of data about viruses in particular, and novel data, data that don't yet exist, that we can make available to responsible researchers on platforms that are secure without making them generally publicly available, and thus available to be used to confer some of these concerning capabilities, including by actors that we wouldn't want to have those concerning capabilities. And like you were saying, Prakash, this is like drawing on privacy law models, not AI models, but rather like frameworks or ways of thinking about things, where we can have all sorts of private data, genetic data, that we want scientists to be able to learn from, but where we don't want that data to be just available for anyone to download and scroll through. And We have seen successful examples where we're able to allow scientists to get access to these data, to bring their questions to the data without exporting the data or making it just generally available. And that's— that is the idea, is that we can have that same type of framework here.

Do you want to talk a little bit about how this was used in the Evo and ESMFold training processes. I think that is a really telling and kind of positive vision for the future, where the pattern that you're describing was actually at least partially realised by folks training frontier models. And it seems like we're all much better for it, especially since they're to some degree open sourcing these models now.

Yeah, yeah. So I think that, um, in, in these examples, um, the creators of these frontier biomodels, so models that are trained on biological data exclusively, uh, voluntarily didn't train or refrain from training their models on data, uh, involving human infecting or, or really eukaryote infecting, uh, viruses, so viruses that can infect all sorts of of life, including plants and animals. And they saw that there were major reductions in capabilities for these models when you compare them with alternate versions of the models, at least in the case of ESM, where the model had been trained on those types of data. And so the idea there was, let's refrain from training on these potentially concerning data in those cases, and then we can see empirically that there are these drops in capabilities. I should just say, like, I guess 2 things about that. One is that our proposal would really apply on a forward-going basis to data that don't yet exist. So in those cases, they were refraining from training on data that are available. And in fact, you can, if you have access to those models when they've been open sourced, you can fine-tune on the data that do exist. And so we do have some evidence that this can actually be done and is not that hard. With Claude Code, for example. So some work from a couple of people at GovAI showed that you could fine-tune Evo2 on data about human-infecting viruses, and you could get jumps in capabilities on prediction tasks associated with human-infecting viruses. So I think that as long as We're talking about data that is publicly available, and we're talking about models that are open weight. The fact that the developers, in the best of interests, refrain from fine-tuning does not preclude people in the future from doing that kind of fine-tuning. And so I think that's part of our— that's kind of what's driving our proposal is thinking like, what can we do thinking a few years down the line when we do want a lot of open-source, fantastic biological models? And we also want scientists to be able to, like, in real time respond to really important developments in virology. But we don't always want the two of those to be meeting in a fully open way if our concerns about these capabilities is realized.

So one of the questions I had for you is that it seems like there's, there's kind of two approaches. One is this Claude-like approach where you have a constitution and Claude basically knows about viruses, et cetera, but decides not to tell you about it. And it says, look, I'm not going to go there. So I know about it, but I'm not going to go there with you. And the other, which is this approach where you actually take out that chunk of data in training itself, and therefore Claude or Claude, whatever you want to call it, does not know that chunk of data. It's kind of a blank space in the kind of mental framework or the map or the weights. It strikes me that the pieces of scientific knowledge are so interconnected in some sense that AI is not a library. It's not just recalling facts. It's putting together facts in various ways and it's inferring things. And therefore, even if you excerpt or block out that chunk of data, it is possible that the AI can just reconstruct it in real time and, um, and know basically the, the, the, the same, the same facts that you sought to avoid it from knowing.

Yeah, so I think that, you know, this is a really important question and gets to, I think, just empirical uncertainty that we are very, I think, straightforward about in our Science article, which is that we don't know whether these types of data are going to be essential to these capabilities or whether they can infer, be inferred from just like general knowledge about how biology works. And I think that you're totally right that, you know, it's amazing to see like cross-domain application from LLMs, right? Where like you can learn like all these languages and then suddenly you might be able to kind of like understand and even translate like languages you haven't seen before. Synthetic languages. And we just see this in many domains. And I think that the bet that we're making, and I think this is one that needs to be empirically tested, is that the complexity of biology is such that we don't, we're not providing these models yet with like a full, complete understanding of physics such that like from their knowledge of like how plants, how plant DNA works, or how genes work in humans, They'll be able to kind of back out everything about what makes a virus infectious in humans or more infectious and more virulent. And so the bet is that we're not handing them a full toolkit of complete stack understanding about how biology works. And I think that that is a pretty sound bet, albeit one that needs to be borne out in evidence, simply because where we've seen the biggest advances in AI and biology, it has come in these arenas where we have copious amounts of data and we are seeing like in-domain efficacy improvements, like with the AlphaFold models of going from amino acids to folded proteins, where we had spent billions of dollars over decades to compile the data, really beautiful data that could be used to train models. And that's where we saw like fantastic leaps. And so some of the bet here is that there is that relationship between data and capabilities that will bear out also in the kind of concerning pathogen domain. But I don't think that precludes the thought that there are other safeguards. We're not claiming this is the silver bullet, and we're not claiming that there's nothing else to be done. In this domain. And I think having a constitution for a biological model might be a little harder if they don't understand English or any human languages. So they may not have personas that really can interpret or kind of take on virtue ethics. But certainly that seems like a kind of fascinating research direction to understand how we might be able to imbue imbue those more virtue-based safeguards into these models. But I think that that may be well in the future.

How does this affect kind of First Amendment-related issues around the models themselves, in the sense that you are preventing the models from consuming, or preventing the companies that build the models from consuming the data that perhaps is not, you know, I mean, the blast radius of data that might need to be excerpted might be larger than, you know, the companies themselves or the researchers think.

Yeah, so I think that there are really interesting First Amendment questions about how we can regulate training generally. Like, you know, could we, could we regulate something like, or demand or mandate something like Claude's constitution? in other domains. In this domain, I think that there are serious First Amendment free speech concerns with saying you cannot use these data in training. That might be because we think about— we're worried about the expressive rights of developers. It may be because, and I think this is maybe the more serious concern, we're worried about listener rights, like the rights of all of us to be able to hear truthful information about things that we all care about. And the Supreme Court has said that the First Amendment extends its protection, extends its coverage to listeners too, and not just to speakers. That said, I do think that, like, when you're thinking about the kind of catastrophic outcomes that we are talking about, and when we're talking about, like, the kind of functional data that we're talking about, where there may be, like, an expressive component, there's also, like, a functional component where, like, something is just being changed in the world. That's not, like, a communication. I think that we can apply probably less scrutiny, so like less demanding tests of the government. And we know that courts have been willing to bless information control regimes when the concern on the other side is grave enough. So, and just to be clear, there's like a decal at the bottom that says training bans. I'm not proposing a training ban. And none of us are proposing training bans. But I think that, I think that if you have a tailored enough regime, and we are trying to propose a very tailored regime that has, like, different tiers of control, and one that gets kind of stress-tested empirically over time, I think that that's the type of thing that can survive First Amendment scrutiny, even though I think it should attract some level of scrutiny. And certainly, like, the kind of actions that we've seen over the past week from the Trump administration and its application of export controls, I think should make all of us think very carefully about the First Amendment implications of how we regulate the distribution of models and how we regulate how they're created and accessed.

A perfect segue. The week as it has unfolded, I think, has left a lot of people with a sort of real politics sense of what's going on in this interaction between the government and Anthropic. And my sense is like, it seems like they're kind of just fucking with them because if they weren't really well justified in what they were doing, we'd probably have heard a little bit more about exactly what is motivating these actions. So the silence is kind of deafening there. At the same time, there is an actual body of law under this that I don't understand very well either. So So leaving aside the sort of capriciousness of the action, help me understand what power the government actually has. When are they supposed to use it? How much discretion do they have around when they're supposed to use it? And what are the avenues that a company like Anthropic has to fight back? If they want to, and then we can maybe analyze their, you know, the kind of strategy that they are playing. I'm kind of confused by their strategy as well, honestly, to have their, you know, to have their new flagship thing yanked and basically just take it for a week and not say much also strikes me as strange. But I might not understand the, the actual moves that they have available to them. So, uh, yeah, I mean, we got a lot to learn.

Yeah, yeah. As you say, I think we're still in the fog of war, and we're kind of learning everything, like, by the hour. Like, right before this, I was checking, like, the Wall Street Journal has some report, like, you know, in within the last hour and a half about, like, discussions are going fine. Who knows? But I think that what we can say is a little bit about the powers of the export control laws. I mean, these are laws that go back to, you know, the Cold War and before it that have been revised many times since. And what they're designed to do is they're designed to control the flow of US technology outside of the United States, as well as inside the United States when it comes to people in the US who don't have green cards and don't have US citizenship. And I think that their essential goal is trying to maintain US technological advantage, relative advantage. That's how they were used in the Cold War, and that's how the first Trump administration and the Biden administration has been using them, did use them fairly aggressively when it came to controlling the spread of semiconductors and semiconductor manufacturing equipment. And the government's discretion here is extremely broad. So the government can issue regulations that control like specific types of hardware, specific commodities, specific types of software. And it can also control what's just called technology, which means information. It can control proprietary information and it can prevent companies from, without a license, sharing those proprietary information with non-US persons. And, you know, the Congress has given so much discretion here to the executive that it has also removed some of the usual review avenues in court that exist, and I can go more into that. But I think that the, you know, one thing to say is that in spite of this very broad discretion, having now looked at the letter, that the Commerce Department issued to Anthropic on Friday, at least the reported contents that Bloomberg obtained, it's not clear that the government has the authority to do what it did here, at least under its stated legal powers, nor is it clear that the letter even actually restricts Anthropic from making its API available, including to foreigners. So there's very broad discretion, but it's not clear that They actually even have the authority to do what they did, at least under their kind of claimed arguments.

Can you just do a double-click on that? I mean, I've heard things like you can't export control services, and I'm not sure if that kind of plays into why they may or may not have the authority. Unpack why you would say they might not, given all the broad discretion that they have, why would they maybe not have Yeah.

So there are a lot of like gritty technical reasons. I think that one of them is like what the law says is an export doesn't cover services. So it can cover information, but it doesn't cover services per se. And the Commerce Department has been explicit about that in its own guidance. It said like, you know, cloud services are not an export. It said that, you know, software as a service is not an export. It said that in its own guidance, and Congress has been working actually to fix this loophole. The House passed a bill in the Remote Access Services Act to try and clean this up, to give Commerce the power to restrict, you know, non-US users from accessing compute or AI models. But those powers don't yet exist. And so saying that Anthropic cannot export a model, I mean, it's not even clear what they mean by that. But the powers of commerce here are not infinite. And if they did try and restrict, which they don't say in the letter, but if they tried to restrict all outputs from these models, that would run into both real problems under just the statute and the regulation, which say that it doesn't apply to published material or fundamental research, both of which, at least like the Fable outputs, probably would, because you and I can buy a subscription to Fable, which means that it falls into this exception in the regulations. And it would also run into, as, you know, we were kind of talking about earlier with respect to biological data, it would also run into, I think, serious First Amendment questions. I don't think those First Amendment questions would be like impossible to get over, if we were talking about a really serious catastrophic risk. But on the level of risk that we've been talking about, especially when they're not doing the same thing for GPT-5.5 or other models that seem to have similar capabilities, I think that the First Amendment issues here loom pretty significant.

One question I had for you is, to what extent can the government even define what a fable or mythos is, because one of the jokes going around in the space is, let's just release Untitled Checkpoint Number 42. And what exactly is fable? If you change a single weight out of the trillion parameters, is that the same model?

Yeah, I think that you could do it. You might need to constantly rejigger it, but I think that one way to do it, which is what the Biden administration did when they They did try to control AI model weights with their AI diffusion rule in January of 2025. They defined it as a proprietary model that had been trained with a certain number of floating point operations. And so we saw that that was the kind of way that the Biden administration went about it, was like models that are trained with a certain amount of FLOPS get certain types of controls. And they also had biological data, like models that were primarily trained on biological data had lower thresholds for some of their Some of their rules. So I think that there are ways of doing this, just like how it was actually kind of hard for the Commerce Department to figure out how do we define the AI chips that we want to control. They had to come up with new ways of talking about chips and using definitions that the industry didn't use because the industry didn't care to define things in the same way that the government does. And so I think that you'd be in a similar situation where you would come up with technical capability definitions or training definitions. And like I said, the department does have very broad discretion to do things like also send individual letters and say, we're defining things in very specific terms, and we're going to say everything that comes off this branch is going to be controlled unless you can show us otherwise. I do think there's something very dangerous about that kind of secret channel governance, which I think we're seeing right now, like, with all the kind of just unclarity and obscurity about, like, what's controlled and under what rule. So I do think there's something dangerous there, but I do think that the government has a lot of flexibility to try to define things, and it can do things in a more public-facing way by using some of those objective criteria.

So if you're Anthropic right now and you've got this sort of mess of a letter and it's not clear at all, it sounds like you would kind of say probably they don't have the authority to do this. And the fact that there's like a bill working its way through may or may not happen that would give them the authority makes it pretty clear that, you know, current Congress thinks they don't have it as well. What would you think? What is the option set for Anthropic? Could they have simply said, sorry, your letter is kind of a mess, like, we're just going to keep doing our thing here? If they do that, what does the government do to escalate? They could obviously sue in some way, shape, or form. One lesson I've learned, you can always sue. You always try to sue, but that might take a long time. And I've also heard arguments around, well, they don't necessarily want to pursue a First Amendment claim because that's sort of Contrary to some of their other ideas about how regulation should happen, like they don't want to set that precedent, even if it you know gets them out of a sticky spot in this case. You know they could potentially try to get an injunction to stop the government from messing with them. What is is that the option set? What options am I missing? And and why do we seem to be getting this like just do nothing for the time being?

Yeah. So I don't know exactly what's going on inside of Dario's head or like, or the leaders of Anthropic. I do think that they could seek an injunction. I think that they would have a plausible case, at least under the authorities that are mentioned here. But I think that the danger of that is like, there are a couple of dangers with that. I mean, one is that there are other authorities that we haven't even talked about that exist in the export control laws. that they could throw the book at here. And I think that they like did a hasty job with this letter. That doesn't mean they couldn't write a tighter one that could hold up more, you know, under, under greater scrutiny. Probably the more serious point is that—

Just as an aside, do you think Fable would write a better letter than they sent?

Yes. That's, I do think that they would write a better letter than they sent.

Sorry to interrupt.

You know, there's, there's some risk that, you know, it might hallucinate a site, but, but I think that, you know, in collaboration with with Fable. Um, uh, you can read—

Government needs Fable.

Yeah. Um, and they have, you know, they, they haven't cut off their own access to Fable, presumably. So, um, uh, uh, or Mythos, really. But I think that, um, the, the problem, the, the broader issue, and I think that that's one that's very concerning about what happens when you have a government that isn't trying to play by the rules, is just that there's so many levers that the, that, that the government can reach that they have power over, that even if you win a particular battle, there are just so many other ways that they can make your life difficult. And I mean, I think this is something that people are in some ways unfairly accusing Anthropic of, of like not recognizing the realpolitik here, which is just that like, fighting the kind of legal question is really secondary when you think about all the other tools that government has to make their life difficult. And I think that that's unfair because I think it's principled to say you don't have this authority or, like, you shouldn't be exercising this authority. But I do think that that's a pretty good reason not to bring suit. And then the kind of third reason, which you mentioned also, is that you might create, you know, a bad law because in the heat of the moment, when a federal court sees what's happening here, and sees this context of all of these negative statements by high government officials about, like, so-called woke AI and kind of attacking Anthropic on ideological grounds, I think that the court might be interested in stopping the government from doing what seems fairly lawless and using its power to attack what apparently it perceives of as an ideological enemy. And that could result in bad case law when it comes to, I think, good faith and well-governed AI regulation, where we actually do want, in spite of some of these First Amendment concerns, I think we do want to live in a world where if we have a sufficiently dangerous model, we are able to have some kind of safeguards in place. And maybe that even looks like preventing release. So I think there's a— it's very dangerous to try and craft law on the basis of outlier situations when that precedent may make problems for kind of good faith efforts in the future.

Yeah, my friend Dean says republics run on virtue, and unfortunately, uh, virtue is in a bit of a short supply at the moment. Does— if this sort of goes on, right, I mean, we've got like now 2 pretty high-profile moments where the government has seemingly taken motivated and kind of, you know, not good faith action.

Right.

With the supply chain designation and now this export control, how, how much does that sort of become relevant or, you know, admissible like in future? You said kind of like, sure, they could write another letter or they've got these other authorities, but in the court of common sense anyway, it would seem like people would start to recognise a pattern here and be like, okay, well, you're just trying to mess with this company and you're reaching for all the levers that you have. But at some point it's clear on a pattern of behaviour basis that you don't really care about any of the laws and you're just trying to do what you want to do.

Yeah.

But do courts think that way, or are they sort of more narrowly constrained to look at just like this one you know, this one law as it applies to this one situation? How broad can they zoom out and consider the government's, you know, apparent motivations and patterns?

We're actually lucky to have, like, a very on-point Supreme Court case from last year, where the Supreme Court said that New York State was going after the NRA on ideological grounds, And that, um, even though, uh, or even if, like, the, the law under which New York was trying to go after the NRA, uh, was itself appropriate— in other words, like, even if all the actions aside from the ideological motivation have been appropriate— if they're using their lawful powers to attack ideological enemies on ideological grounds, then that is a First Amendment violation, and you can prevent the government from taking those steps. And you can look fairly broadly to see that you can look at like, what is the government communicating? What is it saying about its actions? What is it telling other people about why it's making these decisions and how they should proceed? And I think that all the evidence that we've seen of at least some ideological motivation on the part of the Trump administration should at least raise like serious First Amendment hackles, even if we don't think that like the, you know, the models constitute anthropic speech, even if we're not worried about the model output as information that we as listeners have a right to hear, just like going after Anthropic on ideological grounds, even if they were on totally otherwise good legal authority, would itself constitute a serious First Amendment question. And I think that's a challenge that Anthropic could consider bringing, but it's one that would still trigger all the problems that we were talking about earlier, where if Anthropic wants to have an ongoing relationship with this administration, They are faced with a really serious trade-off where there's still all these other tools and just constantly returning to court is just a perilous exercise. So I do think that there are real First Amendment questions about the validity of this action, even aside from all the speech concerns, just like punishing, seemingly going after Anthropic as an ideological adversary presents very serious First Amendment problems on its own, um, to, you know, to begin with.

What's your— what's your over-under on when Fable or a Fable-scale or a Fable-class model, uh, comes back, uh, into circulation for the public by Anthropic or any other, um, you know, firm?

Yeah, well, I have friends who are superforecasters. I'm not one of them. I'd probably just stick with the Calci line on whatever it was of— I think last I saw it was like 55% by July 1st. I don't know what the current betting numbers are, and obviously a lot of what this administration does is challenging to predict at the best of times. I won't give you, I won't give you a number, but, but it does strike me as even if this has worked out in like a somewhat reasonable time, I do think that we are in pretty treacherous waters for the kind of ad hoc AI decision-making that, that we're seeing and using export controls that were not really designed for this purpose being used here. And I think it really is, is a call to action for For Congress and the executive, you know, civil servants who could try to craft a more, I think, sound set of policies that are more reasoned and more principled, and drawing on some of that republican virtue, small r republican virtue, that Nathan was talking about.

Yeah, calling Congress. Might be time to do something, guys.

Step up. You gotta step up. You can hear me.

Professor Donnie Bloomfield, thank you for joining us and helping us make sense of this. Uh, something tells me there might be an opportunity for a return appearance sooner rather than later, the way things are going. Uh, we might have a lawfare beat on our hands, whether we like it or not.

So yeah, everyone, I mean, it's all lawfare all the way down right now. So yeah. Alrighty. Well, thanks so much for having me. Great to be on. All right. Bye-bye.

Thanks for being here. Real quick, just to put numbers on that, I did look up Kalshi. It's honestly, this is sobering. I don't like what I'm seeing here at all. It is as of now, 54% chance that it will be restored before July 1st.

Yeah.

65% chance that it'll be restored before July 10th. And then they've got a July 17th. And that goes up to 68%. So only basically a 2/3 chance over the next month. I'm not feeling that at all. Like, this is terrible. My original over-under was Friday and it's fast approaching. So yuck.

Yeah.

I don't like the thought of— I mean, nobody is going to have something, they can't be that far behind. So the one question that you, or the clause that you added that isn't in that Calshi number and could really make things even more scrambled, confusing, obviously flagrant bad behavior, is if OpenAI gets to launch one. And over the next month, I would expect that they probably will have something in that general class. And if they get to launch it without getting the same harassment, and we're in a really strange spot. So.

So on that note, I'm going to bring up our next guests. We have Sam Paspalak, the co-founder and CEO of Skyfall AI, a startup aiming to build what they call enterprise superintelligence. Sam first made waves in AI as co-founder of Maluuba, a pioneer in natural language AI that Microsoft acquired in January 2017. At Skyfall, he's now focused on world models, AI systems that don't just chat or generate text, but actually understand cause and effect inside a business. Think of an AI that can simulate your company's operations, learn from every new development, and help plan complex decisions, not just answer questions. Sam argues that today's large language models, the kind of AI that writes fluently, are not enough for high-stakes operational decisions. His team is trying go beyond language, building AI that can predict consequences, adapt to changing data, and make long-term plans. It's timely as many companies are now discovering the limits of chatbots and looking for AI that can truly run processes and not just talk about them. Sam, welcome to the show.

Thank you for having me. I appreciate it.

So Sam, imagine you have an AI assistant that can write beautifully crafted emails, but ask it to reschedule your supply chain when a factory shuts down and it's utterly lost. That's the gap that you see in many processes right now. How can that be addressed?

Yeah, I think maybe we can take a step back about what we have seen, the overall success in the last 3 and a half fears first and then go from there. So I think if you think about like what has succeeded since, let's say, November 2022, I think LLMs have succeeded in like, I'd say, 3 broad categories. The first one would be text generation and information retrieval. So you have obviously the ChatGPTs and then the Geminis and such. The second would be the code generation where we have Claude Code and and Cursor just got acquired by SpaceX and whatnot. And the third, to a smaller extent, would be in the video generation paradigm. I think that's much, much smaller, uh, success than the other 2 paradigms. Um, now if we think about, uh, why LLMs have succeeded and stuff, and, and basically, if you, you guys know this, LLMs are trained on the World Wide Web. LLMs are trained on Reddit, Twitter, Wikipedia, everything on the web. But when it comes to the enterprise, I think, uh, all the— like, LLMs are not trained on databases, LLMs are not trained on time series data, LLMs are not trained on, uh, you know, everything that enterprise had to do with on a day-to-day basis. And LLM and enterprises are more so, you know, dynamic in nature. Uh, I Everything changes in enterprise on a day-to-day basis. It's much more complex, the decision-making in an enterprise. So our eventual goal is to make an AI CEO. I think that's the goal that we have, and that can be achieved through a combination of technologies. Yes, LLMs will play their part, but with world models and continual learning as well. That's what we're going for. Essentially, I want to replace the job that I do, which is a lot of complex decision-making under uncertainty and a lot of long-term planning, long-horizon planning and such. And those are things that LLMs can never do because they're simply based on next word prediction, next token prediction. That's the high-level essence of the company.

So can we Talk about what you mean by a world model, because I think there's a very wide range of what people actually instantiate when they use that umbrella term.

Yes.

Just yesterday, I was talking to somebody developing an AI for science agent where the world model is essentially an external, explicit knowledge graph, kind of causal graph type structure, and it's meant to be something that is inspectable by humans or other AIs, whatever. But it's a very sort of tinker toy thing that you can pull apart and explore. Then obviously we have these sort of emergent world models in LLMs and in other architectures, including video generation, where they seem to learn a sort of intuitive physics in the video case. And they learn all kinds of things. that are a little bit hard to characterize what they do and don't know in the LLM case. And then there's some things in between. I noticed on your careers page there's a mention of JEPA, for example. So what sort of world model do you have in mind? What does that actually look like?

So if you think about what a world model is, in essence, it's like, what's the— basically it's a state-action pair. So it's like, given this state, and given this action, what is the next state? That's the very, very high-level version of what a world model is. Now, from our perspective, we want to build an explicit world model of the enterprise. That's how we kind of think about it. So all the decision-making within an enterprise, whether it's connecting to the ERP systems, whether it's connecting to the databases, whether it's connecting to all the data within the enterprise, And then predicting the next step. That's kind of what we mean by a world model. If you think about how it relates to humans, as human beings, we try to kind of imagine what the future steps are going to be before we take an action or a policy in the real world. That's kind of like, that's kind of the conceptual version that's most correlated to humans. And that's what we're trying to kind of mimic via a world model in the enterprise. That's what we're trying to do. In a nutshell.

So does that mean it's making— oh, sorry. Does that mean it's making a, like, high-dimensional— when you said sort of a state and then an action or a perturbation and then a new state, that also maps onto my understanding of how certain, like, cell models work, where you're like, okay, here's the transcriptome or whatever. And then here's, you know, a drug that we're going to introduce. What's the transcriptome going to be after that? Perturbation has been entered into the system, but it's a very high-dimensional prediction.

It's like a latent space abstraction if you think about it. It's like latent space reasoning. It's like a higher level of abstraction and thinking is what we consider world models to be. I think it kind of like 2 similar paradigms in the drug discovery, whatever stage that you're talking about, but for us it's like a latent space representation and abstract thinking. That's how we consider world models in a nutshell to be.

But I guess I'm just trying to understand a little bit better, like, I have this enterprise, right? It has like many characteristics. It has headcount, it has departments, you know, I can, I could potentially shrink this department and grow that department. And I can, you know, add a product line and, you know, discontinue a different product line. And I might cannibalize, you know, for my current product if I introduce a new thing. So When you make this kind of new prediction, how broad is that scope? How do you think about how far out in the ripple effects of a decision AI can go, should go, will ultimately be able to go?

Absolutely. I think obviously we can talk about the present and we can talk about the future. In, in the long term, we want it to be something like Minority Report. If you remember Precog in Minority Report, I think, uh, that's where the future we want to be, where you can predict all the different future simulations and then, and you select the best simulation that fits to the best needs of the business. Uh, in the present state where we are right now, uh, we are still in very, very early developments of of world model right now. So I think, uh, so from the enterprise context, if you think about, let's say, 12 to 18 months from now, what we're going to be building and what we're going to be showcasing in a product is like, if the simplest form of an enterprise is like an e-commerce business, okay? So in an e-commerce business, uh, you have like a— you can have an AI CEO, you can have an AI marketing agent, an AI sales agent, and so on, and they coordinate with each other and you give a goal. Like I need to have $2,000 of sales over the next week or something like that. And these guys go to like figure, they coordinate amongst each other the different sub-goals and sub-parameters and they figure out, okay, I need to like go on Instagram, figure out who the right user set is going to be. Then go on Shopify, try to create the appropriate store for this kind of product. Then figure out a go-to-market plan and then actually execute and go deliver on like getting the $2,000 in sales. That's the most concrete representation of a world model, which we think we can build over the next 12 to 18 months. Now you can go from there to like a fully functional, like let's say Microsoft or something like that. That's the next level that will take many, many years to get there. But we have to first prove that a world model works. In a small-scale setting like an e-commerce business and such. Uh, that's where, uh, that's where they are. That's where we're building our— all our technology and research around right now. Um, again, I also want to mention the fact that current world modeling techniques, they still are very theoretical and there's no practical applications as of yet. So I think around later this summer or early fall will be one of the first companies in the world that's gonna show a practical application of world modeling, uh, in the enterprise context, in the software context. Uh, so I think that, that's— we are very much focused on the digital world modeling. We don't care about like robotics and the physical space. I think there are other players who deal with that because we think there's still so much innovation to be made in the digital space. Uh, and people think that, hey, you know what, LLMs are the be-all end-all, LLMs are going to get you to AGI. But in my opinion, personal opinion, that's impossible because, you know, LLMs cannot do long horizon planning. LLMs cannot figure out like what the automatic sub-goal decomposition can be when you have a specific goal in mind. LLMs cannot deal with dynamic environments and such. And LLMs will always have, like in our opinion, for example, world modeling, since you have the latent space reasoning and stuff, you always try to figure out what the most efficient algorithm is for state space tracking and stuff. Again, that's not possible with LLMs. So I think these are the shortcomings that a combination of world modeling and continual learning will get— will help us get, which we're very much focused. That's why for us, this is like going to be the next big thing after OpenAI and Anthropic, in our opinion.

So, concretely, let's say you do have an e-commerce business.

Yes.

And you apply your world model to it. What is the difference that the client will see? Like, let's say you have this e-commerce business and it sells, I guess, shoes online and it buys the shoes from, designs the shoes, buys the shoes from China, gets it manufactured, comes onshore, sells it on Temu, whatever. Like, what is the concrete thing that this business will see as they implement this?

But you don't need any more employees. That's what we're going to get towards, right? You don't need to hire all these people on contract and stuff. And you, you, the world model should be able to help you run your business by your own. We're trying to democratize the technology that, that is only in the hands of, let's say, a few companies via LLMs and such. The big model companies to like every small SMB in the world. I think that's what we're going for. And I think if you think about the world's GDP, around 70% of the GDP is because of business activities and majority of that is because of small businesses. So we want to democratize the power of world modeling to all these SMBs so that they can run their businesses more efficiently and more effectively over the long, long term. I think that's the overall vision.

And so what that would mean is essentially your model would obviously be using, if it's doing customer service, it's gonna be using a typical LLM sub-model there. And it has access to all the keys and all the bank accounts and all of the accounting systems. And all of the CRM systems, presumably. And it is making long-term predictions and plans on what's going forward. It's choosing product mix. It's selecting suppliers and interacting with them. It's handling its exception handling on returns and, you know, refunds, et cetera, et cetera. So this Yeah, absolutely. Agent is essentially doing all of these items.

But I'm saying that absolutely that is the capability of a world model, but you still need LLMs, for example, because LLMs are very good in when you have a very specific goal for what you need to be done, what you need to do. LLMs will execute on the goals, right? You still need LLMs for that, but world model will be used for all the abstract thinking that's required. to run the entire business. You will never be able to give LLM a billion dollars and be like, hey, run the business for me. It's impossible. And hence you need a combination of LLMs, world model, continual learning, and potentially other technologies in the future for this world that you described to us in the future to be. Yes.

What do you think the interaction model looks like? Like when I'm doing something, for example, Hypothetically, in biology or even ML research, I can sort of prompt my language model agent, which then can use these other things on a tool-calling basis to run the cell world model to simulate something, or call any number of specialist other kinds of models that are trained in a kind of more narrow, but hopefully more effective way for whatever they're trying to do. Right. My intuitive sense of how I would imagine what you're describing folding into a business would be, I, as a business owner, would talk to my language model and it would call to the world model and say, if we do this, what's the future state going to be? If we do this, what's the future state going to be? But I'm not sure if that's what you're envisioning, or it sounds like you may be kind of inverting it where the language model becomes the subsystem, and the world model itself is kind of the thing that I interact with. But then I'm not sure, I'm hurting, struggling to picture like what my interaction with the world model would look like if that's the case.

So in our opinion, again, we're very experiment— we're doing early experiments with this, uh, and we'll have very interesting results that'll come out in August, September, hopefully. Again, it's research, not engineering, but that's a prediction right now. But I think they're working in conjunction right now, uh, is what, what we are— LLMs and the world model in, in, in conjunction. I think my researchers are gonna hate it, hate me for already saying this in this podcast right now, but I'm so excited about the research that we're trying to do. Uh, so LLM is going to be kind of like a policy, uh, and then the world model is gonna make all the prediction and all the decision-making and stuff. So LLM is like the policy that you make in the decision, the final decision that you make. That's how we kind kind of, uh, think about it. Uh, and it will be the kind of like an extrinsic world model is how we think about it, to see all the different simulations for all the different, uh, possible actions that will be, uh, there. Uh, so yeah, I think, yeah, but in the real, the real-world decision-making and the real-world interaction would be through the LLM. That's the succinct answer.

So one of the questions that I had for you is a lot of times in business and economics, you come across this phenomenon where every problem is unique. You only get one Great Depression. You only get one global financial crisis. Yes, there are echoes, but every problem is unique in the sense that the players are different, the timing is different. different. The markets are different. How do you train your models when this kind of the state projection into the future, if you use the historical dataset, is basically unique in every sense of the states? So how do you train? Do you create synthetic data? Do you do simulations of 100 global financial crises? How do you train? train this kind of trajectory going forward and backtest it?

So I think for us right now, obviously we are training on the datasets that do exist, but we do make the simulations of what the future states could be, right? So, so that, that you have to do a combination of both. Uh, you have to do the, uh, yeah, and the simulation and the forecasting of what the future states can be, uh, you correlate some of that with the data, with the training you already did with the existing datasets, uh, but you still have to. So I, I think the answer is not very clear at this point of time, uh, right now, but I would say it will be a combination of the two, uh, is my best guess. Uh, yeah, it'd be very, very difficult to say that this is the clear way that's going to win. But I do think that we're trying to do both right now. We're trying to train the models with the existing data that exists, uh, that we get, and we are also trying to do, uh, future predictions using simulated data and stuff. So I think it's going to be a combination of both. The other thing that I want to add over here is the continual learning piece, because the continual learning piece could be a a solution to partially what you said. Again, it's very— it's research right now. It's not like a coherent solution that exists right now because in the continual learning piece, you basically adapt to the new data that you get, dynamically adapt. So I think that could be a missing— that could be a solution to the missing piece over here. Like, how do you adapt to the new data and stuff? And but again, continual learning is very— even earlier stage research than world modeling at this point of time. Uh, we still have to agree on which kind of continual learning approach will work. Is it the model-based reinforcement learning approach? Is it other approaches? So I think it's very, very early at this point of time.

So over the last couple of years, we've obviously seen dramatic improvements in language models, LLMs themselves, and their capabilities. And to some extent, we're starting to see some levels of not really continual learning, but you can kind of fake it with a longer context window and kind of slotting those things in the context window. And not really proper planning, but again, you kind of fake it with, hey, write me a story or whatever of what happens if this happens, et cetera, et cetera, and then reprocess that. So how does this work with your own role model? Like, are these LLMs improving the capability of your role model, or are they a complement? Are they competing with— have you seen, as the trajectory of the LLM improvement has increased, as thinking models have come out, how has that changed in the last couple of years?

I think LLMs have shown enormous progress. But if you think about like the reasoning aspect and stuff, as humans we do— we always think in abstract level, right? We always have to think in abstract level, and that's something that LLMs cannot do. And this is where if you want really sample-efficient algorithms to work, uh, you need something like a world model. I think that's again where the world modeling aspect comes in. You are not going to carry like a huge— so LLM is Basically, you have to have a RAG system, right? You have to do a, you have to do a RAG system. You have to, where all the information is stored and you are not going to carry so many different hard drives with you as you walk along as a human being, for example, right? So you have to do the abstract level thinking. So I do think that there, sure, you can increase the context window. You can give them a lot of GPUs to run those kinds of algorithms and stuff, but it is not a solution. in the long term. Absolutely not. I think you need a more sample-efficient solution, and world modeling and continual learning are the missing pieces that will help you get there. There's no other way around it. You have to wait for scaling laws to apply to world modeling. Don't get me wrong, like, I'm— what I'm trying to say is, like, in 2018, 2019, no one was talking about LLMs. Like, sure, it was like some researchers I remember back in the day were doing experiments You know, there was BERT and all that that had come about. But before the 2020 scaling laws paper that came out by OpenAI, Dario was part of that paper as well. I think before that, no one was talking about it. So the same thing has to apply for world modeling in, let's say, 2 or 3 years from now. Only then people are going to be like, okay, this is much more efficient for state tracking, for And for example, instead of doing all this information retrieval techniques that we have to do with RAG systems and stuff with LLMs, this is the way to do it. I think we will have to— I will always have to answer this question for the next couple of years is my guess, until the scaling laws apply to world modeling.

So you said earlier that your goal is to have an AI CEO that can run a business and that today Obviously the AIs aren't up to that. We've done a little gonzo journalism talking to our friends at Andan Labs who are trying to do just that. And I'd say their real-world experiments are mostly not super competitive. Their cafe, their Gemini-managed cafe in Stockholm is chronically out of stock of key ingredients and things. And There's just all these sort of obvious mistakes still. But nevertheless, the trend is—

But they're LLM-based. They're pure LLM-based.

Yeah. The trend is positive, although there are some interesting results recently with the Opus 4.5 to 4.8 series and even Fable. They were able to test on too, where they're seeing that the best performance in terms of how much money did you make is correlated with what they describe as ruthless behaviour, various kind of collusion, threats to other models. There's like other models in the simulation that it will try to put pressure on in various ways. And the models that don't do that don't make as much money. So this sort of creates, I think, a pretty interesting tension for us as we go into this next phase of continued scale-up and longer time horizon environments. It's quite clear that your top-performing CEOs are going to have a pretty wide range of tools at their disposal. And even if they're like, broadly law-abiding and ethical. They're not going to be fully honest. They're probably going to be willing to engage in some deception, some bluffs. These kinds of things are just part of what it is to operate in a strategic environment. Is that a problem you are going to have to confront, or do you think that you can sort of make your world models like pure oracles in a sense, and sort of leave the ethics to the LLM makers?

I, I think first of all, LLMs— if we think about how are LLMs trained, LLMs are trained on the data that human beings themselves put into Reddit, Wikipedia, all that kind of stuff, right? And, and LLMs are definitely not grounded. There's no grounding in LLMs. It just comes up with whatever it thinks is the best, next best action is. And if you think about what the concept of world model is, it's like given a state-action pair, what is the next state? And that's why you can ground things in a world model and, and have it more correlated with the source of truth, so as to speak, what you want the source of truth to be. So I think world model is definitely one of the solutions to solve a lot of these problems that you will have, uh, because, because the way they have been built from the ground up is, is very different than how LLMs have been built and trained. So that's, that's why— that's another big reason why for AI safety and such, world models are going to play a huge part. We don't see it yet because they're still in theoretical stage yet, uh, right now, but I think over time you will see, because of the grounding aspect of world modeling, uh, you would be able to solve a lot of these issues, uh, using world models. Again, LLMs can do out the policy or the action that needs to be taken once you decide what the decision is. But, but all the decision— all the decision-making part, all the thinking part should be done, uh, by the world model. Because again, you, you simulate all the different variations of actions that you can take, right? And then you can have like a, let's say, much more super high ethical policy that, that you want to take. Then you tune your world modeling to that aspect and let the LLMs carry out the decision-making or the execution for that. That's how we kind of think of the world modeling versus LLM part in this brave new world, let's say.

Can you give me just a little bit more on how this solves the ethics problem? I might have some misconception here, but I'm imagining that we're consulting a world model and saying, what happens if we do this? What happens if we do this?

we do this.

Yes.

And now, you know, there's sort of a scope question of like, well, if I'm trained to predict, you know, our stock price, or our bank account, or our, you know, total sales, or what have you, then I might get results that suggest that like, certain ruthless actions will pay off, right? And certainly, I think that, like it or not, like, that is a reality sometimes.

But again, it's up to the human to decide, right, what the policy should be, right? The human has to decide what the policy should be, unless, unless you just let the world model— so you can tune it to be like, okay, maximize profitability without caring about like any ethics. If you want the world model to act like that, then the world model will make decisions like that, which you feed into the LLM to make the decision at the end, right? So it depends on how you want, uh, the, the state-action pair construction to be. In my, in my opinion. Yeah. It's all—

Would that mean in terms of like predicting the next state? Like, I'm trying to understand in what way would the world model signal to you that what you are doing is potentially problematic? I'm imagining like a term that's like, your bank account will be this, your stock price will be this, but also you might be indicted, you know?

Like, so it'll give you all sorts of simulations. It'll be like, it'll all sorts of simulations and variations of what the different probabilities are. And then you have to fine-tune and know, and you have to, you have to decide the variables that you want for your policy and your action, right? It depends on what you want the optimization to be on. What level and what scope do you want the optimization to be on? I think that's what you have to—

Oh, he dropped off.

Give him a second. I, I think that is—

I can hear you guys.

Oh, you're back.

I can hear you guys.

Okay, we lost you for 10 seconds, but keep going.

Sorry. Yeah, so, um, yeah, I was just saying that it is up to us as humans as to decide out of all these simulations. Again, think about like precog and in the Tom Cruise movie, right? So from Minority Report, it's up to you to decide what is the future that you want and accordingly tune the future that you want accordingly, right? So I think that's what the concept of world model is, and that's why it's a very, very powerful concept in my personal opinion.

How much data do you think companies are going to need to do this? I mean, Another paradigm that I was really into for a while, and it just doesn't seem to have panned out, is sort of continued pre-training, or you could call it mid-training or whatever. I always had this notion that you've got these big companies like a GE or a 3M that just have this vast history and all these things that they've tried. And if you could just imagine a language model, is usually the way I think about this, that knows as much about 3M or GE as it knows about the rest of the world, how much more useful that would be to those companies to have that kind of depth of knowledge baked into the weights. Now, that hasn't really materialised for one reason or another, and it seems like retrieval has kind of outcompeted so far, especially because models are advancing so fast. It's hard to get that kind of pre-training, continued pre-training cycle to keep up with just the frontier advances and general rising tide of performance. I think that's probably the biggest reason. But those companies have these vast amounts of data that make it worth it. Whereas if you're a small business, it's just like, well, put your stuff into context and it'll probably just work fine at this point, right? So how much data do you think folks will have to have to take advantage of this paradigm at all? How would you kind of map out from small business to the biggest enterprises?

Like, who gets Yeah, I think we have to first go in a step-by-step progression because I also have to be very conscious about what can be built right now, next year, 2 years from now and stuff. So, right now we're just playing with toy research environments and we have to get from that to like— I'll give you a preview of what we're trying to do, for example. So, training on like lots of CUA data, like computer use agent data, right, right now. And we, uh, are training our world models to predict, you know, in the user interfaces you have to fill out a form, click a, click a button. And right now LLMs and VLMs are not doing the best job because it's very high fidelity, uh, uh, and it's very expensive to train as well. So we are trying to do world modeling that's going to do of abstract thinking and abstract reasoning. And so that you can have much more high accuracy because you can predict all the different steps that could be in the future, right? So I think we're trying to do it for like core models. Then we're going to do it for the API models, which is like the backend side of things. So we're trying to kind of reimagine what the software will be, how software has been built kind of in an essence. using world modeling. From there, you would have to first deploy that in production systems, let's say sometime next year in 2027. And from there, you have to work with like these SMBs and stuff and try to see if the data correlate, if you can train this entire world model for the SMB data. And then maybe a few years down the line when scaling laws apply and stuff, then you can do it for like a large enterprise We're not at that stage at all at this point of time. We still— we're still in the very early stage research phases at this moment of time.

Very good. Sam, thank you for joining us here today. It's been a fascinating discussion. And we hope to see you bring the role models into every business within a short time frame.

Thank you so much for having me. Indeed.

Good to meet you, Sam.

Take care.

Bye.

Bye-bye.

One question we didn't get to ask Karina, that I was interested in is, is the solver and the conjecture generator one model, one set of weights just prompted to do different things? Because I have seen that set up in the past for self-improvement plays. You can often just have the same model, you know, it's getting better at, you know, posing and answering the questions. In itself, or you could imagine you have 2 different things. And I, I always think about the same thing when it comes to these sort of domain-specific world models as well. You know, is it going to be a separate entity that gets called as a tool and used to do these sort of structured explorations, or does it ultimately converge in the same way that text and pixels have converged, where the sort of intuition that the specialist domain-trained model has becomes deeply integrated with language and no longer has to happen through this like arm's-length tool call sort of thing? It probably varies. My guess is it probably plays out differently across different domains for probably reasons that are fundamental, like how well does text mash up with some of these other problem areas? And maybe also just in terms of what the developers and users value in terms of like traceability. You know, I I really value the fact that if I put an image into one of the integrated visual reasoners, it sees the image well enough that it can like transform it without completely redoing it. You know, that, that is huge. And I don't really need it to be separate in a way that allows me to— I wouldn't gain much from like inspecting a prompt to the language model. Because I can also just see the image and it's like very intuitive to me if the image that came out is good or not.

Yeah.

But there's probably a lot of other areas where it's much harder to evaluate what came out. And so, you might really want to inspect like how is the reasoning agent attempting to use this model. And so, the bottleneck of an actual call to that tool could be a lot more of a value add, even if maybe in some cases, you know, it sacrifices a little bit in terms of the seamlessness of the integration.

I wonder to what extent you want to have a conjecture machine which generates more diverse statements. And so you actually want it to be a little bit like wild and high-focused and high temperature rather than too constrained. And she did say that the system worked particularly well inside mathlib when you had these definitions already done. And it worked a little bit less well in econlib where you didn't have a lot of these definitions. And so without the definitions, you didn't have this kind of method of posing the conjecture within the mathematical language of the field. So it seems like it also— the other question we didn't get to ask her is how quickly they can do this. If you have a couple hundred million dollars and this much compute, how quickly can you find new knowledge which is useful and economically productive? And that's a hard one because there's a lot of math which is not economically productive. And so there's a lot of questions in the field itself in AI, if any of this kind of math proving is actually going to be economically valuable at all.

I think I'm not worried about Axiom yet. I'd say her charisma and obvious motor is, and the results that they have already today, is on an acquihire basis, probably worth it to several big companies. So I think they probably have at least one more order of magnitude valuation to go before They even really have to be economically additive in a meaningful way. I agree that it is, it's going to be hard for them to outrun the frontier companies, especially because I don't think this is lost on the frontier companies. I mean, the, of everything that we heard from her, I think that probably the least compelling to me was that they won't do it. You know, I'm kind of like, I don't know, you know, especially I can see it being one set of weights where you just prompt the model in a certain way and it just generates Lean. And I can certainly see reinforcement learning recipes being expanded to include all the same sorts of formally verified rewards that they are presumably using. So that Sounds tough to me from their position. But again, at least at this stage, I don't, you know, they're, they haven't exhausted the limit of what people would be inclined to invest, I'm sure.

I'm not, I'm not worried about them because particularly because like what ends up happening is that you have to be focused on a thing that the foundation model companies will focus on in a couple of years, but you got to focus on it a couple of years early. And then it makes more sense for them to buy you out than invest the time in focusing on that area, right? So just gathering all the pieces of data and all the people that you need in one place and having that ready to be taken, ingested by a larger firm is valuable on its own. So that's not a big deal. I think I'm more on the, like, I am more like, I think the mathematical superintelligence proving will work out and they will be able to prove many things. I'm just wondering whether proving all of those things is actually useful and economically productive on the whole, not for their company as an individual company, but On the whole, are we going to see this ramp from 2% GDP growth to 5% or 6% or 7% GDP growth because we actually have more math? Is that actually going to happen or is it just going to be a lot of theoretical stuff or a lot of simple applied stuff which doesn't really move the needle on GDP growth?

I buy it. In the sense that, and I would, I think I'm sure Karina thinks a similar thing and has probably said a similar thing, but I heard it from Vlad Tenev, who's the Robinhood CEO, who's also co-founder of Harmonic.

Yeah.

And Karina has described, has called them our competitors. So it seems like those 2 companies are, in her mind at least, kind of pursuing a fairly similar strategy. And his take was simply, if you're good at math, you're gonna be good at a lot of things. He was like, I was a math student and I found that like working with contracts, for example, later, which you might think is super far afield, is actually like pretty easy because I have these, like, I'm deeply comfortable with these reasoning primitives that, you know, the The conjunction, the disjunction, you know, the logical structure. I can just— I'm so good at that from all this math training that I can bring that to something like analyzing a contract. And it just makes me way better than somebody who, you know, may have even a little bit more exposure to contracts than I do, but just doesn't have the deep grounding in these like core reasoning moves that are second nature to me. And then he also said, and by the way, we've seen a version of this with coding, right? I mean, we've seen that models that are trained in code are generally better at lots of things. So the bet, I think, at core, in terms of an economic return, is math is an even deeper foundation that can generalise in potentially even more profound ways than code has generalised already. And where we're going, we'll probably need/want that. That makes sense to me. I think, again, though, this comes back all the time. I do believe that we are in the world where not anyone, but great ideas can come from anywhere. Breakthroughs could come from anywhere. There's still plenty of opportunity for academic groups, NeoLabs, you name it, to have a new big idea that moves the needle. But who's going to capture the value of that idea? That's where I'm much less sure because it's like, okay, maybe we find a way that this deep training on math. And I actually have notes from years ago where I was like, what if we just pre-trained on tons of math statements, you know, and just, just tried to, as kind of a try to create the circuits in the model to do logic well. And then, you know, train all kinds of stuff on top of that.

Yeah.

You're still gonna need all these other layers too, though, right? I mean, you, it's, it's not to say that you can take the pure math model and go do the contract. Like in Vlad's brain, he also had like a, you know, a broad education. So I do sort of struggle to see how, as these things become known, and we've seen over and over again that even just existence proofs are really powerful in the AI space. So they prove that it can be done. They maybe don't tell all their methods or whatever, but maybe they get acquired. Obviously, that's always one outcome. But if the companies decide we want to incorporate these ideas, it sure feels like they have an inside lane always, you know, to actually recognizing or realizing the upside of that because they've got all these other, you know, abilities already there waiting to be improved. Whereas if you just are, you know, focused on math, then like you've still got an awful lot of work left to do to make that logical reasoning actually useful in the zillion domains that people are using the frontier models already?

So in terms of economic growth for the entire space, as an investor, I've just been trying to figure out how this would work. And the best that I can come up with And to the limits of my imagination is that if you have mathematical superintelligence, you get better algorithms immediately for cryptography, compression, and a couple of these things that have broad application immediately in computer science. And immediately you get upgrades, massive upgrades in those things. And so for compression, for example, if you get a new compression algo, which is much, much better, all of a sudden your memory usage for your models becomes much lower. And all of a sudden you manage to unhobble the models at the current data center footprint dramatically. So that would be something. Cryptography, more secure communications would also lead to probably higher bandwidth. You could start to split bandwidth a little bit better, similar to how we had the transition to DSL in the late '90s, where you had the single phone line all of a sudden multiplexed into multiple bands that you could send more data down. So these were the 2 obvious things that occurred to me that if you had mathematical superintelligence, you would see. And I wonder I wonder, because they look close. I mean, the lean stuff seems to be working, right? So they look close. And at this point, if you have the lean stuff kind of working and you have raised the money, you can now deploy at large scale and you start to do 100,000 machines at once. Do you get some results that start to move the needle on some of these problems? That is the question for me in the next year or so. And the implication, because the implications could be very, very large for investors, very, very large.

Yeah, totally. I think we're sort of seeing this, I think, already with chip prices, if I'm picking up your meaning well. the fact that an A100 costs more today than it did 2 years ago is a strong sign, I think, that we're going to see value pricing on compute, right? Value pricing on inference and the value's gone up. And so the prices are being bid up and under the hood efficiency gains, you know, compound, like every time you can just make it 1% better, that's just like 1% more inventory that you can sell, which seems to be at least for now, like more and those and there's been tons of those, right, since the A100. So it's not like we are running the same, you know, obviously, we're not running the same stuff, but it's also just like a lot more efficient in any number of ways. than it was when the A100 first came out. So it does seem like one kind of limit implication of this would be we might see AI outputs bid up to the level that it would cost to get a human to do them at some point, as everybody realises that There's 1 to 2 orders of cost savings on the table for all these kinds of tasks, and you can go buy your own time back or fill roles you couldn't otherwise fill, whatever. One equilibrium for a while might be it ends up costing very similar to get AI to do a job because it's all used, it's all bought out. And it maybe kind of settles. Now, maybe that's also a little weird in the sense that, of course, the AIs are going to be, given the jagged capability frontier, maybe that works out. Not that they're the same, but some things remain cheaper and other things are just inaccessible. I don't know. But I do think we'll see that trend probably You maybe save less. That'll be a— it'll be a weird narrative if we start to enter into a time where, and of course people will learn the wrong lesson from it, but I can totally see a moment in the future where costs are starting to go up. The savings that you're getting is actually decreasing, and people are like, oh, this AI thing is not really panning out. Look how much we have to spend. And the answer is, well, it's being bid up because there's so many high-value uses. for this, that the use that you're going for is, it's just not a great savings ratio compared to some of these other areas where there's a higher savings ratio. So I guess, yeah, as I think about this more, it's like it wouldn't be, it won't be for everything, right, where you'd have no savings ratio. But you could imagine some things where you maybe have no savings ratio, or even AI could cost more because there's just so many high-value uses that the underlying resources are being bid up to the point where some things are not economical for AI to do anymore, which will be weird and confusing if it happens, but I wouldn't be shocked.

The funny thing is that Waymos cost more in SF than normal cabs because number one, there's still a shortage of Waymos. There's not enough Waymos to go around. And number two, the Waymos are rated of higher quality because you don't have to deal with a driver. And it seems, um, you know, a lot of people don't want to deal with a driver when they're, when they're taking a cab. And so, uh, you could, you could kind of start to see parallels with that in perhaps the other verticals.

Yeah, that's a really, um, great example, I think, of at least a version of that phenomenon. The fact that Waymo is more than a Now, Tesla FSD at $99 a month for unlimited hours is definitely a lot less, or at least it can be if you drive any significant amount of time.

What did you feel about our second guest? I think Donnie was— I think as a law professor, you got to be shocked at this administration. You know, I often say the administration sometimes has its heart in the right place, but they have terrible lawyers. And they could have done a better job of even the export controls, but just, you know, the lawyers were not exactly the best lawyers. And, you know, these guys don't really care. They're like, okay, it's done. I don't really care which legal method we used. You know, the model is blocked. Let's just move on. So I think, I think Donnie was a little bit shocked, but, you know.

Yeah. In going through my son's cancer, uh, journey, there were a few times where we talked to a doctor who specialized in some particular part of the body, and I was really struck by how much love they sometimes have for that part of the body. The one that put in his little, um, med port that goes into the you know, the big vein in the neck. I think it was the VVK that goes into. Um, he was just— couldn't help himself. He's just gushing about the vein itself. He's like, you know, it's, it's such a beautiful plump vein. You put this in there and it was just, you know, you could tell he loves it. And the kidney doctor is the same way. The kidney's such a beautiful organ and it's just so good at what it does. Um, and I feel like a lot of these, you know, people that become law professors, they have sort of a similar feeling with the Rule of law, and this is like not at all what they signed up for. And it's probably yeah pretty disgusting to them in a lot of cases where we had we're like in the process of just actively fumbling the bag of our our civilizational inheritance and what can be done about it is very strange. The other thing I think is strange too is you hear. Like, one thing that has not happened much in the Trump administration is they haven't really ignored the courts too much. So I find it a little odd that they do these things in such a slapdash way sometimes, where they know it's not going to serve them well when they actually get into legal proceedings. And then it seems like, at least for now, they're like, they feel themselves to be bound by the results of those legal proceedings for the most part. Right. I mean, we, we haven't had that many moments where they just sort of say, I don't care what the court says. There was a bit of that in the, in the sort of deportation.

Exactly.

The peak of that cycle, there were people that were like ordered to not be sent that were sent and ordered to, you know, bring back that they were like, well, you know, what can we do? So there, I won't, I don't wanna say there was none of it, but my general sense is still on this sort of thing. If Anthropic goes to court and gets an injunction, you know, I think they'll feel like that is the, you know, that is the outcome. And I, I think that they will probably respect it. But then you would wonder, why not tighten it up a little bit before you send these letters in the first place?

So I see this not as similar to the deportation stuff, but similar to the tariffs. They kind of used various authorities to kind of implement the tariffs over the course of like the year, year and a half. And by the time they had been completely thrown out of court, they'd also kind of made it very clear to business partners everywhere that If you continue importing, you are at risk at any time of being cut off. And they'd kind of already brought that across so strongly that every company had started to migrate its operations, migrate its production into the US, even if it meant another 10 or 20% additional cost, et cetera. And so they'd kind of achieved the economic gains while The economic war, they'd kind of won it while losing the legal battles. And I think to a large extent, that's the same right now because it's already very clear that there's now a licensing regime. OpenAI can't put out its model without going to the administration and asking for permission now.

Right.

They can't put out GPT-5.7 or GPT-5.8 or GPT-6.0 Because the first thing that anyone will do is they're going to repeat the same test on the model and the model is going to be able to break, right? So I think they've already achieved the war goals and they don't really care that much about the battles, the legal battles. The legal battles someone else will sort out and whatever happens happens, but we've gotten what we wanted and it's clearly established now that there's now a licensing regime, and you're going to have to come to them and get permission before you release. And that's, I think, also what Dean Ball really hates about this whole thing, because you've established this kind of soft licensing regime, which eventually might end up in the hands of the Dems or someone else or whatever. And sure, you trust yourself now, you trust your own side now, but at some point the other side will have the knife in their hands. And, you know, the entire policymaking apparatus will flip. So.

Yeah.

I don't like it. I don't want to laugh here, Pete, but I guess, you know, this is the same attitude that you hear from people at the frontier companies increasingly. It's like, you know, it's we didn't choose the war, but here we are, and all we can do is do our best to get through it. And I I feel that way about covering these lawfare issues. I don't really want to mess with this stuff, but you know, it demands our attention. So we'll keep paying attention to it until order is restored. And I'm not so sure on the over take the overrunner on that one. I think it will. be at least into probably early 2029.

To what extent do you think they can— this was in Leopold's situational awareness, the expectation that you're going to get a US-only team, national team that builds models out in the desert, a real Manhattan Project style where they say, look, we have to have a separate and national model. This is of national importance and we're going to set this thing up and we're going to call these people and then we're going to ban all foreign people from working on these models, et cetera. To what extent do we think that will happen in the next year, 18 months maybe before the 2029 timeframe?

I sure hope it doesn't happen, but I can imagine it happening. I think the culture of frontier AI research is in some ways, you know, very incompatible with like military discipline.

discipline, right?

Yeah.

Like we have the famously pink hair libertine, polyamorous, whatever, right? All those kind of cultural dimensions have at least a foothold in the AI research community, if not more. And I certainly don't think people are keen to leave the beautiful Bay Area and move to you know, an undisclosed location in Nevada where they—

Albuquerque, New Mexico—

may or may not have, you know, ability to communicate with, you know, their friends and family in a, in the way they might like, or even, you know, in the extreme cases, like, may not be allowed to leave the facility. And yet, I think a lot of— I think enough people would sign up for that, that they would be able to build the team. If you just went desk to desk at certainly Anthropic and OpenAI and you were like, this is happening, do you want to be a part of it or not? Especially if it was going to be coupled with, by the way, you can't do it out here anymore. You either are able to continue doing frontier research in this way, Or you can't anymore. I think a lot of people would make a lot of compromises and a lot of sacrifices to get into that bunker environment. Just desire to be part of it is so strong. The identity that people have around being a part of this process, this story, this, you know, moment in history.

Mm-hmm.

I think a lot of people wouldn't know what to do with themselves if they didn't have that job in some way, shape, or form, right? And not to say that they went to their specific role at their specific company, but the idea that they would be not involved in a live player project, I think for many of them would just be like, I wouldn't know what to do with myself at all. And so, yeah, you could probably get a lot of people willingly giving up a lot of niceties in life to be part of, you know, whatever underground sprint you might want to put together. I still hope it doesn't happen, to be clear, but I don't think it will be. And, you know, again, in the extreme case, would international researchers, again, if they're faced with the choice of you can't really do this anymore, or you can do this, but even under even more draconian restrictions, like you can't communicate outside to anyone at all, you can't leave, you can't communicate, you're just underground, literally or figuratively, until further notice. I think a lot of them would— I think a lot of the international people would probably also maybe Less, you know, it sounds worse, but I think you'd still get quite a few takers. I think there are a lot of people that are just very— would have a very, very hard time prying themselves away from the science project.

Yeah, it's too, too important. It's probably one of the key moments in history.

So, and they've been all in on it for years.

Mm-hmm.

So that's the other thing. It's not like— that may— what I'm saying may sound strange, and I don't mean to suggest it applies to everyone, and I don't know what percentage it would apply to. But if it sounds like really weird, like who would sign up for that, you got to keep in mind that a lot of these folks basically have no life anyway, you know. They're not— and again, there's broad brushstrokes, you know, all the caveats apply. But you do have a lot of people who are like, thinking about nothing but this already, you know, who are not calling their parents, you know, all that much already, who are maybe not dating, you know, much at all. They're just kind of already locked into this, like, this is all that matters. I don't really have time for anything else. I was speaking to somebody at Anthropic who was like, I honestly said something very similar to what I heard Zelensky say in the last 24 hours. He was asked, I forget exactly what he was asked, something like, you know, what do you miss or whatever? And he said, I miss being a good father. And this person at Anthropic said— this was like a month before the Zelensky quote— said, I miss being a good friend. You know, I'm a bad friend now. And it was kind of like some regret, but not the sort of regret that I'm making the wrong decision. It was just like, this is again, you know, I said that you sound like a world War II era mentality. And they were like, yeah, that's, that's how a lot of us feel.

Wow.

What a time.

Yeah, no doubt.

No doubt.

Should we call it there for today?

Let's call it there for today. Nathan, always a pleasure.

See you tomorrow, Prakash.

Cheers.