Approaching the AI Event Horizon? Part 2, w/ Abhi Mahajan, Helen Toner, Jeremie Harris, @8teAPi

Hello, and welcome back to The Cognitive Revolution.

Coming up, you'll hear part 2 of a marathon live show that I co-hosted with my friend Prakash, also known as @adapie on Twitter, in which we explore AI for science, recursive self-improvement, and geopolitical competition. I love doing full deep dive episodes, but I can only cover so many topics in that way. And so I am experimenting with higher intensity live shows as a way to deliver what I hope is the same high-quality analysis, but in a denser format. In the first half, which hit the feed yesterday, we talked to Professor James Zhou of Stanford about his work on AI for science, Sam Hammond about how well the US administration is doing to manage international AI competition, and Shoshana Tchaikovsky about AI agent behavior in the wild. In this second half, we talked to Abhi Mahajan, also known as Owlposting, about AI for biology and medicine, Including the foundation models he's building at Noetic AI to better predict which patients will respond to which cancer treatments, and why, though he's skeptical of many AI for biology results that have been published to date, he does expect trends to continue to the point where AI is ultimately transformative for the field. Then we talked to Helen Toner about a report that CSET just put out called When AI Builds AI, which summarizes conversations from a closed-door workshop in which participants tried but failed to establish any consensus expectation about the impact of automated AI R&D, ultimately leading to the conclusion that automated AI R&D is simply a major source of potential strategic surprise. Then finally, we have Jeremy Harris talking about the very challenging position we find ourselves in, where we lack both the technical means to reliably control superhuman AI systems And the trust and coordination mechanisms needed for the US and China to address this problem collaboratively. Plus a bit of discussion of how he maintains situational awareness and how our respective personal productivity stacks are evolving. As you'll hear, the challenges of making sense of such massive disagreement among leading AI experts and simply keeping up to date with AI developments coming at us daily comes up repeatedly in these conversations. And to be real, nobody seems to have perfect solutions. One partial solution that I can recommend though is using large language models to help identify blind spots. And for that purpose, I am really enjoying the Blind Spot Finder recipe that I recently created on Granola. Granola works at the operating system level so it can capture all of the audio into and out of your computer, including, if you wish, the contents of this episode. And its recipe feature can work across sessions to identify trends, opportunities, or yes, blind spots that only become apparent with that zoomed-out view. Obviously, this is a tool that grows in value over time, but if you want to try it today, I suggest downloading the app, starting a session while you play this episode, and then asking it to identify blind spots based on this conversation. What is so cool about this feature? for active Granola users at least, is that the blind spots it identifies for you will be different from the ones it identifies for me. As I said last time, this was fun for me, but especially because it is a new format, I would love your feedback. Do you feel that you got as much value from this more time-efficient approach as you usually do from our full deep dive episodes? Or did we miss the mark in some way? Please let me know in the comments. Or if you prefer, by reaching out privately via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. With that, I hope you enjoy the Cognitive Revolution Live from February 11th, co-hosted with Ada Pi.

I'm going to add Abhi Mahajan. Uh, Abhi is owl posting, uh, online, and he, uh, works on AI for cancer at, uh, Noetic AI. Abhi, welcome.

Great to meet you, Vedant. Thanks for having me on.

You have the great distinction of being recommended to me as the Zvi for AI and biology and the intersection of those two. So big shoes to fill, big reputation to live up to, but excited to meet you. This is actually the first time we've properly spoken.

I learned from Ron Alpha that you built an entire competitive intelligence platform, LLM-based, to feed the clinical analysis pipeline. And also that Claude recommends every cancer drug it sees. So, let's talk about that.

Yeah, the typical way that a lot of— increasingly, a lot of biopharmas are interested in asset acquisition as opposed to just developing their drugs from scratch. This is partially because China's pumping out a lot of very interesting preclinical assets. Why not just buy those for a few million dollars? They've already done the optimization. Let's just run those in patients. Most of the time, the way you look for these drugs is either you mine your personal network or you have these clinical trial aggregation platforms that help you do the job. Both of these are obviously lossy and a better way is just scrape the entire semantic web yourself and annotate every single investigational drug you find with your company's priorities, what you think is important to look for, modalities that you're particularly interested in, organize that all into a table, rank it by some metric, And then you give that to the therapeutics team to work off of. Obviously, there's still a human due diligence step. These models still are not perfect, even 5.2, 5.3, not perfect, but it's pretty good.

Do you have an internal eval that you run? And when you swap model engines regularly, do you upgrade every time a new model engine shows up? Do you evaluate and then decide?

It's a pretty hacky process. Our metric for, at least my personal metric for evaluation is like amongst the drugs that our therapeutics team are really interesting and like want to move forwards on, does the next version, next generation of the LLM continue to recommend those drugs as these are like very good? And I don't actually think it was pretty good at the very beginning. I only built this like pipeline a few months ago. It remains pretty good now. I don't think there's been like any dramatic jump. I partially think this is due to like identifying what makes for a good drug is a very like qualitative process and a very vibe-based, like, it depends on the economic status of the company. It depends on like, do we know anyone there? Because oftentimes these companies don't make it easy for you to give them your money. It takes a super long process to figure that out. Yeah, it's pretty good though.

So I definitely recommend your blog, owlposting.com. I've still got quite a bit of archive to work my way through, but I want to throw a couple of what I thought were kind of your more interesting, arguably hot takes, at you and then get you to kind of double-click into some of the intuition and implications of those.

Sure.

One, because we're obviously in a moment now where there's a tremendous amount of interest in creating AI scientists of all kinds. And one of the big bets that companies are making with some serious capital behind them increasingly is that they're going to close the loop by allowing AIs to design and run their own experiments through some sort of automation. feed that data back in, and they're going to get reinforcement learning from basically experimental results. Now, one of the things that you had said in one of your posts is that there's not a lot of verifiable ground truth in biology, and I would love to understand what that means exactly. And then what does that mean in terms of the ability to close that loop? Is there some sort of fundamental messiness or uncertainty that you see as, at least in the near term, being irreducible that would become the functional limit on how much systems could learn from that kind of closed-loop experimentation?

To say that biology has no verifiable ground truth is probably a little bit hyperbolic on my end. But what I will defend is that there's not a lot of verifiable ground truth for the most clinically valuable problems. Yes, there is verifiable ground truth of, does this protein exist in the solution? Is this variant that your NGS sequencer identified true? Those are both verifiable. But I don't think you'll quite see the same explosion of intelligence that happened in math and code as you will in biology because rewards are so cheap and so easy to get in those fields. In biology, it's such a long iterative process to get any iota of information. One easy analog to this is training an RL/VR model on the task of writing the best-selling book. There is technically a verifiable reward. There's book sales, there is the country that the author is writing from, all these sources of data. But it takes 18 months to get that singular data point. And when you get that singular data point, it's very hard to trace it back to any one of these things. One biology-grounded example of this is Let's say you want to do RLVR on toxicology prediction. This is arguably the thing that sinks the vast majority of phase 1 drugs out there. Toxicology sounds like a very simple topic. It does not. A drug can be toxic on the order of seconds, like snake venoms. It can be toxic on the order of months, years. It potentially doesn't kill an animal. Maybe it just leads to cognitive deficits, heart damage. Oftentimes it's dose dependent also. It could be species dependent. All these measures of toxicity, there's no real way to understand them other than just observing them in vivo setting and then seeing what the readout is. There are companies like one SF-based startup called Axiom, which is trying to create a model that can very easily tell, given the small molecule, what is its toxicity impact on hepatocyte cells. It's a very clean, simple problem that probably saves months of time in preclinical settings, but it doesn't poke at the much more important problem of how does this perform in an animal.

Just a segue here. Isomorphic Labs, I think yesterday, announced they have a predictive model which doubles the performance of AlphaFold 3 on key benchmarks, binding affinity, pocket identification, structure prediction. How does that fit into how things go? Is this actually useful or does this just create more targets which need to be validated anyway and it's not that useful?

Yeah, obviously, very incredible piece of work, IsoDDEs. And I'm no longer in the protein engineering field, but I think the benchmark they did, that leftmost plot they're presenting on, that's an incredibly difficult benchmark to get better at and they're 2x better than what was previously done. So, very good. But, I'm sure you've heard the sentiment that the field is already awash with many really good preclinical assets and the bottleneck is actually how well do these work in patients. It sounds perhaps obvious that if you get better at this preclinical design step, you get better at putting it into humans. That's the story that has been told for 10 years. It is not obviously clear that any of it has borne fruit. I imagine at some point it will, but there isn't really strong evidence suggest that it does. There's actually this really great paper, a chemistry paper that came out just a few days ago called The Affinity Advantage. That paper is probably one of the strongest bull cases that being able to optimize every facet of every protein that comes into your preclinical pipeline has nonlinear or superlinear benefits to the drug development process. And it's just a matter of time until these models get even better. even better. It's not an opinion I share, but I'm sympathetic to it.

In Dario Amodei's, I think one of his papers, the blog post that he had, Machines of Living Grace, I think, he tried to kind of map out how he thought developments in biology would work. And he kind of pointed out that a lot of the major developments in biology come from better kind of imaging and sensing techniques that allow you to kind of like look deeper and understand deeper on what's happening in there. And then after that, it becomes easier to do a lot of other things downstream of that, starting with microscopy, which led to all the downstream developments from there, and et cetera, et cetera, et cetera. What do you think are potentially the developments which might be coming up in the next 4 to 5 years that might do something like that?

I guess I would vaguely gesture to building generative models of human in vivo biology. There's layers of discussion to be had. What other instruments do we need? What other modalities do we need? But I think there's a lot of low-hanging fruit in simply collecting a huge amount of highly rich data collected from real human tumors, intestinal lesions, plasma readouts and just feeding a model with that information and not paying attention to any of the in vitro or otherwise biologically unrealistic settings. And then from there, maybe you get access to a genuine bona fide human simulator of biology. And maybe that's a lot helpful for fixing the current state of 97% of oncology trials fail. I think that the Dario pitch of scientists in the data center churning out interesting ideas I think there's already tens of thousands of PhD students churning out very great ideas. Most of them can't be validated because it's too expensive to do it.

Hey, we'll continue our interview in a moment after a word from our sponsors. So that connects pretty directly, it seems like, to what you are doing in your work on cancer at Noetic, right? You guys are focused, first of all, at roughly the clinical stage. and try to predict what drugs will work best for a particular patient, given some relatively deep data about their specific condition. Right. So maybe walk us through what that looks like. I was interested to learn that it is basically a foundation model with lots of different data sources thrown into it. And also, that it sort of is trained with this kind of masking strategy, where the idea is that the model has to learn how to predict from partial data, whatever partial data it might have. And I'm a big believer in that strategy because there just is so much, obviously, so many modalities and so much noise going on inside the system that we don't understand. I've been a big kind of speculator about that. being a driver of how AI can help in health over time. So give me the kind of double-click past what I have been able to learn with online research into what you guys are doing.

Yeah, so let me start with perhaps the economic pitch for Noetic. 97% of oncology trials fail. And you can look at that and say, well, we're awful bad at designing these drugs. Maybe we should get better at designing them. But one interesting phenomenon is that if you look at a lot of the papers that are published after a cancer clinical trial fails, there's usually some patients who did respond to the drug or respond to the regimen they were on. And the researchers try really hard to figure out what is the exact biological archetype that makes up this response population. And they always come up with something super complicated, very heterogeneous. It's like this particular cytokine group or granzyme genes were highly expressed in the response population. It never leads to anything particularly interesting. And so, one argument you could make is that maybe the biomarkers that define patient response for this particular drug is non-human legible. You need a black box biomarker to encapsulate whatever that piece of information is. And so, Knowethic is kind of built around that thesis. We collect vast amounts of human tumor data. We profile them at 4 levels of modality. So, pathology, which is kind of like the blue chip that almost everyone has, spatial proteomics, a 16-plex panel to identify cell types, Whole-plex spatial transcriptome. So this is 19,000 genes over the entire surface of a tumor meant to identify functional state of the tumor. And then exome sequencing to identify genetic alterations. So is this KRAS positive? Is this STK knockout? And so on. And then the ML angle is that you train— yeah, exactly what you said, a self-supervised mass model in the hopes that one, you get a very good representation of any given tumor that walks into the door. So you now have the ability to place In the universe of all the cancers I've seen, where does this patient fall in that embedding space? And so that's what, like, we are doing a lot of. We gather patient samples from, from, like, people who have ran clinical trials. They have patient samples. We profile them. We run that through the model, see if the response population falls in a different area than the non-response population. And if it does, maybe we have access to a biomarker that no human on earth understands, but we uniquely are able to. The more interesting thing you can do with this is use it, use the generative capacity of the model and knock out specific transcripts or specific genes and see how that changes the expression of transcripts within the tumor microenvironment. You can imagine there's this like concept that's appearing in the cancer literature called nudge drugs, which are drugs that don't actually operate on the, like, the immune system or like really the cancer, the cancer site itself. But rather push it in a direction that makes it more sensitized to other drugs. And so you can imagine, oh, I'll knock out this particular transcript and then I will hallucinate what would it be like if I added Keytruda, which is an immune checkpoint blockade that operates on the PD-1 axis, into the site of the tumor. And maybe now you predict, oh, the tumor's highly inflamed. It's hot. There's a high chance that it'll just melt away entirely. Yeah, those are the 2 big economic and ML strategies we're pursuing.

Yeah, that's really super exciting. When you talk about, first of all, identifying or having access to, I think was your phrase, biomarkers that nobody else has access to because you can see a sort of divergence in where different patient populations fall in embedding space. Do you have any means? And if not, maybe I can introduce you to the good folks at GoodFire who just did a version of this with identifying biomarkers for Alzheimer's that had been previously not identified. But do you have any means right now of saying, okay, because it's one thing to say these things are falling, these patient populations are falling into different parts of embedding space. It's another thing to then say, why? What is it actually that is causing that divergence? How far are you guys along in terms of being able to make interpretable what it is that the models have learned from their—

Yeah, the PrimaMente and GoodFire post was very interesting to read. We do have a MechInterb research group internally which is exploring these ideas. I have no doubt they'll find something interesting. But one argument against doing this at all is, why do we care about interpretability? In a clinical setting, we might care about interpretability because the FDA gets very upset with you if you try to do anything that's black box. And maybe that was true a year ago, but circa September or August 2025, there was a pathology AI company called Artera AI which came up with what is basically a companion diagnostic. They intake in the pathology slide of your prostate tumor and they will predict whether you will respond to androgen deprivation therapy. They have no idea why this model works. They've retrospectively validated it on thousands of patients from prior phase 3 trials. And the FDA was fine with that. So one argument against doing backinterpretability is that why spend a ton of resources exploring something that the primary regulatory agency you care most about doesn't really mind whether or not it's white box or black box?

I guess the obvious answer would be because presumably that knowledge would be a great input to further experimental ideas or other knowledge. But maybe you think it's just so hard to I don't know, there's no verifiable ground truth or something that would prevent that from working?

Yeah, I guess what was discovered in the Pyramentic Goodfire post, I forget what exactly it was, but it was something about fragmentomics, something about how the genes are fragmented, is a potential biomarker for Alzheimer's. It's a very interesting piece of work. It sounds very expensive to validate it. And so I imagine we would run into the exact same problem. We have a very good hypothesis for what comes out of the system, but we already have so many other hypotheses, potentially ones that even have higher literature backing. I could imagine a world in which MechInterp as a field gets so good that you can triage. This thing's going to be really easy to validate. This thing's going to be really hard to validate. But right now, the way that MechInterp usually works in biology is you're staring at semantic segmentation plots a lot and trying to think, is this real or is this fake? Is this the model identifying some very spurious correlation? And that time just simply feels better spent elsewhere.

Interesting. Here's another idea of a place that it might be well spent. Continual learning, of course, is a huge theme right now in AI in general. The first conversation we had today with Professor James Zhao from Stanford included a little talk about their recent paper, Learning to Discover at Test Time, where they're using here autoregressive large language models and giving them problems like make a faster CUDA kernel, or find a better solution to this math problem with a lower bound than anybody has previously found, or find a better solution to this math problem with a lower bound than anybody has previously found. And they interestingly kind of flipped the usual model of what we're trying to do when we create an ML model on its head and said, what if we just try to get this model to produce the single best answer that we can and we don't care if it generalises? And in fact, we'll probably throw away this model after this test time fine tuning. What we want is the answer. And they were able to find at relatively reasonable cost, like $500 compute cost, that they were able to actually get some new state-of-the-art answers on some of these highly technical questions. If I'm a cancer patient and you've got a general foundation model, a question that naturally occurs to me is, like, can you fine-tune this on my data? Can we do some test-time tuning? Can we do sort of intensive masking on just my samples and really dial this thing in to understand my particular biology? And then would it be If we did that, would it be more accurate for me? Does that line of thinking have legs? And why or why not?

I actually looked at the paper and they actually have a section for biology. They do single-cell RNA denoising using this test-time training model, which I thought was really interesting. I guess my instinctive answer is it's an interesting idea. It very well might work. And it falls into the bucket of ideas that we would simply have to try it to make sure that it does or does not work. The results for single-cell RNA denoising that's in the James Xiao paper are certainly good. They're better than the state of the art. For each one of these case studies, they attach a note by an actual domain expert saying, how useful is this in practice? And the domain expert in question for the single-cell RNA section did say, this is very cool, but at the end of the day, we don't really care about the results of single-cell RNA denoising. We care about some biological utility that is underlying that. And so maybe you get better at solving this verified task problem, but that doesn't translate to anything actually useful. Maybe it would be for the response/non-response prediction case, but it kind of just sounds easier to fine-tune the model using normal supervised learning. Why go through the RL process if the end result is binary? I think they even call it out in the paper, the setup isn't really meant for binary or sparse learning tasks. It's meant for fuzzier things.

Yeah, they're working on that, but it's not done yet. I guess maybe zoom out and you kind of alluded earlier already to this idea that A lot of people think we just need better ideas for drug candidates. And your consistent position is like, that's probably not really the bottleneck. You made a really interesting point around how the ability to evaluate, more accurate ability to evaluate those candidates drives a lot more value than just throwing a lot more candidates through a pipeline. The quality of the pipeline matters most, more than its scalability. So again, I think you kind of have suggested where you think this can come from with just large-scale foundation model style training. But give us the next level of depth on that. Why are all these other ideas not so exciting? And is this basically just a bitter lesson sort of idea where all your cleverness is going to be washed away by Scale. And so keep your eyes on the prize. You got to data max and compute max until you solve it all. Is it, is it kind of that?

I guess I kind of view things in like 3 ideological camps. The first of which, like, like, like the first one is like maybe us, like we're, we're indexed very heavily on like human data is the only thing that matters. You can't start from like in vitro settings and bootstrap your way up to something more complicated. You need to start with the most complicated thing to begin with. The second camp is very interested in modeling single biomolecules and their interactions in the hopes that maybe you can't bootstrap your way upwards, but you raise the absolute success rate from maybe 5% to 20% and maybe that's all you need. I think the second camp defines the vast majority of ML bio companies that exist today. I think some of them have clinical candidates that are ongoing right now. We'll see what the results are. I think generally it doesn't seem like there has been a massive step change in their ability to design drugs. This is not knocking them. Drug discovery is hard. Everything's a bet at the end of the day. The third camp is maybe it's not really a for-profit thing, but you can just improve the clinical trial process to begin with. And this is arguably the path that China— this is China's main advantage. They're able to run clinical trials far more cheaply than anyone else, partially because of lower cost of human labor, but also because they've just set up the system pretty nicely. such that it's not such a huge regulatory and financial headache to get things going. This has some downsides. Drugs are treated innocent before proven guilty. The FDA is the other way around. But the obvious benefit of doing that is you're betting neither on the AI in human data getting better or the AI in vitro settings data getting better. You're trusting that the typical drug design process, if just made slightly more financially efficient, will improve things on its own. I think all 3 of these are important. And I think it would be, uh, probably like grandiose of me to like assign an unequal weighting to any one of them. Each one feels important to push on.

There've been some interesting— oh, sorry.

What were you thinking about from— I'm going to take a little bit of a segue to something you said earlier, which is that, um, a lot of the new INDs are coming in from China. Like what is, what has happened in the last, like couple of years? Is it an AI thing? Is it— the CEO of Ginkgo Bioworks was on TVPN yesterday. He said they just have more hands. And some people believe it's a regulation thing. Some people believe it's a clinical trial registration. They can just register more people. What is driving— some people believe it's a US cost thing. What is driving this transfer of basic R&D to China at this point?

I, I, like, I, I think like this particular subject is very deep. It's not something I have expertise in. I, like, my, my instinctual thought is that there are many different answers to this. And the one that I'm most like, I think is most interesting is the idea that like China was always like, was always like, like a very good generics manufacturer. And that's like, that's where they started. Slowly they extended their way into like Wuxi having a very good CRO ecosystem. And then at some point, enough talent began to be incubated in China where they began to realize, we have all this infrastructure here, why not just develop our own drugs? And there is something very important about having this very close interplay between both the person who is designing the drug and the person who is actively doing wet lab assays on the drug. Whereas in America, you have a super long feedback loop of I need to get my SAV together. I need to go reach out to VCs. I need to go buy a lab. Whereas in China, that ecosystem is a little bit set up already. Actually, maybe the only missing part is that the VCs are still not super— they're more risk-averse than perhaps VCs in America. But the co-location of the grunt work and the intellectual work is actually surprisingly important. A few months ago, actually last year, I interviewed one of the very few people doing novel biotech research in India, a guy named Soham who runs a company called PopFacts. He said this is the primary reason why he expects not only China to start producing really interesting drugs, but also potentially India, potentially Egypt, places where there is intellectual capital, there's a lot of hands, and it's just those combinations lead to really good compounding results. Indeed.

And does that accelerate with the AI models, this kind of AI co-scientists? Does that mean that even if they don't have that much intellectual capacity yet, They have the hands to carry it out.

I guess this is something that's a little bit opaque to almost everyone as to what exactly is the level of how impressive are the bio AI models coming out of China. I think there's certainly some interesting work that has been done. It's not clear to me that there's anything radically new there that won't be found anywhere else. A fair amount of it is I don't want to say this in a disparaging way, but it is scaling up stuff that was originally developed in either the UK, London, or America. There hasn't been really a deep-sea thing where there's something radically crazy that comes out of any of the Chinese labs. I obviously could be wrong on this though. Whatever the bio AI labs are doing in China, there's much less American visibility around it.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Okay, go ahead.

One big question I have is kind of, I find it very hard to calibrate myself on how excited I should be about all these AI for biology and AI for medicine developments. I know that there's always these kind of headlines. AI does this, AI discovers this drug. I've done episodes of The Cognitive Revolution on it, one with Jim Collins. He has created a bunch of antibiotic candidates. There's a long list, right? Professor Zhao did the nanobodies thing that came out of the virtual lab. To hear him talk about it earlier today, it sounds like those were reasonably well validated. But then you always get this other side too, that's like, well, not so fast. It's all very messy. We got a long way to go. Most of these things don't pan out. And I feel like that sort of parallels the debate that we hear in a lot of different domains where Even in programming, which is one of the more, let's say, legible domains, we've got something like a METER study that showed slowdown of developers, and that was very confusing. I'm still quite confident that it's making me faster, and I kind of want to throw that away. And there's, of course, just a lot of denial and cope out there and all sorts of motivated reasoning. How should one try to ground their worldview? Obviously, subscribing to owlposting is something everyone should do. But what else would you advise me? How can I patch these blind spots in my worldview or get to a better position from which to have my own sense of what really counts or what really matters and what doesn't? Because, again, this happens all over the place where there's this disagreement, even among some of the most informed people, about just how big— can AIs reason? Or how big of a deal is it going to be? But in biology, it's particularly hard for me to make sense of. So I'd love to get some tips for how to climb the learning curve faster.

I've actually written about this in the past, a very long time ago. The title of the article is 5 Things to Keep in Mind When Reading Biology ML Papers. The long and short of it is the evaluations in biology, I think, are very difficult. And so the vast— like you see this thing about like, In more typical wet lab biology of, oh, we cured cancer, but it's in a mouse. Who knows when it'll actually translate to humans? There's a very similar phenomenon in a lot of biology ML papers where they're doing something that feels like it should be useful, but there's a lot of things that they're probably hiding from you when explaining the results that would only be obvious to a domain expert. One, I think, really funny example of this is small molecule binding affinity papers. I've written about one company's work in this, but they found that these— let's say you're able to predict these set of molecules bind to this target, these set of molecules do not bind to the target, and you're very happy with yourself. You publish a Nature article about it. What these folks at a company called LeashBio found is that this can often be confounded by which chemists actually produce the molecule in the first place. Because some chemists are very attached to specific targets. They're very good chemists, so they often produce things. that bind to that specific target. And these chemicals, very importantly, all look very similar to each other. And it is this type of similarity that's a very human vibe space and it's hard to pin down to a singular metric. And so they found that these models are often confounded by author overlap. And these problems just appear over and over again across the in vitro biology biomolecule generation where You can be confounded by variables that you did not know even existed in the dataset. I would probably name that as the thing to be most aware of when reading these papers. There are a few people I trust on Twitter and people in real life who can give a pretty good overview of any arbitrary paper. But I think with LLMs, popular science people often retweet them and say, this is transformative. More often than not, they're probably correct. Opus 4.6 is genuinely crazy. I think when people do that for BioML papers, there's a 50-50 chance that they're completely missing the point because they're not in that field and they don't understand how the failure modes emerge in these models.

Do you think that an opus can help me identify those blind spots? Is it good enough to do that?

I've written an article about this also titled Can O1 Preview find mistakes amongst 56 MLSP papers? MLSP is a structural biology workshop at NeurIPS. It's not very good at it. This was obviously last generation of models. Maybe it would be a lot better. But there's some problems that are going to reoccur in almost every biology ML paper of like, oh, your train sizes aren't large enough. Your test sizes are not stratified correctly. But like, you kind of just like learn to pick your battles in this field and you just move on. There's like a lot of more fundamental problems with article, with these papers that LLMs, in my experience, just like often miss entirely. I think the funnest, like I, in almost every article I've written, I have found that LLMs tell me something about this particular subfield that the domain experts completely disagree with. And they say like, that is not how you should think about this domain. It's like, that's not the real problem we should actually worry about. I don't know why this is the case. It's kind of fun. It's like a domain of science that LLMs still haven't quite captured, human taste.

Yeah, fascinating. Okay, that leaves a lot of work in front of us. Do you want to go back briefly before we break to noetic again? Fortunately, my son is doing well. He recently got cancer 3 months ago. I've had an intensive crash course in cancer, and I hope to be able to close the book on it and return it to a more intellectual and less personal interest going forward. And I think we're on good, solid track to do that. But I think you've demonstrated in this conversation that you're not getting too carried away with the promise of what AI systems can do. We've got the data centre of geniuses, we've got the century of progress compressed into 5 years kind of visions. How much would you shave off of those notions just to describe your own expectations of what Noetic can do specifically and maybe what the field more broadly is going to be able to accomplish?

I think I'm very optimistic that human simulation companies akin to Noetic, but I think there's other players out there as well, will be able to vastly help with the results of at least a few clinical trials within the next few years. That feels like almost like you're not even paying attention too much to the trend lines. I'm almost indexing on what we're capable of today. I think it's pretty obvious. There's papers going back years that are able to show that we've developed an ML model that is better able to stratify patients. The problem has always been an economical one and how do you actually deploy this in a real setting. I think we'll be able to do that just fine. Phase 1 drug— sorry, the failure of phase 1 drugs will go down. And I think this has already been slightly proven out in a McKinsey study that was done a few years ago that showed that AI-designed drugs have a 5 to 10% lower failure rate, which may be noise, may be real. I do expect those trend lines to continue a little bit. Where I'm most unsure of is whether these models will be able to Discover brand new targets entirely, which is ultimately what people care about. I think believing that these models will be able to find new targets far faster than humans would really requires you to index heavily on the trend lines. And I want these indexed on the trend lines. So I believe that these models will be able to deliver very good target finding. But I'm also very sympathetic to the mindset of finding targets is just such an unbelievably hard problem that these models will not make a dent in it because You need this human iteration feedback loop. And unless you build a really good human simulator, which is our bet, you're not going to get close to solving that problem.

The way I put it is usually you can kind of see one order of magnitude ahead, maybe 2. No one can see 3 orders of magnitude ahead. It's just not possible. You have no idea what's going to happen. Avi, thank you so much. I learned a lot from this and hope to see you online. Hope to read more of your blog.

Yeah, absolutely. Thanks for having me on.

Thanks for being here. We'll be working our way through owl posting archives, uh, for some time to come.

Bye-bye.

Our next guest is Helen Toner, who runs CSET at Georgetown. She's a former OpenAI board member, and there's 2 competing views here. She has, on the one hand, the intelligence explosion is coming. On the other hand, AI capabilities may be permanently jagged. So let's add her to the stage. Helen, nice to have you.

Hey, thanks for bringing me in at the end of your marathon. Impressed you guys are still going strong.

There's so much to cover, and we've all got to accelerate our personal productivity timelines and try to pack more information into the same amount of time. So experimenting with ways to do that.

So you want me to talk super fast?

Yes, please. That's honestly one of my reservations about live content is I listen to everything at 2x speed and I'm like, I can't listen to it at 2x speed if it's live.

So, I mean, my, my constant struggle is to talk slower than I naturally want to. So if you want me to talk double speed, I'm here for it.

Please go as fast as you want. Okay. So you guys just put out this report. I think this is obviously a great candidate, if not a shoo-in for the most important question of our moment. What is going on with the possibility of automated AI R&D? Do we have this tipping point where we're starting to hit recursive self-improvement? And if so, how big of a deal is that going to be? You guys brought together a bunch of people that authored this report and some others as well that aren't necessarily authors but contributed to conversations. I understand quite a few people from Frontier Model Developers. And it strikes me that This debate goes back basically to the beginning of AI, right? There was the idea very early on that we could have an intelligence explosion. When I started reading Eliezer in 2007, he was very worried about this. And yet you've written, I think you put your finger on something a lot of people were feeling last year when you said, even though what passes now for long timelines is pretty short, and yet the disagreement on this topic seems to be as fundamental and seems to be kind of as impervious to new evidence as it has ever been. So maybe just for starters, take us inside the workshop. Give us kind of the lay of the land in terms of what are the world models that people have and why are we still just working from so much intuition, despite the fact that we now have what in some circles would even be called AGI out there as products for us to use today?

Yeah. So this was, this workshop was held in, in July last year and was maybe one of my work highlights of the year. It got started. So it was a day and a half. We brought people in. We had people coming from some of the frontier companies, policy, you know, a bunch of great people. To get a sense of kind of what the vibe was like, we started out, the first session was about kind of how is AI being used to automate AI R&D right now. And we had presentations from people who are doing that. And before the first break, we had, uh, Ryan Griebel from Redwood Research, Nicholas Carlini from Anthropic. Ash Kapoor from Princeton of AI as Normal Technology, and Thomas Larson, who's one of the AI 2027 authors. They were all like arguing so fiercely in a friendly and productive way, like before the first break that they just like, everyone else stood up to go and get, you know, coffee and drinks and snacks. And they just kept on arguing, um, right through the break, which is great. It was exactly what we were looking for. But I think did sort of preface something that we knew going in, which was there are really different perspectives here. And that, you know, the workshop was Chatham House. I feel okay giving that anecdote because they ended up writing One thing that came out of that was Nicholas was pushing the others constantly for like, okay, you have such different views about where things are going. Where's the first place that you actually disagree about what we'll see? And they found that as they're looking out for like what we're going to see in 2026, 2027, they actually agree a lot about kind of what we're going to see before we get to that recursive point, which is kind of a bummer. Like it's, you know, it's nice that they agree and can, they wrote a post about that actually, which is the reason I feel fine, you know, sharing that, that anecdote from an otherwise Chatham House workshop. They wrote a post about stuff they agree on. But it sucks that, like, that means that it's actually going to be hard to identify kind of in advance whether we are heading into a recursive loop or whether we're not. 2 big things that I think, so what we were trying to do with the workshop, one was like get this idea of recursive self-improvement out of sort of purely like Silicon Valley, San Francisco, like really AI-filled spaces and make it like, explain it, present it to a wider audience, let people engage with it. But then also actually try and make some progress on like, okay, why do people disagree about this? What is happening? What might happen in the future? You know, what indicators could we gather? Stuff like that. And I came out of it thinking that maybe 2 of the core disagreements here, one is, does AI truly replace all of what humans can do? So does it, does you get to that like fully automated? Because if you're going to have the big, the really like scary recursion, that's probably what you need. It can't be that, you know, ever you have a, we could have much more productive human researchers. You could have, you know, the Alec Radfords and the, you know, Ilya Sutskevers managing fleets of AI researchers. But if it all has to come back to them and they have to process and digest and like think through the research, you're not going to get that like.

Yeah.

massive recursive loop. So that's one piece is like, do you truly get humans being fully replaced? Because if not, then maybe you have some parts of the workflow being really accelerated. We have this diagram in there of like sort of an Amdahl's Law kind of thing, where Amdahl's Law is basically if you have a process that depends on different inputs and there are different potential bottlenecks, then if you speed up one part of the process, the bottlenecks will just bite somewhere else. And so it may be that you speed up the coding part of AI research, but if you don't speed up other parts, And you don't end up speeding up the whole thing very much. Or, you know, do you have, I think another mental model that people who are skeptical that this is going to really go crazy, a mental model they bring is like, okay, we have a long history of computers doing more and more of the lower level work. So, you know, we don't have to do punch cards anymore. We don't have to write assembly code. We have these higher level languages. And so, you know, for example, AI doing more of the coding is just another natural step in that process. And humans, you know, this is kind of a like expanding pie model. The amount of like tasks that we realized can be involved in AI R&D expands. And there's always that outer band that the humans can do while they're automating the inner bands. And I think that is very different from the view that, you know, other people sort of more that AI 2027 authors would have, or, you know, lots of other people in this space, which is no, first you automate some of what the humans can do, and then you automate all of what the humans can do. And then, you know, you kind of, you go until some other bottleneck hits. So then, you know, the other question is like, Uh, okay. What are those, what are those bottlenecks? We can, you know, we can talk about that as well, but I think those are 2 of the biggest questions that came out for me. One was, are you truly going to automate everything, including what all the humans can do? And the other is, if you do, you know, how soon do the bottlenecks bite?

So Sholto Douglas, uh, who is now at Anthropic, had this idea of a, uh, software-only singularity, he calls it, where, um, we get very good at coding and like all of the digital stuff, including AI research, I presume. But not at producing power or copper or all of the physical substrates, which are going to be required in order to support this expansion. How would you feel? How do you think that fits in? Like the fact that maybe the digital stuff happens, but the physical stuff just doesn't.

Yeah, I think there's 2 versions of this. I think some people, when they talk about a software-only singularity, they basically mean like, it turns out that software is enough. to get like absolutely crazy recursive loops. Um, so I think there's like Tom Davidson at Forethought Institute has written about this, for example. And, you know, maybe you can get just like massively more intelligent systems having massive impacts on the world, primarily through sort of software improvements. There's a different thing, which what you're describing of the version from Sholto is a different thing, which I would think of as more like sort of a jagged software-only intelligence explosion, meaning the AI is getting much more capable in certain ways and like But its effects on the world are very limited because it is software only. And this kind of gets to another, another thing that I found really, really helpful from the workshop and really interesting, which is people have very different intuitions about, okay, assume that you have an AI that is very, very good at AI R&D. What does that mean for what the AI can do elsewhere? And I think some people are like, okay, well, if it's very good at AI R&D, then it can like train AI models to do whatever. So it can do whatever. And, you know, maybe there's like, maybe you have to spend like a week gathering data or something, but then if you want to do some arbitrary task. you could do it. Whereas I think other people have an intuition of like, okay, well, even if it gets very, very good at automating AI R&D, this sort of most, you know, software-based task, it's still going to really struggle to, for example, design new biological molecules, or it's going to struggle to like think about geopolitical strategy questions because you have to like actually go out and like see how different countries and decision makers will react and things like that. And that is a piece that I feel like goes under in a lot of these conversations is like, what is that connection between AI that can do incredibly good AI R&D and AI that can affect the world in non-AI R&D specific ways that we also tried to tease apart a little bit in there.

Would you think that's kind of like the connection between, okay, now you have AI doing AI research, that's affecting the economy, it's also affecting the political economy, and then you have to have mitigations to the political economy for this to work out? Does that mean you might need the AI research to go into how to fix the political economy, which is going to be a little bit scary?

Yes. Say more about what you mean by affecting the political economy.

In the sense that, for example, right now you have Bernie Sanders saying that we should have a moratorium because he's scared about jobs. He's very scared about jobs. He wants a moratorium on data centers. I think there's like 6 states with a moratorium now, including New York State. And so one of AI research leading into, hey, how does AI fix the political economy? How do we deal with the humans? How do we mitigate the impact that we have on the humans? Like, is that something that you think would happen with the first, the first configuration of the software-only singularity in the sense that it's not jagged and also affects the political economy?

Yeah.

Yeah. That's, that's the kind of thing. That's right. If you're positing that you can just have a sort of software-only singularity that is going to radically transform the world, then it's going to have to be able to do things like Okay. And then the company like deploys, you know, chatbots that talk to enough people that convince them that data centers are great. The data centers all get built and the moratoriums get rolled back. Yeah. That, that kind of has to be built in. That's right. Which to me intuitively feels like it's sort of a different skillset and also is more dependent on like deployment and rollout and adoption. Yeah. So I tend to be a little, a little more skeptical there, but yeah, I think that's a, that's an example for sure.

I see.

One kind of Odd, seemingly to me, odd pairing of beliefs that I observe and I sort of detect in the report is the idea that among the sort of more sceptical folks, that there's going to be a plateau, and also that plateau is going to be subhuman. And then on the other hand, it's like it's going to not plateau, it's just going to run away and have some sort of singularity. A position that I feel pretty intuitive, attracted to that I don't hear too often is the idea that maybe there will be a plateau, but it could be very easily a superhuman plateau. If I try to zoom out as far as I possibly can and look at life on Earth, I would say it seems like humans are sort of part of maybe an entry into a steep part of an intelligence explosion or an S-curve of capability. And I don't think we're the end of history, but we were clearly better than what came before, and that was enough to take over the world. And so I think I just don't hear too many people say, yeah, it's not necessarily going to be a singularity, it's not necessarily going to go totally beyond comprehension. But in the same way that we were just that much better than Neanderthals, and it might not have been that much, but it was enough to change everything, I kind of feel like there's not too much more room between where the AIs are now and where they will soon presumably be. And even if that doesn't go sort of critical from there, it feels like it's very hard for me to imagine that it's not enough to be transformative. So is that a position that was represented in the workshop, and how do you personally react to it?

Yeah, I think that sounds pretty close to my default expectation, maybe. And if so, then it was represented there because I was there. Maybe to riff on it a little bit, something we didn't put in the report, but that I've definitely found helpful for my own thinking is thinking about, okay, you have an S-curve. We're clearly in some kind of S-curve, I agree. And you have 3, maybe less situating us as humans in the middle of an ongoing S-curve, but we have some kind of S-curve of AI capabilities. And there's 3 segments that are of interest. One is how long is the lead-up period, the first part of the S-curve. One is how steep is that middle of the S, and one is how high is the ceiling. And I think a lot of, mostly when you're hearing people talk about automated AI R&D, either they're sort of, they're in one of two camps on all three of those questions. So either they think the lead up is short and the curve is steep and the ceiling is high, or they think the lead up is long and the curve is gradual and the ceiling is low. And so I also think, find it really interesting to think about, okay, what are different like combinations of those sort of parameters? Where to me it feels like really looking like the lead up is pretty short these days. Like we're not too far from that.

Yeah.

sort of takeoff period. But like, for example, what if the curve is steep, but the ceiling is low, or the curve is gradual, but the ceiling is high? I feel like we don't talk that much about either of those. And maybe also to your point, Nathan, about the, like, superhuman, but not like all-powerful God, like singularity, there's no, you know, no point of return. Yeah. I also, I really think there's room for more thinking about. What does it mean to be superhuman? And what are the domains where there's like tons of headroom above humans and you can easily identify like, you know, yes, it would look, this is what it would look like to be superhuman at, you know, like optimizing a kernel or like, you know, selling things, for example.

You're spewing legalities. Yeah. In previous, you know, parts of this marathon conversation, series of conversations, we've just seen how the ability to interpret the signals that people throwing, are throwing off in sleep to predict disease. It's just a really random, but I think instructive example of how there's an obvious lot of room to be superhuman at some of these tasks. And there's potentially a lot of power to unlock, especially if you can integrate that kind of infinite modality grokking with a kind of basic reasoner.

Yeah.

I really don't see any reason that we're not going to be able to achieve that.

Yeah. Often those things, though, will involve— I think another piece that's underexplored here is people tend to either be in the camp like, the ceiling is high and you're not going to need all that, it's not going to be delayed by real-world adoption, or the ceiling is low and it's going to be delayed by real-world adoption. And to me, I'm sort of like, isn't the obvious combination of these, once you get the real-world integration, for example, you have to collect all that sleep data, or humans are really bad at interpreting scent data, but dogs can smell things we can't. you have to do a bunch of sensors and like all that. So I also feel like there's sort of unexplored questions around how high is that ceiling as you have kind of really increasingly integrated AI to more and more aspects of life in the economy.

I do, I do also wonder to what extent, um, because in, in my, you know, the way I, I might, I might view things as happening is software and mathematics first. And the question for me is that if you get software and mathematics first, You may get things like, I don't need a LiDAR for my self-driving car anymore. I can use cameras. And the cameras can be really bad cameras now because the math does all the work, and I don't need all this sophisticated technology. It could be that your phone could do what those sleep detection machines do with the right software package. Your phone has a lot of sensors in it.

Yeah.

There's an enormous amount of technology within the phone.

Yeah.

And you do wonder whether it would really be application of algorithms to existing frameworks, existing infrastructure, increasing the bandwidth of your communications tech with new encryption and new cryptography, which is how DSL was invented, for example. DSL was really using the existing copper pipes with new algos. So I wonder to what extent the— You don't get a slowdown just because of your physical infrastructure, because you kind of innovate around or with your physical infrastructure.

Um, yeah, I'm sure that, I'm sure that will work in some places. I think that, yeah, I think it'll work in some places and it won't work in others. So I think if we're talking like, you know, there's a huge ongoing struggle with where my mind goes is like cybersecurity for critical infrastructure, where like the systems are, the physical systems are old. They're hooked up, the operational technology, they're hooked up to old information technology because they have to be like. There's going to be a limited amount that you can optimize using smart new algorithms there because it's just, the stuff is old. Likewise for, you know, my center does a lot of work with kind of military technology, same thing there. Like if you have a ship that was built in the 1960s, it's a ship that was built in the 1960s.

Yeah.

Or other pieces of equipment. So yes, I think in some places, yes, in some places, no. To me, that's another, I think Prakash, you mentioned it as I, as I came on the talk I gave on jaggedness. To me, it's like another place where the jaggedness bites. And I think as well, like my default expectation in AI R&D Is that we'll see jaggedness, like the jaggedness is fractal, right? Like you zoom into the sort of quote unquote task of AI R&D or the skill of AI R&D. And actually it's many, many different things. And we'll, so we'll see the AI R&D accelerating in areas that are especially amenable to using AI and lagging more in other areas, not to say they can't ultimately be automated, but it will like take longer.

How far do you think the product which is on the market right now behind what people are using inside the labs?

I don't know. I honestly don't know. That was one of the most trying to be productive sections of the report. Well, that's not true, but most actionable section of the report is a set of indicators. We have a table summarizing the 3 categories of indicators that we have. And that is the biggest category is indicators from inside companies. And one of them is that public-private gap. My sense is that it's not huge right now, but I don't have any inside information. You guys talk to company employees as well.

You believe Arun, he says it's, uh, we have no idea how good we have it and the gap is very small.

Exactly. I'm thinking of things like that exact tweet.

So what I learned in the last few days is that the real gap is that they're using models which are 3 times faster. So it's just the same model. They're running on lower batch size. It's 3 times faster. And that's what they're using in turn. That's, you know, it's the same tokens. It's just a lot faster.

And there's surely also like tooling stuff, right, as well. Like, you know, something we put in the report, a couple of our reviewers who were looking at this, who were less familiar with the idea of automating R&D, some of them were like, oh, haven't you seen that study of like 95% of AI pilots fail? And there's the, you know, that meter study of, you know, that AI slows people down. And so we included in the report, like explicitly noting, you know, yes, productivity boosts from AI are mixed, but these AI researchers are like in the very, very best position to benefit from their technology. Like, they They're the best up to speed on what it can do and what it can't do. They're shaping how it's developed and like what directions it's pushed in. They're in the perfect like setting to be building tooling to squeeze the most juice out of these models. So I'm sure that that is also a piece of it as well.

One of the things you mentioned early on, just a few minutes ago, is the idea that you wanted to kind of bring awareness of these possibilities, you know, outside of the places where they are most often discussed. One of the things I would love to hear your perspective on Is how ideological do you think companies are about this? I mean, this is one of the things that confuses me in the sense that every frontier lab leader has read their Eliezer catechism. I think they've previously, many of them in the past had said things about how we should be extremely careful about this sort of thing and we should not engage in an arms race. dynamic. It's obviously part of the OpenAI charter. Dario has said things like this. And now we're in a world where there is a publicly stated timeline by OpenAI to the AI R&D intern, and then another not too much longer out, 2028, the full AI R&D researcher.

And I mean, Dario, Anthropic as well as Dario's—

Yeah, I would say Anthropic will even seem more committed to it or more resigned to it, maybe, but they believe it.

Jack Clark, June, summer this year. Jimmy Ba, who just left xAI, was a co-founder, 12 months. And OpenAI, this year, research intern, and then full researcher kind of a year later. I think it's this year. So that's my guess.

This year for what specifically?

Like the start of recursive self-improvement. We're going to talk about that.

Oh, but aren't we there already? Wasn't it last year? I mean, you had like, you know, you had like Gemini doing the, what was the evolutionary algorithm stuff where it designed, like co-designed an algorithm that like sped up its own training by 1%. Like, come on, that's recursive. It's really nice. Does this— what I'm talking about, like the, the lead up to that.

Yeah.

So you, but I don't know. I think you think we might be this year where there's no human needed whatsoever. I think it's a high bar.

I think we might be. I think I updated on MultBook. The Moatbook thing took me by surprise, like 1.5 million agents all of a sudden on the web. Yeah, it's all nonsense for sure, but things start off as nonsense. I think what might happen is you get a single model update, which kind of fixes a little bit of the hallucination, a little bit of security issues around like leaking secrets. And I think that might be enough.

Sounds hard. Sounds real hard. Fixing the security.

Yeah, we'll see. Yeah, maybe.

Maybe.

So go, I want to, I do want to give you the chance to talk about the dynamics of this. The, there's, you know, different reads that we might put onto people. They're ideological about it. Elon Musk has said things like pretty much like, I don't know if this is going to be good or bad, but I want to be around to see it. And also like, you know, I'd rather be part of it than a spectator. So that sounds like somebody who is inclined to gamble with humanity in a pretty self-aware way. Others may feel like they're trapped into these dynamics, and they at least will do it as safely as possible. How would you describe that milieu right now? I think it's dramatically underappreciated by people outside the AI bubble where we spend all our time.

Yeah, I mean, I think my impression of it, my sense of it from the people I talk to is there is just a sense of inevitability about AI advancing and a desire to be a part of the future being created because they see this as the future that's being created. And I think there's also, you know, the people who, you mentioned how this has been kind of part of the AI conversation since the very beginning is this I.J. Good was in 19— early 1960s talking about when you create the first ultra intelligent machine, which I feel like is, we always need more terminology in AI. I feel like we should Yeah. Ultra intelligent to make a comeback, that there's this sort of like very natural logic. If you have a computer science kind of brain, it's just a very natural logic to say, okay, we're, we have some level of skill at building computers. When the computers have more skill than we do, then they'll build ones that have more skill than that. And then, you know, you get a loop. And so I think that, that logic is just very appealing and, and seems very natural. And so people think of it as something that's going to happen anyway. And then they may as well be involved. That's not everyone, but I do get the sense that that's sort of the water that most folks are swimming in. And then if you have a different view, then it's in contrast to that. I don't know. Is that your sense as well?

Yeah, I think so. And I think the inevitability is a pretty compelling argument. I mean, I resist it because, and I at least want to make the point that even if some form of this is inevitable, there is still probably important discretion that we can exercise in terms of exactly what flavour. And there's questions like, should we keep chain of thought interpretable, or should we embrace thinking in latent space? And I do think it's important to kind of keep in mind that it's not probably all one or all the other. AI defies all binaries. There's going to be these gradations and these kind of more local decision points. But yeah, I mean, in 2022, I was just trying to make AI work for practical tasks, and with no background in AI research, basically ended up independently inventing a number of the techniques that have gone on to produce great things. I didn't take them past any local plateaus, but just having AIs improve their own outputs, kind of proto-constitutional AI type stuff. I do think the attractor is the sort of gravity well is like, pretty strong. And it's hard to avoid some version of these techniques, because if even a bozo like me lands on them, I don't know how they're not going to happen in the big, broad world, especially as we start to also get dramatic democratisation of training techniques.

Right.

I mean, Prime Intellect just put something out that kind of allows anybody to spin up their own RL environment on a distributed basis, on a community basis. So everything's going to get tried, and I think that's pretty hard to argue against. But again, also, I do want people to still own what exactly it is that they are doing along the way.

Yeah, I think there's something in here which takes me back to long-running conversations about autonomous weapons. There's something in here about the level of human oversight that you can have, where I do think that using AI to accelerate research is, I totally agree, an attractor. But I do, there's a, you know, you'd really hope that there's a meaningful difference between I have a fleet of 10 million AI agents and they're running experiments for me and I'm, I am leading them and I am, you know, guiding them versus I have set something into motion. I have no fucking clue what's going on. And I think there's, there's a boundary somewhere. Is it a boundary that we're able to stay on one side of? I'm, I'm not sure, but I hope it might be. That to me feels like the point to try to intervene, not, you know, we shouldn't use AI for research. That's obviously not going to work.

To what extent do you think policymakers are naive? Because earlier on, we spoke to Sam Hammond, and he advises some policymakers on AI. And he was talking about privacy and the restrictions, the constraints that we could put on, or the regulations. And to me, I think one thing that struck me was that I think a lot of policymakers are not aware, perhaps, that These, like AI with access to existing technology, you know, with access to persistent search, persistent memory, it would basically do a Google stalking of you before, before it even met you. It would know all of those things in the public, public domain, right? Like the amount of access to information that it could have, the persistence of information, listening in on conversations. These things are going to be very powerful in that sense. And I think you can ban the AI from using facial recognition, fine. But then you have network analysis, et cetera, et cetera. You can do metadata analysis on WhatsApp conversations on where the messages are going. You don't need to know the content, right? So there's a lot of these techniques where you can deanonymize traffic, you can deanonymize people. You don't need the facial rec. You can ban the facial rec. You can still do gate analysis and speech analysis, voice analysis, handwriting analysis. There's so many other techniques and all of these things will be available to AI. To what extent does this whole, like, you know, we're going to make sure that we're going to have privacy thing is, are they, are they being naive? Is it going to be possible?

I mean, I think the US has done a worse job of this than pretty much every other country on the planet. So I think there's like some basic rule. I don't think you want to do rules at a level of like no facial recognition. I think you want to do rules at the level of like no data brokers, right? Of like, You can collect data, but if you're going to collect it, the user needs to know and they need to have, you know, notice and consent or— I'm not deep on privacy law, so I don't want to pretend that I have the right, the great privacy proposal here, but I do think there's ways to do it that are better than the U.S. And I do think there's ways to do it that give you that sort of underlying flexibility. Yeah. Yeah. Maybe I'll leave it at that because I, yeah, again, I know privacy law goes real deep and I'm not there.

One more question for you is, you know, we, In the report, you talk about the possibility that the gap that we think is currently small between the models we have and the models that are used internally could open. And you have some recommendations around certain transparency measures. I want to give one quick shout out to the AI Whistleblower Initiative, founded by my friend Carl Koch, who has engaged with OpenAI and at least played some role in their recent updates to their whistleblower policies, which I find amazing that OpenAI is continuing to work in that direction even today. Where do you think we are on the spectrum from secret non-disparagement clauses to where we need to be in terms of insight into what is going on at the labs, other than private philanthropist-funded whistleblower support, which you can avail yourself of, again, via the AI Whistleblower Initiative? should you need that. What other policies do you think the government should be doing? And maybe even more broadly, if you want to zoom out, what do you think a situationally, hypothetically situationally aware US government should be doing in general that it is currently not?

Yeah, I think there's a bunch of things here. I think on transparency, I think we're doing better than we have been. So we have these 2 new state laws, SB 53 in California, RAISE in New York. I think those are good starts. I think something that it would be great, but I think for a lot of this information, we're also just really dependent on what the companies still choose to put out. Now, we're fortunate. I want to give credit to both OpenAI and Anthropic, and to somewhat lesser extent, Google. They do put out pretty proactively a pretty good amount of information. So I think they should get some credit for that. But I don't love that it's almost entirely at their discretion what it is that they put out. I guess that will be shifting as SB 53 and RAIS start to be enforced. So I'm interested to see what that looks like.

Yeah.

that looks like. I think also we need to shift to a bit of a— I think there's been a start, the beginnings of a push to shift from a model release-based schedule to something more continuous, which is partly driven by interest in these internal deployment-type dynamics, not just the external releases. So the idea here is if the risk is not actually purely tied to when you put your model on the market, then all of your risk evaluation shouldn't be tied to that either. And also creating better incentives for the companies around not forcing them to just rush things. out the door, but instead trying to have more of a sort of continuous pulse of updating metrics over time. So I think we could definitely be doing better on transparency and then ideally pairing those requirements as well with some kind of independent audit requirement or independent way to let people come in, external third parties come in and check that things are happening as they're supposed to be happening. That has been in several of these proposals and keeps getting stripped out by industry lobbying. So that I think is a new frontier as well. I think there's various other kind of policy implications that we put in the report, some that are maybe interesting. One is just like this general, like hardening the world sort of recommendation or like, you know, resilience, societal resilience is another way of putting this. So this is like cyber defense, biodefense, biosurveillance, you know, investing in biosurveillance, just meaning like monitoring diseases, not like surveilling people, you know, investing in like epistemic security stuff of trying to have a way to, you know, determine what's real and what's fake, tagging real content, you know, all this sort of like broader societal resilience stuff, which is like, okay, just assume that this is going to get much, much, much better. And then we might see AI R&D, automated AI R&D contributing to increased pace of change. I think also there's been some, this is less of like a policy and more of a mindset. I think there's been a shift over the past year or two to like, oh, actually maybe open models are always going to be pretty close behind. And so, concerns that you might have about there being an access gap or a concentration of power kind of gap, if the closed models are far ahead, maybe we don't have to worry so much about that. And I think if you're taking seriously the possibility that automating AR&D speeds up the closed labs significantly, then we just need to revisit those assumptions about kind of open models and closed models. There's a few others, but yeah, I would point people to the report for kind of the full set.

When AI builds AI, things just might start to get weird. So yeah, definitely check out the full report from Helen and co-authors at CSET and beyond. It's interesting times, for better or worse. Any other closing thoughts before we break?

No, great to be on. Great to chat with you as always. And yeah, look forward to next time.

Indeed.

Cool.

Hello. Very nice to meet you.

Shorten our timeline between now and next time.

See ya.

Cheers.

Bye for now.

So our next guest is Jeremy Harris. He's from Gladstone AI. And they wrote the first ever US government AI threat assessment for the State Department. It's been about 10 months now when they said every American AI data center is compromised. Jeremy, what has changed? Have things gotten better or worse?

Yeah, well, to piggyback off, I think Nathan just said things are getting weird. So things are weird. Yeah, great to be on. What has changed since then is less than one might have hoped and for really interesting reasons. I think one of the things that a lot of people who are concerned about the AI risk story and just the AI threat landscape from a national security perspective, whether it's loss of control or weaponization, I think a big part of the story that's missing is an understanding of the infrastructure buildout. What are the actual bones that we're building on here? Because that's the substrate that underlies everything. And there are all kinds of assumptions that are being made about it. Where we're kind of abstracting away what I really think is like, you know, at least 50% of the problem here. And we think a lot about model reconstruction attacks and all kinds of interesting debate about whether it even makes sense to secure models in a world where you can just reconstruct them if an API is available.

Mm-hmm.

But more fundamentally, when you're building your entire like AI industrial base off of, you know, components that are made in China, with personnel who often are Chinese nationals. I mean, this isn't even like— forget about the Manhattan Project. We're so, so far behind that. I think it's incumbent on us to take a little step back and just ask about what is that chessboard even? What is the board itself? Forget about the pieces, but are we playing on something that's fundamentally stacked in a way that doesn't allow for a winnable outcome? And I'm not saying this to be pessimistic. I think there is actually There are solutions that you come up with very quickly once you take that new perspective. But closing your eyes and not looking at it doesn't address the problem. And I think we're in a space where we're doing a lot of algorithmic-level thinking because that's what so much of the Western economy now is based on— people at keyboards who are used to— we're not making t-shirts anymore. We're not building transformers anymore. We're not doing that stuff. And so we tend to like to pretend that it doesn't exist. So that's kind of like, I guess, my more recent lens on the problem as of the last 2 years. I know that's not quite an answer to your question, but that's kind of the chessboard as I see it.

When you look at it end to end, you have the software piece and the talent piece. 50% of top AI researchers are Chinese nationals. And that includes people working at frontier labs in the US right now. And then you have the infra piece. A lot of stuff is coming from Taiwan, South Korea, Some of it is coming from China too. Uh, and then you have ASML sitting in Holland, which is supplying into TSMC. And then you have ASML suppliers. They have like 3,000-odd suppliers spread across the world. They're buying, I think, neon gas from Ukraine. When Ukraine got invaded, they had a problem. They've got all of this, all of these missing pieces, all of these like pieces all spread out across the place.

Right.

And TSMC has been upfront by saying, You know, we, we are only possible in a safe globalized economy. If we ever got invaded, everything's over.

Yeah.

There's no— we, like, we can't do anything. That's it. It's done. So where do you think— how do you think that fits in with a threat perspective? It seems like someone just has a dead man switch over, you know, TSMC. So how does that— how does that work in terms of, like, security and, you know, securing US Prospects and, you know, the future in the US?

Yeah, I think it's a great question. That whole Taiwanese kind of scenario planning thing is something that everybody has talked about. I'm not so sure everybody's kind of like worked out the implications to full satisfaction. Yeah, I mean, so first of all, yes, if Taiwan gets invaded, TSMC is gone. It's gone whether it's because China takes it or because it's, as I would expect and hope, booby-trapped to the nines. to blow, right? I mean, it takes like, you know, hundreds or thousands of like insane-level PhDs to tweak that. You think of it as like a giant box of 500 dials, each one of which has to be perfectly tuned to keep these things pumping out at the right yields. You're not going to replicate that if you're missing either the equipment or the people. So like, this is extremely fragile, maybe the most fragile production process that primates on this planet perform. So an invasion is unlikely to leave it in China's hands. And so, yeah, the question is then what do you get when you roll that back? What's the number 2 positioned entity? And then you start thinking about, okay, well, what does SMIC do? What can it do? And the SMIC Huawei complex does seem like a very plausible runner-up, especially when you look at scaled production, especially when you look at the emphasis Huawei's placed on networking large numbers of GPUs together. They don't have to be as efficient as ours. They can't be. They don't have the logic. But they can be networked together way better, and that's how they get effectively competitive-scale performance. So this is a real issue. You also, in a funny way, this interacts somewhat positively with the energy bottleneck that we have here anyway, right? Like, we're going to be bottlenecked by energy probably sometime around the end of the year. When that happens, TSMC's ability to outproduce— I mean, it gets complicated because on a per-chip basis, they're way more energy efficient, they're pumping out more flops, but we do have that energy kind of ceiling on our, like that's the main constraint that we're there towards. So the timing matters a lot here, right? That dance between how much does logic matter, how much does energy matter, how much does memory matter, all these things, how much does packaging matter? All 4 of those things have become bottlenecks at different parts of the game in the last few years. Another piece that I think, again, when we think about the actual bones that the AI economy runs on, it's not just chips and not just like the data centers themselves. The power grid is a really just generally vulnerable target, right? So we know that, for example, there have been components in Chinese transformers that have been snuck in, explicitly Trojans, to be able to take down our gear. A very plausible scenario, just based on talking to folks who are working this problem on the IC side, is a Taiwanese invasion begins, and one of the first things that China considers doing is just shutting down the Northern Grid. It's obvious if it's existential, that's massively escalatory. So there are huge question marks there, but it's a scenario that's being taken very seriously for all the reasons you might imagine. So yeah, I mean, I think when that— if that happens, there are questions that suddenly run much deeper than just our ability to literally make chips where, you know, in Arizona or wherever, wherever the next thing is. But literally, like, if we can be kneecapped economically at a more fundamental level, we don't even get to look at the chessboard that we hope to look at. We don't even get to indulge in the, oh, well, what can Samsung do versus what can SMIC do versus, you know, CXNT, like all these things. We don't get to play that game. Like, we literally don't have an economy. Like, like, there's serious implications there. So if we think about this as a game with the stakes that it might have, and this is contingent on what's between Xi Jinping's ears and the Politburo's ears, but this could end up looking like, you know, we're preparing ourselves to take a punch in the face, but then we get, we get kicked in the balls, if you will. I mean, it's This is the kind of scenario, to use the great analogy, that we may be barreling towards. And again, that zoom out, I think, is really important. We've got target fixation here on what could be a pretty narrow part of the chessboard.

You had some ideas on not only do we need to speed up, but we need to slow China down. And what was your concept around slowing China down? Because they have— They're trying their best. They are definitely not there on the chips yet. The Ascend, the Huawei Ascends 910s are, you know, ByteDance doesn't really like them. You know, they, they want to get the H100s in. There is this concept of like building on the US AI stack. It's also revenue denial. If you, if you manage to flow the revenue into NVIDIA versus flowing the revenue into Huawei, Huawei has more revenue to develop those chips. So therefore we should deny them. Like, how does this balance out, this like letting them take on the chips, but not too powerful, and, but, you know, still enough that it doesn't create a market for Huawei? It sounds like a very delicate balance here.

It does sound like a very delicate balance. I personally, I'm sort of less oriented towards the argument that says, you know, if we just let NVIDIA do business in China, then the Chinese will go, Oh, sweet. We have Nvidia that's serving our needs. We don't have to push so hard on the gas on this issue that's been identified for years as the possibly number one national technological priority that we are pouring multiple Apollo moon landing amounts of cash into. This is, to me, a kind of a miscalibrated sense of even just the messaging that the CCP has been putting out.

Right.

I just don't see a world— we don't see, for example, Nvidia, okay, they can ship the H200 or whatever it is now, and then suddenly the CCP goes, oh, okay, forget about that quarter trillion dollar investment that we just made in PPP terms into kind of our national AI chip capacity and infrastructure. Forget about that. We'll stick with the Nvidia play. There's a sense both that the ability to access these Nvidia chips is transient. Because the next administration may just as well pull it down. But also that, like, why not both? I mean, it seems like an insane thing given that AI is a matter of national security importance for China. It would be pretty surprising to me if they just decided to respond that way. Indeed, it seems like they haven't so far. So I guess that's why I think about the export control thing from a slowdown standpoint. They have worked. We know from DeepSeek Like, the public statements of their CEO before DeepSeek was on the radar. And this is actually, I think, really worth noting and kind of like underrecognized and appreciated. Like, before DeepSeek was on the radar, they were coming out and saying, hey, we really think we could do this AGI thing. Though, like, one issue. There's just one problem. We can't get chips. And these export controls are killing us. Then obviously R1 drops and everything has got DeepSeek and they get dragged in front of the Politburo, whatever. and debriefed and suddenly things change. You get these little trickles, these little leaks of similar information that come out the edges of the kind of Chinese AI ecosystem every once in a while. But it's pretty clear that the export controls were working. This is, if nothing else, just look at the massive orders that are going to be coming in for the H200 to show how much demand, pent-up demand there actually is in the AI ecosystem. Of course, we know all about the frustrations of AI companies in China and about current way to win for their chipsets. So, yeah, I don't know. That's kind of like my biased take is very much towards the direction of, I think we got to listen to Chinese companies when they tell us that our export control policy is working.

Maybe I'll come back to some of the frustrating duality of difficulties where on the one hand, we have You have expressed very low hope for the opportunity or the possibility of meaningful, true collaboration between the West and China. And then at the same time, I think you're also not super optimistic about our ability to create a superintelligence that we can actually control and get to do what we want it to do. And I think the way I think about our conversation from a year ago or so and your contribution to the broader discourse, course, with America's Superintelligence Project, is like, those 2 things are both real, they're both true, and you're kind of engaging in motivated reasoning if you try to deny either one of them. With that in mind, we are now also seeing some of the potentially foreshadowing kind of moments on the AI side itself, right? Like, just in the last week, we've had these new models from Anthropic and OpenAI, and they've both kind of said we weren't really able to run the evals like we kind of intended to. Anthropic basically said the eval awareness is pretty high, and so we'll just do a survey and a little internal survey of whether or not this is safe to release. That's probably a bit of a simplification on my part, but I think that is a fair enough summary of their position. Then OpenAI similarly was like, well, The autonomy risk, the part of our preparedness framework, that's also pretty hard to evaluate. We don't really have tasks that are kind of long enough horizon that we can get a real handle on just how autonomously capable a new model like 5.3 Codex is. So that's kind of crazy. And yet, of course, both models are put out there. I don't see China driving the need to do that. It seems like they're doing that and doing it on the same day, notably because their own competition between the two of them and also just sense of rivalry seems to be heating up. They're going at each other in Super Bowl ads to some degree at this point. Not something I thought I would see from Anthropic at the beginning was like a Super Bowl attack ad, but here we are. What do you make of the dynamics between the Western companies? If I were to To put on my slightly pessimist hat for a moment, I would say it seems like we might be racing to the bottom, which was exactly what we were hoping to avoid.

Yeah, I think we are racing to the bottom. I think the only frame that makes any sense, if we're going to talk about, okay, we need to regulate this technology, say, domestically, in the same way that everybody from all leading AI companies have been saying for, I want to say, over a decade, Pretty much. You're never going to do that unless you deal with the outer loop, the outermost loop, which is international competition. There is no version of that. I don't think anyone— I think, again, we can enjoy the indulgence in target fixation of like, oh yeah, let's play the game pretending that other countries don't exist. But in the same way as ignoring the infrastructure target fixation, or sorry, the algorithmic target fixation and not lean infrastructure, This causes us to miss what is really the entire problem. So you are not going to get to a point where, you know, you can have a, a strategic, or I should say a tactical slowdown when you really need it, where you're like, okay, suppose we find that the next version of whatever model can design custom bioweapons, execute catastrophic malware attacks, all these things that are like entirely, they're entirely plausible. And that no, you know, counter jailbreak measures are, are Truly 100% effective against the kind of people we'd be worried about. Yes, you would absolutely in that world need somebody to be able to say, okay, guys, tactical halt. This is insane. We can't be in a universe where you get a nuke and you get a nuke and you get a nuke. We can't do all program free for nukes. Okay, so what are we going to do? We're going to have to have a slowdown. If China still exists and has their program and they are expanding, I mean, I'm repeating all this stuff that everybody said a million times. If they're 12 months away, 6 months away, I don't care. Um, we've got a shot clock now. That's the situation. So we have to start there. Like, we have to start there and say, okay, any serious solution to this problem will involve dealing with China. 2 ways you can do that. One is you have a kumbaya moment with China. Uh, there are a lot of interesting reasons why I think this is just like not gonna work. Uh, one of which is if you think about international treaties, they don't tend to reflect some sort of Star Trek-y commitment to everybody on planet Earth wanting to do the right thing. They tend to reflect the realpolitik kind of like lay of the land in terms of actual power. Like nukes, you have nuke drawdowns when everybody can retain arsenals that can still destroy the entire planet 3 times over. And there's literally no point in building the marginal nuke. You have similar things if you actually look at the history of bioweapon and chemical weapon treaties, is like you find in every case, actually, they don't get you the marginal lift over just killing people with artillery and gunshot. If it looks nice, and they often get adhered to for that reason. But then at the margins, you have like Chinese research labs on American soil doing all kinds of crazy research. You have whatever facilities and all this stuff happens anyway. And so this may sound like super cynical. I think it just reflects the way things work. That's at least my take. I would think that. So the question then is, how do you deal with an adversary like China that's in the position they are that does have the stranglehold? as they do on our infrastructure. They simply, they simply do. So the question is, what are your offensive options? That is it. There's like, you're not going to build the perfect Fort Knox. This is not, not a thing that's possible. And so the question is, what do you do to induce consequence on the other side? That's the only math that will work if my theory of the world is correct. It's not a pretty theory. It's not one that leaves us feel warm and fuzzy inside. It's one that may make you think a little bit about, you know, mutually assured destruction, that sort of thing. And I think I think there are nuances here. Obviously, Dan Hendrycks had his frame on it. But the bottom line is, yeah, I mean, I think you kind of need— and it doesn't need to be an AI-based response, though eventually, you can certainly argue that any offensive option that isn't coupled to the scaling laws is eventually going to be beaten by something that does. So there's kind of an important design principle in these things. But there are offensive options that need to be explored. And this is unfortunate, but it does mean that if you have a situation where your adversary can turn to you at any time and say, hey, watch me turn the power off on your entire grid and have tens of millions of Americans as a reward die of starvation or exposure, you need the ability to say, OK, watch the same thing happen in Beijing, and we can turn it back on, by the way. We need to have this de-escalation option.

Mm-hmm.

I know it's a bit of a grim view, but I think that when I think about what actually gives leverage in this situation, it looks a lot less like let's have a pretty treaty, especially given the history of countries like China, like Russia, with respect to treaty adherence. They sign treaties. We know what it looks like when China signs a treaty. It doesn't end up being pretty in a situation like this where you need perfect adherence at such a high level of precision. There's no version of an international treaty on AI that doesn't involve inspections of compute stockpiles and very precise overwatch of the kinds of algorithms that are being deployed, the kinds of evaluation schemes. The level of cooperation that's required to do something tractable here strikes me as being quite significant. And the trust, just like, I don't see it being there.

So what's your P doom and on what timeline? I mean, we were just talking with Helen about this report that they put out about when AI builds AI and the possibility of recursive self-improvement. It sure seems like all of the vague tweeting that is going on right now out of the Frontier Labs is suggesting that that is happening. And then on top of that, of course, OpenAI has public timelines that they've put out, I guess, to their credit, Maybe. I guess we could see that both ways. The Anthropic people that I talk to are, if anything, always the most firm believers that the recursive self-improvement dynamic is unavoidable. How long do you think we have before these things really start to take on a kind of runaway dynamic? Is there anything that you, if you had power and a lot of power, is there anything that you like you would want to bet on? And where does that leave you in terms of, maybe I should just stop all this and spend more time with my family?

Yeah, in general, I'm a big fan of the happy warrior mindset. I think it's just never constructive to go and hold— first of all, we have to assume that no matter how firmly we might believe in whatever outcome, we may just turn out to be wrong. There's a famous story, right? Richard Feynman walking around New York City, in the '70s, I think it was, looking at all the skyscrapers and being like, wow, isn't it sad that all of this is going to be wiped out by a nuclear war between Russia and the United States sometime in the next few years? And he was just— that's just— that was a fact of the matter. And it reflected a pretty reasonable understanding of the dynamics unfolding between those countries at the time. I'm not saying it's ever quite that simple, but this is an ingredient, if nothing else, that makes you less effective if you're just stuck in a hole all the time. Just as a meta point, I guess, that I think is the first piece. We have to act with agency, and we're going to be most effective doing that if we're not stuck in a deterministic Calvinist frame with this whole thing. In terms of— I'll also not answer your question before I answer it just by saying, regardless of timelines, one thing to focus on is that there's some things that are pure optionality plays.

Right.

And so there are things that you do If you're going to build a frontier AI cluster, right, at scale, that rule out nation-state security at that cluster. Just rule it out. If you don't do these things right on day one, by day, you know, 360, once you've finished building the site, your site is going to be compromisable and there's no going back and undoing that. We think of these as we call like the one-way doors, right, of the data center construction process, figuring out what those one-way doors are, setting standards for them, and actually executing on that, even doing it just voluntarily, right? You think about OpeningEye and Anthropic and so on, they're kind of all independently going, hey, we just want to buy that optionality because at some point—

Can you give me a concrete example of a one-way door?

Yeah.

So there's a bunch that I can't go into, but one that I can, it's pretty easy, is Think about the people that you're getting in the loop to review the site plans and details that would be, let's say, useful to an adversary who is trying to extract information. If those people are Chinese nationals, okay, you're done.

Cool.

Right? You're never going to unfuck that. That's baked in.

Right?

So actually, the interesting thing with these one-way doors is they tend to be surprisingly cheap. And that's the tragedy of it all is, like, You actually could, if you were thoughtful, go through and be like, well, on a fraction of the budget that would be required in CapEx and OpEx for these builds, you just like create pure optionality by implementing. I think that's a really important element. Putting offensive options on the table is a pure optionality play. You don't need to exercise those options. You need to have them on the table. That's what I'm saying. I'm not saying let's go to war with China. That's a crazy thing to say. All things in their context, but you need options. And that's a crucial thing. So having an understanding of mapping out the ecosystems that are relevant, the AI ecosystems that are relevant, and thinking about what might that endgame play out to be, those seem like pure optionality plays regardless of timelines, all cheap, all things you can do quickly. Again, it seems to me kind of— And I'm not saying they're not being done. It's just that often there's a lack of focus on the endgame here. Anyway, without getting into the weeds too much. OK, so PDUM and timelines. Oh, sorry, Prakash.

Yeah.

No, go ahead.

Go ahead.

I thought we had—

sorry.

PDUM and timelines. So I'll almost say PDUM, I don't find it useful. I know what I'm focused on. I know what I got to do. If I start thinking about, like, my generic answer has been for years, like, any number between 10% and 90%, I'll take as, like, that's a reasonable number. I'm not like— I've read the debates. I've seen the post on LessWrong.

Yeah.

So is that your P doom or P loss of control to superintelligence? Because I think in some places you've mentioned it's a loss of control to superintelligence rather than doom.

So, yeah.

You've obviously done your homework really well. Yes, that is more of a loss of control, the superintelligence. I think by virtue of the way that numbers multiply together, I don't know that my answer is that different from like P-Doom in general. Again, this is coming from somebody who, for better or for worse, has almost explicitly not put in that much time to kind of wallow in those numbers, as I think we're sort of all tempted to do, right? Like, I have that temptation. I get it. I mentioned I had a daughter, right? I don't like the landscape that's playing out, but I had a daughter. I chose to have a daughter, and I didn't have her in 2018 before the scaling laws blew up. This is a choice that I made. I think there's a kind of almost like a spiritual risk to getting locked into that kind of thinking. And I say this as somebody who's experienced that. I went through that and felt how it ground my gears. So yeah, so I guess I'll just not answer the question by saying 10% to 90% sounds reasonable. I think if you're below 10%, I just really think that there's homework you got to do because a lot of these scenarios are— maybe sound crazy, but they're a lot less crazy than they seem. And when you get into the nitty-gritty, it's like a lot of these scenarios are already— they're halfway won fully. If you're above 90%, I mean, first of all, if you live as if you're above 90% and you're like, that's going to just make you less effective. I also think, again, Richard Feynman certainly seemed to think he was in that ballpark. There's just like an epistemic question here of how quickly does the world adapt? I think we're constantly surprised by how quickly the world adapts, both how fragile and how resilient it is. And like the 11th chapter of the book, will often involve a new character that comes out of nowhere. And we just need to kind of make sure that we keep uncertainty about our uncertainty factored into this analysis. And I think that buys me 10% pretty easily. I've been wrong on stuff that I thought I was 100% on often enough to be like, okay, I'm not going to push it that much. And I know that's frustrating for a lot of people. Like, no, no, but look at the math, man. And I get the math. What I'm questioning here is just the the process that led to that. And I don't know that I can plausibly ever get fully behind that process and interrogate it with confidence. So last thing is timelines. I thought AI 2027, and contrary to— I think Dan has kind of pulled back a little bit his timelines from that.

He said 2027 always meant 2028, but now it means 2029.

Yeah.

AI is the apocalypse of the future and always will But not actually. But I think there's a sense in which— so when GPT-3 first came out, I was like, oh, man, I've got 2-year timelines. And that was because I didn't understand what the hell would be involved in the infrastructure buildout. And now that I have a much better understanding of that, I'm still kind of like, well, what's the next bottleneck going to be? I'm very uncertain about this. And again, it's one of those things that doesn't really affect what I do just because I'm so focused on just all the low-hanging fruit that we have to pick right now. There's so much stuff that we're just not doing because we're paralyzed by the problem. So I think in terms of what we do, there's pure alpha on the table in the short term. 2027 doesn't sound insane to me. 2030 doesn't sound insane to me. 2035 sounds a bit far. I guess I'll sort of leave it at that as a spread. I think we should be acting as if 2027 is plausible. I think it would be unfortunate if it happened in 2027 and we're like, man, We had a lot of really plausible analyses that pointed to that, and we just didn't do anything. That would be a shame.

Can you give us a little bit more of a hit list in terms of the low-hanging fruit that you want to see us pick? I mean, it's— we've got the one which is like build at least some subset of our data center build-out in a secure way so that we can run hypersensitive projects there as needed. What else is kind of on the— if you're the— replace David Sacks as the next AI czar, what's going to be your priority sheet?

Yeah, I mean, so that first one, by the way, is a lot of things, right? So it bundles together. I mentioned the personnel security issue, insider threat problems. There are a huge number of things in that bucket alone that are necessary and contribute very cheaply to much more optionality on the security side. I think, again, if you zoom out more, so you look at the grid, what could you be doing to introduce redundancies quickly? The supply chains that lead to a lot of these components are very clearly sourcing heavily from China. So here's an easy win. Look at the companies that are offering to build data centers suspiciously fast. And who owns those companies? So so there was actually a letter that came out from the House Select Committee on CCP a while ago, naming Day One Data Centers right as as an entity that is somewhat suspect. And you know, you'll have these data centers where data center building companies, where it's like, oh wow, like you could build stuff like way faster than anybody else. It involves sourcing components from China, and like. Like, my personal opinion is, if I were to see that, I might be asking myself the question— China is kind of a command economy through civil-military fusion. If the CCP wants me to have this very, very rare and precious and backlogged component for my data center in the continental United States, that might tell me something about how much faith I should have in the security and integrity Of that component. There's just not a lot of infrastructure-level attention being paid to these things. And the labs, by the way, they want to do the right thing here. They don't want to be in a position where they're getting a company to build something for them and then it turns out that that thing is compromised and it comes out that that is not good for anybody. So incentives aren't aligned there. Yeah, it's things like that where there's been so little attention paid to the bones that there's just tons of stuff we can improve, including with AI, right? Like looking for malware in old software that's load-bearing for our infrastructure, or not malware, but rather vulnerabilities and finding ways to harden it. So yeah, this is a defocused answer, but it hopefully gives a sense of the venue.

One thing we haven't really given you a chance to flex your ability on in this conversation is just the breadth and depth of your technical understanding of so many AI developments. And I definitely recommend the Last Week in AI podcast, which you usually host, as a great source of, I think, very sophisticated analysis by By both of you, but I tune in for you mostly, to be honest. And I wonder how you are doing it. How are you keeping up? How have your methods evolved so that you're maintaining situational awareness as much as you can?

Well, thank you, first of all. It's very kind of you to say. I have told you this before, but I do actually watch The Cognitive Revolution. And I think it's— anyway, the ecosystem here is really rich. And Interviews are really important because you get stuff that you can't get from the papers. And I tend to focus more on the papers, so I just don't get that kind of analysis. And I just talk to friends from the labs, but it's different from those deep dives.

Yeah.

I mean, so back in— I can't remember when I started on Last Week in AI, but it was maybe 2021 or something.

Yeah.

And back then, I would just read papers and you couldn't use GPT-3 to like help you understand a paper. It just wasn't a thing. Now that's changed. I've had an experience that was kind of frustrating this week in particular because I'm preparing a, like, a state of play briefing for a customer. And, um, basically they want to know, yeah, what happened the last quarter, uh, in the world of AI that we should be tracking. And there was a paper, uh, that I had Gemini help me with. I got to a really good understanding of how it is like a residual, like the dynamics of radiant flow through this residual and whatever. It was pretty complex. What I realized though, after having an interaction with Gemini for long enough, I switched over to Claude and I was like, wait a minute, I just hallucinated my way through that entire conversation, got to an understanding where I was like, oh yeah, I'm pretty smart for figuring this out. Also, I got this down pat, and everything got flipped around. So I'm not saying that always happens, but that has been kind of the most recent update in my process is really be mindful to kind of double-check, especially as you start to get lost in a rabbit hole.

Yeah.

I typically spend, I would say, about 30% to 40% of my time reading the paper and then the rest interacting with a model about— usually it's the implications of the paper or What it is, is it's reinforcement learning versus supervised fine-tuning. Like, if I'm reading the paper, I'm doing SFT. Like, that's what's going on with the models. I get to actually, like, go on policy, and I get to test my own understanding. Like, I would have done this experiment differently. Is that a stupid idea? And often I'll get a pretty good answer, and that— it makes you feel like you're rotating the shape instead of just staring at it. And that, for me, has been just really helpful and empowering. It feels empowering.

Do you have any particular workflows, pipelines, whatever, that try to filter things for you and surface what you really need to spend time on? Because that is as challenging. I mean, it's more challenging than ever, and it seems to be maybe as big of a deal as being able to successfully make sense of any one thing is like, what are you going to choose to spend your time on in the first place? How has that evolved for you?

Yeah, it's a great question. This is that age-old question of taste, right? And one of the things that I've had to come to accept is I can't develop good taste in all the domains that we want to cover on the podcast. I'm never going to— my taste is basically if one of the frontier labs puts out a piece of research or if a researcher I know and have a lot of respect for, appreciation for, put something out or is a co-author on something. I'm going to take a really hard look at that. And then besides that, I have the usual Twitter account, the set of Twitter accounts that I follow, and that's another way. But my passes at these papers are— I'm pretty focused on what's on the critical path to ASI question. Not that I know the answer, but just like I'm trying to find things that to me gesture at that, which is why I don't tend to talk about GANs or the latest in— I was going to say the latest in text-to-video. Now that seems like it could be on the path, so you never know. But I guess part of it is just acceptance. I am reading these papers for the concepts more than the outcomes. And often what will happen is there'll be this paper that'll come out and it might not be the perfect paper to cover from a given topic area. Oh, residual connections and really optimizing the crap out of them to get ultra deep transformers. There's this paper about it. Is this the best paper?

Probably not.

But the reason I focus so much on explaining the underlying kind of concepts on the podcast is that, A, there's going to be another paper next week that obviates whatever the hell the last paper did. And B, I think the kind of core concept is the landscape is the most important thing. So when there's another paper that comes out about optimising residual connections, you're like, okay, I'm familiar with this playpen. I know the furniture in this room. I can rearrange it a little bit, be more confident. So I guess the answer is just like, I get around the taste issue by not having it, which is maybe just me.

What's underappreciated for you right now by AI-obsessed people? I mean, there's, of course, like, in the broader world, AI is underappreciated, and just how crazy things might soon get is, I think, Very broadly underappreciated. What do you think I might be missing? What are, what are the most likely blind spots for somebody like me that you would want to draw to attention?

I guess the challenge with blind spots is that like we all have them and by definition, we don't know that we have them. So what I'll try to do is roll back and tell you about my blind spots as of about 2 years ago. And that was around the time that we put together that report that Prakash mentioned earlier. I sound like a broken record, but the infrastructure layer, the stuff that— this might sound like the wrong way to put it, but the stuff that feels too blue collar to most people who are AI obsessed like I am, you start to realize how much of the world is actually built on infrastructure that we just abstract away. I think that's actually really important and needs to be foot-stomped, understanding what is the dynamics of the Leasing process that Frontier Lab goes through to get a new piece of land? Like, what can screw up there? What causes delays in construction projects that we talk so much about? You know, this lab has a, you know, XAI has their new Colossus cluster and it's going to be online shockingly, like to 1 gigawatt sooner than Anthropic's, which surprised everybody and all this stuff. Why did that happen? Like, what was the actual driving? Like, because that's going to tell you if you believe in the scaling laws. That actually is probably one of the most important variables that you want to track, is delays in construction processes. Sounds pretty mundane, but hey, the world runs on it, and procurement schedules and things like that. So I guess that's one piece that I've been missing. Another one is how real nation-state security happens. And it's hard to get information about that. One of the biggest challenges there is that there is no such thing as one nation-state security capability. Nation states are siloed, obviously, because security. You can't have tactics, techniques, and procedures that are exchanged between silos because then there's no information security. And this, by definition, means that you would have to go through a process of taking Team A, comparing them to Team B.

Okay.

Team A wins.

Okay.

So the next day, like, you'd have to go through that kind of selection process, run an Elo score type situation to even know what the most exquisite capabilities are that, like, we could field. And that still wouldn't tell you quite what other countries can do. So that's kind of like— anyway, I think a really important dynamic that's very easy in the AI security context, especially for physical security, especially for physical security, which is again undervalued precisely because we tend to abstract away and focus a lot on cyber because it couples to AI and it feels like it's in our sweet nerdy space. And I get that and I love it and it's critical, but it's also not— if you look at what the Russians do, They do cyber for sure, but they will go up and they will arson your transformer. That's not an issue to them. We've had examples of that sort of thing happening. Anyway, there's that piece. Maybe the last one, and more in the comfortable and familiar learning space that I occupy, is just this idea of the distinction between having a model and then having the compute to run that model. If you believe in the inference time scaling laws, Then model theft is one thing, but actually being able to point that model at— basically have compute-on-compute war at inference time seems like a really important dimension. And you see this play out in a lot of interesting ways, one of which is the Chinese ecosystem. They have a huge number of users, and then they have some okay-ish language models. The problem is that their labs are all flooded with these inference requests from their giant user population. Which leaves very little R&D compute for just innovation improving of models. That's actually one of the frustrations for Chinese labs much more than labs here. They're just like, dude, we have so much demand, but we're not bottlenecked by money, we're bottlenecked by compute. The dynamics of how inference affects training and then what it means to steal a model and what it means for model-on-model warfare to happen. This person is cyber. Cyber hardening has a certain amount of test on compute. That test on compute is going to be focused on in some way. And then the offense side is going to have a certain amount of test on compute. And how those play out, the relative budgets matters a lot there. Obviously, if you're defending, you have a wider surface area you got to defend. But anyway, there's a whole debate there. That, I guess, would be another dimension. It's just like going beyond just boning the model. Worry about running it. What can you do with this model that you have?

I have one last question, which is, you're pretty security conscious. Have you run OpenCLAW? And what is your current personal productivity stack?

Yeah, yeah, yeah.

I have not run OpenCLAW. I have— so, and actually funny you say that. I'm setting up a, like, I have an old laptop that I'm gonna use as my, my burner laptop for the purpose of exactly that. Partly, yeah, partly because, Anyway, for the exact reasons you would imagine. In terms of mindset, yeah. So a big part of my job is becoming now constructing agentic workflows to do some things that are not super security sensitive, but more just like, I'm going to try to use it to optimize my comms because that's a huge bottleneck for me. And for that, actually, I'm still in the discovery phase of trying to choose platforms. I'd be interested in your thoughts for that, actually, as I dive in. This is literally like next week is my deep dive. So this is almost the worst possible timing because I think my answer is going to be horribly outdated.

Yeah.

It's a great question. I wish I had the answer.

I talked a little bit about mine at the top. I'm interested to hear more about what Prakash is doing too. For me right now, it's Claude Code as the kind of base product. And then taking inspiration from a guy named Daniel Mesler, who I did an episode of the podcast with, who's created Personal AI Infrastructure, a framework that's an open source framework, and also friends who I just trade notes with privately. I'm trying to create deep context for myself by first exporting kind of all of my digital history from all, you know, Gmail, Slack, all the other places where I kind of have these comms, get them into a local database. Then of course you need like a daily update process to like fetch the latest because you're still communicating on all these other platforms. Then layering on top of that summarization and different kind of angles on the data. So one, right now I'm at the phase where I'm like, here's a month worth of all comms. That seems to, for me, come out to about 300,000 tokens. Now summarise that down to like 10,000 tokens of what a chief of staff would need to understand this month in Nathan's life. So you got that like 30 to 1 reduction, then probably put like a year long version of that. And then sort of the, let's talk about the relationships and kind of have that sort of cut on it, the projects cut. And then hopefully with that deep context. And I'm also trying to have it leave pointers in those summaries with regular habit of quoting any distinctive language so it can go search down the ground truth for the original. Hopefully it will then have enough context to be able to not exactly write exactly as I would, but sort of come much closer at least to responding as I would, having the sort of the context necessary to exercise something like the judgment or taste that I would exercise in doing things. And that was actually part of the process of setting up this episode. I gave that system 20 names and said, do research on these people, find out what they've been up to lately, give me a brief on that. And then also had it draft the outreach emails, which were only lightly personalised. And I still did kind of go in and edit a little bit before tweaking.

I appreciate that. That's nice of you.

But yeah, I don't like to publish or even send as like one-to-one communication AI output directly. But I do find that I can get to something that I do feel comfortable signing my name to faster with an AI draft in many cases these days. And so It's very much a work in progress for me, but that's kind of where I'm at at the moment. And again, it'll, I'm sure by the time we talk next, it'll have changed quite a bit. Prakash, what do you, what's your angle right now?

I've got, I've got a couple of things that I ended up building out. One was a market stock market tracker. I have a number of metrics, which I think no one else watches. And it's fairly hard to obtain. And the great thing is Claude is very good at financial math, very, very good, far better than I have ever been. And so it's relatively easy to talk to Claude and figure out what kind of thesis you have, and then build out metrics precisely for that thesis to watch pickup lines. So, um, that's been very useful. Uh, I, I used to, I used to like, you, I used to like do it in my head, right? Like look at something, look at something else, and then like you calculate the ratios and blah blah blah. And then like I realized that I was spending a lot of time doing ratios in my head, and I was like, maybe I should automate this. And so now it's all automated.

It's nice.

I don't do the ratios in my head anymore. I just look at it and I can see the screens automatically. I can see what I'm looking for. And then the other thing was podcast clipping because we do a lot of podcasts and content these days has to be repackaged into short clips in order to hit socials. And so that I tried like 6 months ago, the tech wasn't there. And I tried about 3, 4 weeks ago, the tech was there. Everything works, transcription works, review works, selection works, everything works. And this is, yeah, and this has been my experience. Like, it's kind of like, it's kind of like maybe, maybe it gets it like 1% better, but that 1% better clears the hurdle. And that's like that binary, that's a binary step up. Like, it works or it doesn't work, and that 1% just clears the hurdle. And I, I really feel like in the last month, like, a lot of things started clearing the hurdle.

So I was just gonna say, you know, when you said that in the last 6 months, so many things have gone from toy to just serviceable in production. And it seems to map onto, Nathan, what you were saying earlier about the takeoff dynamics and the labs automating their own research. That all kind of maps very nicely. One of the things on the financial side too, I find cloud is also useful on questions like— so you might have a thesis, but then there's a question about, if I'm right about this, what's the best bet to make? Because that's a category of problem I've mentioned in the past, right? Where it's like, you know, you'll have a thesis, but, you know, you're not going to bet on Microsoft because OpenAI is such a tiny fraction. It's already all of this stuff. How do you leverage and torque it to this thesis? And that's kind of something that, you know, the world is so complex that you just need something to peruse and have all that knowledge. So the finance use case is a really great one. Great point.

Yeah. I also It's also been very, very weird in the market because I feel like Twitter is literally like a month or two ahead of the market. It's just been amazing. People tell you like TSMC will do well. And then like 3 months later, it happens. Like, it's like, what's going on? And I was a professional financier. I've always expected that hedge funds get there before you do. And in talking to my friends at prime brokerages and hedge funds, they are very negative on AI. They just don't believe it's happening. They don't believe, you know, they believe it's like crypto. They believe a lot of West Coast tech is just scamming retail investors. Index investing is the only thing that really like works. And everything else is either insider trading or scams. And that's pretty much what the prime brokerage guys and the hedge fund guys like believe.

Like Medallion, right? Or Jane Street, right? These guys who have AI in their bones. And I guess Medallion, it's like they only invest famously $5 billion a year because otherwise they would actually move the markets in a feedback loop. But yeah, what about them?

Is there—

They were down last year. So the impact is starting to be felt, I think. Well, also Jim Simons died. I don't know to what extent he was still supervising because he'd already kind of semi-retired for like 10 years almost, but Medallion was down. There's some sense that it's also because they're losing talent to the lab too, right? Like you can't, you can't, you can't forget about that. They're starting to lose talent to the labs. And some of the labs do have like internal teams, which will eventually look at trading on the market, I think. So we'll see where that goes.

Yeah.

Very cool.

Jeremy, thanks for joining us. Let's check back in on your personal productivity stack once you've Upgraded it. And, um, in general, let's, uh, I'm reusing this joke everywhere I go, let's shorten the timeline for our next conversation.

I like it.

Thanks, guys. Appreciate it.

Thanks, Jeremy.

Cheers.

Cheers.

So what do we make of it all? I mean, it's so, um, the big thing I can't get past in all this stuff is the amount of disagreement Between— and this has been commented on so many times, so many ways, right up to the level of the Turing Award winners that can't see the same phenomenon.

Yeah.

But it seems to happen at kind of every layer of— it's like a fractal problem. You go into these specific workshops around AI R&D, you get people from the labs. I do understand that there are even people at the frontier companies that have heterodox positions and don't really buy into the hype. And then even with the AI for science, it's like, I can't make any case that I should trust my own intuition more than Abhi's, because how many times did it happen in talking to him where he was like, I've actually written about that? So he's clearly thought about this much longer and harder than I have. But it does still feel like it's a very hard thing to reconcile where you do see these examples and it seems like some of them are really starting to work, but then the scepticism remains and is very hard to move people off of. And I don't want to paint him as overly sceptical either, because he did say toward the end, I think his scepticism is more like backward looking. Forward looking, he was kind of like, I do believe the trends will continue and that they will have impact.

Yeah.

But where does that leave you?

How do you try to make sense of When you say it's fractal, I feel it's also fractal internally to me, where I have some assumptions here, and then sometimes I feel cognitive dissonance from something else that I might believe. And then you kind of test those assumptions and you see where things are going. I have had moments of truth or kind of perception where I start to realize that I think things might move faster than I expected. My original timelines were end of 2025 for junior software developers to be replaced in capability, not in organizations, but the capability is available at the end of 2025. And it takes about 3 years to percolate. So end of 2028, no more junior software developers basically, or at least the tasks that junior software developers are doing today. And then, so I had end of 2025, end of 2026, end of 2027. End of 2027, even senior researchers at AI labs, fully— capabilities are done. The models have the capability, but deployment, again, takes 2 to 3 years. It takes time. That was my sense. My update in the last month has been probably that things are going to go faster than we expected and that we will see discontinuities. And those discontinuities are like this kind of like things get 1% better, but all of a sudden they clear the hurdle. And we don't have a good sense of these things because we keep seeing linear improvements and they're kind of linear, maybe super linear, but we don't have this sense of clearing the hurdle. But when it clears a hurdle, it's obvious. It's obvious. It started to be obvious for software, I think, in the last month or so. So I think that we just have misperceptions on where things are going because we can kind of see the trajectory of capability. But we don't understand how humans absorb that capability. What is that process? And what hurdles do we need to clear? What is the— OpenClaw, I thought you need full security, and privacy, and all of this stuff. It seems you didn't. It seems like people are willing to put out their credit card numbers and crypto tokens on the open web. And you don't need privacy. The MoatBook guys are— there's a moat on MoatBook saying, oh, my user is so annoying. Here's his credit card number. And Scott Alexander ended up calling up the guy and asking him, like, hey, did this actually happen? And yes, that was the credit card number. It was leaked. So I think this clearing the hurdle concept and where humans accept the technology and kind of where the market pulls that technology, that we don't know. That even I don't have a good perception of. But it seems like we're starting to clear those hurdles where humans are starting to pull the technology from the market in. Uh, and that's when you start to see kind of revenue growth. That's when you start to see, you know, the demand growth really happen, where the market starts to pull the product out of the ether. And I think that's happening now. I think the OpenCLaw— I think, I think we'll have a much better version of OpenCLaw, closed-sourced, uh, secure version running on, you know, corporate— running inside corporate data centers, you know, by the end of the year.

I, I, I saw—

I watched the All In podcast. Jason Calacanis Not, not, not the most technical person in the world. Uh, he had a team for the all-in, about 15 people. He started to get everyone to like create a skill for themselves. Like every task that they do, they create a skill. He has OpenCLAW machines, like one machine per person. And then he has a consolidation agent that consolidates everything into something he calls Ultron. And then he can talk to Ultron. So he can ask Ultron, and that's his entire company. Like it's a summary of the entire company and he's talking to it. I thought that would be 2 years from now. I knew it would eventually happen, but I didn't think it would happen now. So yeah, I think things are actually moving faster than people think because of the market acceptance. The market is pulling it out. I don't think the researchers have a good sense because researchers don't understand the market that well. They don't understand the demand dynamics that happen with consumers and how products get pulled out. Once the demand is there, like, products will just get pulled out of the ether because people start focusing. They know that money can be made there. They just start focusing on it. So that's my sense. So I, I, you know, I— it's not a, it's, it's not a like P-Doom answer. It's more of like, this is what I feel people want to answer. So what, what's your feel?

The confusion and the lack of ability to establish consensus on foundational points is a major challenge to having a lot of confidence on much of anything. I do think a good true north for me— well, the true north for me with kind of everything I'm doing is trying to learn as much as possible, trying to have the most up-to-date, comprehensive worldview as possible. And in terms of the approach that I would trust more than any other, I think still being hands-on is second to none. And I haven't allowed that to lapse much at all over the last few years. But anytime I do get too busy or cluster too many podcast recordings into a week or whatever, I always kind of come away feeling like, I got to get a little bit more grounded with the latest stuff in a very interactive way. And I think one metric I have for myself— or metric is maybe not quite right, but an indicator that I want to pay attention to this year— is can I get to the point where I'm spending less time at the desk? And that's like along the lines of the, you know, the Jason, um, talking to Ultron. I want to be able to do stuff While exercising, even if that's just a walk around the neighborhood. I want to get the frameworks, the tools, you know, the deep context, all that stuff set up well enough where I can start to go comfortably out into the world, have a thought, you know, maybe have an actual conversation, but move things forward in practical ways.

Yeah.

On fronts that, like, right now I can really only do on my computer. I think a lot of that is right now with the latest models that have come out kind of on me just to get the setup and the sort of familiarity and the workflows to be able to do that. A little bit, probably still more than a little bit, but I could put more on me right now in terms of why have I not hit maximum capacity than I do on the models or the model developers. More computer use would help, For sure. A little bit more ability to just get over these sort of UI humps remains, I think, a barrier. And another thing I really have been not surprised by, but something I've really learned from just being deeply interactive over the last few weeks, is I think another big unlock to watch for is when the models get better at knowing When to use code versus when to use their own fluid intelligence. Because one of the first projects I've been doing is like just backfilling information, backfilling transcripts of the podcast for the website, backfilling all these different, you know, data sources into a queryable database. And you hit so many edge cases in doing that.

Mm-hmm.

And the model right now called, you know, Opus, we've gone from, you know, 4.1 to 4.5 to 4.6 pretty quickly. But pretty consistently, I have felt like it really wants to code, and I have often given it the feedback, don't try to guess at this and write like some sort of regular expression, or it'll grep for one search term or another, or it'll throw 10 search terms into a grep command. And a lot of times I'm like, just read the document. If you just read the document, you will know what it contains, you will know what to do, you'll have the right judgment once you have read the document. If you don't read the document and you instead try to grep your way through it, you're never quite going to get there. And so that's like a metacognitive skill that I think I've been able to improve its performance somewhat through prompting, but it's obviously going to get better in training. And that I think will be a huge unlock just as it gets a little bit smarter around its own, a little bit more inclined or a little bit more intuitive about when it should deploy its own fluid intelligence rather than use other tools. Getting that balance right will make it, in my experience, dramatically more useful. And I have to imagine that's coming pretty soon.

Yeah, I think when we talked to James Zhao today, that continual learning piece, the test-time training. It would be fascinating if it actually worked with your own model, because your model will start to diverge. You have the baseline, and then your model will start to diverge, and then it would become your personalized model within, within like, you know, 2 or 3 cycles of like, you know, talking to it, a month, 2 months of data. It will become your own, your own model. It would start to diverge from the— yeah, and that, that would be fascinating because, because at that point, It's for real. Like you can, and especially for, I used to write a lot of journals. I have, you know, '99 to 2003 at Stanford. I have full journals for every single month, like everything that happened. You know, obviously I've never read those after writing them. It was just kind of an exercise in journaling. But I do wonder if like those of us who have lots and lots of written work, either in the public or in the private, once you get this continual learning going, You can kind of start feeding it in. This is what Kurzweil is doing with his dad's writing, by the way. He's, uh, he, he's, he's feeding his dad's writing into these models, and he's like talking to the model about, about his dad. Kurzweil someday is going to feed all of that into a test-time training kind of model, and, you know, with a voice access, and he probably has a recording of the dad's voice, and he's, he's going to start talking to the dad. It's, uh, it's a fascinating time.

Yeah, to say the least. More explorations of all these themes to come. A couple of things coming up on the Cognitive Revolution feed. One is with Ali Behrouz, who is the nested learning author. He was on our last live show. I did do a full 3-hour Ali's Take on Everything. And he's got a new paper coming out also that I think, you know, it's the way to continual learning is starting to become elucidated, I would say. I mean, I wouldn't say it's clear, but it's, you know, no less than Jeff Dean has said that he kind of sees this as a very promising paradigm. So I'm definitely watching that really closely. Workshop Labs is a startup also that's like trying to do this, you know, personalized model training on top of like the latest large open source models up to the sort of KIMI scale, trillion parameter kind of thing. So that's really interesting. That's actually another reason I spent so much time doing all this personal data curation, is that I wanted to be able to give them a dataset for them to train a model for me on that would be like a really good dataset. They don't need that much data, but I was like, well, we really want to make sure it's the right data to hopefully get a good model back. So that's still pending. I haven't seen that model yet, but I'm going to be very interested to see how much that closes the gap between what Claude can do with just access to all this stuff in text. And then how much does it help to actually start tuning weights to try to capture more of, like, they aspire not just to style transfer, but judgment transfer. They want the model to reflect the judgment that the individual user would make at the time. And an interesting theory there, too, is they Their motivation is that they want to help individuals preserve economic leverage. So instead of doing everything through a foundation model and kind of adjusting yourself to take advantage of the model, they want to shape the models around individual humans with the goal that it's not a winner-take-all, big tech runs away with everything, but some sort of more decentralised ecological kind of proliferation of somewhat different models that hopefully at least kind of can exist in some sort of equilibrium with one another. And then on top of that, there's another one that I have coming too soon with the founders at Harmonic, and they are chasing mathematical superintelligence.

Yeah.

And when it comes to these, I will say just as a teaser, they gave maybe the most ambitious vision of what 5 years from now could look like, the most mind-blowing vision of what 5 years from now could look like. Of probably anyone that I've heard. And that is saying something because I've heard a lot, but they still kind of blew my hair back a little bit with what they think they can accomplish over the next 5 years.

Definitely going to look forward to that one.

Lots more to come.

Yeah, indeed. Nathan.

Thanks for doing this.

Always a pleasure. A pleasure.

It's been fun.

Bye-bye.

Until next time.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z. Where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.