AI:AM — Build, Measure, Heal: AI's Three Frontiers · June 4, 2026

So, uh, good morning. It is, uh, Thursday, June 4th, uh, 2026. It is the 4th day of our, um, marathon, like, daily live streaming. And, um, we have— it's been an interesting, uh, overnight. So let me just, let me just start off with the markets. Uh, so overnight, um, the major market news is Broadcom had its earnings and our friend Hock Tan, who is the CEO, made a little bit of an error and he started reading and he read the Q2 2025 results. So he was like, we did a max, we did a record revenue of $15 billion or something like that. And immediately the market tanked because everyone listening in on it is a bot. And the bots were like, oh my gosh, the revenue is not what we expected. Market tanked. Broadcom stock down 15%. And this is a trillion-dollar-plus stock. It's $150 billion of value, like, just erased. And then he's like, you know, and literally it's like one sentence. And then he's like, oh, oops. Actually, it's Q2 2026, and we had a record revenue of $26 billion. And it was just too late. The stock was down 15% immediately. And then later on, people tried to make a narrative to justify it saying like, oh, Google is diversifying away from Broadcom. They're going to do chips with MediaTek. And, uh, you know, Hock Tan's pointing out that— Hock Tan also pointed out that, look, our margins are not going to stay this high because the, the amount of spending per megawatt doesn't look like it's going to increase. But the number of megawatts, the number of gigawatts is going to expand dramatically. People are underestimating. So he's like, look, our market size is getting bigger, but our margins are not necessarily getting— going to get— go higher, and the margins are probably going to decline. And so the market kind of justified the reaction with that later statement, but the damage had been done. Like just that kind of miscue, coding last year's numbers instead of this year's numbers drove the stock down. So I thought that was pretty interesting. And our friend Jensen Huang at Computex, Nvidia announced a 10 or 20% reduction in the memory that they use per chip for the latest chips. So they're actively trying to push down the amount of memory that they're using, which is kind of what I alluded to yesterday, that people would start to design around the memory. And so those were the 2 interesting things that happened. And as a result, immediately what the market does is often it tries to find ways, because Traders in the market, especially the big banks, they want the volatility. They want people to like, oh my gosh, the stock is down, I gotta sell. Oh my gosh, the stock is up, I gotta buy.

Right.

And so the moment the stock— the moment there was some sign the stock was going down, immediately some bank puts out a note saying that, hey, Micron margins are gonna decline. It's gonna go down 66%. So Micron stock is somewhere there, and they're like, it's gonna go down 66% in the long run. So immediately Micron is down like 6% today, another $60 billion loss. These companies are so big right now that every 5% fluctuation is S&P 500, like new company joining. And so that is the state of the market overnight. And just to kick us off, I think I will share The what we were thinking of doing today. This is the the DNA screening. Let's there we go, and let me let me pull up the DNA screening. And there we go. So this was the. Top AI CEOs call for law protecting against biological weapons. It sounds so crazy. Nathan, I'm not sure if you read this, but what did you have that you thought about this one?

Studio experience is lagging a little bit, so just want to make sure Sure, I'm coming through. Seems like not—

you—

I'm—

yeah, it is. Okay, cool.

I'm here.

Yeah.

Um, yeah, well, this— so yeah, a couple threads I wanted to follow up there. I'm sorry, I kind of shorted out on you for a second, I think.

Yep.

Um, this call for A law protecting against of DNA sequences. I think it's a very good idea, and people go to public access, and we either don't have a great way to control them, or and or haven't hardened the world around them. So my understanding is that this has been done. By an executive order in the Biden White House, reinforced with an executive order in the Trump White House, and now, you know, they're pushing to take it all the way to force of law. And the basic idea is just that there are still companies out there in the DNA synthesis business who don't bother to screen your order against a database of known pathogens before they go ahead and produce it for you. And I think it's ultimately probably going to be quite a bit more sophisticated than just against a known database. It'll also be guesses probably made by frontier models to make sure that even if you've intentionally tried to get around detection, that they can hopefully detect as well as possible. But this seems like a very good idea. If there's one thing I know for sure it's, uh, a pandemic is really hard to put back in the bottle once it's out. And, you know, things like anything that can self-propagate, you know, anything that, uh, finds a, an uncolonized world just waiting for it to take over the world is something that we should definitely be very careful about. So I support this initiative, and it's kind of striking, honestly, that it, it needs to come to this. It's a little bit like, hey, guys, you know, couldn't you have been shamed into doing this on your own? There were already multiple executive orders around it. I guess I don't know for sure how many companies are not doing it. You know, this, this may be codifying something that has become standard or pretty close to universally standard practice in light of those executive orders. And it's just a matter of shoring it up. That'd be an interesting question for somebody who's deeper in the weeds than I am around, you know, just how many companies are still not doing it. But I think it's absolutely a good idea. And I think we can, you know, watch this space for all kinds of stories. You know, the sort of mythos moment for biotech is coming not too long behind the mythos moment for cybersecurity.

Indeed. You know, as you know, I'm a little bit of an accelerationist. Believing that the future should come faster. And one of the things that has changed my opinion over the years is I started reading about, you know, how technologically sophisticated terrorism has worked. And to date, actually, there's only been one group that has been technologically sophisticated, and that was the Aum Shinrikyo group in Japan. And they did the sarin gas attack in the Tokyo subway, killed about, I think, 47 people. Those guys were a billion-dollar enterprise. And they ended up using— reading through it, I was like, this is not very dissimilar from what a private equity group would do because they ended up using a bunch of shell companies to acquire chemical plants. In Japan. So they were $100 million chemical plants that could produce sarin gas precursors in bulk. And they had the engineering talent too, being Japan. They had the engineering talent to do this. And what struck me was that You always think of the Al-Qaeda types who are in a cave in Afghanistan, and the best that they could do is that they could come to the US and learn how to fly a plane. They didn't manufacture the plane, they didn't make it. They exploited basically a social loophole there in order to get access to these tools, and then they used those tools. And the Japanese attack was very different because those guys actually bought a plant and modified it, bought the precursors, and they did their actual manufacturing themselves. And it struck me that that kind of attack would be much more easy to carry out if the people carrying it out were uplifted, basically, in a sense, by this kind of greater intelligence. So that has kind of changed my My viewpoint a little bit. And, but that strikes me that the answer to that is probably much greater kind of controls and surveillance, which is, I think, one of these things is, you know, this law protecting against biological weapons is really about identifying people or labs that could produce in bulk, in mass. It's not about stopping the weekend chemist in the backyard.

Right.

but it's about identifying and stopping people or entities from building up the technological capability and the sophistication and the industrial capacity to actually carry out a mass attack. And so that strikes me as quite different from just saying like the weekend chemist cannot, is not allowed to do this.

Yeah, I don't know how hard it would be to do a DNA synthesis on your own in the garage, so to speak. It does seem like this is a fairly important choke point. I mean, there's no doubt that if you are well-resourced, motivated, sophisticated, you know, have a coherent team, that it would still be possible to Do something like this. But it is hard to build those, you know, those organizations. We do have a lot of surveillance. It is hard to build those organizations without somebody becoming wise to you. It's hard to maintain the coherence of a group that wants to do something like that without somebody deciding they think better of it and going and, you know, telling law enforcement or whatever. So the— this does seem like a pretty important choke point where if you are a lone actor and you're not super well resourced, at a pretty affordable price point, you could send in a sequence and get a small amount of whatever sequence back and then do something with it. And if you didn't have that, I'm not sure how much harder it would be to, to do it on your own, but I think it would be like quite a lot harder. I think synthesizing these DNA sequences is still not easy. It would take you a long time. You would probably have a lot lower fidelity if you could manage to do it at all in your garage. So it does seem like nothing is foolproof. This, of course, wouldn't catch every possible scenario, but it does seem like one of those things where you got to believe you get at least an order of magnitude risk reduction along this specific dimension of somebody wants to cause a pandemic and they have a sequence, can they actually get that to a live result if you could actually close down their ability to use the companies that do this on a, you know, small order basis? Because most of these things are coming— most of these orders are coming from legitimate academic labs, right? So they're not like buying high scale, uh, most of the time. Yeah, it seems like— I think, I think it's a pretty Strong idea, and I do think you can get a lot of value without having massive surveillance, although, you know, we might end up there, we might need to go there for any number of reasons as well. You know, order of magnitude risk reduction in pandemic preparedness may not be enough where we're going. I was really struck in the course of my son's cancer treatment We bought an EUV light. There's a couple different frequencies that have been demonstrated to work pretty well for killing pathogens. And we took one that you can just shine into ambient space into the hospital room every time we went in there just to try to protect him while he was so vulnerable from whatever, you know, infections that it was flu season and all that. And nobody at the hospital even recognized what it was or had ever really heard of it. And we got a couple interesting, you know, questions from people who, you know, were curious enough to ask, and their responses were like, oh, it seems like that should be in every room. I was like, yeah, it really does. We could— and there are other versions that you would maybe put more into like the, you know, the forced air systems so that— because those are more harmful to skin, harmful to eyes, but in a confined space they can still work really effectively. It really is, I think, a tough commentary on human affairs that so little has been done on this front.

Mm-hmm.

Wastewater monitoring is another one in terms of surveillance that really doesn't harm anyone. You know, it doesn't really restrict your freedom of speech at all for people to be monitoring the wastewater in major cities. But my understanding is that isn't really happening nearly as much as it probably should be as well. So I'm glad to see this. I hope this is what, you know, the first in, um, what'll probably be, you know, a 10-plank, uh, platform of defense in depth against pandemics. I didn't enjoy the last one very much, and I certainly don't think the next one will be all that much fun either. So I would like to— what do they say— delay, detect, and defend as much as we possibly can.

Indeed.

Speaking of delay, detect, and defend, so I'm going to switch gears very quickly to the issue of Sherlocking. So yesterday, OpenAI dropped OpenAI Sites, You can build, build, share websites directly from, you know, ChatGPT. So you can basically, once you build an idea inside ChatGPT, a revenue dashboard or something like that, you can publish it as a mini app. And you can even do authorization on it. So you can decide who gets to see it. And they can come and they can log in, they can see it, they can use the app. And yeah, so this is obviously something that people have been putting together.

The few—

the companies that would be involved are Lovable on the consumer side and Replit on the enterprise side. So as you noted before, what do you feel, you know, the site— what do you think of the site's idea, first of all?

Well, it's— they're really starting to run the same playbook that we've seen platforms run in the past, from Microsoft to Facebook to you name it, right? It's kind of inevitable. I think they're— as they staff up, there's like nobody really to blame, and yet it is kind of a gross outcome, I think, in some ways.

Mm-hmm.

But I experienced this with Facebook directly, where, you know, at some point they were pretty open about like, yeah, the reason we have this platform is that so all of you can go out and do R&D for us and take the social risk that we don't really want to take. But then when we see things that work, like, we've got an absolute army of product managers here who are to varying degrees resting and vesting or motivated to advance. And regardless, you know, either way, it kind of makes sense for them to pick the low-hanging fruit of I saw something out there in the world that looks like it's working.

Yeah.

So, I mean, that's a little bit unkind, I'd say, to OpenAI. This is like a pretty obvious thing to do. It's not like, you know, if Lovable had never existed, would they still do this? Like, yeah, probably at some point. I wish I had a better answer for why I think They're not— they being the top however many intertwined cap table intertwined AI megacorps aren't going to suck up all the value. But looking back so far on the conversations we've had in this week one of this marathon sprint, I don't think I've really heard much that has convinced me that margins are going to be sustainable for people that are building wrappers, you know, I guess is still basically what we're talking about around frontier models. I have a real hard time seeing it. It doesn't mean they don't have a business at all. But it does mean it's gonna be hard to sustain margins. The barriers to entry there are gonna be really low.

Mm-hmm.

It's gonna be really tough to compete with, I already get this for free in Codex or CloudMax or whatever. So yeah, I think it's going to be tough. I can't say I've— I haven't been bullish on the app layer for a while, and I can't say I'm any more bullish coming out of this week. And these announcements only reinforce that point of view, I think.

Let me segue a little bit there. And so Satya Sachin Nadella yesterday saying OpenAI is, let's face it, Anthropic over time or OpenAI over time will build their own clouds. It makes sense. They would use, I'm not saying that they won't use other cloud providers, but they are gonna build their own clouds. And so now you see from that, from the model layer, you see going upwards into the app layer, but also going downwards into— the data center layer, Microsoft is thinking that, you know, they are eventually going to end up competing directly with OpenAI and Anthropic on the data center side as well. So in terms of this platform, one of the, one of the things that you see here is that the platform is not only going upwards and absorbing margin from Lovable, but also going downwards and absorbing margin from AWS and Microsoft and, and the rest. Which I thought is really a commentary. The fact of the matter is in order to get a return on investment on this trillion-dollar valuation, you only have 2 pathways. And one is create new value. And the 2 is eat up existing value. So eat up existing value means you either have to go horizontal and absorb your competitors in the model layer, which means fighting against OpenAI fighting against Anthropic, or you have to go vertical and you have to absorb people who are using and people who are providing you services. And so I think this is where this kind of model, the AGI firms are basically like spreading outwards in every direction, like vertically upwards, vertically downwards, horizontally against each other. It's just like a virus almost, right? It's just spreading in every single And it's just a commentary that how much capital that they have taken in at what price means that the urgency to grow revenue at all costs is definitely there. And you can see that on a day-to-day basis per se.

Yeah.

They're going to go down to chips as well, right? I mean, that's Kind of already announced and pretty obvious, but there's really no end to it. Sam Altman was just in Michigan, not too far from where I live, I think yesterday at the groundbreaking of a gigawatt-scale data center, which I don't think has been super well received. I haven't been too, you know, I haven't been involved in the local politics, but there have been some Kind of shady maneuvers there where like, uh, it's been very, um, the details of what they're building have been very hard for the public to get access to. A ton of stuff has been redacted. Um, it seems like they've kind of gone through some unusual channels to get these things, uh, fast-tracked, which I'm sympathetic to frustration or objection to that. Um, if only on the basis that like everybody should probably play by the same rules, even if those rules should be changed. But as it stands, you know, they're— I don't think they're super welcome. But yeah, they're going to be building out every layer of the stack.

Yeah.

Again, I think it just speaks to the value of the intelligence layer just being so high. And who has a better chance at eating the other one's opportunity You've got Nvidia yesterday putting out this new model that— let me get the actual number on it here. It's 550 billion parameters, whatever. I'm not quite sure at this point which benchmark score, if I had to pick one, would be my absolute go-to, but I think it would be GDPvalAA from Artificial Analysis. I don't know if you have a different take on this, but interestingly, their latest and greatest model from Nvidia comes in with a score of 1448, and that would put it somewhere in the range of maybe the 20th best model out there, maybe a little higher than that. But for comparison, Opus 4.8 is, at least with Max Reasoning, is now 1890. So it's like 400 and some points ahead, which translates to a pretty high preference rate. I think that's like a— let me, let me just Google it real quick. Elo score advantage, 450 points. Win rate. Um, yeah, so you're expected to win 93% if you have a 450-point advantage.

Mm-hmm.

So NVIDIA is trying to, you know, commoditize their complements, right, and make a model that they'll give away, and then you don't have to pay the Anthropic premium, and then all the, the value accrues to the Trip. But if you're only winning 7% of the time compared to the latest Opus, yeah, you haven't really succeeded in commoditizing that complement. And it does feel to me like I would bet on Anthropic to make a good chip before I would bet on Nvidia to catch up with Anthropic at this point.

I, I, I, I think Perhaps the catching up will not be that necessary because, you know, I end up— so I had this experience where I was setting up the OpenCLaw and I started off with Codex as the underlying because OpenAI allows you to use the Codex inside OpenCLaw. And Codex was so-so, but it was also like I knew I was coming to the end of my, you know, OpenAI had given me a 10x, uh, you know, bonus for the month. So I had like extra tokens. And so I was like, okay, I need to switch this out. Let's see, let's see what happens if I switch it out. So I tried, uh, DeepSeek, and I asked Claude, and I asked ChatGPT which one I should use. They told me, you know, start off with DeepSeek. So I started off with DeepSeek Flash, uh, V4 Flash. V4 Flash was okay, but a little bit so-so. But some, some parts of it were already better than Codex, which was interesting. And then after a while, I was like, okay, you know what, I want to switch to something higher. So I went for the DeepSeek 4 Max and a DeepSeek 4 V4 Pro Max. And that is genuinely good and is actually, I think, on some tasks, better than Codex. And especially, you know, the recommendation given by Claude ChatGPT was to use V4 Pro Max for difficult coding tasks. I'm not using that for difficult coding tasks. I'm using it for very general kind of API MCP usage in order to extract information and put in information. So for Codex, Codex I'm using for complex coding. So I think if you start off with a model which people are using for complex coding, and then you get a model, an open source model, which on that kind of index is maybe only 30 or 40% of that index, fine. But then if you take that open source model and then you apply it to a very simple task, like organizing your calendar, you have this kind of overhang of capability on the intelligence side. And I think more of the tasks will be in that pretty simple to do over time, I think, especially if we really imagine this kind of intelligence explosion. In 2 years, a single model will be smarter than all of humanity. You don't require that kind of model for your day-to-day life. And so perhaps we have this kind of edge frontier where things are very well developed and those are people who have frontier-level tasks will need them. And you can imagine paying whatever amount if you're going to get a a model which is able to deliver a Nobel Prize every month or whatever, that's amazing, right? But day-to-day, you don't need that. The average person doesn't need that. So maybe that average person is okay with a model at 1% of the capability at 1% of the price. That sounds like a good kind of trade-off, right? And I think that's what NVIDIA is doing. NVIDIA is ensuring that they have a baseline of customers because their biggest problem is that they have, their customers are very consolidated and concentrated. Like 60 or 70% of their revenue is just like 5 customers. And the problem with that is that you can kind of see an avenue for collusion where the other companies find a way to force Nvidia prices down. And to some extent, The whole AI safety thing about not giving chips to China is really a price negotiation on Nvidia and TSMC. It's about, you have limited capacity, this capacity should be going to US customers. And then the US customers have their own price points. The US customers do not have to compete with Chinese money. And I think that's an underlying subtext, which people don't really want to talk about because it's a little bit collusive. And so the bigger companies to some extent have used this kind of chips ban idea to constrain Nvidia's market. And so that they don't have to compete with other people, the Chinese for the chips. So I see that too. And Nvidia is now trying to get out of that hole. And they also announced Windows laptops with smaller chips that you could have AI laptops. They're trying to get out of this constrained, 70% of their customer base is just like 5 customers and trying to get a wide, the widest base of customers possible so that they're not squeezed out in the future. So I find that interesting move by Jensen that he doesn't, he's very well aware of that and he's trying not to get price negotiated.

Okay.

Well, tell us about your company. You're building for solopreneurs and you guys are running BackOffice. We were just talking before you joined about the NVIDIA model yesterday and how it compares on benchmarks like GDP, Val, AA, from Artificial Analysis to the Frontier models. And we're debating whether or not these open source models are starting to become competitive. Give us the 101 on the company, and then I'm really interested in your take on whether your customers demand frontier models or if you're able to, at this point, lower their costs and provide still the quality of service that they need with some of these open models like the one Nvidia just put out yesterday.

Great.

Well, let me tell you what I do quickly. I'll keep it brief, and we can jump into some of the more fun topics like that. So I'm the founder and CEO of Collective. Collective serves the 30 million solopreneurs in the US. It's the largest group of founders, largest group of workers. And so right now, 80% of people need a finance department. And so in AI parlance, we are trying to build the autonomous finance department. And so we are combining the power and outcomes of what you would expect from an accounting firm, everything from formation to tax filing with the efficiency and convenience of software, you know, single application, single conversation, all your workflows in one place, and also the prices you would expect from software. So we've been doing this for a bit. I think we're the only AI-native platform right now in the country who is at scale for accounting automation and actually proven this thesis out. So it's, it's pretty amazing. We've really been working with AI tools since day zero and have a pretty strong perspective on it. I would say it's a very applied perspective and a grounded perspective because our customer base is America, you know what I mean? So I'm in the Valley right now in the Bay Area, and I think all of us here think, you know, put a .ai on your business and, you know, really talk about it on the website. That's the best thing. I'm not sure it is. I'm not sure it is for everyone else in the country. So we can have that conversation. But with regard to the models conversation, You know, I've been talking to a lot of these different folks, whether it be, you know, the OpenAIs or the NVIDIAs of the world. I do think, number one, the open source models are getting materially better. I think the open question for the community, in particular the US community, is what do you do with Chinese models? Because they're really good. I don't know if you guys have tested those out. They're really good. And I think inevitably those open source models, they won't be top tier for every use case. And a lot of it depends on where you're hosting it, the optimization on inference providers. Like if you're going, for example, who are you hosting with, how are you tuning it and everything else? But it has to be good enough for most applications because the idea that, I mean, if you look at the token costs right now, in particular for Anthropic and OpenAI, I think there's a great apocalypse that's about to occur where everyone is going to have to switch plans inside of enterprises because, I mean, it's just getting way too expensive. And so it is working. I think it has to work. On our end, we look at it differently because a lot of the AI that we develop, we do have front-facing AI and conversational experiences with the application. But a lot of the power for us is what you don't see. Whether it be categorization of your expenses, you know, as a business owner to do your bookkeeping or reconciliation, you know, whether it's 1,000 monkeys or 1,000 robots, you don't care, right? We do because we want it to be delivered well for you and we want to make sure we do it efficiently so we can stay in business. And so whether we use Model A or B is irrelevant to our customers. I don't even think they know what Model A or B is. In fact, I think the vast majority of America could care less.

So, so the intro price point for Collective is $200 a month, right?

Correct.

And that feels like it's, you know, obviously right in line with like a Claude Max plan or a Codex Pro plan.

Is—

how many monkeys do you have in the, in the loop? How has that mix of delivery method changed and where does it sit right now? Where do you think it's going to be in, say, another 6 months or a year?

I mean, it's been fairly drastic, candidly. I think our business is at an inflection point, which not surprisingly is correlated with the inflection point that's happening in the marketplace. We're a full-stack AI accounting solution, right? And I think a lot of the full-stack solutions, and I don't want to speak for them because they don't operate, Crosby, for example, or Corgi. There's a lot of these solutions that their thesis is, I'm going to architect some service from the ground up using AI at the core. Now, ultimately, they're going to use some people in human loop. In the best case, it's for evals and exceptions, right? But most likely, they're doing some level of unit work. Now, whether that unit work is reusable in training the system, that's a different story, and that's an architecture question. But for us, we've just been doing it for long enough and we're far enough, we're at a pretty significant scale. The way I would answer is this, like a standard bookkeeper, which is one element of our service, if you were to go use QuickBooks or Xero, for example, you would probably be able to support our client size, our client type, again, solopreneur, handful of employees at best, right? Maybe 30, 40 clients.

Yeah.

For a bookkeeper. We literally can support 250 and it's doubling. So we're getting quite good. Our efficiency is pretty outstanding and it's starting to get there. Our margins are software margins and we are the first company to be able to do that. Everyone else, their gross margin is pretty poor. And in part that's because, you know, the way you implement and use tokens, the way you're designed to use CPU versus GPU, there's a lot of implementation choices which are going to lead to people— I think people are overusing inference. In a lot of use cases, right, we have deterministic models. Like, why are you paying all this money, like 1,000 times more than compute, uh, just to do it? So I think you have to be artful, especially because the token prices have not fallen in the way that I think people had hoped.

So, um, one of the questions that I had for you is that—

Gosh, we can't hear you at all. I don't know if you can hear us, but we can't hear you.

I think maybe you can call in. We can go old school.

All right, all right, you get— you, you guys go ahead. Let me, let me, let me, uh, fix that.

I can add.

Yeah, this is fun, man. We're literally talking about AI and all this crazy stuff and models, and we cannot get, you know—

troubleshooting at the same time.

Yep.

Yep. How much we want to be with you.

All right. So, we're merged.

Let's go.

When you mentioned— so first of all, just to recap that kind of ratio, you're at something like in the 5 to 10x range in terms of number of customers you can support per human.

Correct. Yep.

Which is pretty notable and definitely suggests major market restructuring to come, perhaps. You also mentioned using canonical systems like QuickBooks, Xero, et cetera. Obviously those come with a cost as well. Do your customers have to have those or do you see yourself replacing those, if not already, at some point in the future? Like, are you gonna vibe code your way to a QuickBooks replacement so that all they need is you truly and, and don't even need these other things? At all?

Look, I think today, and our customer segment is very unique but very large. It is a prosumer segment. They're consumers, so they're left between 2 horrible choices. Okay? They're spending 1 out of 4 hours on finance administration. And with AI rising, remember, AI is benefiting them as well. We have a customer that went from $200,000 in ARR to $700,000 in 6 months. by using AI, right? And they think they can get to $1 million. I'm seeing more and more of this. So there's this whole billion-dollar business of one discussion that's been occurring, and I'd like to point out for the record that we said it first. But regardless, those— that is, it's almost irrelevant. What do you— really soon you're going to start seeing $30 million, $40 million businesses of 3. Now, those folks don't want to hire a finance department. If you can, what's your N+1 hire? You don't want to hire a controller, you don't want to hire an accountant, right? And so they basically are going to use those systems to record. Now, our solution— the other solution, the alternative— is that they can drive outcomes with accounting and tax firms, right? But those accounting tax firms don't require you to go buy QuickBooks and all that stuff, right? I mean, you, you, they, you, they do that. That's obscured, obfuscated from you. It's the same as us. From the user perspective, for all intents and purposes, We, we serve as an accounting firm, right? They don't have to bring anything to us. We will handle it. There are things they have to do because they are the accountable party running the business, right? I cannot, unfortunately, you know, go and administer like certain things on bank functions and whatnot. But, you know, it's— I'm a member, by the way. I use it. I'm still a Venture Partner X, but I use the platform and it is about a third of the cost that I was quoted from my accountant. to use the same, to drive the same outcomes with software-driven solution. So to me, it's an inevitability that the space is heading towards like a really, really strong disruption. There's 50,000 small practice accountants right now that are serving 30 million people. Like, so in a couple of years, this is, this is really hitting it because I'm telling you today, not in the future, we've already— we finished it. We're done. And, you know, it's kind of like the person that breaks the 4-minute mile. I think once it's been done and the word gets out, it's going to happen again and again and again.

Yeah.

How do you think about maintaining enterprise value at the app layer? This has been a big discussion for us, and I know you're an angel investor who's had some very notable deals, including, I believe, OpenAI, Stripe, SpaceX, Uber. You're on both sides of this, right? As you're building at the app layer, but also I'm sure you're getting pitched all the time.

Yeah.

When people come to you, How often do you find yourself thinking, geez, this is cool, but like it could be a Claude skill? And then how do you make sure you're not ultimately a Claude skill?

I think everything's about scale of time, right? And I think if you want to be strict academic about it, if we take a really long scale of time, it's difficult to discern when and how this market's gonna play out. There is a possibility, and I think people are turning it into like a high probability, that a company who is a foundational model provider at super scale, where these things can do cogent, will start generating agents and replace the workforce. And so in theory, if Anthropic could generate all the agents that we can and then orchestrate those agents and the people willing to trust them data, could they do what we do? There's like, again, inevitability. We can see there's 4 companies in the world that you can— everyone talks about that happening. Now, what scale of time could that happen? I personally am not willing to underwrite that's going to happen within the next 2 years. I can give you some reasons why. And so for me, what I think you have to do as a business owner is you have to operate— of course, if you're in an obvious strike zone, like for example, I personally would not want to underwrite into a company that is in the core strike zone of OpenAI or Anthropic. Like I'm seeing a lot of AI assistant type work that are consumer facing and they're like, we're going to be better at managing X, Y, and Z, like your inbox, tasks, whatever. I'm not suggesting those companies won't do well. I'm not suggesting that they don't even have value. But Claude and, you know, OpenAI's ChatGPT most certainly have to develop that competency. It'd be the equivalent of saying when Google was nascent and rising in search, you're going to go do a search engine or do something in search. They have to win that. And they have the capitalization to win that. They don't have to win MySpace. They just don't have to. And in fact, what I'm going to argue to you— and this is not a technical argument, I'm giving you an economic argument— when these guys go public and their financials are there for everyone to see, I think there are going to be 2 priorities, simply. And you saw this with other waves. Priority one, and you saw with Uber, you saw with— they have to get their house in order and make this an attractive, profitable company. Right. Can they do it? Probably. Right. Anthropic looks like they're better, but it's non-trivial actually, because the transformer models are not conducive to, you know, a trivial amount of CapEx. I mean, you saw Google just did that crazy— I mean, it was like a first— it was like a dot-com move. We're in unprecedented times. Okay. And I don't know if Anthropic has that power, the public company, to go say, oh, I'm going to go sell a little bit of stock and do the same thing Google. We'll see. So when they do that, they're going to have to do— Google is a cash machine, Microsoft is a cash machine, and ultimately investors are going to value that cash machine at some point. So they have to work on that, priority one. That's going to take resources. Priority two is they're going to be at $1 trillion. Okay, there aren't a lot of markets that are going to move you up. You have to be winning the biggest markets, whether it be enterprise cloud computing, right, where you see a lot of competition in Microsoft, Oracle, you know, consumer search, e-commerce. Like, these markets are so big. that you can rationalize building in them. And if you win them, you can then move from $1 trillion to $2 trillion and $2 trillion and so forth. Like, right? How many markets are like that? And so those 2 forces, when they go public, are going to constrain Anthropic and OpenAI the same way they did at Google, same with everyone else. Because until they have self-generating code that also can build the product, that also can make sure there's product-market fit, and like all these, you know, combinatorics, everyone's safe. So I think there's this illusion. I mean, there's always something. When I started my first company, why can't Microsoft do it? Then we got bigger. Why can't Google do it? Then Facebook came. Why can't Facebook do it? And so at some point, again, in the long run, sure, sure. If we're going to this period of singularity, we can all sit there and argue that, but then let's not do anything. Everyone just stop. Let's go home and either buy a bunch of guns or celebrate because The end is near.

Permaculture gardens. I prefer permaculture gardens to guns personally.

You know what I mean, man? Like, it's kind of bullshit, just to be honest with you. Like, the whole argument is like, then let's not go to work. And it's funny because the same venture capitalists that are arguing this are backing up the truck into these new startups as fast as they can, you know, because they don't know either. Nobody really knows what's going to happen. But I think as a builder, you can chart the course It used to be probably if you were really, really strong, you might be able to go out 5 years. I don't think that's the case anymore, unfortunately. I think you have to be really, really nimble, and AI gives us that agility, and you have to walk around. But I think they'll be constrained by economics, and they have to make that market cap go up and driving down costs, going into big markets. Do you think, I don't know, Sam building a— I don't know, what's that thing they did on finance, the thing that they launched? Is that going to make the company get another half a trillion dollars in market cap going into that segment? And he has to answer that question. That's tough. Google killed a bunch of stuff to make it. But then what are they in? They're in Waymo. They're like, I'm going to take driving. What else is going to make that market cap go up? There's only so many trillions of dollar markets. That's why I'm less concerned. It's not the tech. Now, there'll be a point when the tech gets so good, but it's just not actually that good. If you guys have tried Claude for small business, it's like, imagine giving your mom DOS back in the day and being like, hey, there's these things. I have all this stuff. You can upload skills and you can do all this and you just give it to your mom and you tell them to go run their business that way. That's literally what they're expecting right now. It's actually laughable.

Maybe, Prakash, we might need to do a little reset here before our next guest. Why don't you work on that? I'll ask maybe a couple more questions and then we'll break and try to get our tech house in order.

Right on.

On— I think one big reason, I mean, so the big market, of course, that like, I think that the frontier companies would answer with is simply labor, right? I mean, labor is a $20 trillion market in the US or something like that. And they maybe don't have to compete niche by niche if they can succeed in creating the drop-in knowledge worker that you can just hire by the minute rather than, you know, by the hour or by the, you know, the year and meet your needs that way. Right. And I think that's the play that, that seems to be why these companies are—

I think that is absolutely the play. I agree with that. But let me give you an example. Okay. We already have labor. And it's at a unit cost. So what you're saying is they're going to give it to you at a better unit cost, right?

Yeah.

Like, okay, say I go hire 4 ICs today. Like, imagine I came to you, I'm going to pitch you, and I'm like, going to venture caps. I'm going to go start a full-stack accounting firm that's going to handle 4 or 5 different functions. And I have the labor, by the way, it's a 10th of the cost, but there's still IC2s and IC3s. So how are their workflows going to work? How are the data going to move between them? What— how do we manage them? How are they going to communicate with clients? What's that experience going to look like? So labor is a unit that's important, but ultimately labor drives an experience, and that experience needs to be built, managed, and it needs to be secure and provisioned. And so I think that, you know, inevitably that's going to be a large part in it. And look, if they have an agent that can work better on particular and all that, that's great. I I can't remember who I was talking to about this, but actually I think it was one of my investors, Darian Shirazi, at Gradient. We were talking about Salesforce and he had a really funny but I think good adage. Salesforce, their future is not CRM. Their future is Salesforce. They literally should provide sales agents. You know what I mean? Like, that's what they should be. Every company should be moving from the tools to enable labor market into the labor market. Well, but you still ultimately have different levels of abstraction of work, right? So I have ICs, I have managers, I have people who are up there. Like, I need a CFO. My CFO can't go hire 1,000 people and manage them, right? And so there needs to be some orchestration, some workflows, some trust the way the data is going to flow between. And so it's— I think there's a lot of opportunities staying in the outcome space, and in particular, trusted and regulated outcomes where, like, Is Anthropic going to want to be the one who signs a tax return and says this is good to go? Probably not. They'll look at your tax return. They might give you a, hey, I did it for you. And then they'll put at the bottom of the thing like, this might be wrong. Good luck. Right? And so you just have to decide if that cost is worth it. But my argument to you is TurboTax is really cheap. It's really cheap. It's $99. And not everyone in America uses TurboTax. Why? Because they want someone's ass on the line. They want a trusted outcome where they can be like, hey, you know what? You looked at this. I can ask you questions. Let's try to do A, B, C, and D. And so again, we're talking about unit economics and the prices changing. I think it will have a disruptive effect. Most certainly. Most certainly. I just think people are willing to pay for certainty and underwrite that particular business when you're talking about $349 versus like, okay, I can go on the cheap and use DOS and maybe it'll work. There's a price to the risk that I think they're willing to ascribe.

Yeah.

Okay.

Last question. And I apologize for rough technical experience here.

No, that's all right.

You mentioned unit costs, and I think one of the things that is making it most hard for app layer folks like yourself these days is the dramatic difference in per token cost that is offered to people directly through a Cloud Max versus via the API.

Oh, yeah.

I am still not quite sure I trust these numbers, but if my Claude is to be believed, it says that I've spent upwards of $1,000 in the first 4 days of June API cost equivalent on my Claude Max plan, which I'm getting for $200 and which refreshes every week, and I haven't even hit my limit. So it would suggest that I'm something like more than 20 times subsidized if I max out my plan, which of course not everybody does. So that's a little bit— makes this a little complicated. But one policy move that you could imagine folks making in an antitrust sort of way would be like, you must offer some sort of pricing parity to your first-party customers versus your API customers. Do you think that is something that if the government wants to get serious about, you know, protecting diversity and having a sort of, you know, rich ecology of innovation, is that something that you think the government should start to think about?

I think there are— they probably won't because— and I'll tell you my underlying drivers for this. If you look at the first order bid that they want prior to consumers, because consumers vote, I think businesses have a larger say also because of the incentives with PACs and whatnot. So I can see like, who are you protecting if you go over tokens? So you're just saying, you're like, hey, I'm free riding the system in theory, right? Nathan, you don't care. Like, you're good. Like, where it hurts is in enterprise. And this is actually happening. I think this is going to be a massive problem. That's about to occur is I jokingly say Anthropic is like a drug dealer. Okay, so they're like, here, go use this thing, get addicted to it, and then they give it to you for free. And then everyone gets addicted, overuses it, and then everyone hits plan. And then it's in order of managing our costs. And this is going to happen to more organizations. Like, I can tell you in our organization, it's a great product, right? It's not bad. But now I have to sit down and be like, who's going to get this? Because we're going to hit the limit and I'm not going to pay 10x the cost. And so now you have to rank order developers, but even within developers, how are you going to do that? So it plays into your OpEx, it plays into your R&D costs. I think there's a lot of poor engineering going on and people are overusing the AI tools, again, using inference instead of compute. There's deterministic patterns that are much better that they should use, and they should, and that'll— people will swing to that. There's also the fact that you can use— you mentioned at the start of the call— these off-the-shelf open source kind of models. And so I think people are going to start getting— they're going to have to engineer. You can't just go and, you know, like beat the shit out of these things and have venture subsidies. You're going to actually be an engineer and understand it. But I don't think the government is going to care that big businesses are going to get fleeced by Anthropic. I think they tend to care in the reverse, where consumers get fleeced. And right now consumers are getting a free ride, at least today. What I do think is going to happen is these consumer plans are going to change and they're going to be potentially more expensive if they want to be profitable on the consumer side, if that math works out. Where they do need to regulate, and I'll kind of end on this note, is I think, and this is very difficult, but I mean, we started the scale of time. These models, like you look at Mythos and you look at some of these leaps that are occurring, there needs to be some sort of way, and regulation is, let's leave it a blank word, but some sort of way that as a group, and I'll call the group people are technologists, are looking at this and saying, wait a minute, hold on. If we don't— there's one, there's one version of this model that is a version too far. Nobody knows what it is. Nobody knows when it's going to happen. But there's this— the next version becomes too far. Mythos was our first taste of like, wow, if I open that up, the level of attacks are going to go out of control. Maybe I have to think about it. And so that to me is where I would spend the time regulatory-wise, not the cost and price of tokens. That'll— the market will suss that out, right? Because, you know, Anthropic messes up, OpenAI will fix it. You know what I mean? That's a market force that'll fix that. But the race itself, the armed war that's happening, that is actually the result of capitalism and that incentive to go to the next model being better. That is where you have to regulate because the current incentive structure is not for them to build safe, you know, aligned models that are aligned with humanity. I mean, the one I think about all the time is Anduril. Like Jesus Christ, like I don't know if you guys watched Terminator recently. Like we clearly have just forgotten James Cameron's teachings. But I mean, those guys are running as fast as they can to autonomous drones. It's a good company. I don't actually have any political problem with it. But like, it's scary. Like, you know, if you take humans out of the loop, which they're debating, they're not even— it's, you know, on those, these are decisions to kill people. Like, well, that's pretty strong. And then all these models are now accessing the internet and they're working together. And so there's just— the combinatorics are going to be staggeringly bad for an incident. So that's where I would spend the regulatory time, is that next move and just making sure that there's people in line. Even— I don't even think that they're going to do a good job regulating. We're just shitty at it as humans. Forget government. Let's just make a step where we take a breath when these things are occurring, just to take a look to make sure we're not making the step too far.

Yeah.

Well, I definitely agree with that. The arms race is a huge problem and you can count me, it sounds like you as well, with the Pope in calling for the general disarming of AI. Really appreciate your perspective. Again, sorry for the technical difficulties, but this has been a great conversation.

I hope you guys figure it out. I hope you figure it out.

Thank you.

We will get there.

The company is Collective. It's the back office for the entrepreneur, solopreneur business of one. Check it out. Thank you, man.

All right, thank you.

Hello?

Hey, how are you? Can you hear me? Yes.

All right, we're gonna do it, uh, on speakerphone.

Um, I think we can—

we can— tell me, tell me who you are. I know who you are, but tell me your name, uh, so I make sure I'm gonna say it right in the future.

Yeah, so my name is Taras, and I'm a founder of a company that is called Illumia Health.

Yeah. So fascinating. And we see your picture now too.

Okay, that's cool.

So yeah, I got multiple audios now coming my way.

This is gonna be interesting.

Yeah, because I had to mute my mic. Has to be better now.

Okay, we'll carry on. So your company caught our attention because you're deploying AI for mental health, which is obviously hot topic in the first place. But not only are you deploying it, you're deploying it in some of the highest difficulty situations, I would say, in the world today, including in Ukraine, your home country, if I understand correctly. I believe you started this country—

Yeah.

This company in Ukraine some years ago. And, you know, I don't know too many people who I feel have a harder mental health situation right now than guys on the front lines on either side there who are just having an incredibly difficult time even rotating out. So the burden that those poor guys are under is ridiculous, and anything that can be done to give them relief sounds great. And then the same thing also with some pilot programmes that you're running in United States prisons, where I have to say, I guess I might rather be in a US prison than on the front line in Ukraine, but— It's debatable. So again, a very high difficulty environment. Tell us about how you're doing it and what the findings are from these frontline experiments, so to speak.

Yeah, no, I actually, I would personally be on the frontline than in the US prison because if you are there, like we have the right to defend our freedom and, you know, that's an honor. So yeah, we're not really comparing, but yeah, so like in terms of what we do, we started 7 years ago basically, and we specialized at the intersection of AI and mental health. And currently the way we position ourselves is like Codex or like Claude Code, but if it was built for mental health. So unlike a regular chatbot, it is much more powerful, under the hood and it uses all these agents and it really relies on the agentic architecture to power the safety and to power the responses. And I think the safety is an important one here and that's why we had a chance to work within those environments including in the US prisons, including the mental health clinics, healthcare organizations and on the front line in Ukraine, yes, in hospitals where in rehabilitation centers there as well.

So tell me more about the architecture, the safeguards. I mean, I— we know the basics in terms of like content filtering, you know, classifiers to, to raise alarm bells when needed. Um, but I guess my— I haven't really used chatbots much for any sort of therapeutic purpose, but my naive sense would be that, you know, they probably do a pretty good job doing cognitive behavioral therapy out of the box. And that, you know, I think a lot of people are using them for that. Where do you find they fall short? And, and tell me more about what you've built that isn't immediately visible to the user to improve on those weaknesses.

Yeah. So safety, I would say that it is the most important part of what we do. And obviously a lot of companies would want to build something for their patients, for their customers using APIs like OpenAI API or Anthropic API, but they simply cannot because they do not have an in-house expertise to steer those models towards working the way They would want them to behave, and then there is like this regulatory risk that you would have to address there, and safety risk, of course. Yeah, I think an interesting kind of like fact about us is that we spend around 70% of tokens on safety, which makes the system like very, very expensive. But yeah, like we on the technical level, we have a bunch of classifiers that are constantly running in the background. So it's in real time but also in the background, especially when you need to think longer about certain issues or about certain risks. And then we, like why I say that it is similar to Cloud Code, because it is able to use different tools. And for example, if you want to plan a session ahead or to reflect on what happened in the past week when, like, I mean, from the chatbot perspective or from the agent perspective, what happened in the last 7 days in the conversation with this user and how this should inform our plans and our actions going forward. So the agent is able to deploy tools or maybe sub-agents that do stuff like that and then kind of feed back the information into the main context and then it continues. So as a result of that, you have this very powerful memory and very powerful planning capabilities that are there. And as a side benefit, it also has the chances that if you miss something that is important for the user safety, they are dramatically lower than without these kind of systems.

How are people accessing This? And is the user experience kind of the intuitive chatbot one that we're familiar with? Or due to— I can imagine due to constraints on, you know, devices on the frontline or in prisons, you know, there may be very different system-level requirements or oddities that you have to work around. Also, with all these things running in the background, if you want to be maximally safe, you have to increase latency quite a bit to let all those things run to completion before you, you know, can get to the output that's going to face the user. And that obviously also lends itself to somewhat of a different user experience. So what does the end-user experience look like starting with the device? And then, you know, how do all those safety layers impact it?

Yeah, actually, the latency is a very interesting question because if you just use normal large language model like Claude Opus, maybe it will think for 20 seconds, for 30 seconds or more if you would do stuff like we do. And of course, that would create a terrible user experience. So the way we build it on the technical side is that we have these thinking agents in the background, but then they insert their, I would say, instructions or guidance for the chatbot. So the user talks to the chatbot and it responds normally, but as conversation continues, it acquires more and more context. Does it make sense?

Yeah, that reminds me of the—

you can actually—

yeah, go ahead.

I wanted to say that you can actually try it because we also have this B2C app, consumer app that we have and that anybody can download from from the App Store and from Google Play. And from our experience, we had this situation a few times where a person comes and they say, hey, I was using your app for 4 weeks now and I've recommended it to all my friends, I've recommended it to my wife, and please let's build something together for my clinic, or something like that.

So the architecture reminds me a little bit of the Thinking Machine, voice real-time models that they recently described. I guess they've sort of soft launched it to some early partners, the sort of deeper thinking being in the background and then a kind of more superficial, faster response model being the thing that you actually hear from. I suppose that so far today, correct me if I'm wrong, but I'm guessing so far, your main modality has been text. Have you looked into— I mean, maybe you have even done some voice. Do other voice models support the kind of interaction that you want to create? And how excited are you for that more time-aware thinking machines thing to come more available?

Yeah. So, voice is really important for the consumer Obviously, the main use case that we see is that you can talk to it while you are driving your car.

Huh.

That's like the main— no, but it's not really, you know, because you would actually, if you talk to people, what they would want to see is like a chatbot or like an agent or AI that sounds like AI from the movie Her, right? That is like super realistic and you can maybe fall in love with it. Now, that is not something that we want to happen here. So yeah, we try to support that hands-free capability so people can use it, but not more than that because we also try to— I think because if you think about safety, that's maybe one of the most unsafe things that you can do if you deploy this very real, hyper-realistic voice capability that people will become dependent on. And I mean, there's different kind of arguments on both sides of it, but if you talk to real users who use it like once or maybe 2 times, you will understand that it's just, you should not build hyper-realistic AI. It's not going to do anything good. And I mean, the movie Her kind of shows it.

Yeah, that's interesting. Do you think that the market will stay there on its own? I mean, one thing I have kind of been a little surprised by so far is how little we've seen in terms of like highly polished products for kids. You know, I've got a couple little AI math tutor apps and stuff. But relative to what seems like it's probably possible, the market hasn't gone as far as I would have expected with kind of AI embodied or stuffed animal AI friends for kids. But I still kind of think that's coming, and I think it's going to be up to me as a parent to sort that out. For the most part, you're in a more regulated, protected, licensed, kind of space, but still it's all kind of Wild West right now. What do you think the competitive environment is gonna look like if somebody pops up and is willing to go where you're not willing to go with a hyper-realistic voice? Will they win in the market? And will, you know, do you think this is the sort of thing that ultimately the government needs to put rules around? Or maybe the market can sort it out. I don't know. It seems like it's a— I have a hard time imagining that we sort of stay in the current state where we refrain in a voluntary way from making things hyper-realistic because it just strikes me that that's what people will gravitate toward. But how do you see that playing out? And what, if any, intervention do you think would be wise?

I don't know the future. So, but I can kind of speculate, right? So the way the market currently looks like is that you can win the consumer with something hyper-realistic. You can win the consumer with a product that praises you, that tells you that you are always right. I know we already see that happening with ChatGPT. The question is usually like, what do they win? Because the way I see it, they only win a bunch of lawsuits against them.

Right.

I don't know, maybe. When we talk to healthcare organizations, for example, they never ask for something hyper-realistic or for something that may create even the slightest risk for the user. But in terms of talking about and thinking about the future, and if I were to speculate, I would say that Yeah, that's a good question. I would say that, you know, like with many things, you know, I heard that they were using cocaine to treat the flu back in the day, right? But then you kind of learned that it's not really healthy and maybe that's not the best medication out there. And I think that something similar to that may happen where we as a society develop this kind of knowledge that is embedded in our culture that that is something that you should not do even if it feels really cool at first. I mean, I would bet on that. I think in terms of regulations, yeah, some people say that it's a wild west. I would say that it's a wild west only for those who did not really read these documents that Because the way I see them, they are pretty clear, and especially the latest edition that we have from January, both from MHRA and from FDA. I would say they're pretty clear, so I would not say Wild West. At the same time, the way I see it is that the FDA may lack resources to enforce the regulation at the moment, and that's essentially what the so-called gray area is. It's the lack of funding maybe to enforce the regulation. Now, it looks like that in the United States. It looks to be different in the United Kingdom where MHRA has resources and has funding to enforce the regulation. And, you know, that's actually what happens to Ash, for example. And I think Ash is a really great product and I know that a lot of people love it, but yeah, they, you know, they were removed from the market in the United Kingdom. And I would not be surprised if something like that happens in the United States as well.

Is there a clean line to draw? I mean, the one challenge is going to be, of course, resources for enforcement, although we may be soon entering the AI-powered panopticon. So There is certainly a chance that everything will be enforced beyond our wildest imaginations. But then I also wonder about what is really the difference. If there— is there a sharp difference that we can use to distinguish a mental health app versus just sort of an AI friend and companionship app? Because it sort of strikes me that it may come down to specific marketing claims. I mean, is that kind of where the distinction is going to be drawn? And isn't there always a way for an AI friend or companion app to be a little bit more the cocaine that the consumer may demand or at least crave? without explicitly saying, you know, that we are a mental health, you know, built purpose, purpose-built product. But still, you could even imagine doing like, if a friend applies some like cognitive behavioral therapy techniques in conversation with another friend, that's like not regulated, right? And it's also, I think it'd be very interesting to see where we—

A friend is not a device. Because they regulate medical devices and a friend is a human. He's not a medical device. So that's kind of like—

But I mean, but if I'm positioning my AI app as a friend, do you see a clean line that a regulator can come in and say, you have crossed the line from friend to something more than a friend in a way that enters our jurisdiction? What is that line?

The way I see it, and I do not have any insight, but the way I see it, I think they would look at how people are actually using your product. So you can say that it's a friend, or you can say that it's like, I don't know, math tutor or whatever. But if they see people talking about your product like it's a therapist, or they say things online, like, I don't know, like I stopped visiting my therapist and now I use ChatGPT and it's much better. That's what they will use as evidence to kind of sue you. But what I said previously, now I'm not sure if there is anybody who knows for sure, but so the lack of funding is like an obvious reason, right, why they would not want to enforce it at the moment. But another reason that I think is also true is that they are still waiting to see how it develops. Because I think that essentially the last thing that they would want to do is to prevent really helpful and useful products from appearing. So maybe that's part of the reason as well. But in terms of the documents and the regulations that were published, I think it's pretty clear that— I mean, I think even— yeah, I think it's pretty clear where the line is.

Interesting.

Okay.

Is there, I guess, one more question? I know we're running late, and I apologize for our rough technical experience here. How does privacy play into this? Obviously, in the US, we have, I would say, kind of in some ways problematic medical privacy laws, because I think certainly with AI to process all the information, the public at large stands to benefit greatly, I think, from better information sharing, more ability to mine information that's currently locked up behind HIPAA and whatever other kind of constraints. What you're saying in terms of people just observing how people are using the product, obviously some of that will kind of make its way onto Reddit and whatnot. But the real— where the rubber really hits the road is going to be show us your logs and OpenAI is pushing for a sort of right to privacy in terms of your chats with AIs that would, in their mind, rise close to, if not all the way to the level of the same privilege that you have with a licensed attorney, that we may not get quite that far. But I think there are a lot of reasons to value that.

Right.

I don't want the government panopticoning all of my AI chats. When I hear about, you know, things like what you're describing, I do think, yeah, well, some oversight, you know, definitely could be prudent. Do you have a sense for how we should think about striking the right balance there between privacy and visibility for authorities to police this stuff?

To be honest, I do not feel like there is any issues from the governmental side because if you are talking about health data, it has been held by vendors, right? So now if they may not have an incentive to share this data with ChatGPT and I think it's more of that because we also work with healthcare providers and obviously when we work with them, we operate in a HIPAA-compliant manner, we are HIPAA-compliant. And yeah, I mean, I did not see any complications in that regard. So yeah, and I think, you know, If you would ask what's my take on this future, I think that a few years will pass and it will all become open just because patients would want that to be open and they would want to export their data and upload it into ChatGPT. And I think that OpenAI does a really great job with creating specialized models for healthcare. So I think that that's where we're kind of moving into because that's how the market works as well, right? If there is a company, a medical provider that says it's really easy to export your data into ChatGPT if you are our patient, of course, if there is a You meet consumers and patients that do prefer to use that medical provider, and that medical provider gets a competitive edge. The other companies would be forced to do the same. And I think it will just play out.

I guess maybe one more just to wrap us up. Are there any anecdotes from your deployments in Ukraine or in U.S. prison populations that you think are particularly memorable or inspiring that you would leave people with? Or failing that, just anything else you would want to leave people with in terms of a positive vision of the future?

Yeah, positive. Yeah, I mean, the Many. There are many stories like that. Like in the correctional environment, we heard numerous times from medical teams there that our AI helped them identify people who they did not know were suicidal, and because of that, they were able to provide better care and to potentially maybe potentially save their lives. In Ukraine, it's also very interesting because we, like, before we started working in Ukraine, we thought of ourselves as like these big safety experts. And that's until you meet the first veteran and you get exposed to what they face and how their lives look like. And I would say that our work with the veterans and with the soldiers It shaped a lot of what we do in safety because if you test that under those extreme conditions and it works well there, of course, you know, like, of course, you know, very confident in your safety when it comes to normal consumers. And like, I think that more and more people realize that they do not have to be left alone and that they can use AI. And that's a good thing. And even ChatGPT, you can, there are still people on X who write about GPT-4o and all the things around that. But even GPT-4o, if you were talking to it, it would say, hey, go and get professional help. And it would help you, it would maybe help you to be prepared for that and answer some questions about that so you are more ready. And especially now, it's even better. GPT-4o is an outdated model right now. I think it's executed much better than that.

GPT-4o, RIP, I guess, soon if not already. Thank you, Charles. Fascinating conversation, and thank you for coming in. Thank you for serving some of the people who need it most in this world. Obviously, the I hope the Ukraine war ends soon, and that that you know extreme market circumstance dries up for you. But even if it does, the U.S. prison population, I suspect, is going to be here and is going to be you know. in bad need for the foreseeable future. So keep up the good work and we'll be following Illumia Health.

Thank you. Appreciate it.

Great to meet you. Bye for now. But we've got it fixed. So you're Peter Jansen. You're joining us from Allen Institute. You're also at the University of Arizona. And you have been, if I kind of read the trajectory right, on a couple-year journey from measuring the deficiencies in models' ability to do science to now— and correct me if this is not the right characterization— but now it seems like you're kind of saying, yeah, they're kind of getting there and it's time to actually start to take advantage. So tell me the story of your last couple of years and then we'll really dive into the frontier.

Yeah, that's certainly a way to place it. So when I started in this AI for science space about 10 15 years ago, we were all trying to teach the best models we could how to do 4th grade science, right? These are really basic tasks. You know, if you put water on a stove, what's going to happen? And, you know, every, every kid knows what's going to happen. But the models back then were just absolutely terrible at these things, right? And then fast forward, um, to certainly the, the sort of ChatGPT revolution when that came out, all of us who had been working on 4th grade, 8th grade, 12th grade science had the models sort of overnight be able to go from 30%, 40%, 50%, 60% to 80%, 90% very, very quickly. And so then we really started to wonder, hey, I wonder if we could start applying these things to the sorts of problems that we care about every day as scientists. Certainly, Today, one of the tasks, or backing up a little bit, AI for science isn't a new field. AI for science has been very seriously studied since the 1980s, which is sort of when it was formulated by a fellow named Herbert Simon, this very famous computer scientist. And before the 1980s, people used to think that scientific reasoning was this sort of very special kind of reasoning because it seemed to involve lots of creativity and lots of specialized domain knowledge. And then Herbert Simon came around and said, well, wait a second, maybe we can use existing AI techniques like searching through large spaces or optimizing problems or whatnot in order to model this productively. And the '80s was a great time for AI for science, and then it kind of settled down a little bit, even though it had a lot of big successes back then. There was equation discovery, so fitting data to equations. There were really famous results in trying to discover something called undiscovered public knowledge. So you can picture scientific papers as sort of these puzzle pieces, right? And what happens if something from paper A and paper B could fit together, but nobody's noticed it yet. And a classic example of this is that my wife suffers from migraines. And one of the classic treatments for this is to give the patient magnesium. And that was an AI for science discovery that we use today because they discovered that the pattern of metabolic deficiencies in both magnesium deficiency and migraine happened to be the same thing. Fast forward to today, There's sort of this reinvigoration of AI for science people. There's no researcher I don't know who doesn't fire up ChatGPT all the time, right? And say, hey, can you help me with finding papers for my research or sort of any other task or even just bouncing ideas off of? And so I think there's certainly been this reinvigoration where people have said, wait a second, if there's this all, very powerful tool. Can we use it to do those kinds of things we do every day as scientists?

So where are we now? You just put out this project called Theoriser, and I've kind of had to— I try to maintain what I call the AI scouting report, which is usually a breathless 1 hour of trying to tell people, to the best of my ability, here's the current state of affairs in AI. And of course, that needs regular updating. I used to say things like, no eureka moments. As of GPT-4 era, it was like, AIs are closing in on professional performance on a lot of small and medium-sized tasks, but no eureka moments. Well, that's certainly been blown away, right? We have novel math problems getting solved, including increasingly important ones. And so we do have eureka moments, even from pretty vanilla LLM long rollout systems. There's a lot more, obviously, I think, on the horizon in terms of integrating with wet labs and closing the loop that way and training on other modalities and integrating those. And maybe I want to get your take on all of that as we look ahead into the future. But as it stands right now, it seems like we're like, no eureka, or yes, eureka moments, you are getting eureka moments, but maybe the next thing is like, but no paradigm changes, which is, that's a pretty high bar.

Yeah.

But I get the sense that this is what this kind of theoriser project is really probing the difference between, right? Like solving an unsolved problem within the paradigm versus actually kind of shifting the paradigm. Where do you see the frontier of possibility today?

Yeah, that's a great question. So I'll push back a tiny bit on one point that you said first. And I'm just realizing as I said that pushing back is something that the AI models all say these days. I'm really starting to sound like one, which is terrifying. So that big question that you started off with, which is, have there been any eureka moments right now? And certainly you open up your Twitter feed every morning like I do and you just see this giant sort of deluge of people saying that they've used AI for this and that and whatnot. My, I'm not a mathematician or a number theorist. My best understanding is that most of the, like the Erdős unsolved problems and whatnot that have been solved, that not too many of them are sort of mind-bendingly, you know, obvious paradigm shifts or, or big eureka moments, but certainly they're making discoveries. Right? In the past 2 years, we've seen them make what I would probably call a lot of incremental discoveries. Certainly at AI2, we've had a bunch of these sort of code generation systems where they come up with ideas, they run the code for them, we validate the code. And yeah, it seems like they've made some pretty incremental discoveries sort of at the level of like a master's student. But to your point about sort of paradigm shifts, which is I think the really exciting thing, right? So, you know, when we wake up in the morning, we're probably, as AI scientists who do AI science, we're probably not hoping that we're going to get, you know, the next incremental result. You know, wow, this model is 0.1% better today than it was yesterday. That's not really super exciting, but we're hoping to really change how science is done or find new problems or new ways of thinking about problems. And certainly if an AI scientist can help with that, that would be— Fantastic. So there's this AI scientist, Arvind Narayan, and he has this criticism that I really like about AI science right now in his book, which I think is AI Snake Oil. And he says that if you have a lot of AI scientists or a lot of anybody sort of working on doing what we would call sort of experimental science, which is, I think, what most of us would call sort of bread and butter everyday science. You get up in the morning, you come up with an idea, you run it in the lab, and you write up that experiment. He says that if you have a huge amount of experimental science, then it actually counterintuitively is slow for— or it slows science because you wake up every day and you don't know what to attend to. Certainly I can't attend to everything that's going on even in my sub-area. And so he says that this slows science because science isn't just about doing experiments. It's also about coming up with theories or making paradigm shifts, these sort of higher-level scientific activities, which were framed by Thomas Kuhn, this famous philosopher of science back in the '70s. And so what Theorizer gets at is it says, well, wait a second, could we take data, could we take scientific papers by the hundreds and sort of shovel them into this theorization-making system and say, okay, tell me, human scientists, what kinds of problems you're interested in. And then I'm going to go off and read hundreds or thousands of papers and try and come up with theories that automatically organize all the experimental results that have been found in that area in a way that would be really hard or really effortful for humans to do. That's what Theorizer has done. We ran this on, I think, 100 different sort of, we call them theory queries, just make me theories about X that were this broad cross-section of AI topics that we automatically sampled from the literature. And it came up with 3,000 theories on topics that were ranging from sort of everything you can imagine, intelligent tutoring systems to algorithms. And by reading, I think, about 14,000 papers. And so that seems like a big shift, right? That seems like if it can help us discover not just a new experiment to run, but better experiments, better questions to ask, better research questions to ask, better ways of framing things, it'd be fantastic. A classic example I can give you of this is that my classic theorization example and paradigm shift example is that if we had lived about 100 years ago and we were geologists, then there would be all these people studying volcanoes and all these people studying earthquakes. And maybe some people in geography would've noticed that, hey, if you sort of look at the continents on a world map, they kind of fit together. Doesn't that look weird? And there were theories about all of these things sort of separately. And then someone eventually came along And said, well, wait a second. If you imagine that the Earth is built on these big sort of shells or plates called tectonic plates that kind of rub together every now and again, it actually explains volcanism and earthquakes and continental drift and a bunch of other things, but also allows you to ask a lot more productive research questions. And it completely changed the field. So wouldn't it be wonderful if AI could help us do that?

Uh, yes, as long as we can keep the recursive self-improvement process on the rails as we go. Do you, um, have any highlights from the Theorizer work that you would say were like particularly, you know, useful theories? And I guess, how easy is it to evaluate theories? Because this seems like a key bottleneck, right? If the theories are hard to evaluate, you have a hard time getting that flywheel turning. So how do you sort of structure them to be as easy to evaluate as possible? And are there any that really rose above the rest and stood out as meaningful contributions?

Yes, certainly. So there's a bunch there, one of which is how on earth do you go about evaluating a theory? And then number 2, is, have we found any sort of exciting discoveries yet? I'll sort of front-load it with the last one, which is, I'll give you 2 quick stories. Story number one is when we designed Theorizer, I had these ideas about how I would apply it in the sub-area that I focus on a lot, which is sort of agents and virtual environments and bolting on memories to them and whatnot. And so when I was testing it, I would give it a lot of these sort of high-level queries, and it would come back with theories that seemed very reasonable and are things that I myself had sort of hypothesized, but I had never written down. Things like, if you're making an agent that's for a specialized task and you're bolting on a memory to it, which is a very popular idea these days, that you might want to match the kind of memory to the kind of task. And that if you have task-matched versus task-mismatched, mismatched memories, then you'd get different performance, things of that nature. There's sort of lots of theories. Some of them aren't necessarily mind-blowing, but they sort of help you formalize a discipline that until now has sort of been really sort of— haphazard isn't a really good word, but experiment-driven is maybe a better word to use there. I can't share any super exciting discoveries yet, but I will say that we've been working with a bunch of partners in the medical domain and they seem to be getting a lot of utility out of it. So hopefully we'll have some exciting stuff to share on there in the near future. Back to how you evaluate theories. Well, one of the funny parts about Arvind Narayan's AI snake oil criticism that if you have too much experimental science, it makes it hard to see what's going on and sort of filter through it to do the higher-level scientific activities like theory building. It turns out if you have a machine that you can just press go, and in a day you can get 3,000 theories by reading the entire literature of an entire subfield, it turns out now you almost have that same problem. You have so many theories and it's effortful to go through theories and look through them. You sort of have to look at them and they're designed to be as easy as possible to digest. They give you the theory or the law that they've sort of proposed. They give you a long list of supporting evidence and all the sort of papers that they've come from. They propose experiments for you to think about. Some experiments are sort of easy experiments that are, you know, if this were true, what would you expect to be true. So those are sort of confirmatory experiments. For me, the ones I get the most utility out of are the ones that we call high entropy experiments. And those are things where it's experiments where if this theory kind of held, or if you really wanted to sort of test this theory to see if it broke, you would give it this research question. And it's sort of an edge case, and you really don't know what would happen when you— if you were to test it in that way. And those are super exciting. So yeah, it's definitely an effortful process though. And I don't think we've ever had a machine, a system that could produce theories. And so learning how to best present them to users and humans in a way that they can digest is an open research question.

Yeah, there was a really interesting experiment that you're reminding me of around AI's ability to come up with new ML research direction ideas. And I think this came out of Stanford. I, I did an episode of the podcast on the first leg of it, um, and basically the finding was like the AIs could come up with better— yes, with Changlei C. from Stanford, can AIs generate novel research ideas? And the first evaluation was having humans go through and look at all these different research directions and evaluate them. And the headline was the AI ideas were judged to be more promising than the human ideas.

Better.

To their credit, they followed up on that and actually ran a bunch of these experimental directions and then went back to the human expert panel with results. And then it was like, okay, well now which ones have actually yielded better results? And the human ones, though they were judged less promising upfront, did in fact generate more interesting results later. So it is clear that we still have some both sneaky advantage, I guess, in that space, at least for now, and also a very difficult time, it seems, separating the quality of the core ideas from the surrounding presentation trappings, all that sort of stuff. The AI is obviously really good at all that. So it's like, They even took steps, I believe, to correct that by having AI kind of polish up the human writing so that they even tried to control that effect. But it seems like maybe didn't fully manage to control for that effect. I'm not sure exactly what the mechanism is to explain why that reversal happened, but there's definitely something really interesting there. When you think about kind of moving to paradigm shifts, On the one hand, yes, it's like very effortful to wade through all these new things. On the other hand, you can tell maybe a story that's like, we need all these complications of the current paradigm in order to build up enough, you know, what do they call them? The little epo— Epicycles. Epicycles, right? Was the, the thing that gradually accrued in the, in the motion of the planets that finally led people to think there's got to be a cleaner explanation for this. So I wonder if we're in a phase of, you know, we're just fundamentally labor limited and we're also maybe, maybe incentive limited in some ways because, as you said, you know, scientists don't want to get that 1% improvement. But first of all, there's just probably a lot of value in grinding out the epicycles that we can grind out within the current paradigm. And then second, I wonder if that is a really kind of necessary step to create the data to then feed into the big brains, whether they be human, AI, or some sort of hybrid at that point, to have that really kind of paradigm-shifting insight. How would you react to that story? Oh, wow.

There's a lot there. So yes, is the short answer. The longer answer is, you know, back when I worked in 4th grade science, my life was a lot easier because you put the water on the stove and what happens? It boils, right? There's a clear answer to it. The most exciting— I love hard problems. The most exciting part about my day as a scientist is when I get to work on something where I don't know the answer and it's going to take me a while to figure it out. And I know that there's things that are just unknown about this space that I get to discover. Every theory is wrong. And that's one of the first things they teach you in grad school, right? Every theory you've ever heard is wrong, but a lot of them are useful. And as long as we can create better, more useful abstractions that help us really work through the problems better, I think you get utility out of them. Another favorite historical example of this is that for a really long time, we thought that the reason why people got sick was this theory called miasma theory, that it was bad air, right? And it actually explains lots of really wonderful things about sickness. So, oh, there were a bunch of people who were together in a room and then they all got sick a couple of days later. Oh, what happened? It was bad air, right? And you can see a bunch of reasons why that is explanatory and it's actually predictive, right? It explains a subset of why people get sick, the idea of air-based transmission of viruses, but it doesn't explain a lot of it, right? And eventually, After people chugged away on that theory for a while and iterated on it, they came up with a germ theory of disease, particularly when better instrumentation was available, like a microscope, and you could see these little things crawling around. So certainly chugging away on theories, sort of banging on them as these sort of abstractions of problems that— I think the layperson might expect that a theory is this immovable object that we sort of believe just as is. That's not how scientists use theories. We use theories as this sort of living abstraction that we sort of beat against in the hopes that it leads us to a better abstraction tomorrow or a year down the road that helps us ask better questions and make better progress on the problem. And so in that way, certainly given how effortful that process is, How effortful the theory-building process is. If we had a way of mechanizing it, I think that allows you to do 2 things. Number one, make faster progress. And then number two, certainly on something like medical domain research where you're really in a resource-constrained environment, needs of the many outweigh the needs of the few. If 90% of people suffer from X manifestation of the disease and 10% suffer from Y, you probably gotta focus your money on the X rather than the Y. But if it now is faster, cheaper, uh, to be able to do underexplored research, I think we can gain a lot from that.

Yeah, that reminds me of that story of the guy who used ChatGPT to find a cancer treatment that saved his dog's life, uh, when obviously that was not something that was gonna mobilize great scientific resources. if it wasn't for the AI to make it accessible. I guess another story I'd be interested in your take on is integrating other modalities with reasoning models, which is something that kind of one eureka moment arguably that we've had in AI for science is like the AlphaFold line of work, where we used to have to do a whole PhD's worth of tedious crystallography to get a protein structure, and now we just guess them and it seems to work roughly as well. Amazing. Obviously, models already can use those things as arm's length tools, but we're also seeing with— and I think Google's Omni model and Luma has a— there's various examples of this. Runway, I think, is kind of doing this too. But you've got these deeper integrations of reasoning and so far pixel-based modalities, image and video. But you can see clearly that there is this ability to reason and ability to operate in pixel space that is connected at a very deep level, right? So it's not a lossy arm's-length tool call anymore, but really some sort of joint understanding. And I'm expecting that this is going to be a huge deal for a ton of different areas, right? Because what we don't have as humans is the intuition for how this protein is going to fold or what the band gap is going to look like in this, you know, new semiconductor doping recipe or, you know, any of another— any of a million other, you know, super esoteric things. But the models generally do seem to be able to form that. And we do have enough proof point that it seems like those intuitions can be integrated with reasoning. That I guess my default vision of superintelligence is a reasoning model like what we have today that has a Gemini Omni sort of integration with all these other physical science domains. such that it can kind of run thought experiments that are actually really well grounded in a sort of intuitive physics, and then also reason about them in a kind of higher-order language deductive sort of way, like the best human scientists do. What's your take? You could call me out for AI snake oil if you want to, but my bet is that that's coming. What do you think?

Yeah, I don't think that you're wrong there, and I think that that's a very reasonable way to expect that the near future might pan out. Certainly historically, language models have lived mostly in text. And then in the past few years, there've been just wild advances in making them multimodal. And a bunch of that training has happened at AI2. And we certainly have just wildly impressive open and fully open multimodal models. I think that there's certainly ways that make this promising and there's ways that also make this challenging. Right now, for most of the work, for example, that I do, a lot of the content of a scientific paper is just unavailable to me. So if I wanted to look at the figures of a paper, especially for complicated material science figures that I, as a human, have a Great deal of trouble figuring out. It's very, very hard to interpret those. And figure interpretation is a very open problem. It's been an open problem for a long time. I think we're making pretty good progress on it, but certainly it's going to take a while. I don't think that's going to be solved next week or anything. To the AlphaFold example that you gave, where the models have to work in these really high-dimensional spaces, I think that what you're actually getting at there is a deeper intuition of what's going on in the field right now, which is this transition from sort of problem-specific methods, is what we call them, to sort of domain-general or problem-general methods. And so AlphaFold is a great example of this. So it was tremendous work. I think it won the Nobel Prize recently, and they basically built a machine that could fold proteins and predict protein structures with wildly better accuracy than anybody had ever done before. And this is a wildly useful thing that researchers in biology use every day. But the way that they solved it is that they solved it by building a machine that could fold proteins, that could represent those proteins and only represent those proteins. If you asked it a question about geology or anything else, it's just not going to work. It's like chess playing, Deep Blue. If you asked it how to play chess, it would be fantastic at it. But if you asked it any other thing, then it wouldn't be good. Certainly these sort of multimodal spaces, if you want to think about it that way, representing a protein as this sort of high-dimensional space, and you got to figure out how to represent it in the best way that helps solve whatever problem you're trying to solve. But then you also sort of have to work through it. That's certainly a big win, right? If you could imagine a language model, you know, Olmo 10.0 or something that could dynamically represent any kind of high-dimensional problem and work through that space, you would probably— it would probably be a big win. I will say that right now what we do is almost an intermediate of that, which is instead of the model thinking in that high-dimensional space, or at least I'm making that assertion or that assumption right now. I can't reach in and look at what it's thinking. What we do is we use code as an intermediate. And so the model will output Python code, for example, that represents the problem in some computable form. And then either it'll make an algorithm or it'll call some external library for protein folding or materials, this or that, that solves it. And so that's probably a pretty good intermediate form. And that's certainly where people are making a lot of progress these days. And we get a lot of utility by sort of having an interface to these domain-specific simulators and solvers for proteins and whatnot, and sort of the human interpretation of those.

So just today, while we've been Live on the show, Anthropic put out a statement saying that they are starting to see recursive self-improvement happening. Exactly what form, I have to read deeper into their statement to see what they've disclosed. But you can sort of imagine all of these things happening there. One big challenge, of course, is you got to have the raw data to build some of these other modality models. But it does strike me that the frontier companies are going to be in a pretty interesting space when it comes to maybe even teaching models to sort of natively understand ML architecture space, right? They could— you could imagine them training on like experimental setup predict loss, something along those lines. And if they get good at that and that becomes sort of an arm's length tool, you could really imagine things accelerating at a few companies that have the depth of the experimental record in pretty interesting ways and potentially pretty exciting ways, potentially pretty scary ways. I'm always reminded too of just how small the dataset was that was used for AlphaFold. tens of thousands of proteins, not millions, not billions, not trillions, but just tens of thousands, maybe upwards of 100,000, but a small number by any sort of big data definition. So plausibly in the ML experiment space, you could imagine a similar thing where you may not need a zillion experiments to get your models to I sometimes visualize like vacuum pulling a wrapper around some conceptual space until you just get tighter and tighter, tighter down to you really understand the shape of that space. I could see that happening in ML. This obviously takes us to strange places pretty quickly. How fast do you think this is going to go? Are you buying into the country of geniuses in a data center 2027, 2028 timeframe? And if not, what would be the core reason not?

That's a great question. So I think that sort of like a lot of other AI terms, the idea of recursive self-improvement is potentially like a repackaged version of fairly old ideas that aren't necessarily super shocking to AI researchers. So one of the oldest programming languages that was used for generating AI code for decades, which is called Lisp, actually lets the code itself sort of run code that it is itself generated. And so people have been exploring sort of recursive code generation, the code, the AI model generating its own code for longer than I've been alive. More recent stuff in auto machine learning is what it's usually called, or AutoML, has models that sort of try and make better models themselves. And this has been around, it's been explored. Usually it looks like a search space problem where there's 20 different knobs you can turn and they each have a large number of possible values. And so you just got to hope that you turn all the knobs in the right direction enough times that it gets you to something that's better than before. I can certainly imagine that with language models, people have been doing this work for at least a few years. My colleague Kyle Richardson has been doing a lot of automated— he calls it language modeling for language models that recursively does this and tries to get them better. So that in a way isn't surprising to me. I think that the improvements are largely fairly incremental. Certainly the models have been primed with synthetic data that they themselves have generated for a long time. So, well, it sounds sort of super exciting and you can sort of imagine turning on the sci-fi channel when you were a kid and watching The Outer Limits and that somebody would plug in this model that itself sort of gained intelligence over a day. Or something like that. I think that that's maybe a little bit distant from the reality in that there's sort of an established trajectory over the past couple of decades that we're comfortable doing this. And I don't think it's going to be a huge game changer, but I've been wrong about a lot of things before.

Were you on the Kurzweil prediction path? Because I do feel like I am living in a sci-fi future right now. I don't, I don't, uh, and I was, you know, Kurzweil aware 20 years ago, so I wasn't— it's not like I wasn't even exposed to those ideas, and I was open-minded enough to take them somewhat seriously. But even having seen 20 years ago the Kurzweil graph that puts, you know, human-level intelligence right around now, I still feel like I am kind of living in a surprising reality. Has 2026 AI progress not surprised you?

Oh, it surprises me and doesn't surprise me all at the same time. Some days I wake up and I feel like I'm living in the future, and other days I wake up and I feel like I'm living in this sort of strange reality with all these agents that can't do the things that I want them to do. I'll give an example that's really grounded in AI and scientific discovery. So we have this project Code Scientist, which looks very similar to a lot of the projects that sort of you pull up on Twitter every day where people say, I made this AI agent, which is like a thin wrapper on some OpenAI or Claude model or whatnot. And it generates code automatically and it generates ideas automatically and runs in a loop and away we go, writes papers. And so we gave it 50 research ideas and let it chug away for a couple of days. And after a few days it came back and it said, well, I've discovered 19 new things. And we were very excited, right? Wow, 19 new things. We live in the future. Life is great. All that jazz. And so it wrote papers on those 19 new things. And we gave those 19 papers to 3 colleagues at AI2 who hadn't seen it before and said, tell me if this is a real discovery. Look through these papers. Here we go. And they went through them and I think it was 70 or 80% of the papers, they said, yeah, it's probably at least incrementally novel and minimally scientifically sound and whatnot. And then we convinced somebody, me, to go through And spend days and days and days looking at the thousands upon thousands of lines of code that these models were generating to support their discoveries. And it went down to like 30% of the discoveries were probably real. And the things that you see are absolutely all over the place. One fun example is the AI came up with some fancy idea for making a new neural network architecture with some fancy new kind of attention, I don't know. It wrote hundreds of lines of Python code with all this neural network code that I have absolutely no idea what it was doing or can't understand any of it. I'm going through it and I'm like, how on earth am I going to review this? This isn't my domain area. Then I get to the end of a couple hundred lines of code and there's just this comment that says, insert rest of neural network code here. And then it picked a random number and returned a random number from that function. And so this model, this paper, this entire paper was analyzing the values of a random number generator. And that isn't shown to the— nobody knows that if you're reading the paper, but the science itself, it's hard to evaluate it. It's hard to be sound. And so a lot of the You know, when you see it do something amazing, it's easy to be very impressed. But then when you use a standard benchmark, like we have Science World and Discovery World, these sort of virtual environment benchmarks— Science World does 4th grade science, Discovery World does sort of like a master's or PhD level science— the best models right now are getting something like 80% on the 4th grade science. So you ask them to go in this environment and boil water and they can't do it 20% of the time, right? That's wild. Or you ask them to go in and give them a toy task, you know, the colonists on Planet X are getting sick, figure out why and solve it. They're really terrible at that, right? They can't solve most of those. Whereas you give those to real human scientists and they get most of them. So The summary of that is it's really easy to be excited when they work well, but you got to pay attention to all the really simple ways that they break before you get too excited. I think that's not to say they don't have utility and there's lots of places that they do have lots of very near-term utility, but I think my job is safe for a little while.

Yeah. That again reminds us of the importance of verifiability. When you have to spend days and days going through all the papers, it becomes a real challenge. But if you can find ways to surface the moments of, of brilliance without getting, uh, sucked into the, um, the fakery, then you can really get some, some cool stuff out. I don't know if there's anything else you want to talk about on the science front. I'd welcome anything else you think I should have asked about, or things that you think are kind of underappreciated. I also wanted to ask one question at least about AI2, and just kind of get your take on what the future is for it. I saw a friend of the show, Nathan Lambert, just announced his departure the other day. And so people are kind of wondering what the future of the institute is going to be. Are we going to still see large language models open sourced? Is it going to go in a more AI for science direction? Just kind of curious about the overall trajectory of AI2 and the role that it will play in the ecosystem as you see it going forward.

Oh, gotcha. So the first half of your question, which is what haven't we talked about that's relevant for AI for science? I think that probably, not that we haven't talked about it, but a summary would be That right now there's a huge amount of hype in AI for science. Again, I open up my Twitter feed every morning and I see more things that I'm excited to read than I can read. At the same time, it's never been easier to get started in it, in that just for fun the other day, I sat down and I tried to write sort of a minimal AI for science bots, sort of something that generates code and runs it based on an experiment. And it took me less than an hour without any coding agents at all, just sort of bare Python, bare sort of notebook editor to write a minimal science agent. And so anybody can do this, undergrads can do this. It's not that hard to get started. At the same time, Twitter isn't peer-reviewed science. And even peer-reviewed science, right? You put this paper through 3 very tired reviewers who have 10 other papers that they have to review. They do their best job, but reviewers are looking at papers. They're not sort of looking at code. They usually don't look at sort of detailed data analysis to make sure that you did everything right. And so we're really suffering from this evaluation problem right now where it's really easy to make a claim, But it's really hard to verify the claim. And previously the claims were all made by people who, they'd gone through 30 years of school and were really heavily invested in this. And if you make a bad claim, your reputation kind of takes a pretty big hit, right? Whereas the barrier to entry right now is so low and the normal institutional things like peer review to verify these discoveries are sometimes not there, certainly if it's just a hyped result on Twitter. So I'd be wary of that. I would be, as a research method scientist, preaching the virtues of how science is not just about ideas and it's not about implementations, it's about doing the science right. Richard Feynman, a Nobel laureate physicist, who invented quantum electrodynamics and was also a drum player, had all these wonderful quotes. And one of the quotes that he said that's one of my favorites is, science is about not fooling yourself when you're the easiest one to fool. And so, you know, having wonderful research methods and really excellent confidence in the results that you output is super important.

Great. Tell us about the future of AI2.

So you picked the lowest guy on the food chain to ask this question. And so I'll just give you the high level, which is that AI2 is one of the most wonderful places that I've ever worked. It is just, everybody there is the brightest person you've ever met and the nicest person you've ever met. They do amazing work. And you just sort of wake up in the morning and you're super excited to be there and work with those folks working on impactful problems. At the same time, AI2 is a technology company and there's regularly, like every tech company, people come and go all the time. I do know what you're talking about, that there was a few people on another team, the open language model team, that I think moved to another company. Things like that happen all the time at tech companies. I know that there was some speculation on Twitter that people thought that we were no longer doing language models. I have no idea where that came from. AI2 is big in language models. We have just incredible people working on that team. And I think just recently had a massive investment from National Science Foundation to work in part on AI models Or language models for science. So it's sort of never been healthier, never more resources poured into it. And I think that there's just fantastic stuff ahead. Cool.

That's great. Thank you for the update.

Thanks for having me.

Peter Jansen, sorry we started late, but appreciate you staying with us. And great to get your overview of all things AI for science. We will absolutely be watching this space.

Great. Thanks.

Thank you very much.

I found Peter didn't want to comment too much about the AI2 thing, which I found interesting because that's obviously, as Nathan Lambert leaves, the obvious question is what happens next to the firm? It must be tough because they must have like, every researcher must be getting like million-dollar offers. And the biggest game right now is how do you retain your talent? Because it's not easy to retain talent right now. And There are very few tasks that you could do that could justify paying a researcher $1 million a year. Because if you're going to pay someone $1 million a year, you need that person to produce like $10 million of output. The way VCs work is that they say like, okay, does this person have a 10% chance of producing like $100 million of output, right? And if there's a 1% chance, if there's a 10% chance of $100 million of output, that's $10 million. Like, okay, then we can afford to pay them a million, right? So I think it's tough. And Peter didn't really want to talk about that, I guess. So I respect that. Nathan Lambert was one who was very prominent, I think, both on Substack and on Twitter. So yeah, let's see what happens with that. And what did you think about the overall thrust of AI for science? It's obviously, I mean, AI2 is Peter's particular thing because a lot of what Peter's done has been that these models are not that competent at science. And it's quite a like opposing thrust to what OpenAI and DeepMind have been doing. Not, not so much Anthropic, but yeah, OpenAI and DeepMind, what they've been doing or saying is quite different, I think.

Yeah, I think I have to take the bitter pill perspective on this, as always, probably. It is hard to reconcile, as always, the flagrant mistakes and failure to even complete the task, as he described the one example of returning a random number. So I hear that for sure as a— all the scientists aren't immediately at risk. But I don't know. I kind of still feel like all these things are headed in the same direction. I need to get deeper. Apparently, Ben Todd from 80,000 Hours, they just put out the 80,000 Hours 10-year anniversary edition of the book, you know, all rewritten for the AI era. And he has an analysis which I need to revisit on why the prediction that you shouldn't be a radiologist, which was made some years ago, hasn't come true. And in the end, I'm just kind of like, I think it just hasn't come true yet. I don't really see it for all that much longer.

Yeah.

So maybe I'll end up looking stupid. But I think the original analysis really seems like it holds to me. And in science, I do think it remains to be seen whether we will get paradigm-changing discoveries out of models. That's arguably the quadrillion-dollar question, I suppose. Pretty soon you're talking real money. But yeah, I don't know. A lot of the stuff that scientists do, I don't think is really all that, you know, it is a grind. A lot of it is a grind.

Yeah.

The idea that, like, you know, I think what he expressed was an aspirational perspective on what scientists want to do. You know, they don't want to just grind out a little improvement. They want to do these sort of conceptual things. True for sure, at least early in their careers, you know, when they're, you know, when they're dreaming big dreams. In reality, there's a ton of incremental work, and I think a lot of it's probably really valuable. And what little contribution I made to science was over the course of a year as an undergrad chemistry research assistant. I don't know if I ever told you this story. I've told it other times on the podcast, but I worked for a full year, not full-time, but you know, probably 20 hours a week as a research assistant. We were developing one reaction type.

Mm-hmm.

And it was already a very esoteric thing that, you know, hydrocarbon oxidation under mild conditions, you know, in theory it had value, but it was like the kind of thing that, you know, is already deep down a particular rabbit hole. And my one discovery was based on an observation that an experiment that we thought was gonna work one way seemed to be working the other way. We thought by adding more acid to this one reaction that we would drive it forward and get more of the product out. And I just ran a, you know, little basically parameter sweep, right? It's the same thing that people do in ML all the time. But in my case, it was like just literally a row of vials with everything the same except for more acid in each vial going down the row. row, put them all into the same hot bath, measure them all at the same timestamp. And it was going the opposite way. We were getting less product for everything that we— every, you know, incremental bit of acid that we added. And so I was like, well, maybe we should try less. That was my— I had one actual insight that led to new knowledge and a new way of doing things over a year.

Mm-hmm.

That was enough to get me, you know, an authorship on that paper. And, you know, it's not nothing. But I just don't really believe that things that we call science are going to be beyond the reach of AIs for all that much longer. And then truly elite stuff, you know, we'll see. But I do think we're going to see a paradigm shift in the way science gets done, if only at the So let me give you like a framework and see whether it makes sense.

A lot of what we used to do for science was we used to get kind of a hypothesis, like this is going to happen because of this. And then we would kind of create experiments to test that hypothesis. And then after that, we'd say the hypothesis is correct or the hypothesis is not correct. is either— the null hypothesis is either proven or disproven, basically. And what AI has done is that instead of doing that, you go ahead and you collect the data first. And then you kind of run the data through this analysis machine, which then kind of p-hacks its way into what is the finding? What is the hypothesis at the end? And then that is the cause. So the causality is discovered basically, rather than proposed and then tested. The causality is kind of discovered. And so instead humans become these kind of like meat robots, like doing the experiments, getting the data, and then feeding it into this machine and kind of asking the machine to find the kind of like hypothesis in effect, which is also why it feels a little bit illegal because that's why a lot of people kind of say ML shouldn't work, but it does work. It's not supposed to, I guess. So I guess that's— tell me what you think about that way of thinking about things. Right.

Yeah, I think that's going to be huge. I'm doing an episode of the Cognitive Revolution coming up with Alex Rivas, who's the, the head of the Chan Zuckerberg Initiative Biohub, whatever that's officially called now. And they seem to be investing a ton in just scaling out datasets with this basic approach in mind.

Yeah.

And I think with interpretability techniques, that's also going to be a real way to close the loop and look at, well, what has this thing actually learned? Can we make sense of it? And can we, you know, add our own layer of conceptual refinement to it? We've only seen, I'd say, like a modest number of examples of that really working so far, but we have at least seen enough proof points that I'm pretty convinced that that paradigm will work. And it seems like it's probably right now just limited by datasets. If you imagine scaling out with Zuckerberg-scale resources, the biodatasets that we have, training models to make these predictions, Mm-hmm. and then interrogating model internals for what is driving these predictions, I think we will see, especially in biology where we're, you know, still in so many ways in the dark, I think we will really start to illuminate a lot of that space. I would expect with a phase change rate of speed. I mean, it just seems like it's the power of the intuition. I mean, I think in a way it's sort of p-hacking, but in a way it's like I really come back to this visualization of shrink-wrapping reality.

Yeah.

And the tighter, you know, the bigger the dataset, the tighter the wrap, and the better that intuition is. And when that intuition really starts to work, it's just something that we cannot match. So I see that being a pretty key threshold, and we've crossed it a little bit in a couple domains, but I think the, you know, a whole bunch of companies, right, have been started to do this kind of stuff in the last 1 or 2 years, and they're really just kind of getting started.

One thing that I think a lot of people do not understand, like a lot of people even in ML don't understand really, is that for biological data, it is very expensive. And I'll give you an example. I have friends doing cancer immunotherapy kind of models. And in order to get, let's say, 20 samples of a certain kind of pancreatic cancer, and you ask, like, let's say you go and ask like Stanford Health. Stanford Health will tell you, okay, in order to onboard, you have to pay $100 grand. And then for each sample, you're going to have to pay like $5 grand per sample. So if you want 100 samples, that's $600 grand for a particular kind of cancer. And it's not even— they're not making a profit off it. It's just, you know, it ends up in the NIH kind of funding bucket and it goes to the research organization. It's not, but, you know, because it costs money to like, and they're not trading it. They're not like, none of the researchers are like, you know, making money off selling cancer samples. It's not, it's not happening, right? But in order to get all of these samples, like, that's the cost that you have to go through. There's the HIPAA, and then like, there's making sure that it was properly collected, storing it, refrigerating it. Like, it has to have like, um, a chain of custody from the point it was collected to the point it's been stored, for how long has it been stored, the whole cycle of things that has to happen. And let's say, you know, if an ML researcher comes out and says, look, if you give us like, you know, 100,000 samples of this particular kind of cancer, we can cure it. It's not happening, right? It's just not happening. Like, you're fighting over like 100 samples here or 500 samples there. And it's, it's when you see people, especially when you see people like from Google who enter for like from the web space and they go into bio, bio ML, and then they're just shocked because you go from like billions of data points, you know, a day to like, here, you're gonna get 200 samples this year, right? Like, that's it. You're gonna have to cure cancer with that.

Right?

And that's the sad part, right? Like, we don't have— we just simply don't have enough data in the biospace for a lot of things that I always say, like, look, the algorithms that we're going to use to cure cancer already exist, but the data that we need to cure that is not there yet, right? And the real hope is that we get to some level of sample efficiency, that 100 samples is enough for us to do something with it, rather, or we