The AI Product Going Viral With Doctors: OpenEvidence, with CEO Daniel Nadler

One of the things we hear so frequently from doctors about Open evidence is, you know, I used it to look up this thing for a patient case that is maybe a patient case that I would have seen one or two times in my career. And then the same doctors are saying that about a different patient case and then about a different patient case and then about a different patient case. And then you realize just how long the tail of this thing is, where it's like, well, if the majority of your uses of open evidence are patient cases that you would see once or twice in your career, then that really captures what the thing is doing.

One of the questions on everybody's mind as it relates to AI is will this actually be good for humanity? One of the use cases where AI is inarguably good is medicine. We have Daniel Nadler, co founder of Open Evidence, on the show today. Open Evidence is trained on peer reviewed medical literature to provide an AI copilot that helps doctors make better decisions at the point of care. This is inarguably good for humanity. It doesn't take crazy assumptions to believe that Open Evidence will save a million lives over the next decade. Today we'll hear from Daniel on how they built the product, what's different versus some of the other application layer AI products out there, and some real life examples of how it's being used in the field to the benefit of everybody.

I hope you enjoy.

Daniel, welcome to Training Data. Thanks for coming on the show.

Thanks Pat, for having me.

All right, how many doctors will use Open Evidence today?

Today, probably over a hundred thousand physicians in the United States and more globally.

And what would that number have been.

About a year ago? A thousand? Fewer than zero. You know, look, most people aren't aware of this, but there's only about a million doctors in the United States serving a population of over 340 million people, which is itself an issue that we'll probably talk about later on. But there's only a population, about a million doctors today, maybe a hundred thousand of them are going to use Open Evidence on a monthly basis. About 3, 400,000 monthly active users touch our system, including over 200,000 that log in and ask questions. So, you know, you're talking about 10, 15, 20%, maybe even 25% of doctors in the United States that are using Open Evidence in some form or another.

And we're, you know, we're not healthcare investors, we're technology investors here at Sequoia. But my understanding is that it's not normal for 100,000 doctors to be using Something, you know, overnight. Like my understanding is that in healthcare it normally takes a lot longer to get to that sort of scale. So what, what did you guys do?

Right? Well, that's why we get along so well, because you're not health care and you're not health care investors. And our approach has not been a health care approach. What we got right is we realized that, that doctors are people too. Doctors are consumers. In fact, everyone's a consumer. I think that's what you get right in your investment strategy, which is you don't sort of think about consumer Internet applications as one category and then, you know, a bunch of stuff over here as its own kind of weird, quirky, opaque, siloed industry categories over there. You think about everything in terms of consumer, consumer Internet growth curves. And that's exactly how we thought of it. You know, obviously if you try to go top down, so everyone says, you know, it's, it's such a refrain, you know, healthcare is impossible to break into. Healthcare is so hard, don't start healthcare.

And I think the evidence generally supports.

That, of course, because they're all trying to do the same. They're all trying to bang their head against the wall in the exact same way. They're all trying to go to the top of some integrated delivery network, which is like a large network of hospital systems. Score a meeting with the CMIO or the CTO or the, you know, all these acronyms. It'll take them three months to four months. Even if, you know, if they're properly networked and connected, it would take them three or four months to get meeting one on the calendar. Meeting one will be great. They'll high five each other. They'll be like, that was a great meeting. We got great feedback. What's the follow up? Well, they're going to schedule a follow up meeting which will probably be with a responsible AI committee. That will take three, three months after the first meeting. Then you'll present to the responsible AI committee, you know, at that hospital. Along the way the committee members will have changed their strategy around. Responsible AI will have changed, maybe the presidential administration will have changed. And now JD Vance is saying actually you have to change your approach to responsible AI. So that gets delayed and you're a year into it before you're, you know, in meeting three or meeting four. And by the way, no doctors there along the way are getting the benefit of using the application. I had a lot of experience, you know, this wasn't my first rodeo. I had sold a very successful AI company before starting Open evidence Kensho. And so I was very familiar with how corporate America works and how large organizations work. It's not specific to the healthcare system, it's just called large organizations. And I was very familiar with that. I had pattern recognition to that. And so I realized that that wasn't really a viable path forward for us especially. This is my second company was very mission driven, it was very impact driven. But it would not have been much sense of accomplishment for me to have nominally started a second company that was mission driven or impact driven in healthcare, but then have no doctors using it because we're waiting for meeting number six or seven or eight with the hospital system. So long story short, we just took a radically different approach. Doctors are people too. Doctors are consumers too. And if you make something awesome that is life changing and game changing and profession changing for knowledge workers and it's really good enough and you put it out on the app store, it sounds 101, but it really works, people, you make something awesome, you put it out on the app store for free. And it turns out that without fancy marketing campaigns or large marketing budgets or anything like that, all of our growth is word of mouth, doctor to doctor to doctor to doctor. People will start using it and people will discover it and then they'll start using it. And when they start using it and if it's really awesome, they'll tell other people about it. And then you get just sort of the network effects that, you know, you got with Tesla early on. Tesla early on famously spent, you know, almost no money in marketing. They didn't, you know, as I understood, car commercials were kind of one of the largest category of advertising. It was, it was taken as dogma that if you wanted to have even a remotely successful automobile, you needed to spend enormous sums of money on advertising those cars. And Tesla early on said, we're just going to make a product that's so awesome that people will tell other people about it, someone will drive it, they'll be like, oh my God, this is so much better than other cars and tell other people about it. And that sort of word of mouth network effect that Tesla had happened with open evidence, with doctors, where doctors downloaded from the app store. As I said a year ago, it would have been, I don't know, somewhere between zero and a thousand doctors using it. Today, it's hundreds of thousands. It's, you know, 10, 15, 20, maybe 25% of active physicians in the United States. It really the denominator, it depends on how you calculate it because there are more physicians that have a medical license than are necessarily active at any one time. But it's as little as 10% and as high as 25% of active physicians in the United States use open evidence today. And all that is just this sort of Tesla logic of like make an awesome car. It'll spread through word of mouth.

And I think it's intuitive to people what it means to have an awesome car. It's probably not intuitive to people what it means to have an awesome app for doctors. What are doctors is actually doing in this app? Why do they like it so much?

So I think something is awesome if first and foremost it solves a real need and a real pain point. Right? So much of technology is a solution in search of a problem. It's important to be able toward problems that, that, that are real and, and have a solution that addresses a real problem. So you gotta begin from well, what's hard about being a doctor. One of the hardest things about being a doctor, besides the hours having to go through medical school, all the rest, the fact that they're stretched too thin and there's too few of them for the population, is that they're expected to keep up with a fire hose of medical information. So this is really not appreciated by people who are not Doctors. But there's two new medical papers published every minute, 24 hours per day. Two new medical papers published every minute,24 hours per day, seven days a week. There, there was a, a study done in, in published in Nature that said that medical knowledge doubles every 73 days. That's probably, that methodology was probably a little aggressive. We did our own internal studied open evidence of the rate of doubling of medical knowledge. We came up with a much more conservative number, but that's still five years. Our conservative number is medical knowledge doubles every five years. There's a lot of methodology in how you think about counting, you know, all citations. If it's all citations, yeah, it doubles every 73 days. But not all citations in medicine are equal. Even if you really just count what you should count, which is the stuff that doctors really need to know. Like top tier journals that have the highest impact factors, let's say the top third of journal, something like that. Even with that very conservative methodology, you're talking about medical knowledge doubling every five years. So if you think about the math of that for a second. So in 1950, medical knowledge doubles every 50 years. Today it doubles every, every five years. What that means is if you graduate Medical School in 1950, the Half Life of what you've learned in medical school, meaning the time it takes for half of what you learn in medical school to kind of be out of date. And we're not talking about anatomy or, you know, that anatomy, cellular biology, that stuff doesn't change or doesn't change that quickly, but the rest of it actually does change. And so in 1950, the growth of medical knowledge, meaning new treatments that they need to be aware of, was every 50 years, which means the half life of what they learned in medical nicely overlapped with the length of their career. So by the time they retired, maybe half of what they learned in medical school in terms of the efficacy of new treatments and new treatments that were available was kind of out of date, but they were retiring, so that's kind of okay, right? And they tried to keep up along the way, and it was easier to keep up along the way because it was so much slower. Today, if you think about that same framework, by the time they're like, depending on their specialty, going through their residency and fellowship, half of what they learned in medical school is now out of date in terms of new treatments, the efficacy of those treatments and so on. So it's just, it's impossible for doctors to keep up with that because medical school, which is what most people are not doctors assume is the information transmission mechanism for medical knowledge, only sustains a doctor for a couple of years these days in terms of the half life of their knowledge. You have a patient coming to the dermatologist who has psoriasis, okay, maybe you say, fine, just read up on the new classes of biologics and pick one based on efficacy and safety. And in this example, you see that the specialization of medicine becomes very challenging because the dermatologist maybe is keeping up with the dermatology journals and all the new biologics for psoriasis that are published in the dermatology journals. But Ms. Is a neurology condition, and it's ludicrous to expect the dermatologist to also read every single page of every single neurology journal, let alone the border region or the interaction between those two specialties. So the dermatologist in that situation, in that scenario, is in a real pickle because they don't want to make the Ms. Worse. They don't want to just send the patient away and say, hey, everything's too risky, so we're not going to treat your psoriasis because that's a quality of life issue for the patient. And they need to sort of figure out what is the latest evidence, hence open evidence, what is the latest Evidence on the efficacy of, yes, IL17 inhibitors and IL23 inhibitors, but specifically with a lens to what is the efficacy and safety of those inhibitors for patients who also have the comorbidity of Ms. And that information hunt and gathering exercise is in a traditional, you know, pre open evidence is just very painful. The traditional way of doing that. You would, you'd go on Google, try to Google this, you'd go on PubMed. None of those searches work particularly well. They'll sort of give you the titles of the articles. But this is such a specific question. This is a very specific question. This is not like a generic article title, like what is the efficacy of IL17 inhibitors? This is a very specific question. What is the safety of IL17 inhibitors versus IL23 inhibitors for a patient with both psoriasis and MS? And that was a real need to come back to your question. What makes something awesome for doctors? Not just this one specific case, but as you can imagine, for every example you can give like this, of, you know, a comorbidity, psoriasis, MS, seeing a dermatologist. The surface area of medicine is so enormous that you have millions and millions and millions of cases like this. Engineers that are listening to this can immediately understand this. Everything is an edge case, everything is a corner case, right? So from an engineering perspective, the way to think about medicine is the surface area is it's not truly infinite, but it's for all intents and purposes enormous. And everything is an edge case, and everything is a corner case. And you're trying to always look up the edge case or the corner case. That's the experience of being a doctor, put in engineering terms. And so if you can solve the experience of being a doctor framed in engineering terms of everything is an edge case, everything is a corner case, go find, solve the lookup for that edge case and that corner case, meaning go find the reference somewhere in a medical journal that is a peer reviewed top tier medical journal that answers the question of the comparative safety of IL17s versus IL23s in a patient with psoriasis and MS, which is never going to be in the title, which is what PubMed or Google could find. It's always going to be buried, you know, on page five or six or seven of that 30 page medical journal article. Then you have made the experience of being a doctor that much better. You have made the lives of doctors that much better. And most importantly, you've improved the life of the patients that they're treating because then you've prevented a scenario where The Ms. Is getting worse because the doctor didn't know. Hey, you know, IL17s are very promising generally for psoriasis, but if a patient has MS, IL23s are actually a lot safer. Right? If, if the doctor didn't know that, which is no reason for them to know because the average, you know, given the average age of a doctor, not. Neither of those two things existed when they went to medical school. They couldn't have, is not even like a study question. They could not have learned that in Medical School. IL23 inhibitors came out in 2017, 2018, 2019. There's no way even if doctors my age and I'd be a young looking doctor, most doctors you see are older than I am. And even if a doctor were my age, I am old enough that were I a doctor, I would not have learned that in medical school. So there's no way a doctor can learn that in medical school. They would need to sort of keep up with it post medical school. And because it's an edge case or corner case, and because for every one example like that example, there's 10,000 other examples that they would also need to stay on top of, they probably would have had a hard time keeping up with that data point pre open evidence. And that would have resulted in a worse outcome for patients. And one of the things we hear so frequently from doctors about open evidence is, is, you know, I used it to look up this thing for a patient case that is maybe a patient case that I would have seen one or two times in my career. And then the same doctors are saying that about a different patient case and then about a different patient case and then about a different patient case. And then you realize just how long the tail of this thing is where it's like, well, if the majority of your uses of open evidence are patient cases that you would see once or twice in your career, then that really captures what the thing is doing. It's, it's essentially running search and discovery and, and knowledge retrieval on a tail that is not nothing is infinitely long, but, but is so long that it, for all intents and purposes to a, to, to a wet human brain might as well be infinitely long.

Yeah, yeah. The, you know, the data on the rate at which medical knowledge is increasing, it's a very, it's a very positive data point. Right. Like it's great that medical research and medical knowledge is increasing so quickly. It's like this keg being filled with potential energy that hasn't converted into kinetic energy because we have this choke point, which is a human's ability to ingest and make sense of all this information. So it makes sense that now that AI is here, AI is great at looking over enormous amounts of text and doing reasoning across enormous amounts of text. It makes sense that AI is a little bit of an unlocked. It's sort of convert that into kinetic energy with the, with the system level view. The question, we talked a bit about some of this evidence that's going into the system. Decompose the name of the company real quick. Open evidence. Why is being open important and what exactly is all the evidence that's going into the system?

Sure. So the evidence is peer reviewed medical literature. And it's most important to say what it's not. One of the reasons you got so many egg on face situations from large publicly traded tech companies that put out AI systems in the area of medicine, you got these sort of egg on face situations. I won't say specific examples, but I think we all know what they are is because they were doing retrieval across the public Internet, which means they're doing retrieval across health blogs. Health blogs on the public Internet. They, they might have been doing retrieval at the time on Twitter, before Elon cut off access to that. There's a lot of stuff on Twitter that you wouldn't want to be the basis for for a doctor's decision. But even if you just take the health blog thing, it's amazing how many health blog writers there are on the Internet. I only discovered that when I sort of started working on this problem. And most people assume that the people writing health blogs are doctors or at least have some connection to medicine. That's actually not the case. These are not bad people. They're really well meaning people. But what you'll find is a lot of these people are only part time health blog writers and they're also part time travel writers or they're part time cooking recipe writers. They're basically sort of journalists bloggers where their core skill set is writing. Very good at writing. They don't have any domain expertise in medicine. They don't know any more about medicine necessarily than they know about planning an itinerary for a trip to Mexico. And that's what goes in to the training data. This thing's called training data. And then we're shocked when in the early days of large language models they said all sorts of crazy things. Well, they didn't say crazy things. They regurgitated what was in the training data. And those things didn't intend to be crazy, but they were Just not written by experts. So all of that's to say, you know, where open evidence really right in its name. And then in the early days took a hard turn in the other direction from that is we said all the models that we're going to train do not have a connection to the Internet. They literally are not connected to the public Internet. You don't even have to go so far as like what's in, what's out. There's no connection to the public Internet. None of that stuff goes into the open evidence models that we train. What does go into the open evidence models that we train is the New England Journal of Medicine, which we've achieved through a strategic partnership with the New England Journal of Medicine.

Say a word about that for a minute. Because my understanding is that New England Journal of Medicine doesn't just give all of its research to everybody for free to go train on.

They don't. To the best of my knowledge, we're the only AI company that they've done this with and, and to the, and without getting into the specifics, we're not the only AI company that's asked, they've said no many times.

Why do they trust, seemingly trust open evidence when they don't trust other folks? Like what, what makes you particular appealing to them?

Well, looking into the exact play by play, what happened was a number of other well known AI companies showed up at their door and said, can we train on the England Journal of Medicine? And the England Journal of Medicine said no. I won't even get to the reasons of why they said no and I'm not them, so I don't speak on the behalf, but they said no. In our case, we didn't show up at their door. A number of the very senior people on the editorial board New England Journal of Medicine were power users of open evidence and they wanted their content to show up in the thing that they were using. So it's beautiful, right? And so they came to us and then we spent a lot of time really getting right what a framework for cooperation, collaboration which prioritizes and privileges the importance of their brand and the sanctity of their brand and their. They're the pinnacle to the Apex Medical Journal and the Massachusetts Medical. And they're nonprofit, they're not commercially motivated. There's no amount of money you could throw at them that's going to make them make a perfectly mercenary decision. And in fact, without getting the specifics of it, some of these really well funded AI companies threw enormous amounts of money at them and they Said no. If they're a private company, they probably would have said yes, but they're a nonprofit. So they said no. Because the Massachusetts Medical Society, which is a nonprofit organization, cared more about the sanctity and the pristineness of their mission as a nonprofit than they did about just trying to score some sort of quick commercial contract. In the case of open evidence, again, it was beautiful. Very senior people, there were users. And then this sort of circles back to what we talked about right at the start, which is, had we waited, had we taken a top down approach, had we taken an enterprise SaaS approach and been, you know, in waiting mode for meeting number 17 with the hospital system and no one was using it. Well, if no one was using it, that would include no one at the New England Journal of Medicine using it, which would have meant that they wouldn't have fallen in love with it, which meant that they, we wouldn't have never had an opportunity to strike a content partnership with them. That would have been, you know, that in turn made the whole thing that much better and more awesome to the people using it. So you get into vicious cycles versus virtuous cycles. In our case, the whole thing was a virtuous cycle. We put it out there, people downloaded it for free. Some of those people included very senior people at the New England Journal of Medicine. They started using it, they fell in love with it, they reached out to us, we did this deal and now the thing is 10,000 times better because it's trained on the full tax Journal of Medicine, which no AI today in the market. I can tell you with certainty, no way. I today in the market is trained on the full text in England Journal of Medicine other than open evidence.

So we talked about the evidence part. Let's talk about the open part a little bit. What, what is, what does the open piece of open evidence mean and why is that part important?

Open meant a lot of things to me in the early days. One of the things that it meant was capturing that go to market strategy that we talked about, which is for me, open was almost a reminder to myself that this was not an enterprise SaaS company. You know, my first company was Enterprise SaaS company. Those can be great businesses. I don't need to tell you you had phenomenal success with enterprise SaaS companies. They can be great businesses. But for my second company, I didn't want to just have a mission driven, impact driven company. I also wanted to have a company that was different from my first company in every respect because I don't like repeating myself. Specifically, I want it to be sort of direct to consumer, in our case, direct to prosumer. So the open for me sort of symbolize that, that we would go directly to doctors, that we wouldn't go to their gatekeepers, we wouldn't allow people to be gatekeepers to doctors, we would go directly to doctors, we'd appeal directly to doctors, would appeal directly to the pain points that they experience on a daily basis, which is that they're overstretched, they're overworked, they're not enough of them, they're stretched far too thin in terms of the number of patients that they need to see and treat, but also that they're forced to drink from a medical information fire hose. And we were going to make that better and we're going to help tame that medical information fire hose and we're going to make that appeal directly to doctors as consumers and as people specifically. And that's, that was a big part of what open meant. You know, there's, there's enormous inequality in the healthcare system in the United States, as there is in everything in, in the West. And they're the haves and they're the have nots. And the best hospital systems in the United States, which have enormous endowments and unlimited funding, can afford to buy not just like one tool or two tools or three tools, they can, they can afford to buy every tool in the category and try them all out. And they can afford to have like doctors not really use most of them. And that's okay. Because you know, if, if, if Doge or Elon went and did an audit of the SaaS spend of some of these really well endowed hospitals, he would have a field day because what he would find is they're buying everything and they're using almost nothing.

Yep.

And that's happening over here. While over here you have doctors in rural parts of the country or urban parts of the country that just are socioeconomically disadvantaged with, or you have doctors that are in private practice or in groups of, you know, 10 or fewer practices of 10 or fewer doctors. Most people don't realize about doctors. A lot of doctors are small business owners. They don't work for these enormous hospital systems with a lot of money. A lot of doctors are, are, are self proprietors, they're small business owners. They, they have a, a practice. It's kind of almost like the 1950s, but it, it persists. They have a practice. They might have one or two people helping them in an administrative capacity or a secretary or something like that, but they're the principal and they have to deal with all the administrative stuff plus see patients plus plus plus plus. And they don't have these enormous technology budgets. They certainly don't have endowments like university style endowments like some hospitals do. Then they can't afford to go pay $10,000 for a Enterprise SaaS subscription to some software product. So all of that is what open meant. We got a letter from a doctor in Albany, Georgia who said he's the director of a cancer center in Georgia and he's in community practice as a community oncology practice. And that open evidence has become a lifeline to his daily practice in his cancer center and has been a game changer for patients in treating his cancer patients. And like most people, I didn't know much about Albany, Georgia, so I looked up Albany, Georgia and very quickly on Wikipedia, I discovered that Albany, Georgia is in southwestern Georgia. It's 75% African American. It has a median household income of $43,000 a year. And I started to piece together the situation. You know, this doctor's probably the only oncologist in a 50 mile radius. Maybe there's a second one. And they're serving an enormous geography of fairly poor people. And there's no way that this doctor has the resources to pay 10 or $20,000 SaaS subscription software rates for anything. And that to me is what open means. It means the doctor in Fairbanks, Alaska who wrote us a letter saying that she practices in again, a community setting in Fairbanks, Alaska, very limited access to subspecialists. And open evidence has been a game changer for her in allowing her to sort of access sub specialty level medical knowledge without direct access to human sub specialists in Fairbanks, Alaska. Again, if you look up that situation or just even think through that situation, she's not paying, you know, she's in a community practice. She doesn't work for a hospital that can afford this kind of stuff. So that's what open means. It's every doctor in the country. We're really proud that we're used not just at Mayo Clinic in Cleveland. We love the Mayo Clinic. We were partly incubated at the Mayo Clinic. We love these sort of elite hospital settings. We have a lot of users at these elite hospital settings that also use it for free. But we're not just used at the Mayo Clinic. We're not just used at the Cleveland Clinic. We're used across the country. We're used in the middle of the country, were used at Walter Reed, where our nation's warriors and veterans are treated without the government having to go through a three year procurement process to decide whether they want to use open evidence. You know, that's another example. So one of the biggest health systems is the va. To me, that's one of the most important health systems because it treats our warriors and our veterans. For the VA to decide to do anything is probably a three year exercise. And were open evidence not open to answer your question, we would probably still be in year 1.5 of three to four years of a government procurement process to decide whether doctors in the VA could treat their patients who are warriors and veterans with the benefit of open evidence. And thankfully we didn't go that route. And we're getting letters from, from doctors in the VA talking about using open evidence to make a treatment decision at the point of care for a wounded warrior. You know, that's this. I mean, it energizes me waking up every day. So that's what open means.

Okay, so you, you guys, you've built something of a killer app for medicine. And it's working, it's working really well. And we have a lot of people typically who listen to the show who are themselves trying to build killer apps of some sort for AI. So I'm curious, using AI.

So I'm.

Curious, how did you build it? Like, what, you know, is this a wrapper on top of GPT3 or GPT4? Like, what is the, what's going on under the hood? How'd you build it?

So I think what I'm gonna, I'm gonna sort of have two parts to what I say. I'm gonna sort of talk about what we did and then talk about what is applicable for a lot of people listening to us. Now, I'm guessing a lot of people listening to us are building applications that don't necessarily have the same requirements that medicine has. So I do want to address that. In the case of medicine, the way we attack the problem is by bringing together a team of PhD level scientists who are working in the field. My co founder, Zachary Ziegler is a brilliant computer scientist from Harvard, studied with Alexander Rush in his natural language processing lab at Harvard. That was one of the leading labs before ChatGPT even came out. Evan Hernandez comes from Jacob Andreas lab at mit, so I could go on. Eric Lehman from MIT as well. I mean, we put together a team of sort of elite scientists who are working at the frontier of language models at the top. You know, two of the top three, maybe labs, if not two of the top two labs at the time in the country in the world actually. And we needed to do that because we're trying to solve medicine and we're trying to solve the application of language models in medicine. And that was a very high standard and a very high bar and it hadn't been solved. And what the very large consumer companies were putting out was creating all sorts of embarrassing moments in medicine for those companies at the time. Sort of recent enough that we probably all remember this. And so we needed sort of that level of almost the intersection of academia and engineering to go after this problem. And we needed to in our case, produce original research and original knowledge. So we attacked the problem in a different way. Everyone was trying to at the time focus on us, you know, scaling these language models larger and larger and larger and larger. And we were very, very early. You know, now this is sort of consensus with Deep SEQ and all this stuff, but Rewind, this is 2022. We were very early to the insight that smaller, highly specialized models over trained on in domain data would outperform much larger models on those in domain tasks. They were very rigid, they wouldn't write you a poem, they'd fall over very quickly the second you go outside the domain. But in the domain they were beautiful. They outperformed. And we published our work. We approached this very academically. My background is academic as well. I did my PhD at Harvard and we were all academics by background. And we published our work in this paper. Do we still need Clinical Language Models? Which was awarded the best paper in machine learning in 2023 at the leading conference in machine learning and healthcare and attracted a lot of attention, was really the first paper in the field altogether that showed that in medicine the best way to attack the problem was these smaller, more specialized models. Again that's become consensus today. But you have to sort of pretend you're not listening to this today. Pretend you're listening to this in 2022. Pre chat GPT it wasn't obvious because what was, what was coming out at the time was like the chinchilla paper from DeepMind and everything was about larger, larger, larger scale, scale, scale. And we just took this very different approach. And in a way like it's very, with the benefit of hindsight, everything's obvious, but with the benefit of hindsight it's kind of obvious, right? If you, if you, if you go and listen to Jensen's interview of Ilia, you know, Ilya uses the metaphor of JPEG compression. Basically it's like these language models, basically like a JPEG compression of the world. And that's okay, well, if they're a JPEG compression of the world, what's the world. What's the world that you're compressing? And it goes back to what we talked about, Public Internet. Yep. If you're, if you're doing a JPEG compression of the public Internet, then what you're. Which is what these large language models that we're focusing on scale, where basically they're token limited, they're like, give me as many tokens as possible for me, for me to train on. And so. Well, where do you find all the tokens in the world? You find them on the public Internet. Well, to go back to Ilya's point, well, what are you compressing then? What are you JPEG compressing? You're compressing the public Internet and then you get all these sort of embarrassing outputs that, that were sort of of that late 2022, early 2023 vintage. In our case, we sort of said let's make a JPEG compression of medicine. And so let's, let's over train on again. Peer reviewed medical science stuff that comes out of the fda, the cdc. We had the advantage, this is way before the New England Journal of Medicine partnership, but we had the advantage of that under copyright law, anything created by the US government is public domain. That's how Wikipedia does a lot of what it does. So in the early days, we started with sort of like Creative Commons public domain stuff that was available. And we were very lucky that in medicine this wouldn't work in every other field, because in other fields like law or accounting or tax, there's a lot of stuff that's behind walls. But in medicine, it turned out that a lot of the really great stuff was created by the US government in the form of the FDA and the CDC and what they had put out. So we overtrained on that stuff and solved the copyright issue that way early on, which then allowed us to bootstrap something awesome enough that people could download it, which then won over users at places where there were copyright considerations, like the Journal of Medicine, which then started the flywheel of them reaching out to us and then now us having the benefit of the stuff that is under copyright of the New England Journal of Medicine. But that, that was our approach. It was very technical and it was very academic and it was very scientific because accuracy mattered that much given our domain in medicine.

Yeah, I was gonna ask you about that. So you know, you have doctors using open evidence to make clinical decisions at the point of care.

Yeah.

So in most applications, a Hallucination is annoying. With open evidence, a hallucination could be literally life threatening, given how it's being used. So how do you, how do you deal with hallucinations in particular?

Well, and just to reiterate the first thing you said, you know, and maybe this is advice for entrepreneurs or engineers who are listening to this, there are scenarios where hallucination is not even annoying. There are scenarios where hallucination is a feature. One of my favorite applications is mid journey. Yeah, hallucination is a feature.

Yeah, yeah.

Of mid journey, maybe one takeaway is find applications where the biggest hesitation actually gets judo moved into a feature as opposed to a limitation. And just a total aside, an example. And someone should go start this company again. My first company was in finance and I feel like I kind of grew up on Wall street and I think that way. And if I wasn't running open evidence, I wouldn't run another company because I'm going to work in medicine, I think for the rest of my career because of the impact. But if I were pressed to sort of like, what would I do in finance now that large language models exist? I would actually begin from thinking about hallucination as a feature as opposed to a limitation. So where is hallucination a feature in finance? Well, it's certainly not in like doing retrieval on the P ratio. You need that to be right. Well, where else could it be used? What about risk management? A lot of finance is about figuring out what the black swans are. A lot of finance is what could go wrong. And then at the extreme tail, there's a lot of money at stake. What could. What is very unlikely, but that could go really, really wrong. And the first two questions most people can reason through without the aid of computers, what can go right and what can go wrong? You can kind of reason through that with your wet brain. But in terms of like, what could go really, really, really wrong at the level of the 2008 financial crisis? That's harder for, for the. On for. For most unaided brains to imagine that. But these language models would be pretty good. And I've tested some of this in my own portfolio management at hallucinating those things. I've actually, as an experiment, done this in my own portfolio management. Just use these language models in a way that the hallucination is a feature where I give it certain details of my portfolio, I talk about the company, I give it in the context window, enough information so that it understands what the company does and so on. And it comes up with all sorts of scenarios that are kind of on the long tail of what could go wrong. And I'm like, huh, I never thought that I, I love Nvidia, but I never thought about that happening to Nvidia. That's interesting. So at a high level, I think there's enormous opportunity. I think we're like 1% of the market captured today in 2025 in terms of applications built that even begin to think about hallucination or riffing as an advantage as opposed to as a limitation. So that's for all the entrepreneurs listening. There's, there's 99% is still up for grabs. Yes. In my specific domain, medicine, none of that is fair game. And so the way, you know, we had to deal with that is by not connecting open evidence and models that we trained to the public Internet is by only training on peer reviewed medical knowledge. And go back to Ilya's point. The JPEG compression that we made in our models, in our, in our smaller specialized retrieval models and ranking models, you know, we don't use one model without getting into our sort of trade secrets. Like, it's an ensemble architecture. There's multiple models that do different things. There's half a dozen models, they hand off tasks to each other. You can't get the accuracy level of open evidence by training one large language model. It's under the hood, going to be this sort of cooperative ensemble architecture. And those are made up largely of smaller models that do very specialized things like retrieval and ranking and other things. And for those models, the JPEG compression is exclusively of peer reviewed medical literature. So it's never going to be at risk of regurgitating or surfacing something that is not in the peer reviewed medical literature, which is, as they say in GI Joe, more than half the battle. Right. And then the other half the battle is allowing transparency and interrogation of the answer. And we were very early to that. I've seen now that ChatGPT and others start to do that, but we were probably the first application. I would go on a limb and say we're like the first application that grounded our answers in references that you could drill down and drill through to see the underlying sources of. We did that in early 2023, long before chat and others started to come out with similar features. And that's how we won over users in the early days, because doctors saw that not only is this not saying anything egregiously wrong in terms of regurgitating something on the public Internet, but it's also even within the domain of the answers that it gives based on peer reviewed medical literature, allowing me as the physician to go interrogate where it's getting that thing that it's saying where it's getting that source of information from, and drill down all the way to the reference and then go and read the reference. Which by the way, also created a beautifully symbiotic relationship with the publishers, because instead of just compressing all their knowledge and then giving it away, as some folks did as a result of just trying to make this as accurate as possible, inadvertently stumbled upon a model of cooperation with publishers that ended up being very good for them too. Because we send an enormous amount of traffic to medical journals. We send millions and tens of millions, tens and tens of millions of visits from doctors to medical journal pages hosted by those medical journals, including traffic they might not have otherwise have gotten. Because the doctor is going to that journal because of some detail deep in the methodology section that the doctor would have never known to go to that journal for, really this virtuous circle kind of all around. And then we had medical societies that write guidelines reaching out to us, saying, hey, we noticed that you index this other society's guidelines. Can you go index our guidelines? Because we want the traffic. It's beautiful, right? So you get accuracy, you get a symbiotic and a mutually beneficial relationship with medical journal owners. But critically, you get the right information back to the doctor using open evidence, who then will make the better decision as a result of having better information for their patient at the point of care.

And it's been a decade or so since Kinsho got going and obviously there's been a lot of progress in the field of AI and machine learning since then. If we were to inspect the underlying architectures of Kinsho and Open Evidence, how much is the same and how much is different? And I guess part of the question behind the question is how much of what goes into making an AI application that actually works is recent breakthroughs and how much of it is more sort of classical engineering, machine learning principles.

Kencha was pre large language models, pre language models, pre small language models, pre Bert, pre anything, almost pre Fire. So it's hard to compare, right? I mean, Kensho was very early nlp, not when I sold the company. By the time I sold, the company was much more sophisticated. But I'm talking about when I founded Kencho in 2013. So it's very different today. What they have in common is there's an enormous infrastructure component to building this stuff. So, you know, we train our own models. And I talked about that over the last few minutes. And. But even if you're not training your own models, even if you're just like using an API to one of the usual suspects, you know that's going to fall over at some point if you're successful and you want to be successful and you want to get to the point where it falls over and it will fall over. And at that point you need to have all the traditional things that you have in traditional software engineering, like infrastructure and really good infrastructure. And that is very similar to Kencho, because both were critical systems. You know, in the case of finance, there's enormous amounts of money being moved around on the basis of this information. You can't have it just stop or fall over in the middle of a trade. And I think that's a good thing. I think that one of the things that everybody was concerned about post the ChatGPT moment was that all the rules of the game had changed. And, you know, I'm here to sort of tell you that they haven't. Yeah, the technology's better, but that's a continuum. The technology has always been better. The technology was better from 1982 or 83 to 1987. Right. And from 93 to 97. Right. The technology's always gone better. Yes, there's a step function now. Yes, there's non linearity. Yes, there's a exponential rate of increase. Yes, everything Ray Kurzweil said is correct turned out to be correct. But. But it's a continuum. You know, even Ray Kurzweil thinks about this stuff as a continuum. And when you think about something as a continuum, it's very, it's a relief in a way, because a continuum is something where the laws of physics aren't changing along the continuum. Even if you think about the metaphor of travel toward the speed of light. Yes, the technology that would get us from 1/10 the speed of light to one half the speed of light in a spacecraft is highly nonlinear in its sophistication. But the laws of physics in that acceleration are not changing. It's just the technology's on a non linear continuum, but it's still a continuum. And that same continuum nonlinear. But continuum exists in engineering and entrepreneurship more broadly, but specifically in AI, where everything that mattered at Kensho continues to matter today. And the intelligence level of the people that you're bringing to bear on the task matters. Kensho and open evidence are identical in that we were able to be successful because we brought people with really high IQs to bear on the task.

Let's talk about that yeah, you mentioned Zach and Evan and Eric and Micah. How do you. How do you attract people like that? Why do they choose for all the options they have, why do they choose to work on open evidence?

It's impossible for me to answer that without just repeating what Steve Jobs has said, which itself has been repeated so many times, but I don't have a better way of phrasing it. A players want to work with a players. It's that simple. Elite people want to work with elite people. A lot of people who sign up to Buds, which is the sort of screening process for Navy SEALs, do so because they just want to see if they can keep up with the other people that are doing it. They want to test themselves. They want to see what their limits are. That's as old as Achilles. That's not new. It doesn't matter whether it's warfare or, or engineering or sports or any other domain, finance, the very best people in the world want to see just how good they are. They want to see what they're made of. And the only way to do that, the only way to learn that, is to put yourself around other elite people and see how you stack up against those people. So that's the common denominator to what I did at Kensho that worked for very well to what I'm doing here that's working out very well, which is, you know, and it. And it's kind of controversial, or at least was controversial for a minute. You know, you couldn't talk about iq. You couldn't say out loud for a while, like, I just want people with really high IQs. I don't care about anything else. That's. I don't care who you are, what your background is, what you look. I, I just want someone with a really, really high iq. But that's the honest truth. I just don't know. I don't know how to sugarcoat it. I don't know how to say it differently. I don't know how to talk around that fact. And so, you know, if you think about the people on, you know, the first four or five people, Zachary Ziegler, Jonas Wolf, Evan Hernandez, Eric Lehman, Micah Smith, that came together, sort of senior people on my team initially. Yeah, every one of them, if, if I have to sort of classify this way, you know, came from a PhD program at Harvard or MIT, but that's not because I'm like, I'm only going to recruit from Harvard and mit. It was because I had the Kensho experience, and I learned from that experience that if you bring very high IQ people with very high velocity of learning to bear on a very difficult program problem, they make more progress far more quickly than a team 100 times that size. That's a more normal team. And I think the really reassuring thing for everybody listening in this moment is the rules of the game haven't changed. The physics haven't changed. All the things that used to matter still matter. An elite team, high IQ people, high velocity people, hungry people, very motivated people, people with very high neuroplasticity. And by the way, when I say high iq, what I mean is high neuroplasticity, I mean something very neurologically specific. I don't mean speed at solving a Rubik's cube, which actually doesn't necessarily correlate very highly type ui.

It's the Francois Chalet definition of the ability to efficiently acquire new skills.

Absolutely. It's the ability at which you can learn completely new information and assimilate that new information. That's what I mean by very high iq. And, and, and guess what? That mattered a thousand years ago, mattered 3,000 years ago. The domain was different. It showed up in warfare and tactics and Sun Tzu and you know, those sorts of things. But it, whatever humans were doing at any moment in history, what mattered was neuroplasticity. You know, I spent a lot of time reading in my personal life von Clausewitz and Machiavelli and Sun Tzu and the history of warfare. It's a subject I'm very interested in. And the history, of course, Napoleon and Alexander the Great and these folks. And it's all just neuroplasticity. If you had to sort of say in a few words, what differentiated these people? I mean, none of these people were the physically largest people in their armies. Not even close. What they all had in common is you. The, the, the facts on the ground could change very rapidly, as tends to happen in war. And their thought, not just their decisions, but their entire frameworks for thinking would just like adapt. That's the quality that like a Napoleon or an Alexander the Great had, which is, you know, yeah, they over prepared for the battles that they went into and thought through every single thing that the adversary could do. But then none of that preparation would exactly match to what happened in the battle. And what differentiated them from even very good generals or very good military leaders is they would just completely adapt their way of thinking about the battle to the facts on the ground that we're developing in real time in the battle. And that's, you know, the, the standard way of describing that as neuroplasticity, or at least in, in some branches of cognitive science, it's just very high neuroplasticity individuals. So what humans have been doing over the last, let's say 3,000 years has kind of changed a lot. Right? We're, I mean, unfortunately there are elements of what humans were doing 3,000 years ago that persist to this day. And war does persist. But not Everybody today in 2025 is engaged in warfare in a way that might have been the case during the time of the Greek city states. You know, people thankfully today that are engaged in things other than city state warfare. But what hasn't changed are sort of the neuroprocesses and cognitive qualities that are required for outlier success.

Awesome. Let's jump into a lightning round. Sure. Okay, question number one. I know this number is impossible to measure because it requires a counterfactual, but we suspect that the way open evidence is being used, it's saving lives. Like it's helping doctors make better clinical decisions. On what date will we be able to say that open evidence has saved a million lives?

A million lives? Well, in a way this feels like a McKinsey interview because you got to sort of reason through like, well, if you have 150,000 or 500,000 doctors using it and those doctors each see a certain number of patients and what percentage of those patients are in life threatening or situations. I'm doing the sort of.

Can I tell you my math?

Yeah, yeah, you tell me your math.

My math is, you know, it depends on where you look, but kind of 300,000 to 800,000 lives per year lost due to just straight up medical mistakes. Not all of those are going to be attributed to doctors making decisions at the point of care. There could be other things that happen. But let's take the low end of that. 300,000. Let's cut it in half. That's 150,000. That says it's about six and a half years, you know, until you get there. And that's a fully ramped open evidence. So we'll give you a couple years to keep growing. I don't know, maybe eight or nine years from now. So we'll call it like 20. We'll call it November 4th, 2034.

I'm going to use this as an interview question. The only thing I would add is, you know, Maybe in the2030s you have a million lives saved through the use of open evidence. But what that doesn't count is the patient today whose Ms. Didn't get worse because the dermatologist used the wrong biologic. And that's happening today. Right. And for every black and white life saved, you know, there are those examples. There was a doctor who uses open evidence in Rhode island who wrote us that he saved his patients life by using open evidence to reason through whether the patient's presentation of symptoms was consistent with a pulmonary embolism. And literally used open evidence as like a curbside consult to sort of reason through that patient's presentation of symptoms. Realizing that actually it was the patient's presentation symptoms was consistent with a pulmonary embolism, they rushed the patient back to the emergency room and saved that patient's life. So lives have already been saved through the use of open evidence. And we know that because doctors tell us that. But for every one of those things that happens, it's just, you know, the Ms. Not getting worse or it's some comorbidity not getting worse. That's in the order of millions today.

Yeah, yeah, yeah. Broad domain general purpose foundation models, they are commoditizing, yes or no?

I think they're getting better and better. I think the costs are coming down. I think everything Ray Kurzweil says is typically always right. So, you know, the frontier of the frontier doesn't get commoditized. It's always the frontier. But you know, they, yeah, the costs are going to. The cost of the wow factor that the first chat GPT produced. Yes. Are going to converge to zero, which is why I think all the interesting stuff is going to increase. From a business perspective, there's still phenomenally interesting work being done at the foundation model layer. Intellectually, academically, scientifically, from a business perspective. I think so much of the interesting work, the great companies, to be blunt, are going to be at the application layer.

Yeah, well put, well put AGI. On what date did we, or will we reach AGI?

I think we've already reached. We keep moving the goalposts. We've reached AGI, we've passed Turing, we've passed the Turing test. And even so we just keep moving the goalposts on what AGI is. What people really mean when they talk about AGI is consciousness. And they don't know how to say that. So what they really. Because like they try to sort of say, well, it's this thing or it's this other thing and then the AI does that thing. And like, well, it's the ability to have high school level expertise in multiple different fields. Okay, it reaches that. Fine, fine. It's not that AGI is college level expertise. And then it reaches that. And now it's like, you know, PhD level expertise in everything from coding to medicine. That would be AGI. That's never going to. And then it reaches that. What they really mean is consciousness. That's what people, I think, underneath it all mean. When does. When do you get what you get in the movies? When an AI becomes aware and conscious. I'm not sure that ever happens because I don't know that consciousness is an emergent property of sufficient density of a neural network. That's a philosophical question.

Yep. Yep. For AI founders, AI builders, AI fans. Other than this podcast, what one piece of content should they consume?

Ted Chang's novella Understand.

Okay, tell us why.

I want you to have the joy of experiencing it. I won't. I won't spoil the story. It's a. It's a. Ted Chang's one of the great science fiction authors of all time. He wrote Arrival, which became a major motion picture. And the. The novella Understand was written in the early 90s. And it is the best encapsulation of what. Without any spoilers, it is the best encapsulation of what nonlinear acceleration and intelligence looks like. It's my touchstone for everything that I do. Most people expect a non fiction answer. There's great nonfiction. There's tons of nonfiction reading, including just like go read the chinchilla paper. Great paper if you want. Really just a touchstone for what it is that's happening in our civilization right now. It's this Ted Chang story. Understand. Because that just captures narratively, really the feeling of nonlinear acceleration. Yes.

Awesome.

Love it.

All right, last question. What is the most optimistic or positive thing that you can imagine AI bringing to the world in the next couple of decades? How are. How are all of our lives be better thanks to AI?

I have to go with a kind of an extrapolation of the field that I'm kind of obsessed with, but which is not truly possible today, which is personalized medicine, by which I mean. So personalized medicine has been just over the horizon. It's kind of like quantum computing. It's kind of like fusion. You know, it will happen on some civilization scale. It's been just over the horizon for a very long time. What it means changes in a way. We've been talking about it this whole time. Because using open evidence to say that, hey, if you have psoriasis and Ms. Means you should use this biologic versus this other biologic. That is the beginning of personalized medicine. That's personalized to your comorbidity versus any other person with psoriasis. But that's just scratching the surface of what personalized medicine can be. You know, I think in 10 years from now, whether it's open evidence or whether it's a constellation of These types of AIs, the exact specific fact pattern of your specific medical case is going to be matched to everything that is known in the entirety of medical knowledge about everything that is relevant to your case. And a plan of care is going to be formulated that is hyper tailored to everything that is specific to you and that is specific about your case. And I mean, to me that's just enormously exhilarating. And I think that is just over the horizon. But that is feasible and that will change. That's how you really start to push the ceiling on life expectancy. That's how you start to get into maybe 120, 130 is no longer the ceiling anymore. And you get into these sort of ancient Greek metaphors and paradoxes of Theseus's ship and replacing every plank on the ship to the point where there's no plank in the ship anymore. That was the original plank. But you're walking around and you still have your memory of your wife and your child and your relationship to them and you're still alive to, to watch your own child turn 100 and those things. And I'm an optimist in that regard. I have an atomistic view of human biology and I think the Theseus ship approach to human biology is just over the horizon. And a lot of it turns on the sort of personalized medicine stuff.

Awesome. Daniel, thanks for joining us.

Thanks, Pat. Sa.