EP 287: Harnessing AI Technologies for Next-Generation Clinical Trials

This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life.

Clinical trials can be slow and antiquated and expensive, and sometimes it can be a lot of guesswork. But guess what? Large language models and generative AI have completely changed the way that companies can discover new drugs and bring new treatments to the market for all of us. So I think a lot of times when we think about generative AI, we think about productivity and we think about, oh, you know, now we can have all of this unstructured data and we can use it in ways to, you know, boost our company or, you know, grow our career. But it's about more than that, right? It's also about improving the quality of our lives and offering new opportunities to people. So that's what we're going to be talking about today on Everyday AI. So what's going on, y'? All? Thanks for joining. My name is Jordan Wilson and I am the host and this is for you. Everyday AI is your daily guide to grow your company and grow your career to keep up with everything that's happening in the world of generative AI. So if you're listening on the podcast, thank you. As always, make sure to check out your show notes. We always have related episodes. Way to reach out to us. Send us a message if you want. And if you're joining us live, thank you as well. So before we get into today's conversation, let's first start by going over the AI news. And as a reminder, we will be recapping today's show and everything that's happening in the world of AI in our free daily newsletter. So make sure you go to you go to your everydayai.com and sign up for that free daily newsletter. All right, so let's talk about what's happening in the world of AI news today. So first, AI employees are banning together to warn the public. So a group of current and former employees at top AI companies just published a letter voicing concerns about the dangers of unregulated advanced AI and are calling for more oversight and transparency from companies. So this is signed by some current and former employees from companies such as OpenAI and Google DeepMind. But this group is concerned about the potential for AI to cause serious harm if not properly regulated, including inequality, manipulation and even human extinction. Wow. All right. So they believe that employees are in a unique position to hold companies accountable, but are often restricted by non disclosure agreements and lack of whistleblower. Protections. Yeah, pretty, pretty big topic there. So interesting that this group of employees is coming out and speaking about it. All right, next in AI News. Well, Elon Musk has confirmed why he diverted so many Nvidia GPU chips from Tesla to. To his other company, xai. So Elon Musk has confirmed that he did redirect thousands of GPU AI chips from Tesla to his companies, X Corp and xai. So the move was made what he said due to a lack of space for the chips and potential conflicts with Nvidia. So he redirected 12,000 Nvidia GPU chips from Tesla to Xai, and he said it was due to a lack of space for chips. But many are talking that it is due to his ongoing battle to have a little bit more voting control at Tesla. So interesting that he's, you know, taking from his left hand and feeding the right. All right, last but not least, the U.S. treasury Secretary is sending warnings about AI as well. A lot of AI warnings today. So Treasury Secretary Janet Yellen warns of both opportunities and risks associated with the use of AI in the financial system. She also noted that current AI models can be opaque, produce biased results, and lack proper risk management frameworks. And regulators, including the Financial Stability Oversight Council, are monitoring the impact of AI on financial stability. So, yeah, we will have a lot more on those stories and everything else in our newsletter. So make sure you go to your everydayai.com and sign up for, for that free daily newsletter. And you know, always hit me with a reply. I, I read them all, so let me know. All right. But today we're not here to talk about the AI news. Well, we at least got that out of the way, but we are here to talk about how AI is changing and creating really the next generation of clinical trials. So I am excited today to bring on my guest. There we go. We have Saurabh Jane, who is the executive chair Chairman of Trial Key AI. Thank you so much for joining the show.

Absolute pleasure. Thank you, Jordan.

All right, so hey, you know, for everyone out there, could you tell us a little bit about yourself and what you all do at Trial Key AI?

Yeah, totally. Look, I'm software engineer by trade, spend a lot of my time on the tech side of things, and then these days a lot more on the commercialization of tech. So right now I'm in about half a dozen company boards and one of the companies I'm super interested in is Tralky. So the problem that we try to solve is what we did is we got 350,000 clinical trials. We built our own custom large language model to get about 700 variables of each trial. So these are things like the condition, the patient, the sponsor, the sites, the medical components within that clinical trial. So it turned all this unstructured data into structured data. And then we use that to try to predict the outcome of clinical trials. And three, three and a half years later, with a whole bunch of investment, we can do that now with about 90 accuracy. So it sounds super crazy, but we can predict whether a trial is going to succeed or not before it starts with 90% accuracy.

Yeah. And maybe before we dive into that a little, a bit, talk about, you know, things like unstructured data and improving, you know, clinical trial accuracy, maybe can we talk a little bit about what this has looked like traditionally? Right, so kind of before generative AI and before large language models, I mean, can you tell us a little bit about. Well, first of all, why do we need clinical trials? What do they do and how have they historically worked in years or decades past?

Yeah, totally. So just some of the real foundational things around clinical trials. So you go back about 100 years ago, a couple hundred years ago, anyone could claim anything about any medication, pop it on a bottle and some stuff works, some stuff didn't, some people died and the government thought, hey, this, we should actually totally regulate this. And this is when medicine started to really come together, when it became like an evidence based pursuit. So you'll have, you know, animal trials and you'll have first in human trials and you'll have phase one. Phase one is much more around safety. A is what we're doing giving this drug or treatment to somebody causing any harm. Phase two is more on efficacy. Hey, does it actually work? And then phase three is efficacy, but at a much, much larger scale. Then sometimes you have phase four or you go direct to market. So the idea is this rigorous process, which takes about a decade for most trials, for most drugs from start to finish, is designed to weed out stuff that doesn't work, that causes harm to people. So then, you know, the stuff that does get into people, more often than not, it works quite well and it doesn't cause any unknown side effects.

That's, that's wild to me to think that, you know, sometimes it can take a decade or more. Right. So you know, what we're saying is in theory, you know, a pharmaceutical company or a group of scientists, you know, might actually have something that could help today, but it could take 10, 10 years, right?

Oh, totally. And, and look, I'm generally Quite an optimistic person. So one of the great silver linings around Covid was it was kind of awesome. Like all of humanity banded together to solve a single problem. And it had like a thousand labs around the world trying different covert trials and we kind of got something out to market in a tenth of the time. That normally would happen. Sure, it probably took like insane amount of resources and insane, insane motivation to do, but that's the only time that there's actually been like a quick drug to market. Otherwise it's just a decade or problem. It takes a decade from inception to something being on the shelf that's actually treating people.

So, you know, speaking of the kind of time around Covid that kind of coincided, you know, a little bit after, you know, kind of this surgeons of large language models, you know, so GPT technology became, you know, available in 2020, started rolling out in popular customer facing applications such as ChatGPT in 2022. So you know, how, you know, I'm curious because you do have a background, you know, in this, in this area in general. I mean, I think some people saw ChatGPT and they're like, oh, this is kind of a toy for, you know, writing blog posts or something like that. How would you say that in your field? Generative AI and large language models were initially perceived?

Totally. So first I say like ChatGPT, I did not think in my life that would be a solved problem. I didn't think that someone would be able to solve the Turing Test. I thought that would be like something my kids would do, my kids, kids would do. So when I did my thesis about 20, 25 years ago, it was all about predicting cancer in patients. Back in the day, Java 2.0, you had to hand roll everything, 80,000 lines of code, and you could build a model that was about 50% accurate because we only had structured data. So it's only structured data in fields that a clinician actually wrote into, like an Excel spreadsheet. Like this field means this, this field means that. And the really cool thing that ChatGPT did, it let us go from unstructured data to structured data. So for example, you upload a clinical trial to ChatGPT and you ask it, what, what compound was used? And it goes, does its thing and it comes back and says greatness of the compound that was used. You'd ask how many patients were in this trial, where were this, where were the sites, where the locations, who funded it. So all of a sudden you kind of have access to all of humanity's knowledge that you can now put into a format that you can actually put into a, into a decision engine.

And you know, what were some of the early concerns around using large language models as well? You know, I, you know, obviously there's things that we all deal with, right, like hallucinations and, you know, is, is this model actually, you know, using the data that we give it, you know, context, windows, et cetera. You know, what were some of main, you know, just the industry kind of went through in general when it came to implementing tools like large language models in clinical trials.

So a couple of things. One is all around the hallucination or having blanks, having incomplete data sets that you don't, that's kind of hard to solve on the LLM side. You can kind of hypertune them and make them more focused, but you're always going to have a bit of that. So what you want to do is you want to solve that on the actual precision engine side, the predictor side. The other kind of concerns I guess we had or the issue had was how does this work for novel drugs? Like, how does this work in a data set which, you know, the world has never seen before? And then we did a whole bunch of work and I'll kind of tell you some of the interesting stuff that we found out around that. But they're all the things like how do you deal with scenarios that have not happened before and then how do you deal with kind of incorrect data.

Coming through and how do you think so far, you know, because those are, you know, obviously things that you have to take into consideration. How has the industry kind of responded to some of those concerns?

Yeah, totally. I mean, keep in mind that probably no one that we know is kind of doing AI clinical trial prediction design to the extent that we are. So it's still very new to the industry. Like we only launched to market like two or three months ago, so it's super early days. We only got our first set of customers using this. But generally what you, what you'll find, it's a completely different approach. So the way it traditionally works is like a lot of these things are based on like a clinician with years worth of experience about, you know, certain drug and certain disease. Now the downside of that is no single clinician can have like a complete knowledge set. Like no one can actually know everything that's happened in every clinical trial, over 350,000 trials over 20 years. But an AI tech computer totally can. So AI will find connections that a person will never see. So when we show people our tech. The first thing is, surely that's not right, surely that's not believable, that does not make sense, that's even a solvable thing. But when you bring up their trial and talk to their experience and some of the stuff the data shows, that's kind of when you get them on side very, very quickly.

Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on gen AI. Hey, this is Jordan Wilson, host of this video very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use gen AI. So whether you're looking for chat, GPT training for thousands or just need help building your front end AI strategy, you, you can partner with us too. Just like some of the biggest companies in the world do. Go to your everydayai.com partner to get in contact with our team or you can just click on the partner section of our website. We'll help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on gen.

Yeah.

And you know, speaking, speaking of, you know, you mentioned some of those things, you know, taking in this data from, you know, all of these, you know, 35, 35,000 clinical trial. Right, so all of these, or, sorry, 350,000 clinical trials, all these different variables and you know, improving the accuracy of clinical trials. What does this ultimately mean for, you know, on the front end? Right, like how can this influence, you know, whether it's pharmaceutical companies, you know, research organizations, how can this data and kind of what your company is working on, how can this influence them? And maybe in good ways or potentially bad ways on what they maybe should or shouldn't focus on?

Certainly. Well, I mean, probably even the thesis that we have is that everything about how humans interact with drugs and devices is already known, that knowledge already exists in the world, except it just happens to be in a crappy kind of way across 350,000 clinical trials. We've just bought that together in a single AI model. So now if I'm a pharmaceutical company, I can get an insight into things that I actually just didn't know before. So our big claim to fame now is we can Predict whether a trial succeeds with about 90% accuracy before you start. Super crazy, super not credible, but that's actually what we can do now. So if you think about that from a pharmaceutical perspective, like we think there's three main use cases. One use case is that we'll probably just make free because the world is for patients. If you have a loved one who needs, who wants to go on a clinical trial, we'll help you figure out which trial to get on which is most likely going to succeed. So then they have the best chance. If you're a pharmaceutical company and you've got 100 trials across different drugs, we'll help you figure out which trials are most likely going to succeed. So you can over index and over invest in them and you might call out the ones that are least likely to succeed. Or if you're doing a new trial, we'll help you through generative AI design the best possible trial. Then the last use case, which I didn't see coming, but it's so a thing is for the investment community. So we did this for a company last week. They're going out to funding osteoarthritis company. We did a model for them is to predict their likelihood of success of completing that trial. So when they go off to investors, they can say, great, we've got an X percentage chance of succeeding. This much more than all our competitors said, please invest in us. And then on the flip side of that, something the CTO and I do for fun right now is we invest in pharmaceuticals with our own cash. You buy before a company announces a clinical trial, inflection share price goes up and you sell afterwards. Yeah.

And you know, something, something I'm curious about and we talked about this, you know, briefly before the show. You know, this, this can obviously help companies save a lot of money. Number one, yes, it does bring drugs and treatments faster to the market, which, which helps humanity. Right. It helps us all, you know, hopefully live higher quality of lives. But what about, you know, companies that may be too focused on the commercialization piece? You know, are there, are there going to be maybe now, you know, drugs or types of diseases that maybe just go undiscovered or unresearched? Mainly because, you know, hey, we are able to now make more money or we're able to just focus on, you know, these, these bigger problems that have a larger pool of potential customers and clients. So I guess how can companies balance that? And is that something where, you know, smaller or rarer diseases maybe aren't going to get explored at the same rate.

Totally. So what tech like ours does that makes it like a, like a near perfect capital allocation in the market where money goes, it's like meritocracy, right? If you're most likely going to succeed, you'll invariably get more investment and more funding. And most pharmaceutical companies are not philanthropic enterprises. Right? They're there for a profit motive. So the kind of things you want to figure out is how likely am I to succeed? How does that compare against my competitor set? And so we've applied machine learning to figure out who's your competitor set. Like who's trying to solve the same problem, albeit with different ways. Some people might have surgery, some might have devices, some might have drugs, some might have any other interventions. And who's most likely going to succeed and when do you think they're going to succeed? Because the theory is you want to get out to market with the most. If you have an invasive intervention model or ingestion method, I. E. You have to take a needle versus the tablet versus a cream, you're going to find it's going to be harder to get to market. If someone else solves that problem with a much easier ingestion method or another model kind of thing, they'll probably succeed. So allow you to see, okay, I'm actually going to succeed after these guys, they're more likely to succeed than I am so likely to kind of choose. Well, is this the best use of my capital, what that means for rare diseases? So I'll use Covid as an example. So this company started about a year before COVID and there were about 400 vaccine candidates at the time. And we predicted Pfizer and Moderna is number one and two and Ashley's Enico is number seven. And history has shown we published that out, we've sent those results out, got published and that's kind of what happened. So what that meant is not because of us, because we weren't really known back then, but those two, three succeeded. Now they've been over indexed in terms of funding. The Moderna is investing a hell of a lot in MRNA and they've been able to get a lot more kind of cash because they've done that the other part around kind of rare conditions. I think what's going to happen unfortunately is the money will go to the problem that's easiest to solve, that has the biggest payback versus Its competitors. So you'll have things like diabetes, weight loss, cardiovascular cancer will be over indexed in terms of probably investment. And some of the rare genetic Disorders will probably just get less attention.

I guess, what new problems then might arise? Right, like might there, you know, kind of before generative AI and before large language models could predict the likelihood of something working, it seemed like. I don't know from an outsider's perspective, I don't know much about the space, but you would think that companies would be investing equally in all of these different areas and all of these different problems and diseases to kind of see which one worked. So if large language models can actually take out some of that guesswork, I see how it's a good thing. But then how can you still, you know, whether it's the industry, your company or, you know, society, how can we still make sure that people are researching and investing in these areas that are these rarer and maybe less profitable or less. With a smaller kind of pool of people that are suffering from these diseases?

Yeah. So what you probably find is that clinical trials with tech like us will iterate a lot quicker. So as you said, it takes right now a decade, right, to go from start to finish. What you'll find is it might still take that long, might have a few years off, but you'll be doing less trials. So right now humanity does about 35,000 trials a year. I'll probably drop to 5 or 10,000 trials. But it leaves lots of bandwidth to go and solve other things. And as long as there's a good capital model or incentive or some way that incents people to go solve some of the other problems, that will probably still happen. Because luckily not everyone does this for pure financial motive. People have personal experience in those kind of things. But you'll find some of those rare conditions. They'll probably get funding now when they couldn't otherwise get funding, because they'll be able to run a clinical trial and AI simulator be like, awesome. I think I'm going to solve this problem. 72% chance of success, baseline is 10%.

You know, one thing, you know, you know, I'm curious about. So, you know, you said that, you know, your company has essentially, you know, built a model that helps you, you know, pull out and extract all this data from 350,000 clinical trials, you know, 700 different variables. What are some of the things, kind of regardless of, you know, how this can be used in the future, what are some of the things that your company has found, you know, when looking at all of that data, you know, because I'm sure that there's kind of like what you said earlier, I'm sure there's connections that Large language models can make that humans can't. So what are some of the things that you've found in, in your process of creating this model for others to use?

Let me, let me ask you a question, and it's a trick question. What do you think is more important in a phase three trial? The actual drug or the way the trolley succeeded? The way the trial is designed. What do you think is more important on whether, whether a drug works and gets through approval or not?

I'm gonna, well, trick question. Love trick question. Yeah, so I, I'm gonna guess the second, I'm gonna say how it's used because regardless of the drug, if it's not used correctly, it's maybe not relevant. That's, that's my guess.

What's the, that is, that is totally it, right? Look, it makes no sense. Like I'd actually say, oh my God, the drug is so much more important because if the drug doesn't work, nothing else matters. But you're fine. There's about 80% of the probability of success is about how the trial was succeeded. So how the trial was designed, the drug only accounts to about 20% of the likelihood of success. Because keep in mind in phase two, you, you kind of prove some level of efficacy. That's how you kind of get to phase three. It's, it's not the main dish and that's totally crazy, totally not credible, but that's actually what the data shows. The other thing that I reckon the data has shown us as well is there's lots of awesome drugs, drugs that actually were really good. We're going to be super efficient, have a lot of efficacy, but they've never, never got taken to market because they had a crappy trial design. Is the principal investigator, the doctor, the chief medical officer, didn't design a great trial or what, didn't have enough funding or couldn't do what they wanted to do and they never got some amazing drug to market.

You know. So one thing, one thing I'm curious about is, you know, I, I kind of think right now people who are doing knowledge work, right, so they're, they're reading, you know, PDFs and they're creating outlines and pitches and presentations. To me it seems kind of wild if people aren't using generative AI in their day to day work when it comes to clinical trials and the future, might that be how we view clinical trials in a couple of years, that if companies aren't using large language models, if they're not using, you know, digital twin simulations, like, how do you kind of see the future of this playing out?

Yeah, totally. Look, as I said, there's about 35,000 trials that happen in the world. I reckon that's going to be dropped to like, 5,000. And there'll be millions and millions of trials that happen in a. In an AI simulator. And at some point in time, you can have ethics committees that will be like, it's completely unethical to test a drug in a human person unless it's been done in an AI simulator. So the analogy I gave you beforehand was it's a bit like learning how to drive, right? I Learned drive, like, 20, 25 years ago. It was super dangerous. I'm so lucky. That never killed anybody. But everybody who's 17 and learns to drive are horrible, right? And my kids will probably never learn to drive in a real car. They'll go to the dmv, they'll learn how to drive in a simulator. They'll have to pass the test in a simulator. And we'll talk about how back in the olden days it was super unsafe and let people learn on the roads. And the exact same thing will happen for clinical trials. At some point. It'll be like, that's crazy, right? In the Dark Ages, we actually tested drugs in people before in the AI world.

Yeah, it's kind of crazy to think about it, but I can see both sides of that, right? So I can see how, you know, oh, it's. It seems wild to not test certain drugs on humans if humans are the ones that are ultimately taking them. But kind of the AI side of me says, okay, it might be a complete waste of time and you might be withholding, you know, great new discoveries from the population that can help if we are using humans. You know, I'm curious. You know, you said that you, you know, 20 some years ago wrote your thesis on, you know, AI and being able to detect cancer. So, you know, where do you even fall personally on that? Do you ever struggle with, you know, balancing this concept of using generative AI and using large language models with the human in you? That's been doing this for decades.

Oh, totally. Look, I mean, I think. So you always still do a troll in a person because, like, you can't simulate everything in the human body in AI and humans are so different, it's so unique. But I think what will happen is before it's done in a person, will be done in AI, and that will cull out a lot of trials that don't, should never happen or not be safe. Or probably not succeed and then only the best ones will be done in people. So it just kind of means that you put less people through unnecessary trials, right? You put less people in harm's way of drugs that actually should, shouldn't take or drugs that are probably not going to succeed. The other thing that it does do for people as well is one of the models that we've built. It's all around trying to figure out, hey, I have this drug candidate, what are the conditions this could solve. So I might have a drug that works great for, I don't know, hair loss for example. But hey, it helps with acne as well. So before you do a trial in people for acne, you can actually run that through an AI simulator and we'll recommend all the other conditions we think this therapeutic could actually work for. So it might actually really expand out the scope of different treatments in different areas as well.

That's, that's, that, that's, you know, super, super interesting. And, and maybe could you, for those of us unaware, because, you know, we talked about, oh, it could take maybe a decade in some cases for, you know, a good discovery or a good drug to finally go to market. So where are we at today, you know, for companies who are, you know, using, whether they're, you know, using your service or one of your competitors, what does that look like today in terms of, you know, months, years and also cost and then what can you see that, you know, being in the future? So kind of walk us, you know, current day companies that are using it and then in the future what that looks like in time and money.

Yeah, totally. Look, unfortunately it still takes a decade. Like FDA legislation, regulation is not kind of caught up to AI. For what it means is when you start the journey, you can be more confident that you've got a model or a clinical trial that's going to succeed. So theoretically there'll be less failures along the way. Like we'll get that meritocracy that we spoke about. But I think what will happen over time is as we get more and more generative AI, we design more trials from scratch for doctors or for clinicians, we'll hopefully speed up the process a bit, call off a few months, few months here, a few months there. But I reckon it's going to be 10 years, maybe 15 before you get like the FDA to consider clinical trials in AI as a, as a proxy for what actually happens in people. The one exception I'll actually give is one of the things we're working on now with the client. It's what they call real world evidence. So instead of running a trial, go collect every, all the evidence, what's happened in this area across the last 10 years, collate that into a virtual trial and submit that through for approval. So the tga, that's the authority in Australia, similar to the fda, they've started to accept some real world evidence trials that they don't have the same kind of bar as a clinical trial. But it's a great way to start for, you know, for some alternatives, vitamins and some of those kind of things.

So that's, yeah, it's interesting, right, and also a little disheartening maybe, right, that, that you know, all of these, you know, technologies are, are helping this field advance. But you know, still the, the slow turning wheel of, you know, a big organization like an FDA can still slow down things, things for everyone else. So, you know, we've, we, we've talked about a lot of things here, but you know, as we kind of wrap up today's show, we've explored a lot of different ways that, you know, AI technologies can really help in the future of clinical trials. And we've talked a little bit about, you know, some of the old school and antiquated processes. But you know, maybe what's the one takeaway that you hope that people can keep in mind as they look at the future of how AI is used in clinical trials?

Yes, if I take a step back, not just in clinical trials, but like you know, any kind of similar problem set. So what we've actually done, and this is what the co founder's genius was, he's, he's found the world clinical trials in this example, where there's 1,000 variables or 700 variables, in our case, you have a known outcome and you want to figure out what variable leads that known outcome. So we just happen to apply that kind of to clinical trial. And basically anywhere where you've got a whole bunch of variables that lead to an outcome, you can now for the first time ever using AI, actually figure out what variables lead to that outcome. And I think that generic solution set will now solve a lot more problems around the world, I think.

I love, love to see it. You know, another great example today on how generative AI is really shaping our future in a good way. So this, this was a great conversation. So thank you so much, Sharab, for joining the Everyday AI show. We really appreciate your time. All right, and hey, as a reminder to everyone, we covered a lot today, so make sure to go to your everydayai.com Sign up for the free daily newsletter. We'll be recapping everything that we talked about and more. You can find out a little bit more about the company there that we talked about today, Trial Key AI as well as keeping up with everything that's going on in the world of Generative AI. So thank you for joining us today. We hope to see you back tomorrow and every day for more Everyday AI.

Thanks y'. All. And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit youreverydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.