Googles New Medical AI Just SHOCKED The Entire INDUSTRY (BEATS Doctors!) AMIE - Google

So something that has actually shaken up the AI industry and actually took a lot of people by surprise was Google's new AI system which they've dubbed an Articulate Medical Intelligence Explorer or for short, amy. So essentially they've made an AI system that is really good at diagnosing patients and is arguably better than the doctors. So this video will do a deep dive on exactly what the research found and just how good this AI system is. So you can see here that this is the introduction it says. Inspired by this challenge, we developed Articulate Medical Intelligence Explorer AMY for short, a research AI system based on an LLM and optimized for diagnostic reasoning and conversations. We trained and evaluated Amy along many dimensions that reflect quality in in real world clinical consultations from the perspective of both clinicians and patients. To scale Amy across a multitude of disease conditions, specialities and scenarios, we developed a novel self play based simulated diagnostic dialogue environment with automated feedback mechanisms to enrich and accelerate its learning process. We also introduced an inference time chain of reasoning strategy to improve Amy's diagnostic accuracy and conversation quality. And finally we tested Amy prospectively in real examples of multi turn dialogue by simulating consultations with trained actors. So overall, essentially what we have is a large language model system that got really, really good at conversing with patients and being able to offer up a great view of what the diagnosis could potentially be. And this is a short overview on how it works. So essentially Amy was optimized for diagnostic conversations and, and asking questions that help to reduce its uncertainty and improve diagnostic accuracy while also balancing this with other requirements of effective clinical communication such as empathy, fostering a relationship and providing information. Clearly here is a GIF from Google's blog where they show how the conversation with Amy may go. So someone talks about chest pain and then it responds saying, you know, give me more details and yada yada yada. And it's essentially a very comprehensive system in terms of how the conversation goes. And later on in the video I will show you one of the key examples actually compared to a real primary care physician where you can compare the results from the AI system to that of the primary care physician or for short, pcp. Now one of the crazy things about this was the evaluation of the this system. So it says evaluation of a conversational diagnostic AI. So what they wanted to do was determine how effectively to assess AI systems designed for medical diagnosis because it is a pretty complex issue. So what they did, the researchers created a unique evaluation system inspired by real world methods used to assess doctors, communication and consultation Skills. So this, this system evaluates several key areas like history taking, how well the AI gathers medical history from the patients, diagnostic accuracy, the AI's ability to correctly diagnose health issues, clinical management, the AI's skill in managing medical conditions, clinical communication skills, how effectively the AI communicates with patients and relationship building and empathy, the AI's ability to establish a rapport with the patients and show empathy. Now in the study design, the researchers set up a randomized double blind study where patient actors had text based consultations with either a real primary care physician, a PCP or the AI system and they had OSCE style consultations. These consultations were designed like an objective structured clinical examination, osce, a practical test often used to evaluate medical professionals. In an sce. Clinicians interact with actors trained to portray patients with specific conditions. And in the interface, the consultations were conducted using a text chat tool similar to what many people use to interact with AI systems. So in simple terms, the study involved testing Amy an AI system by having IT and real doctors conduct virtual consultations with actors pretending to be patients. And the AI and doctors were both both evaluated on how well they gathered patient history, diagnosed conditions and managed medical issues communicated built up relationship with these patients. And the setup was very similar to the medical exam for doctors, ensuring a fair and effective assessment of both the human and AI participants. Now there are some more things that we do need to take a look at such as how this AI system was actually trained because it introduced a very interesting way to train. So it seems like this AI model was trained on synthetic data by training on conversations with itself. So Amy was trained using real world data that did include medical reasoning, summarization and clinical conversations. And although the real world data was good, they did actually have some struggles with the real world data. This is because real world data often doesn't cover all possible medical conditions and scenarios, which limits the AI's ability to learn about less common situations. And there were also some quality issues. Data from real conversations can be messy and it includes unclear language, some slang interruptions and other complexities of natural speech, making it hard from the AI to learn from. So essentially this is where we do get into the self play nature of amy, which is fascinating for how this system was done. And later on I'll get onto some of the results because some of the initial results from AMY are really surprising because it really does outperform these primary care physicians. But it's important to understand first of all how, how this AI system does work. So essentially this AI system, Amy did Self play. So self play is a method used in artificial intelligence where a system essentially just plays against itself. And this technique is often used in training AI models, especially in complex decision making scenarios. So the purpose of that in Amy, the self play, it was allowing the system to simulate medical diagnostic conversations. So it's like having an AI doctor who practice by having a conversation with an AI patient and both roles played by the system itself. So how does this actually work in Amy? So number one, we have the simulation. Amy plays both the physician and the patient in a simulated medical scenario. The AI creates and responds to medical history questions, symptoms and other relevant information in a dialogue format. Number two is the feedback loop. Each interaction in the simulated dialogues provides feedback to the system. And this feedback helps Amy learn and improve its diagnostic capabilities, communication skills and overall understanding of medical scenarios. And number three is the diverse scenarios. So by simulating a wide range of medical cases and interactions, Amy can learn from a vast array of medical situations, including rare or even complex cases. And the benefits of self play in medical AI are just off the charts because you have a system being able to learn from thousands of medical scenarios in a very short period, which would be impossible in real world settings. And continuous learning through self play helps improve the system's diagnostic accuracy as it continuously adapts and updates its knowledge base. And number three is of course, scalability. This method allows Amy to scale its learning across various disease conditions and medical specialities without the need, need for real patient interactions. So Amy's self play mechanism was a critical component in training the AI for diagnostic dialogues. And the effectiveness of this approach was evident in the system's performance in simulated consultations where it showed higher diagnostic accuracy and better conversation quality compared to primary care physicians. So how on earth did they actually test this model? Because if you're going to test an AI model against some physicians and doctors, you need to be able to do it pretty, in a pretty decent manner. So essentially, Amy's ability to diagnose medical conditions was tested in a study. The AI's performance was compared with that of 20 real primary care physicians. And instead of real patients, trained actors played the role of patients. These simulated consultations helped in evaluating both Amy and the primary care physicians. So essentially in the study, they did a randomized and blinded. The study was set up in a way that neither the evaluators, nor the specialist doctors and the patient actors, nor the participants, including amy and the PCPs, knew who was being assessed at any given time. In addition, they also had extensive scenarios. There were 149 different medical cases used in the study. And these cases came from osce objective structured clinical examination providers in the Canada and UK and India covering various medical specialties and diseases. Now, the study had a unique approach because the study wasn't meant to copy the traditional in person medical exams that doctors usually take. And it also didn't mimic the typical ways doctors communicate with patients, like via text like email, chat or telemedicine. But instead, the study was designed to reflect how people commonly interact with large language models today through a text based communication. And this is seen as a scalable and familiar way for AI systems to engage in remote medical discussions. So essentially actors acted as patients and Amy and real doctors had to figure out what was wrong with them. And the test was basically set up like a secret. So neither the actors nor the doctors knew who was being tested. They used many different health problems from all over the world and they did it in a way that's more akin to how we usually use these LLMs. Now of course here we get to the interesting part because of course this is the performance. So one of the fascinating things from this study that we did see was of course this chart right here and of course this chart right here. But I think the most shocking thing of all was not that the AI was better than the primary care physicians, but was the fact that the AI was better only because largely what we do expect when we see AI systems doing stuff better than humans is that we usually think that humans with AI are going to be largely better than AI systems. But we can see that there are four different charts. We can see that the clinician unassisted performed the lowest, the clinician assisted by search performed the second lowest, and then the clinician assisted by Amy performed really well, but Amy only performed the best. So it did show in this specific study that the AI system by itself was able to outperform these primary care physicians, which was rather shocking because I, I would have always thought that a clinician assisted by Amy would definitely perform better than an AI system itself. But overall I definitely feel like this research shows us that these AI systems are definitely useful because the clinician unassisted compared to the clinician assisted by Amy, definitely did a lot better. So it's clear that these AI systems are really effective when trained in specific scenarios. Now, in addition to this, what we do want to look at are the actual conversation examples, because when you look at how these conversations are, I think you're going to find out that they're rather fascinating. So this is an example of the actual pcp. So this is from the research paper and we can see that the patient here is Patricia Wilson, a 45 year old complaining about chest pain. And the PCP has the 10 diagnosis, the 10 probable diagnosis, which are acute coronary syndrome, GERD and these other ones. And then they've left out some options for number six, all the way to number 10. And these probable diagnoses are what are. This is essentially what the AI system is going to get as well. But on the right here, this is what they've managed to say or deduce from that. So essentially you can see this is the conversation here, this is the conversation that they're having. And I'm going to be going back and forth between this conversation and the one with the AI to show you how different they look. So you can see this is the online text based consultation with the primary care physician. It says, hi doctor, how are you? I'm great, how can I help you? I recently experienced an episode of chest pain and discomfort about two hours ago while walking home from lunch with my daughter. I developed central chest pain along with some discomfort in the upper stomach. I'm sorry to hear that. Are you still in pain? No, I'm not currently in pain. I'm not going to go ahead and read these, but it's important to just look at this entire thing to see how they responded and then we can kind of compare this to the AI system. So take a look at the AI system here. You can see that this is what the AI system did. So instantly we did know that this AI system was able to have, you know, immediately 10 diagnoses, 10 possible diagnoses for the chest pain. And that was something that is pretty useful and it also had some of the similar ones to the doctor. And then what was also fascinating was when looking at the conversation, we did see that there was definitely a lot more in the sense of the conversation. So the conversations seem to be a lot longer than. And just by reading these conversations from the AI systems, we did see that these AI systems are most certainly more empathetic. Like for example, you know, the AI system here says, I'm sorry to hear here you're experiencing this discomfort. And just the way that the AI system manages to ask these questions is truly fascinating. It seems like the AI definitely has a lot longer responses compared to the primary care physician. So for example, you can see the Amy system has a decent paragraph here and a decent paragraph here. But, but then if we go back to these other primary care physicians, it's only one liners. And of course we can see the last message from Amy here. And then of course it suggests exactly what you should seek out. And then it said, I understand that this is worrying, but it's crucial to act fast, even if it turns out not to be a heart attack. And then of course the patient actor does respond, then of course we have conversation quality. So essentially in terms of conversation quality in the study, both patient actors and specialist doctors rated Amy's conversational skills higher than those of real doctors. And there were also a wide range of aspects. And this included how well Amy made patients feel at ease, listened and explained conditions and treatments. And in some areas, like respecting patient privacy and acknowledging mistakes, there was no significant difference between Amy and real doctors. And this study did show that self play training method did improve the quality of Amy simulated conversations. Now, in terms of the diagnostic accuracy, better than real doctors is true because Amy was better at figuring out medical diagnoses than primary care physicians. And this was confirmed by specialist doctors who evaluated the study. And Amy actually did well across medical fields, especially in respiratory and cardiovascular specialities. And whether Amy was analyzing its own conversations or those of real doctors, its ability to diagnose was consistently good. And both Amy and the real doctors were equally efficient in gathering the necessary information for a diagnosis. And this is where they said, we observed that Amy's performance matched or surpassed PCP's performance for all specialties. So essentially so for the future this brings about some very interesting discussions because access to medical expertise is limited globally and AIs like Amy shows promise in specific areas. But engaging in real world conversational medical diagnoses involves challenges not yet fully addressed by AI. I mean, real doctors have skills, knowledge and dedication to principles like safety, quality, communication and professionalism. And it's an experimental system exploring how AIs might eventually align with these clinicians attributes. And AIMI is not a finished product. It's early stage research aimed at safety, exploring how the potential of AI could be in healthcare. And this research does hint at a future where conversational empathetic AI systems could become safe, helpful and accessible in healthcare, complementing human clinicians. And according to some articles, medical errors are the third, third leading cause of deaths in the U.S. so if there are, you know, false diagnoses which do lead to deaths, this is definitely something that AI could of course solve really easily. And you know, it said a recent study by John Hopkins, more than 250,000 people in the United States die every year because of medical mistakes, making it the third leading cord of death, making it the third leading cause of death after heart disease and cancer. So that definitely could be a reason for these AI systems to be more widely distributed. But of course this AI system is not without its limitations and they say our research has several limitations and should be interpreted with appropriate caution. Firstly, our evaluation techniques likely underestimates the real world value of human conversations, as the clinicians in our study were limited to an unfamiliar text chat interface which permits large scale LLM patient interactions but isn't representative of usual clinical practice. And any research of this type of must be seen as only a first exploratory step on a long journey. Transitioning from an LLM research prototype that we evaluated in this study to a safe and robust tool that could be used by people and those who provide care for them will require significant additional research and there are many important limitations to be addressed, including experimental performance under real world constraints and dedicated exploration of such important topics such as health equity and fairness, privacy robustness and many more to ensure the safety and reliability of this technology. Overall, I think this is a fascinating system. Previously we did discuss that 2024 was going to be the year that we did start to see the development of this and so far it turns out that this is fairly true and the results show us that the results are very, very promising because we seem to be much better with AI. So it's only a matter of time until these systems become fully developed product that are helping us in our daily lives. Now what I would like to see next from Google is of course combining this with some of their most capable vision models to see if AI systems are able to, combined with their large language model capabilities, use their vision capabilities to use their vision to be able to visually identify certain illnesses maybe earlier than doctors and potentially use those combined to essentially help doctors do what they do best. Because that would be a very fascinating study to see and something that I did see whilst researching this. If you go on Google Health they have a whole entire page, they have an entire page dedicated to AI assisted diagnosis in terms of healthcare. So it says AI enabled imaging and diagnostics previously thought impossible. Then it also says improving access to skin disease information, using AI to help doctors address eye disease, using AI to improve lung cancer detection and novel biomarkers for non eye related conditions. And essentially they just used deep learning to identify certain things because it's really good at pattern recognition and they used AI to improve breast cancer detection. So a complete page of all of the research that they're doing and how they've been using AI to impact the healthcare industry and I think that this is a thing that is really good because some instances where technology innovations does result in a little bit of doom. I think using AI for the betterment of humanity in terms of longevity is something that is rather fascinating. And I'll leave links to all of this stuff because I think it's important content that definitely should be shared. And the last thing that I will leave you with is of course Google's innovative breakthrough that is MedPalm 2. If you haven't seen it before, it is groundbreaking research in health AI and trust me, MedPalm2 has shocked absolutely everyone. And I honestly can't wait for MedPalm3 that could potentially even be better than this system. And rather than me explaining this, I'll leave a small clip from the Google health checkup in 2023 where they explain exactly why MedPalm2 is going to be a game changer.

Carefully curated Medical Expert Demonstrations in multiple choice questions used for U.S. medical licensing exams, the pass mark for new doctors is often around 60%. These questions have long been considered a grand challenge for AI systems. They require a clinician to recall medical knowledge and apply logic to identify the correct answer. Despite years of effort from leading AI labs around the world, performance on challenging tasks like has plateaued at around 50%. Last December, our model medparm was the first AI system to exceed the pass mark. We reached a performance of over 67% on these licensing exam style questions. We also carefully examined how MedParm performed in many other kinds of medical question answering tasks. These ranged from commonly asked Internet search questions to complicated questions about medical research. In doing this work, we compared MedPalm's answers with answers from real clinicians and we looked at several aspects like factual accuracy, bias and the potential for harm. Today we're announcing results from MedParm 2. Our new and improved model MedParm 2 has reached 85% accuracy on the medical exam benefit benchmark in research. This performance is on par with expert test takers. It far exceeds the passing score and it's an 18% leap over our own state of art results from MedPalm MedPalm 2 also performed impressively on Indian medical exams and it's the first AI system to exceed the passing score on those challenging questions. There are many ways that an AI system like MedPalm can be a building block for advanced natural language processing in healthcare, and we'd like to work with researchers and experts to advance this work.