Googles NEW "Med-Gemini" SURPRISES Doctors! (Googles New Medical AI)

Google DeepMind and Google Research have released a very interesting research paper called the Capabilities of Gemini Models in Medicine. Essentially what they've done is they've made a paper discussing and showing how Google's Gemini model can be fine tuned and turned into something that can be used very, very effectively for helping out in the medical industry. So this is quite surprising because I didn't expect such a system from Google just yet. But they also released something earlier this year which was actually pretty similar. If you remember, this is something that I spoke about. And this was amy, which was Articulate Medical Intelligence Explorer. It was an advanced AI research system developed by Google. And this one, which was released around three to four weeks ago, it was basically designed to handle diagnostic reasoning and, and engage in meaningful conversations within a medical context, aiming to enhance the interactions between physicians and patients as well as improve the quality and accessibility of consultations. And basically the reason that AMY was so good is because it was able to use a simulated learning environment to enhance its learning. And it engaged in diagnostic dialogues with AI patient simulators, allowing it to practice and redefine and refine its conversational and diagnostic skills continually. And it was actually trained on a huge diverse set of medical data, including real world clinical conversations with medical reasoning scenarios. And one of the key things about aami, that when it was actually pitted against clinicians, it showed us that this was something that was far effective if humans used it in the loop. And you can see right here on this graph, we can literally see that with Amy, only this, the system performed increasingly better than the clinician unassisted. And the clinician was assisted by search. And search is essentially just the Internet, which shows a huge improvement in the gaps. And then we can see that assisted by Amy is a far stark increase from just the clinician unassisted. And then of course, Amy only compared to the clinician assisted by Amy shows us that Amy actually did surpass the actual clinician. So basically what this showed us, okay, and I know this isn't Med Gemini just yet, but this is basically showing us that Google's increasing their efforts for medical health in terms of research because they're showing us that these AI systems like Amy are far superior than just the clinicians. Now, essentially, like I said, this is of course Med Gemini. So what we have here is we have the initial Gemini system that exists here. And Gemini is a family of powerful AI systems that are completely multimodal. You can see that there are the inherited capabilities such as the advanced reasoning, the multimodal understanding and the long Context processing. Now this is where they decided to have the development for Med Gemini. So they did medical specialization with self training, with web search integration and of course the multimodal understanding, they did fine tuning and customized encoders and of course with the long context processing, they did chain of reasoning, prompting. And with all of these skills combined, that's where we now get of course Med Gemini, this version of Gemini which is specialized for medical applications. And it's very, very fascinating because if you're someone that hasn't been paying attention to this space, this is an industry that is truly about to be disrupted by. Because the applications and the things that we're seeing show us that the benchmarks are looking pretty incredible in terms of the applications. Now one of the things that was there before was of course the previous state of the art. So the previous state of the art in terms of, in the industry for medical AI systems that are able to talk and that are able to answer certain questions and queries in terms of the accuracy. So you can see that here we can see from September 21st all the way to September 2023, there has been a huge increase in terms of what these AI systems have been able to do noticeably. The jump from DPT 3.5 to Google's Med Palm, then of course to GPT 4 and MedPalm 2 and the previous state of the art model which was before Med Gemini. And of course now what we do have is, is we do have a state of the art system which is Medical Gemini. The state of the art system before the one that was released today with Google. Well, not actually released, but the paper was released was actually GPT4, not just the base version, which the GPT4 base version was very close to Google's medical one. It's actually GPT4 with a fine tuned version that gets it 19.2% on the MedQA, which is a decent benchmark for these AI systems. So GPT4 with the Med Prompt actually had a very high benchmark. But once again Google has beaten them now with GPT4 and Med Prompt. The reason I'm showing you guys this infrastructure is because this shows us how crazy it actually is. So this takes us from the base level of GPT4 and you can see all of the different things that They've added to GPT4 in order to get the system to perform a lot better. And what's crazy is that Med Gemini, the reason why it's so effective and we're about to get into that is because it doesn't use all of these crazy techniques like the ensemble with choice shuffle. And if you don't know what that is, basically, in multiple choice questions, sometimes there is bias rated towards the first question in terms of like, the answer. So for example, if you were to ask someone what is the primary gas found in Earth's atmosphere? And if you were to have four answers, the first one, a lot of people subconsciously might think that that one is correct. And essentially what you do is you shuffle them and, and then when the most common answer is picked among these, you can then find out that answer and then that's the one that you use. So it's pretty, pretty crazy. You can see how many different iterations that they did on top of GPT4 to get to 90.2. And surprisingly they managed to surpass this benchmark. And you can see right here that this is where Google's Med Gemini comes in on the med QA at 91.0.1%, which is very, very interesting in terms of the increase. And I guess some people could argue that maybe we are starting to peter out in terms of the capabilities of large language models on these benchmarks, but I would certainly disagree because whilst yes, this might be the truth, Med Gemini is pretty great. And one of the key things you do need to know about the benchmarks is that you can see here, it says relabeling with expert clinicians suggests that 7.4% of the questions in the data set have quality issues or ambiguous ground truth. Essentially, one of the things that have been consistently problematic in the AI benchmarking industry was the fact that these kind of systems unfortunately have to go with benchmarks that are pretty standard. But these benchmarks contain thousands and thousands of of questions. But some of these questions are quite ambiguous and they have quality issues, meaning that the AI systems can't really even get any of the answers correct because the questions literally don't make sense. I wish I did have some examples to show you, but just trust me when I tell you some of the questions are completely insane. So what they're stating is that 7.4% of this might not even be that great because the benchmarks are potentially facing several quality issues. And whilst you might think that this is going to be something that peters out in the future, we do know that more advanced reasoning systems could take this to 100% in terms of not only the system being good, but of course the benchmarks performing a lot better when those quality issues do get fixed. So one of the things we could see as well about the medical benchmarking here is we can see that there are several categories in which Med Gemini actually surpasses the previously state of the art. The blue one here is of course the main focus and we can see that the blue one is Gemini and this surpasses the previous state of the art, which is GPT4 with Med prompt and every single category, well, nearly every single category. There are some ones right here where it is pretty much on par and only this one where it's on long context video. But you can even see right Here it says GPT4 results not available due to context length limitations and the advanced text reasoning, you can see here that this is done pretty well. The multimodal understanding, it's done pretty well as well. And essentially, if you just want to take a look at this in terms of how much better it is, if we just take a look at this line, we can see that anything above this line is something that is an improvement. And we can see that in pretty much near all of these categories we do have a decent improvement. So one of the things here is the Medical Gemini on advanced text based reasoning tasks. And you can see here that based on the previous state of the art, it surpasses it in many different categories. And one of the things that they actually did talk about is that they actually did talk about how this system compared with GPT4. In some scenarios, GPT4 just didn't actually have the context length to support that on long context reasoning. So it's important to know that long context reasoning, with Google's new context length, it's actually a pretty important feature for the future because it allows us to process more information. And the thing is, with the medical industry, the more data you have, the more comprehensive of a picture you do have. Because a human body is made up of so many different intricate parts and because they all connect together, if you have a long context window, you're able to fit more data in and arguably get to a better conclusion about what the diagnosis may be or what's going wrong with a certain individual's body. So it's important to have that for the future, especially if you in terms of the needle in the haystack, which is where you're trying to get that data from a long piece of context and use it correctly and of course reason with that correctly. So we can see Gemini. Well, Med Gemini surpasses state of the art and we can of course see it here as well, compared to Amy, where it does surpass that as well. Now of Course, some of these aren't that crazy, but it seems like maybe there's going to be some advanced reasoning techniques. Now, one of the things that I did actually see that was pretty cool was that the advanced reasoning that we did see was a little bit different. So they had two kind of ways that they did advanced reasoning with this. And I think that this is probably what we're going to see in terms of different models that are specialized for different use cases. So in the case of Med Gemini, they essentially had self training and search, and these were leveraged to enhance its capabilities in handling complex medical data and queries. So let's actually dive into how these features were used by Med Gemini. So the self training aspect with Med Gemini was where you use the model's own outputs to generate new training examples, which are then used to further improve the model. Apologies for that coming off. And then this method is particularly beneficial for refining the model's capabilities in an area where the initial training data might be limited or lack diversity. So, so essentially what they do is the first thing that they do is they generate synthetic examples. So for example, Med Gemini processes medical data or queries and then generates responses based on its current understanding. These responses, along with the context in which they were made, serve as new training examples. Then we have iterative refinement. So these generated examples are fed back into the training cycle, allowing Med Gemini to learn from its own outputs. And this iterative process helps the model to continually refine its reasoning and decision making capabilities, especially in handling complex medical scenarios. Then we have enhanced learning from simulators. And this is where for Med Gemini, simulations could involve creating scenarios where the model must interpret complex medical data from text images or even long medical records. And feedback from the simulations helps the model adjust its methods from for better accuracy and reliability. And then of course we have the search for Med Gemini, which is where we have Med Gemini. When it actually encounters a question that it might struggle with, has low confidence or insufficient internal data, it can perform a web search to gather additional information. Like here you can see it says confident no. Then of course we go to here. Then of course we go to here, which is web search. And then of course there's an uncertainty guided search where essentially the Med Gemini employs an uncertainty guided search strategy where the model calculates that its predictions have high uncertainty and it proactively searches for more information before finalizing its response. And this method helps in the accuracy of the reliability of its outputs. So this is the loop that it conducts in order to get better results. Then of course, we have continuous update of the knowledge. The ability to search and integrate information from external sources means that Med Gemini can continuously update its knowledge base without the need for actually frequent retraining. And this is actually pretty crucial in the medical field, where new research and clinical practices can change standard care protocols. So by combining these two approaches that we have here, it can better adapt to new or rare medical scenarios. It has access to the latest medical information via search, which means that its data is not only based on the initial training data, but, but also the latest studies, clinical trials and guidelines. And then through the continuous learning and adaptation, it becomes more proficient at handling diverse and complex medical queries, making it a valuable tool for medical professionals seeking AI support. Then, of course, here are the video benchmarks. And this is something where it's pretty, pretty good. And we do know that Google's Gemini 1.5 Pro can actually take in, I think, an hour of video, which means it also can look at these videos and analyze the scene in a medical way. So you can see right here the state here you can see the previous state of the art where we've got med palm and GPT4 vision. And I'm wondering what would happen when GPT4 release the video, how it's going to compete with Medical Gemini on this as well, which is going to be pretty crazy. And then of course, here is where we have the capabilities of Gemini models in medicine and this is where we have the actual benchmarks. So you can see right here, this is where we have the benchmark. And you can see the accuracy from the clinician, then the clinician and search, then the prior state of the art, then of course the Med Gemini, then the Med Gemini plus search. So we can see a stark improvement, like I showed you guys before with of course Amy and, and this is the Amy stuff right here is what I'm guessing. And then this is of course the GPT4 or Amy, I'm not sure which one. But what we do have here is a clear conclusion from this graph. Just on this part, we can see that the clinician is actually the lowest rated one in terms of the accuracy. And then if we go all the way up here, we can see that Med Gemini plus Surge gives us a huge, huge increase in terms of performance. So there is a huge gap right here in terms of what's being done. And I think that that is rather, rather impressive on how AI is able to bridge this knowledge gap for clinicians and it's able to help them. So I definitely think that this is something that could provide a lot of details and a lot of help to people. The only thing that I do think that could be problematic here is that hopefully humans don't completely rely on the AI, because sometimes the AI might miss certain things in diagnoses. And I think humans always, always, always have a lot more data points than an AI system, because something that I've, you know, and this is a little bit off tangent, but it does make sense here, is that if you are prompting GPT4 or any other AI system, sometimes you'll think, why can't this system get what I want to go out? But if you give the system every single piece of information that you can, like for example, if you're asking it how to write a certain essay and if you say when the deadline is, if you say what your teachers like, if you say, if you say the style, if you say the length, if you say a few things, you need, the more information you give these systems, the much better they are at essentially giving you the output that you want. That's something personally that I've seen, and that's why I state that the reasons humans would have a lot more information is because a human might go to a system and be like, oh, I have a runny nose, but if you do, I have a runny nose. To an AI system, that's the only piece of information it has. Whereas a human who's in there, the room with you, can see how your skin looks, it can see how you're moving around, it can see if you're fatigued, it knows your age, it knows what you look like, it knows your family tree. So the point here is that I do think that this is going to definitely improve as we move on in the future and hopefully this can be a very good tool for clinicians in the future. Now, like I said before, whilst, yes, this is doing well, there are some problematic benchmarks you can see here. As I spoke about previously revisiting the medqa benchmarks, there were some probabilities, it says, however, some MEDQA test questions have missing information such as figures or lab results, and potentially outdated ground truth answers. To address these concerns, they had at least three US doctors review each question to answer the question themselves, check if the original answers were still correct and note any missing details or ambiguous elements in these questions. And they used a method actually called bootstrapping, where a committee of three reviewers decided if a question should be excluded due to its flaws. Apologies for that little glitch there. And the findings were that 3.8% of questions were missing information, so 2.9% had incorrect answers and 0.7% were ambiguous. So most reviewers agreed on these assessments. Showing strong consensus and removing flawed questions helped improve the AI's test score from 91.1% accuracy to 91.8%. And if they used majority decisions, which are the more relaxed criteria, instead of unanimous ones, the accuracy further increased to 92, but by dropping about one fifth of the problematic questions. So in simple terms here, by cleaning up the test and removing or fixing flawed questions, they made the test a better tool for accurately assessing the AI's ability to handle medical queries. Now, here's where we actually get into some of the dialogue examples for this AI system. And I think this is where we can see how this multimodal system works. So it's not something that's too crazy, but I think you guys can all truly understand exactly how this works. You can see you have the dialogue examples, and this is where the person actually decides to leave a comment down where they're stating exactly what's going on. And then Med Gemini says, I understand your concern. Can you send me a picture of whatever it is you're dealing with? And then it sends them a picture. Then it queries and asks for more information. Then, then it says, this is what I think it is. Thanks. Could you explain what this is? Then it, you know, tells you exactly what it is. It's a diagnosis. And it says, only a definitive diagnosis can be made. And it says, okay, will you advise me on how to treat this? And then you can see here, it's able to give out a lot of these pieces of information that could help this person to, you know, solve this issue. And I think, like I said before, this stuff is really good because doctors, you have an appointment, you maybe have 15 to 30 minutes, but with an AI system, you could ask it a million or a billion questions, and it's going to be patient, it's going to understand you, and it's going to be able to talk to you in many different ways, that it could easily help you more than a person. Because whilst, yes, I do appreciate what doctors do and I know how good they are, the only problem is, and I know this sounds crazy, that I'm about to say this, is that they are human, which means they have human limitations. So, such as the time. And this is something that AI systems can help. So imagine a virtual AI doctor where it would just talk to you about this kinds of thing and you could quickly get diagnosed. Now, There was some feedback from an actual dermatologist where it says that it's impressive diagnostic accuracy for this condition, which is relatively rare and a speciality specific condition based on limited data of one photo and a brief description. So the reason they included this is because this is something that is relatively rare and it's speciality specific condition. And the fact is is that this was just one picture. Remember I've always stated how in the future AI systems trained on millions and millions of different images are going to be far, far superior than any human because they're going to have seen so much data that they're going to instantly know when something is like that. And I think in the future with more advanced systems, these kind of things are going to becoming pretty, pretty normal. However, there were some cons. It says additional photos of representative lesions on different extremities would strengthen the diagnosis and it says they could include other things. And then of course it does say whilst there's no cure, it could emphasize the possibility for symptom improvement and, and management. There was also another one here and I do want to state that there are several examples in the paper that I simply just didn't want to take a look at because the pictures were pretty graphic considering they were from medical examinations, like people doing open surgeries. So it was a little bit graphic and I just didn't want to include that. But you can see here, this is where we have another dialogue example of someone getting their picture and it says, hello, I'm a primary care physician and this is an X ray for a patient of mine. The formal radiology report is still pending and I would like some help to understand the X ray. Please write a radiology report for me. And then it talks about that, then you can query it and then it says, my patient has a history of xyz. But I do think in the future that it would be better if you could just somehow load the patient's data with this. Because trying to prompt it and say, could this be the back pain? I do think that whilst yes, doctors are good to do that, I think it would be better in the future if these AI systems are going to have access to that user's complete medical history because it's going to allow for a much more comprehensive diagnosis. And then of course this is where we had feedback which was a rather interesting example where it said, you are a helpful medical video assistant. You, you are given a video and a corresponding subtitle with a start time and duration followed by a question. Your task is to Extract the precise video timestamps and then answer the question given below. So provide one single timestamp that spans the entire length of the answer while considering the entire video. It's better to be exhaustive and providing the longest time span for the answer. So how do we relieve calf strain with 4 foam roller massage. And you can see that this is exactly where the footage is. It says the start and the end and it says that the ground truth time span annotation is the same as what Gemini had answered. So overall I think some people at the end of this might be a little bit confused between Med Gemini and Amy, but essentially they are quite different. Whereas the purpose of these two systems, AMY is primarily designed for improving diagnostic dialogues and and reasoning within medical consultations. It aims to simulate and support the interactive conversation part of a medical consultation, focus on history taking, diagnostic accuracy and patient communication. But Med Gemini is a more generalized AI model that excels in processing complex multimodal medical data such as text and long medical records, and is specialized in understanding and integrating broad medical knowledge across across various formats to assist in diagnostics and treatment planning. Beyond the conversational capabilities and the strength of course of Med Gemini is the fact that it could be used of course in some instances for the long context processing with long history records to enable more accurate diagnoses. And AMY is actually optimized on focusing on engaging patients in meaningful dialogues. The future goals for these two kind of systems are quite different because Amy aims to become a virtual assistant in medical consultations, enhancing the quality of care through better communication and diagnostic support. And it seeks to basically address the conversational and empathetic aspects of medical practice. Whereas Med Gemini is positioned to aid in a more analytical way, involving vast amounts of data, aiming to support medical professionals by providing comprehensive integrative analyses of patient information per potentially leading to more informed decisions. And I think something that you also need to take into account is that once these AI systems are trained on vast different languages, it's also going to break down the barrier in terms of people who struggle with certain languages get the right medical care that they need, because I can't imagine trying to explain something to a doctor that doesn't speak my language. The nuances in certain things do make the difference in ensuring that you get the right medical treatment.