OpenAIs New Model Stuns Even DOCTORS!

So today we're diving into a fascinating study that could change about how we think about AI in medicine. Researchers tested one of the newest AI systems called O1 Preview, against both human doctors and previous models like GPT4, just to see how good AI has become at medical diagnoses and decision making. Now, this wasn't a simple test. The researchers put the AI through four or five different intense challenges, ranging from diagnosing complex medical cases that have stumped doctors, to suggesting treatment plans, to identifying critical conditions that absolutely can't be missed. They used real medical cases from the prestigious New England Journal of Medicine. These are the kind of complex cases that even experienced doctors find challenging. And what makes this study particularly interesting is that they didn't just use multiple choice questions. Instead they tested the abilities of the AI to think and reason like a doctor would in real world scenarios. Because they wanted to see if, if the AI could handle complex multi step thinking that doctors use every day when treating patients. And in this video, I'll break down exactly what they tested, what they found and what this could mean for the future of healthcare. Now, one of the things that they found in this research was the fact that this AI model, the OpenAI's 01 model, was actually really, really impressive in comparison to GPT4. They actually showcase around three different cases where GPT4 cannot solve a complex case, it can't diagnose it and it manages to get it completely wrong, whereas O1 gets the diagnoses completely right. So case one, they had some really complex disease, GPT4 got it completely wrong with a bond score of 0. Then O1 Preview managed to get it completely right and identified the exact condition. Case number two, there was another complex task which GPT4 completely missed and listed common conditions and said then 01 preview completely nailed it and got the rare condition completely right. Then we had case 3, there was an actual condition and GPT4 was close, which managed to get a bond score of 3. It listed some correct information but incorrect conditions, whereas 01 preview got it exactly right again. And in this, what's particularly interesting is that the Bond score shows how close each AI got. 0 is completely wrong, 5 is exactly right. And these were actually really tough cases. So these were like Medical Mysteries and GPT4 tended to guess more common conditions, but O1 Preview was able to identify rare and complex conditions pretty accurately. And this just basically shows us that with each improvement of AI and of course with this new series of models, whilst, yes, you might use this AI on a day to day basis when we are tackling complex scenarios like this, this is where these thinking models really do shine. Now, there was also this image right here. And this image shows a comparison of how well different diagnostic systems, both AI and human, perform at correctly diagnosing medical conditions. Using cases from the New England Journal of Medicine. And this is from 2012 to 2015. So now, the types of systems shown in the blue colors are of course, the modern AI systems and the light blue. Blue is where you have the older diagnostic systems that required doctors to manually input symptoms. And of course, in the brown bar at the bottom, that is where you can see the human clinician's performance. Now, overall, what we can see here is that there is of course, a stark improvement when we look at the O1 preview compared to GPT4. Then when we look at these older AI systems, we can see that they're not as good. And of course we can see compared to the clinician, there is a large increase in terms of the percentage correct diagnosis. From here you can see it's around 30%, whereas with these LLMs it's around 60 to above 75%, which is rather surprising. And this really goes to show us just how powerful these AI systems are. I know a lot of people give these generative AI systems flack because, oh, they're just regurgitating stuff, but when you apply them to medical use cases, you can see that these tools are remarkably powerful for diagnosing different diseases or diagnosing different things in a variety of different scenarios. Processing complex bits of medical information and arriving at correct diagnosis is the kind of thing that AI is exactly designed for, or should I say uniquely designed for. Now, we can see here Figure 5. Comparison of GPT4.01 preview and physicians for Management and Diagnostic Reasoning. And we can see here that this image shows how well different groups performed when managing medical cases called gray Matters Management cases. Comparing scores between O1 Preview by itself, which scores a remarkable 85 to 90% GPT4, AI scoring around 40 to 50%, and human physicians using GPT4 as a tool scoring around 40 to 50%. And then of course, human physicians using standard traditional medical resources scoring a whopping 30 to 40%. So this is rather fascinating. Once again, the scores ranging from 0 to 100 show us that O1 preview clearly outperformed all other options by a large margin. And this is fascinating because this performs significantly better than both GPT4 and the human physicians. Interestingly, there wasn't much difference alone between GPT4 and the physicians using GPT4. But this visualization powerfully demonstrates how much more capable O1 preview is at medical management reasoning compared to both earlier AI systems and human physicians, even when those physicians have access to AI or traditional resource. Now, in addition to this, I do want to caveat this by saying this is O1 preview. This isn't even the full 01, nor is it even O3, which was recently released by OpenAI demoed. And we know that that model is even smarter. So imagine what kinds of results that would get if this preview model is getting around 80 to 90%. We can also see this in terms of the landmark diagnostic cases. And these cases are basically the greatest medical mysteries that have been solved. They're like famous cases that have become teaching classics in medicine, kind of like the greatest hits of medical diagnosis. Now, these are real patient cases from the past that were particularly challenging or groundbreaking. They helped doctors learn something new about a disease or condition, and they often changed how doctors approach diagnosing similar problems. Now, what makes these landmark cases is that they're usually complex cases that weren't obvious to solve. They often involved unusual combinations of symptoms and the final diagnosis was essentially surprising or taught doctors something new. And they become standard teaching tools in medical schools. Now, when they managed to test these AI systems on this, we can see once again that Zero1 Preview manages to get a extremely high score on the left hand side. And we can see that GPT4 only also manages, interestingly, to outperform physicians with GPT4 and physicians. The GPT4 does perform better than physicians and resources. Now, interestingly here we can see that the AI didn't manage to supersede humans that much because there were several cases where humans managed to get this stuff. But we can see here that the AI is definitely really effective when it does come to these landmark diagnostic cases. I mean, whether or not you could say that this is a training data thing, I still think that this is remarkably impressive considering that physicians are seeming better off with these AI tools rather than without them. Now, this graph right here shows how often different groups caught the most critical diagnosis. And this is what they call cannot miss diagnoses. These are the diagnosis conditions that if they are missed, they could be life threatening for patients. So we have four different categories. So we've got the residents in pink, which are junior doctors in training. We've got the attending physicians in green, which are experienced, fully qualified doctors. Then we've got GPT4 in blue, which is the previous AI model, and 01 preview in purple, the newest AI model. Now, what the graph shows is a scale that goes from 0 to 1 or 0% to 100%. And the boxes show where the majority of these scores were. And the black lines show the full range of different scores. And of course, the dots show the individual results. Now, all groups perform similarly around a 50% to 100% rate. But we can see once again that O1 preview was more slightly consistent and residents showed more variation in performance. Experienced doctors performed about as well as these AI systems. And this was rather fascinating because once again we see that AI manages to perform really well in these scenarios. Now let me break down this table which shows how 01 preview planned medical tests compared to what actually happened in the case. If we take a look at this first case, you can see, you know, there was a certain plan which the doctors actually planned. And Then interestingly, the O1 preview managed to suggest another plan which was actually very similar to exactly what these doctors suggested. So you can see here, in this case, it managed to get a 2 score, which is a completely correct score when it comes to planning certain things in terms of the range of tests that you would conduct when you're trying to figure out what kind of diagnosis that you would have. Now, there were some things here that were rather interesting. It was impressive that the AI didn't just suggest random tests. It laid out a comprehensive step by step plan that included backup plans and alternatives. It explained why each test was needed and it matched what expert doctors actually did in real life. And this was rather fascinating because there are complex steps that go into doing this. And it's important to understand that all of those reasoning steps have to be completed successfully for the AI to get the right answer. Now, there were certain areas where the AI was wrong. There were two other scenarios where the AI got half the answer right and then the other one I got completely incorrect. But I think the most fascinating thing about this is that this is an AI system which isn't just purely medically based, like it isn't fine tuned on medical issues. But remarkably, we can see that when we're looking at these diagnosis, we're seeing these suggested plans, we're seeing that it's able to sometimes get the right suggested plan and the right steps to take, which is rather impressive. And we can only imagine what's going to happen in the next five years, the kinds of models that we're going to be get, and just how accurate they are in terms of diagnosing conditions and of course, suggesting plans. Of course I would say though that I hope Humans don't become too reliant on this because, of course, with hallucinations, you wouldn't want to have, you know, a tired dentist that is overworked, or a tired doctor that is overworked, or a tired clinician or physician that just uses what the AI says, and then next thing you know, a hallucination manages to mess up a person. So, of course, I do think that humans will always have a role to play when it comes to diagnosing individuals. We can also see here that this individual said that I had 01 analyze a very specific immune disease for my friend, who happens to be one of the top scientists in the field. And after Heist sent the results, his response was, oh, my God, I just read it. This is breathtaking. This is insanely good. So we can see also that the qualitative results from individuals using this at the top of their field does seem to be one that proves that these models are also rather fascinating. So with that being said, what do you guys think is the future of AI and humans when it comes to the medical industry? I think it's really fascinating that we're now starting to explore this in further detail. I do think that with rules and regulations, it's going to be pretty hard to actually get these models out into a real sort of practice. But I do think we're going to start to see more and more cases where doctors may have missed certain things, but users taking it into their own hands to consult with a model like O1 or even O3 and get remarkable results that doctors simply would have missed. This is something that I've discussed before, that literally millions of Americans die each year because doctors manage to make mistakes. We all make mistakes. We're humans. But the only problem is, is that in the medical industry, sometimes there are situations that are simply life or death, and those mistakes do cost lives. So maybe having an AI system review every single decision made, maybe we could catch those rare conditions or diseases that we otherwise would have missed. And then, of course, having humans check over and run the necessary test to ensure that what the AI suggested is potentially factual. With that being said, would you be open to having an AI doctor? I personally think that within the next 15 to 20 years, we're certainly going to have maybe some pods or something where you prick your finger, you get an instant blood test, you get an AI doctor that tells you everything wrong in your body, you get an instant diagnosis, you get an AI that reasons over all of your personal data. Maybe it knows everything you've done, everything you've seen, it knows everything you've eaten. And it's able to conduct probably the most effective plan for you because it understands your emotional state, your physical state, your water levels, how much you've been drinking. And it could probably suggest the most accurate thing. Context is, of course, key, and I find that the more context you give these models, and of course, your doctors, the better they become. And if we look at how AI is going to be integrated into our lives, I wouldn't be surprised if we're going to be sharing that AI data with our doctors very soon. A very interesting world for those of you who are trying to live forever. With that being said, if you enjoyed this video, I would like to see you in the next one.