The Ouroboros Effect: How AI Threatens Human Intellect. Guest: Prof Blackford Middleton

In an exciting announcement from Daedalus, they are soon going to be launching a new version of their Pharmacy Management Solution, iFarmacy in Australia and E Pharmacy in New Zealand. This platform enables hospitals to manage all aspects of pharmacy practice within a single integrated system. From dispensing and inventory management to cost centre accounting. These platforms streamline complex pharmacy workflows. Now, Daedalus began this work by listening to the healthcare community. And after hearing directly from pharmacists, clinicians and healthcare leaders about your evolving challenges, they've responded with significant action. The modernization of iFarmacy brings it into a contemporary platform that will deliver even simpler, more efficient workflows for healthcare professionals because they're committed to supporting pharmacy teams with cutting edge technology. Now, the launch is approaching, so stay tuned.

Foreign.

Hello and welcome to Pulse, your podcast delving into all things health, innovation, digital and data, proudly brought to you by Pulse IT News. I'm your host, Louise Schaper.

And I'm your other host, George Margielis.

Now, George, in the last episode we sort of hinted to people that you might have some big personal news. Are you ready to let listeners know what's been happening?

Yes. It's all happened since last time we met. So the biggest thing that happened to me over the last fortnight was the birth of my granddaughter Evie.

Yay.

Our first grandchild, who's pretty much taken up all of our attention over here.

Congratulations. The first grandkid. That's so exciting. It is very exciting, of course, absolutely. And you know, she's born into this generation where her electronic record, depending on where she was born, I guess she would already have one. That's to follow her for life if she chooses to. That's exciting.

Absolutely. Although I did note that when she left the hospital they gave her a big printed blue book with all her details in it. I do understand and I will be following up. I know New South Wales Health was working on a online child record, so that is something I will follow up. But yeah, she did come home with the classical blue book that my kids came home with. And I understand even I came home with some 60 or so years ago. So things do move slowly.

Okay. You know, 15 years ago now I think I was with my sister in law in America and she was pregnant and she was just going for her checkup for her first pregnancy and I asked if I could come along and she was in Kaiser. So you know, it's very closed loop system, but been digital forever. And so the doctor was talking to her about, you know, the stuff you talk to a pregnant woman about and how things are, tracking and doing the tests and I was just looking at the electronic record the whole time and going, oh my God. And so like my niece who's now in her teens, yeah. She has an electronic record from the time before she was actually born. Like it's pretty cool.

I must admit. We do have lots of images pre birth. We have a lot more images post birth, I have to admit. But yeah, yeah, definitely. Her father is a medico as well and has been involved in digital health in the past as well. So the poor kid has no choice. She will have a digital record whether she likes it or not.

Oh, well, that's wonderful news. Thank you for sharing. That's very cool. So I guess anything else you've been up to in the last fortnight sort of pales in comparison.

It does. It's been a fairly busy week. We had a webinar that business New South Wales ran on aged care technology which was pretty good. The Microsoft had a public service hackathon and I have to point out the clear winner in this hackathon was a digital health idea. So yeah, it just goes to show that health is still up there. And we did have the New South Wales Smart Sensor Network aging forum which I was the opening speaker at and that was pretty interesting. And there were a whole bunch of AI things pop in and of course the most important thing I did was I joined bluesky.

Yes, yes. Well done. And yeah, so shout out to all Pulse Potties if you're not yet on Blue sky but you used to be on Twitter back in the day when it was actually Twitter and it was pretty good. Yeah, there's a whole movement now of digital health people moving to bluesky. So you will see in the show notes, George, and my new handles which you can follow, please do. As well as Pulse it. And I've already found it so valuable, George. Like I'm finding articles and things that get lost in LinkedIn and yeah, I've found a lot of resources sources on bluesky just in the last week and.

There'S some people have been posting some amazing issues because one of the things I joined while I was on there, I'm a bit of a space nut, so I joined a space group and there's some amazing images which I've been setting up as a wallpaper. So it's really quite good. How about you, Louise? How's your fortnight been?

It's been good. Probably my personal news is I think I might have lost my dog. Well, I know where she is. It's going to be a while, I think, before I see her again. So. You know how I went to the US and we went for three weeks and. And I've also mentioned before that my dad is not in great health and he loves dogs. Like, you know, he'd be more excited about seeing the dog than his grandchildren. And no offense to the grandkids, but that's just how my dad's wired anyway, so we thought, oh, well, we're going away for three weeks, we'll take the dog over to wa and she's flown over there lots of times. We've always flown with our dogs. So we took the dog to WA and left her there and said, you know, we'd organize a return flight for when we get back from America. And I kid you not, that dog and my dad have bonded so much. I've never in my whole life. Me and my sisters are talking about it like we don't think we've ever heard my dad be so happy. Like, he's just happy all the time. The dog follows him into the shower and waits until he gets out of the shower. And apparently, like, the only place my dad goes is there in a small country town, so he goes to the local pub a few times a week. And apparently she doesn't even hang out with my mum. When dad goes to the pub, which is the only time he leaves the house, she actually waits outside on the back veranda and waits for him to come back. So Ryan and I have decided that she should stay there. So we made up like an excuse and said, oh, we're so busy and, you know, we're going to be traveling. Could she stay a little bit longer? It'll be a great favor to us. But, yeah, I'm not sure when she's coming home.

Well, look, I mean, it's a happy and sad story, but I'm glad she's found a great home and she's having a lot of fun in Perth and it gives you an excuse to go visit your dog as well as your parents now.

Well, that's right. As well as my dad. Yeah. And, you know, my sister saw her last weekend and she was like, yeah, I think Dad's latest pill is the best one, or his latest medicine. And I was like, what, is he on a new drug? I'm not in the loop on this one. And she said, yeah, it's called Ripley and it's got four legs and it barks occasionally because it's just put him in such positive frame of mind. That's been good.

So as an occupational therapy, you've prescribed pet therapy and it works. So well done.

Oh, pet therapy, definitely. Big fan of pet therapy. Anyway, that's probably my big news. Should we get to our hot topics?

George sure, let's get through to it.

Well, George, three episodes ago we discussed a small study that was published in jama, the Journal of the American Medical association, which showed that ChatGPT was better at medical diagnosis than doctors, even doctors who had access to ChatGPT. And we did discuss it briefly, as I said, a few episodes ago, but last week the New York Times wrote about that study. So look, we did get in first, but the Times does have a slightly more readers than we have listeners. So now it has completely blown up and I thought we should revisit it. There's a heap of commentary and postings discussing the ramifications of this study, especially how it could be possible that the AI model did not improve human performance because doctors with the tool performed worse than the tool did on its own. And this was like a year ago when the study was done, when the models weren't as good as they are today. And it's probably going to be true, even if it's not quite today, that I will be better at diagnosis than doctors. So what does it mean to be a good clinician, a good doctor, when AI is better at diagnosing than you are? How do you feel about that, Dr.

George look, it has been an interesting week. That article has literally blown up my LinkedIn feed. There's been opinion pieces for and against it from various sources. They've seen some doctor bashing and the return of the dreaded doctors will be replaced by computers debate as well as some engineer bashing about how they don't understand healthcare. And even our good patient, e patient Dave, joined the debate, highlighting how the patients can use AI now as such an important topic and how this article supports it. So it's good we had that little bit of blip when it first came out last week. It was being astronomical. So obviously more people read New York Times and JAMA Open Network. Look, it is a pretty important paper. It has to be noted that it does have some technical thoughts and it's not the highest level of evidence, but it is a great start and it does show a generative AI is a very powerful tool and clinical decision making can benefit from the use. One thing I did note when I actually went back and reread the paper and actually looked at some of the references the clinical vignettes they use come from a 1994 paper that evaluated the state of the art diagnostic tools of the time. And the clinical vignettes they used good ones. My main criticism, if you can call it that, is that it does rely on these clinically crafted vignettes, which are well designed and structured clinical descriptions of a patient's issues crafted by an expert clinician to provide the AI tool with the information it needs to make the diagnosis. And to be honest, that's the hard part. Taking the information from the patient through scrupulous history taking, deciding what tests you need to do to confirm you, deny your suspicions and build up that big net, that clinical case to base your diagnosis on is very much where the art of medicine lies. So William Oslo's the grand old Duke of clinical medicine. He's got two great quotes which really link in here. The good physician treats the disease, the great physician treats the patient, who has the disease. And his second quote is, listen to the patient. He is telling you the diagnosis. And in AI terms it suggests that it all comes down to the prompt. I wonder how good the diagnosis would have been if the prompt was, hey, I'm a patient and I don't feel very well, what's wrong with me? Which is what we normally get in medicine. So we need to think about those sort of things, about how it fits in. But critical paper. Interestingly, one of my very first medical informatics projects was developing an online history taking tool which linked into diagnosis. And this was back in the early 2000s. And what it was trying to do is work out how to take a history online using a whole bunch of questions. And it got really complicated because there were a number of branching logic loops and it got really quite hard. So that is the core skill that the clinicians bring to the table. There's that ability to capture that information and then using the AI. If there's one thing I would come out of that paper thinking about is that I would recommend that all young clinicians learn how to fine tune their AI prompting skills, because that's going to be the skill set that's going to make a difference.

Well, not just young clinicians. It's sort of like everybody and whether you're a doctor or other type of healthcare professional. And that paper made it clear, didn't it, that the training was really important. So how do you actually get the best out of the model? And so, yes, what clinical information do you give the model? How do you write the prompt? And I think our guest today is going to talk about these topics as well.

Absolutely. And like I said, it demonstrates that to use new available tools, we need to develop new skills to use those tools. AI is one of those tools we really need to get our heads around because like it or not, it's going to be a critical part of everything we do. The other interesting I did find, of course, is they talk about their pre print article about what's coming ahead. I think they're doing a similar study looking at clinical reasoning and I think that's a really interesting paper as well.

Oh yes, I read that last night actually, but I thought, oh, too many topics today. We'll put that in the next episode.

It did make me think, you know, it's the progression from Dr. McCoy from early star Trek to the emergency medical hologram in Star Trek Voyager. We are following that science fiction theme as we go forward.

Yeah, it does feel like that. And what do you think? Because it was always thought that surely the person that brings all of their experience and wisdom and knowledge to the fore but uses these tools really well, that's going to be a better outcome. But what the study showed, look, there was only 50 clinicians. Certainly that was one of the limitations. It was only small scale. But we, we always thought that people working in partnership with these tools would be better than the same person unassisted. And yet these studies showed that if the AI agreed with the human, then the humans were like, of course I'm great, but if the AI said oh, maybe you should consider this instead, the doctors ignored it. So I'm wondering about what this means for professional identity. If we think our professional identity as clinicians is. I know all the stuff and I'm the only one who can do this. This is a different way that we need conceptual actualize ourselves as clinicians.

Ego is a very negative force when it comes to practicing medicine. So I think it's really important. I think there was an ancient Greek philosopher who said, the only thing I know is that I know nothing. And I think we really need to go into these discussions with that mindset we're constantly learning. But yeah, we've all seen the movies of the old British doctors who never listens to anyone else's opinions. That is an old mindset which we need to get rid of in medicine.

Yeah, absolutely. Well, it'll be interesting to see there'll be more studies done on this topic and we will discuss them and bring them back to the pod for a chat. But thanks, George, because I think it was useful to revisit that I'm sure.

We'Ll hear a lot about it at the AI Care conference we're both attending later this week.

Yes, that's right. Well, onto our next topic now. The health startup world has been abuzz with the news that a startup called Forward Health announced they were closing down. Now, for the pulse potties who haven't heard of these guys, we're basically talking about Doc in a box is how I like to think of it. Forward Health launched eight years ago in the United States with a mission to provide the world's best affordable and accessible preventative care. But they had expensive clinics in some big US cities, they were only available for members, they didn't accept health insurance. And then last year they launched these care pods. These were actual physical boxes, very futuristic looking, that you would step into for these AI automated medical checkups. And the idea was that they would put them in shopping centres and other common places where humans like to go. Now, investors gave them more money to roll out 3,200 of these futuristic boxes. And I recall at the time my social media feeds were full of wow, you know, oohs and ahs and. But then also skepticism by informaticians and people with a deep understanding of healthcare. But investors gave forward health $650 million in total. And that's us. And in 2021 they reached a 1 billion DOL evaluation. Well, last week Forward Health turned off their app, they closed their locations and they cancelled all scheduled visits, effective immediately. And their announcement to patients said that they would work out how to get them access to their health records. And 200 employees are now looking for a new job. What do you reckon went wrong here, George?

Dock in a box. And $650 million later, no dock and no box. No. It is sad, especially for those who invested in it. Interesting. Again, looking back at my old intel days, the smart guys in Oregon office had come up with a very similar idea which went nowhere then either. So unfortunately it wasn't well documented. So had the poor folk at Forward Health had the opportunity to look backwards, they may have realized that been there, done that, didn't work before. We've had the closure of Babylon, Olive, Amazon Care, Theranos and Litiny others. And I guess it just demonstrates that that famous Donald J. Trump quote, nobody knew healthcare could be so complicated. We may actually find that President Trump knows more than he's talking about now. Fortunately for the United States, he now has RFK Jr. And Dr. Oz to help him. So that should solve the problem. And just in case, isn't Picked up in the podcast. Yes, I am being sarcastic.

I know. When I saw the Dr. Oz announcement, I was in another meeting and, you know, I had my screens open and this thing popped up. I actually thought it was the chaser or the onion or something, and I was like, oh, dear. Okay. But anyway, we're not talking politics, but yes, that's right. That is a famous quote that Donald Trump did say. But yes, healthcare is complicated. But then how do these investors. Because it seems to me that investors. And you know, this is not the first time this has happened. These unsubstantiated hubris from Tech Bros. It gets funded by the tune of, like, a lot of money. Do we need more clinician investors and clinician and patient advisors to help investors? Because what I saw, I was thinking, thinking, well, yeah, if I'm a tech bro and I'm an investor from tech bro land, then of course I'm like, wow, this is great. I want this. But without actually thinking, what do actual individuals want, which is tech enabled personalized care. Humans don't actually want to go into a box, Star Trek style.

Interesting. Yeah. I mean, if we're here to give tech bro some investment advice, I'd say invest in Pulsepod. You'll get much more out of that than you will get anywhere else. We'll help you make better decisions. I mean, that's what we're here for. But I've been rereading the works of Sir William Osler over the last few weeks, as I do every now and then, and he's got a great quote which is really relevant here. The practice of medicine is an art, not a trade, a calling, not a business. A calling in which your heart will be exercised equally with your head. So think about healthcare as much more than just having a bunch of devices and gadgets, but use those devices and gadgets to support that, calling that art, to enable people to provide better care. So, William, Oslo is now almost 200 years ago, but these sort of key things come back consistently in healthcare more complicated than what we thought it was.

Yeah. And the other thing that I thought was really flawed, they built a proprietary EMR which isn't interoperable, and, you know, shouts out to everyone leading the Sparked community, which we talk about most podcasts. I was at a Sparked event in Melbourne last week, which was excellent. Thousands of clinicians and tech people and policymakers all getting together to actually make sure that we have proper standards that are going to work for clinicians, that are going to work for healthcare administrators, not put everything in these proprietary boxes where information isn't shared. And I just couldn't believe they're like, sorry, folks, we're going out of business now they have real life patients, like people who rely on them and they don't seem to realise their responsibility to patients. It wasn't like, hey, sorry George, we have to cancel you appointment, but we've done a deal with this other clinic in your area, so you can go see a physical doctor there or here's a telehealth appointment. Just like abandoned all these people who are relying on them. Just blew my mind.

What's the old Silicon Valley motto? You know, move forward, break things and keep on moving. Well, this is the sign of that, but I guess this is one of those real examples where they really should have been thinking outside the box, shouldn't they?

All right, I love it, man, we should just end the podcast there. That was a really good pun, George, but we've got other topics to cover today. Now this is going to have just a brief intro for me because meanwhile, while this, all this has been happening and we've been talking about New Zealand a lot and the substantial amount of like half a billion dollars that's been stripped out of their digital health program, well, there was an announcement that the New Zealand government made. They're aiming for 10% of all care to be delivered digitally by mid-2025. And I do note that because I checked, this includes telehealth by telephone, so not just video. What do you think, George? Is this a when other health systems are aspiring towards digital first? I mean, 10%, it's pretty low, isn't it?

First of all, New Zealand, yeah, you've got to love New Zealand. They shift from one strategy to the next in their attempt to be quite as good as Australia, but doing it at half the price. Oh, and it doesn't quite get there, does it?

So look, he's also being sarcastic, everybody. Like, we do love New Zealand. We're just teasing.

We do love you and, but like I said, yeah, these idea of setting an arbitrary goal, it's nice bureaucratic exercise, but. But that's not the way healthcare works. You need to think about what the goal should be. The goal should be deliver care digitally where you can improve access, quality and safety. And I think that's something most people would get across. But using a goal as a target is a good first step. Does it lead to sustainable change? I'm not sure. We need to see how it develops, but at least they're highlighting the need for it. But then they need to make sure they've developed the infrastructure, the training, the tools to enable digital delivery of care. And if memory serves me correct, that's why they imported our good friend Lee Donahue last year to try and do that. Maybe they should have some consistency in their policy.

Absolutely. Because I was like, where's the strategy? Like mid-2025. So we're talking, let's say generously, seven months for that to happen. And where's the strategy to make this happen? Common sense dictates focus should be on improving health outcomes for all. And yes, doing digital, of course, on PulsePod, we're always going to think that's a good idea to digitally enable the care that you provide for a whole bunch of reasons. But yeah, there was no strategy, just this announcement struck me as a little strange.

We'll see how it all rolls up. But like I said, every little bit helps, I guess. So I guess piece by piece we'll watch with great interest.

Yes, and let's hope in seven months we're reporting that, wow, they've done it.

And if they can do it at half the price, maybe Australia will start following them.

Well, that's right, yeah. So we will keep an eye on that story. All right, well, moving over to the UK now, we know the uk, they love a good health review report over there. Well, the newest is a report called uniting the UK's health data. It's authored by Professor Kathy Sudlow, so it's therefore referred to as the Sudlow Review. The report highlights a transformative moment for healthcare data in the uk. It calls for a more coordinated and efficient health data ecosystem where health data is treated as critical national infrastructure and used to transform patient care and outcomes at not just an individual level, but also a public health level. There's five key recommendations in this report, including a national strategy for health data, establishment of a national health data service with strong governance, widespread engagement, as you would expect, a UK wide system for standards and accreditation of secure data environments and that all of it should be a for UK nations wide approach. Now George, I know you love a good UK health review and the idea of looking at health data as a national asset, which should be carefully collected, managed and used for the benefit of individuals and society, is something surely all health systems should be reflecting on at the very least?

Absolutely. Look, I love a good report and yeah, the UK does some amazing reports. I've got a hard drive full of reports from the uk which we use and they've had some fairly amazing people from Eric Topol, and many others have been involved doing those reports. It was a good report. It's a bit of a motherhood statement, the need for a coordinated and efficient data ecosystem. I do note that Europe has a European health data space, so maybe if they had stayed in the European Union, they wouldn't have to go and reinvent the wheel. But hey, that's what happens. The recommendations are pretty good. Leveraging the Biobank in particular, I thought was a very interesting initiative, that it can basically tie data to physical results from records and that ability to go back and re craft that data going forward. The challenge which the UK often faces is that they have to translate this wonderfully crafted documented reality. And that's where it gets hard. Nevertheless, Australia has a very similar challenge. I mean, we've talked about our health data strategy. We know we have some real challenges around secondary uses of data here in Australia, so maybe there's an opportunity for us to learn here. It is important that we keep on saying data is the new oil. We need to make sure that data is in the right formats, is standardized, is easy to access, all those sort of things with the privacy restrictions are all in place. The good thing about these UK reports, of course, is that when they do struggle to implement them, a lot of their people come over here and help us. So there may be another influx of British colleagues coming out to give us an idea.

Yeah, well, and I was reflecting too, in our interview at last episode with Onna and Robyn and the Health Information Exchange. Onna gave a really good case for how useful the data can be, not just for clinical care, but to research and to startups as well, to think of new innovations that they could solve some of these problems.

Absolutely, yeah. The key thing here, and we're having this struggle in aged care, and we've gone through it over many decades in healthcare, is to have a culture of data, the culture that respects the value of data and realizes that it is a essential tool going forward, rather than thinking of it as just a compliance and regulatory requirement.

I know the Tony Blair Institute in the uk. I think they've been laying a lot of the groundwork as well for this idea of treating their data assets as an asset. I mean, I've never worked with or know of any hospitals that actually have a line item in their budget. Data doesn't get recognised its value at all. And the IT enabled to support the data is seen as an expense, an IT expense item. The idea of, I'd love to see if the UK do this. Well, how health service services can actually start looking at how you could actually put a value on that data as well.

Having data in your balance sheet as an asset, which is. Yeah. Differentiates you from other organizations and gives you commercial advantages as well as the advantages we've seen in quality and safety that we get from the use of good data.

Yeah, absolutely. All right, well, we'll keep an eye on that story. Our last hot topic is in Australia, where it's so exciting and pulse potties will probably already know this, but we didn't want to have an episode go by that people in other countries didn't hear that in Australia. Last week, Health Minister Butler introduced legislation to Parliament to mandate sharing of pathology results and medical imaging to patients through the My Health Record portal. We call it Share by default. It's pretty big news. I've been responding to a lot of comments on LinkedIn, on other people's posts where clinicians who aren't aware of the health information exchange, they're not aware of the fact that my health record, which, let's face it, currently sucks in the way it's architected, but that's getting. And so I've been like letting everybody know and giving people links to the people we've had on the POD to talk about these topics. What are your thoughts, George?

It's very exciting seeing it actually in Parliament. So I was fortunate enough to be at the Victorian Healthcare Week when the Minister made the original announcements about this. And I have to admit, he was extremely passionate about how important it is to move away from what he called the shoebox of PDFs to a integrated record and really highlighted how important it was to share by default, Fault pathology reports and diagnostic imaging is first cabs off the rank. I think they're very important. They're already 99.9% digital nowadays. The fact that they weren't being shared with the patients through mechanisms like my health record, I think is a real fault. When I went for a nice MRI a few months ago, I could access that report through a separate portal in digital format. So the idea of having seven or eight different portals I need to access my results from is bizarre. When we have have my health records, I think it's really good. We're watching with great interest how it goes through Parliament. I would suspect this is a fairly unambiguous vote for everyone. I mean, why would you vote against it?

Yeah, I think it's got bipartisan support.

So I think we should be fine there. Then comes to the next stage about what are the other documents? And we've seen referral letters, reports back from specialists. The idea of GPs developing summarized clinical records that we can utilise in other uses, Allied Health, health bringing them into the play. I mean this is the first step in what I hope is a great journey. As we noted, my little granddaughter will have a much richer my health record than I do, which just basically tells me when I pick up drugs from the pharmacy, which is pretty much all I have on it at the moment. So it is exciting. I think it's a great first step. Maybe 20 years late, but hey, better late than never. We watch with great interest. I did note Simon Cleverly and Daniel McCabe were at the recent Connected Care Council meeting and they were talking about it and all the privacy. If people don't want it uploaded, they have that option. So it still is patient centered. So those options, the fact that we'll be able to look up results and not necessarily imaging is a classic example. I mean it's all well and good looking at it at an X ray but unless you have something to compare it with, you don't know whether things are getting better or worse.

That's right, yes. It's not rocket science and Grok doesn't know if it's a testicle or a breast.

Exactly, exactly.

So, yeah, well we'll keep postpone in the loop on this. My only, I mean not about the legislation itself is just the timing. So it's probably not going to be debated in the House until next year because I don't think there's not enough sitting days left and the Parliament has a jam packed agenda and then it's got to go to the Senate and then we're on track to have an election next year, a federal election. And for those outside of Australia, there's lots of really good things about our democracy here. One of the things that I find strange is we don't just have a set date for when the election is. So if you're unfamiliar with that, the Prime Minister of the day, basically he or she, but it's usually a. He gets to decide on when it is going to be and we only have short election periods here, but it does have to be done by I think the end of May. Is that right, George?

End of May, Yep. And usually because, I mean the Aged Care act is the one that obviously I've been following quite closely as well and that's now passed through the centre that's gone through a few hiccups along the way. That one is particularly important because it's due to be implemented on the 1st of July next year and a lot of people in aged care are sort of scratching their head saying we probably need a little bit longer lead time. So yes, we need to go through. Watch with interest when the election is called.

All right, well, after this brief break, we will be back with our special guest of Pulse, who is both a philosopher, a doctor, an innovator and a professor of informatics.

Looking forward to that one.

When you're building a healthcare company, compliance is crucial to earning trust, winning bigger deals and proving that you take security seriously. Vanta helps healthcare companies achieve compliance quickly and painlessly by automating over 35 frameworks, including ISO 27001, SoC2, HIPAA and HiTrust. And with Vanta continuously monitoring your security posture, your team can focus on growth. Get started@vanta.com PulseIt for $1,000 off Vanta, the leading security compliance solution trusted by over 12,000 customers. Care Monitor is a digital health platform designed to make care more connected and scalable. They support healthcare organisations across Australia to deliver a wide range of programs including remote monitoring, chronic disease management, hospital in the home, perioperative care and more. By giving providers a secure and flexible way to deliver services, Care Monitor helps teams focus on what matters most caring for patients. Learn more@caremonitor.com au. A couple of weeks ago, George sent me a substack written by an esteemed health informatician that we both know and and it was such a gripping read that we decided we would reach out to the author and we're thrilled to have him in the Pulse Pod studio today. Now, as Pulse Potties know, I usually introduce our guests, but I interviewed this person back in 2017 for a podcast I used to host where I would get the guests to introduce themselves. I always remember his response back then because it was, let's say, unique. So I thought I would do the same this time around. So, guest, welcome to Polls and I'll start by asking, who are you?

You well, Louise George, it's a pleasure.

To be with you.

My name is Blackford Middleton. I am a semi retired professor of medicine, professor of Biomedical Informatics. My wife says I've failed retirement twice and I'm still having too much fun. And the advent of generative AI and other AI tools has really re stimulated my interest in how we can improve decision making for doctors and patients. I've been involved in academic research settings and decision support and knowledge engineering, knowledge sharing and I've been involved in a few startups that have aimed to produce computable knowledge artifacts and democratize them if you will, so that every clinician and every patient everywhere can have best care or best care everywhere. Lifelong dream. I want to be sure that's in place before I actually need it.

Absolutely.

Sounds fantastic. I can relate to the semi retirement. I think doctors never retire, they just keep on shifting focus until the last minute.

Indeed.

So Blackford, the piece you penned which we want to talk about today is entitled the Ouroboros How AI Threatens Human Intellect. And at the top of the page is a massive picture of a sinister looking snake eating its own tail. So what is an Ouroboros and what was the genesis behind this piece? You know what inspired you to tap out these 1200 words?

Woods the Ouroboros is a mythical and historical figure which is a snake eating its own tail and it's supposed to represent rebirth and rejuvenation. Perhaps, but it's also self consumption and can have an untoward interpretation or even malevolent interpretation as well. It struck me when I was sitting in Australia in Sydney at MedInfo listening to a discussion about AI, which I just loved. And it struck me as we use more and more synthetic data to train our AI systems, what's going to happen? And I did a little research, got into it, found a few papers and decided this should be named. It's the Ouroboros Effect because it's what happens when AI consumes AI output, whether it be synthetic data or other outputs from AI. I worry about the homogenization and the loss of our critical reasoning skills as humans.

Excellent. So medinfocidney was a very inspirational event. So I'm glad to see that it helped inspire you to think about the Ouroboros effect. In your article you talk about the concept of model collapse. What is model collapse?

This is a very important concept and I refer people to the paper by Chomalyov, if I'm getting that name right. Just this year came out describing how models can collapse, I.e. performance degrades substantially and discrimination capability or answering capability can degrade substantially. When the models are trained on their own output or synthetic data, what happens is they become less and less reliable, perhaps more and more faulty, and the maths are beyond the conversation today. But I encourage folks to take a look at that paper.

Paper.

It really is interesting you say in.

Your article that AI won't necessarily take over, so that's good news for all the dystopian science fiction fans out there. But you do say that while they won't take over that human intellect is literally under threat. So how so?

Well, this is sort of the critical issue. What I've been thinking about is when we use AI, the critical issue is to get the question right. What I mean by that is like my old professor of medicine teaching me in medical school said, you have to know what you're asking and to get the question right before you can even have any hope of finding an answer. And that was sage and useful advice. I think the same will apply in AI using generative AI technologies because the quality of the prompt, as many have described, is going to determine how good the output is, number one. Number two, as we use more and more data generated by the AI synthetic data, I worry there's an asymptotic curve which will result in the artificial intelligence produced by the synthetic data becomes completely homogenized and vanilla and lacks discriminant capabilities or is not informative because it's only usable its own data. The phenomenon in model collapse is when the answer is no longer appropriate for the model output.

Is it also a matter of by using these tools that aren't necessarily in this scenario going to give us the best output, that we also lose our ability to think critically and to even recognize that the output is not helpful?

Yes, indeed. So the concern is, is will generative AI replace clinicians? Will it replace professionals? Others have written about this in Clinical Informatics historically. Randy Miller, for example, described the Greek oracle that an early system known as Quick Medical Reference built from the Internist 1 expert system at Pittsburgh many, many years ago, really shouldn't be viewed as a Greek oracle to provide provide the definitive and absolutely true answer. It should be used more as a tool to stimulate thinking and reflection and iterate on questions or discussions, perhaps to really get at what is the right answer. I am convinced now, having played with dozens of these tools, that if I did not have the knowledge base I have in medicine in informatics, I wouldn't know how to use the tool or use the system. Not everyone has to be a doctor or an informatician, but all of us as citizens using these tools, I think need to have some assurance that we can rely on them to be credible, where appropriate, authoritative and accurate. And that's the thing about human reasoning. If we rely completely on these tools, I do worry that our own internal discriminant ability or critical thinking ability could be damaged.

Interesting. So the concept of a dystopian future that Luis touched upon comes in a different way. So it's not Arnold Schwarzenegger and Terminator model. But it's basically humans losing the ability to critically think, artificial intelligence tools which decrease in their value in their accuracy, and a world where we just accept that as a given. So all of a sudden we have a world where we lose critical thinking, rely on poor data, make poor decisions, and it just becomes worse and worse. You talk about the trajectory of AI development, how you compare that to the Uber as a self destructive cycle. I think we've sort of discussed it a bit, but yeah, elaborating on that metaphor. How do we avoid that state? And how do we avoid it in particular in healthcare? Because again, you know, if it tells us what movie to watch on Netflix and it tells us the wrong movie to watch, no big deal. But if it tells us in healthcare and tells us to remove the spleen instead of the kidney, yeah, that could be a challenge.

It could be problematic. George, excellent thought. An analogy I like to draw is between what's happened to humans and their own spatial reasoning in getting from point A to point B. Back in the day I used to have to bring my world Atlas or my U.S. national Atlas with state by state maps and figure out how to get from point A to point B. And you know, I got good at it. I was able to get around. Nowadays I will not go hardly anywhere without turning on Mr. Google or one of the other map tools that gives me turn by turn guidance. So have I lost map reading capabilities? Have I lost wayfinding capabilities? I think the ideal way to consider AI is as an adjunct to my human reasoning. That is a support or buttress or tool to use with the topics I have of interest to explore or determine, et cetera, but not to replace.

That's a great example. I know a lot of teenagers now who are getting their driver's license and the idea of having this book that they have to have in the car and learn and memorize, they're just like, how did you do that? And people in my generation go, actually I don't really remember how I did that, but I did.

I can relate to that. My first road trip to the United States back In the late 80s, early 90s, you know, we had great big fold out maps and every time you'd pull into a town you'd buy a new map to find out where you're going. But like I said, it also gave you a different perception. You knew you could understand in context where you were, rather than, yes, I need to turn right at the street with a blue building on a corner.

Yeah, absolutely. So back to Model collapse Blackford in healthcare, we are increasingly using AI for diagnostics and decision support. There's certainly a lot being written about it it and you know, the odds are that within a couple of years it'll become quite ubiquitous throughout healthcare everywhere. So how can we detect early signs of model collapse?

This is a very hot topic and I'm not well equipped to dive into the mathematics of evaluation and whatnot for all these tools. But clearly we need to have some reliable performance metrics. That is some way to assess the validity of the outputs from these generative AI tools, to assess all the typical components, face validity, content validity, bias that might arise in the outputs due to bias in the data or even something else which worries me a bit is kind of the hidden values that might be sort of inherent in some of these systems given the value structure of our supply of data from the Internet. An easy case in point is generative AI tools are largely trained on Western experience simply because Internet, you know, is not very prevalent. It's increasing, but in less developed areas of the world we simply do not have the same supply of data. So I think the idea of metrics, the idea of oversight and audit, particularly for sensitive decisions or decisions of high risk risk, is the way the US FDA likes to look at it. But I think we need to have the user be completely aware, buyer or user beware. And the idea of model cards that describe what a particular model is and does and to which population it might apply and those to which it may not apply are all useful, along with transparency, public education, all the kinds of things one might imagine imagine in Australia.

There's lots of discussion in the US as well around regulation and accountability and we keep on talking about how we control AI, but without a deep understanding of the potential for it to go astray. Given your background in healthcare informatics, have you seen any concrete examples of how AI has potentially diminishing rather than enhancing decision making? Are we not there yet? Is it still early days and we're still on that rising part of the hype curve?

Well, the most interesting thing to talk about is the very recent paper, paper Jonathan Chen and others in JAMA Open, that looked at evaluating the way clinicians reason on difficult diagnostic cases aided by generative AI or not. And the result is that the generative AI outperformed not only individual clinicians, but outperformed individual clinicians using generative AI. So that's very curious to me. What it means is that we, we have to think about getting the question right again and thoroughly. That is that we train users how to use these generative AI tools. We train them in what are limitations, what's appropriate, what's not appropriate, and when to double check, triple check or confer with a colleague. If something seems, you know, that doesn't sound quite right, just remember, and this is the human in the loop idea, you know, humans, well, docs in the US will always be sued for malpractice or untoward outcomes and the like. And you know, they are the final arbiter of decision making with the patient in healthcare, not the AI.

When we look back at our classical medical training, we were taught history taking is the core part of what we do. And obviously when we're talking about the prompt, it's only as good as the history we've collected and entered into the prompt. How important is it then for us to not only be able to develop the prompt, but make sure we actually give it the relevant clinical information?

Well, George, you've touched upon the rat's nest, so to speak. How do we get the data in a coherent, well structured and standardized form into an analytic environment, whatever it might be, a predictive analytic, a quality measure or even a big generative AI? So number one, we need to continue to standardize our data, harmonize data and make sure that they are accessible, if you will, to the tools and not burden the clinician with re entering every single thing into a generative AI. Back in the day, I remember using expert systems for differential diagnosis like internist 1 or QMR. You had to retype the entire case finding in the appropriate syntax and semantics of the tool to get it to work, otherwise it just didn't work. I think that's part of the answer here. A second part of it though is not only how do we qualify the data, I think we need to work hard on ensuring that the learning being done in generative AI, encoding of world's knowledge, if you will, and vector databases and the log like is buttressed by structured knowledge that is human curated. What I mean by that is my current area of research is around knowledge graphs, working with the BMJ incidentally, but the idea of using a knowledge graph to help encode information into a large model or to help decode information from the model, interpret and verify or validate as well as prompt engineering, could we use structured knowledge from a knowledge graph to improve the quality of the prompt itself? And I think all three of these are useful avenues of research and discovery.

And there's so much work to do in that space. And thank you for coming on the show Blackford, because so many of our listeners will be excited about these, but also concerned about things like the topics we've discussed today. So knowing that there's so much more research that needs to be done at the outset before these tools are universal across healthcare, to get things like the prompting correct as much as possible. As you said, training our data sets is a massive issue because the data quality across healthcare, no matter what system you're using, even if you're using a paper based system, it's pretty bad. All right, let's move to solutions. At the end of your article, you did talk about strategies to prevent the Ouroboros effect and you suggested specifically for policy and technology strategies. Now, the first one is AI transparency regulations. So as it relates to healthcare data, who do you think is going to require these explicit labels on all generated material? Like can we get that done and do we need a global consensus on this or is it going to have to be jurisdiction by jurisdiction? Which I understand why that might be the case. From a policy perspective, that might be easier. But then we're using systems that are global. They're not just a specific system for a specific hospital. So how do you see that transparency regulations working thinking?

It's a great question, Louise. You know, the digital rights management technology space has been growing and developing for years and allows us to, for example, buy digital media from music stores or art and whatnot. But everything is clearly labeled. You know its source, you know its owner, if you will, and there's no infringement on that IP or copyright. So I think labeling, ideally the Internet drives us toward global solutions, but we may have to start in a more regional way given differences in regulatory approaches and languages and all the rest of it. But I imagine a day when it'll be perfectly clear when I'm using one of these models, how it was trained, where it was trained, how current and up to date is the model, and then making sure that those data input to it are labeled in a way that's appropriate. That's a big tall order and it's going to take a long time, but I see efforts in that direction already.

At the moment, we're seeing the war between the various providers about making their models bigger and not necessarily better, but they're containing more data. What's your thoughts around a curated LLM versus a free for all? Something that's taken everything good, bad and indifferent from the Internet and added it into the LLM? Some of the data suggests they have better or worse outcomes, but in healthcare, I'm not sure I'd be overly confident of a LLM which has a huge homeopathic section embedded in it.

Yeah, and I do love my vitamins, George, but it's a great thought. On the one hand, I think using a generative AI tool, when using them one discovers their strengths and weaknesses. I've almost thought of late that these tools almost have personalities in a way. Way. I know which one to go to for kind of a fact check. I know which one to go to for rewriting and helping me expound on a thesis that I've developed. I know which ones are really good at fact checking. So I think we have to view these generative AI tools almost like drugs. Now, I'm not suggesting they be FDA certified or reviewed and qualified, but at some level clinicians like I know how to use different drugs for hypertension. When is it beta blockade, when is it calcium channel blocker, when is it something else? I need to know where these tools apply or my colleagues need to help me understand where these tools might apply and that gets then to governance at the local level. I think effective use of generative AI is going to be really determined by effective local training, implementation, support and understanding. And if that really happens on a local basis, then there won't be too much risk, hopefully from a clinician or a patient getting advice from a generative AI tool. That just doesn't make sense.

There's a health equity issue there as well. The last thing we want is you have a lot of money to go to the expensive hospital that's got the best LLMs and the clinicians have the best tools. There's a lot of work to be done to ensure that there's an equity argument here that we take seriously and it's implemented well. The second strategy that you came up with in your solutions here is that we must implement human in the loop systems. So how does this work optimally in a clinical setting? Blackfoot.

You know, in a way my thinking here just borrows from existing clinical decision support systems kind of thinking. That is when you make a recommendation of a drug dose to a pediatrician caring for a sick infant in the neonatal icu, that drug dosing advice is reviewed by the clinician and of course the clinician already knows pharmacology and physiology and the rest of it. We don't all have to become AI data science specialists, but we need to know how the tools work and what they're good for. So to me it comes to this idea of not only getting the prompt right, getting the question, right, as my old attending physician used to say, but also educating both clinicians and patients in the appropriate use as well as the inappropriate use of these tools, another question arises. Is there a certification step here? Do we need to have these things reviewed by FDA as software, as a medical device, or by professional societies or whatnot? I think that's still evolving, and I'm not sure what's the right answer, but probably some form of, of local review. Before I implement this at my hospital, I should have a committee that understands how to look at it, how to evaluate it, reasonable performance characteristics to use in the kinds of patients we have at that hospital or clinic and the like. And then the model cards, an idea that a generative AI will have a model card describing what it's good for, what it's not good for, and ideally how to use it effectively.

It does produce one interesting challenge. We hear a lot about burnout and the overwork on our clinical workforce. Adding the accountability for AI and then adding that requirement that they do that assessment obviously adds to the workforce. Do we need to train our clinicians differently to be able to do that? Because, I mean, my son just recently graduated from medicine. I don't think they were taught how to evaluate AI, and that's 40 years after I graduated.

George, I hope your son teases you like I used to tease my dad. He was a doc, and when he finished training, you know, I used to say, oh my gosh, well, how many drugs do you have to know? A few hundred. And nowadays, of course, it can be hundreds to thousands, even within a, you know, subspecialty of practice. So I think the answer is actually, yes, we do have to have training for clinicians using these tools. That doesn't mean we have to become necessarily programmers or a data scientist per se, but just like we all learned how to do, you know, critical evidence review back in the day and evidence synthesis, knowing which papers apply and which sources can help you answer questions. I think that's where we have to go with clinicians. Expose them to tools, teach them how to use tools, tell them where the boundaries are for safely using those tools, and have them always retain or maintain their own personal accountability authority using these tools.

100%.

The skills we developed with evidence based medicine are still very relevant.

Yes, basically we need AI literacy education, but specifically, what's the role for professional bodies? So, you know, all the bodies that exist are for every single discipline. There's hundreds and hundreds of them. And also the universities, the medical schools, what role do the professional Bodies and the universities play here in preparing healthcare professionals for this because it's happening now. It's not something that we're going to have to encounter in five or ten years time.

Absolutely, Louise. I think individual clinicians and the educational side of things, if you will, medical school and residency training, fellowship, what have you, specialty training necessarily now will include generative AI and related tools in the armamentarium. So just like I need to learn physiology pharmacology, I need to learn about generative AI and decision support to be sure. Sure I don't get misled or I don't use tools inappropriately. So I think that's number one. But number two, there's this ongoing exercise. How do we make sure that all the good principles of knowledge management will apply to generative AI? What should be the update frequency, who should be doing updates, how are they done, how are they validated or vetted? And what's the appropriate combination of this structured knowledge and, and unstructured knowledge of generative AI? So the professional societies, that's a very good question. On the one hand, they certainly have been producers of guidelines, therefore they're producing content to train generative AI. Whether or not the same professional societies want to go back and then look at a generative AI tool and said, well, how do you do with congestive heart failure on these cases and are you abiding by the guideline or is there erroneous or, you know, hallucination occurring?

A couple of decades ago the BMJ published a number of articles around how to read a paper, how to read an rct. I guess there may be an opportunity and Blackford, you've got some association there. How to use an LLM, how to define an LLM. I look forward to those articles in BMJ written by Blackford Middleton.

George, I might need your help, Louise.

Okay, much needed, but it is an issue. A lot of the professional bodies, this is so hard for them and I get it, but whether they need to set up up committees and maybe if you can afford to pay people to be on these committees so you can get the work done quicker rather than relying on volunteer assistance. Because it concerns me greatly that a lot of the professional societies are either not saying anything publicly because they don't know what to say and I think we need to support them or what they are saying is hey guys, this is way too experimental, don't do anything yet. It's really mostly radiology that seems to be, as a professional society sense leading this and putting out guidance guidelines and I think there's a lot of work to be done to actually support these professional bodies, that this is a new world for them. Have you seen other examples, Blackford, of organisations that are doing that well, or do you feel they still need a lot of support?

Not so much from the clinical professional societies or the academic societies I'm engaged with. Everybody's talking about it for sure. You can't go to a meeting now without 50% or more of talks being about AI and some.

We can't do a podcast episode without talking about AI.

Yeah. So I guess the idea is that I'm committed as a clinician to human in the loop. I think these tools are going to be additive to our clinical armamentarium. They're not replacing the doctor as a patient. I know when I go see the clinician, I want to have a human to talk to. I think the therapeutic effect of hands on and social dialogue and all the rest, the of of it is very important in establishing trust and that's the way I think we could use these tools effectively.

Vinod Kozler told us a few years ago that the next wave of AI is going to replace us. Do you think we will be replaced?

I think of AI in three buckets in particular. One is really the difficult diagnostic dilemmas or differential diagnosis or complicated treatment planning. That's really cognitive support at a level, level that might need to be largely remaining in my hands or supported by an AI tool but never replaced. In the middle bucket there's a whole bunch of procedural stuff. Can I get this patient referred? Can I write the referral note? Can I measure the quality of this clinical encounter or can I measure the quality of this population of patients under my care? All of that middle bucket stuff I think is immense to support by generative AI tools and others that can simplify and reduce that clinician burden on stuff which we don't really need to have to do, but needs to be done on the last bucket. I think there are ideas that are coming to the fore about surveillance. How do we have agents or technologies or bots that are monitoring my patients? Or as a patient, how do I have a bottle lot that's monitoring my own health, observing and suggesting where I could do more? Now there's a lot of this out there today, but I do think we're going to see dramatic improvement in that individual level of understanding for a person and their care as we aggregate data, as we assimilate methods and agents around the individual patient for the benefit of the patient, but also for the benefit of the doc. I will refer people to Robert Pearl's book, ChatGPT MD. Dr. Pearl is a Stanford physician. He's a notable clinician, but he wrote this book with generative AI with ChatGPT and says this is inevitable. We really can't fight this. We just have to learn and prepare and get ready and help equip clinicians and equip patients with the insights, the knowledge and the methods to. To use these kinds of tools safely and effectively. And I think that's labeling the data, labeling what is AI Transparency, as we've discussed training and ultimately getting the question right.

Brilliant. What a great place to end as well. Blackford, you've been a fabulous guest. I can imagine everyone listening to this as they're going on their walks, doing their laundry, driving in their car, and they'll probably be pressing pause and going, what did he just say? Like, this has been really fascinating. Thank you so much your time today.

It's been my pleasure, George. Louise, I look forward to our next rendezvous.

Fantastic. Thank you very much. Blackford.

George. Now, do you know this word? Because Blackford used this word twice when we were talking to him and I must say, it was a whole new word. I was looking it up on my phone. Armamentarium, which means the medicines, equipment and techniques available to a medical practitioner. Practitioner. I'd never heard of that word before. Is that something that you doctors know?

That's part of that secret doctor's business that we keep hidden from everyone. We like to either use ancient Latin or ancient Greek words to make sure it sounds like we know what we're talking about. And Blackford really knows what he's talking about. I mean, that was an amazing interview.

Oh, he does. Oh, so good. He talked about his, you know, failing at being retired and he's actually listed on LinkedIn with nine concurrent appointments. Plus he also does that subscription stack as well as, you know, other things.

Actually, I've known him as long as I've known you. I met him in 2004 at MedInfo as well. Amazing guy. He actually invited me back then to go visit his lab over in Boston, which I did and been following his career ever since. When I did the Harvard Technology in Healthcare, he was one of our lecturers.

Oh yeah, you did that course at Harvard.

I highly recommend people keep their eyes on Blackford Middleton. Definitely one of the gurus of digital health.

Yes, I loved his interview. We talked with Blackford and it's come up on the POD before about, well, how do you know if the AI AI is Good enough or not to apply in healthcare. Well, that problem might hopefully soon be resolved because I see that in the US a group of health systems led by Boston's Mass General Brigham, they've launched something called the Healthcare AI Challenge Collaborative. And what they're doing is they're allowing participating clinicians to test the latest AI offerings in simulated clinical settings. So basically they'll be directly comparing AI products and there's going to be a competition and they're going to produce public rankings of these commercial tools apparently by the end of the year. The idea of having health systems that actually review and publicly produce the results, these comparisons of AI products, that could be something really of value and be fascinating to see what they come up with.

The plethora of tools now becoming available is quite exciting. And actually having real life testing by real life people in real life environments is critical for us to make sure we go forward. So I'll be watching that with.

Interesting.

The fact that there's actually now a dedicated AI in medicine podcast publication and training course over in Boston suggests that that's something we need to keep our eye on.

I know that'll be great. And George, just finally before we wrap up, I'm excited I get to see you in person in a couple of days time.

Yes, yes. We've moved from the virtual world to the real world as we descend on Melbourne for the AI Care Conference. Really looking forward to that. Whilst AI has been the big buzzword of this year, the reality is it's going to become a core part of what we do. So, looking forward to meeting you and Emily and hundreds of our friends who are into AI in healthcare in Melbourne later this week.

Yeah, I look forward to it. All right, George, well, until then, can you please read us out?

Thank you for listening to Pulse, brought to you by Pulse IT News. Pulse. This is produced by Ivan Durich of Octopod Productions. Thanks to Nigel Chartres, Emily Mara and Kate McDonnell for contributing to today's episode. Subscribe, follow and rate Pulse wherever you get your podcasts. You can subscribe to Pulse IT magazine at www.pulseit.news and you can send your feedback and questions to PulsePodulseIT news. And don't forget to rate us on your podcast service.

Absolutely.