How Generative AI is Elevating Testing Quality in Healthcare with Richard Kedziora

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way O the Test Illumination Testing podcast Guiding testers with our nation awesomeness. From ancient realms to modern days, they lead the way. Oh, the Testicle Automation Testing podcast.

Hey, do you want to learn more about how generative AI can really elevate your testing quality in healthcare and AI in general? Well, you're in for a special treat because we have RJ joining us who's the co founder of Astender Solutions, a leader in custom software and data analysis for healthcare. With over 30 years of experience in designing and deploying software software projects. He's passionate about creating cost effective solutions that really improve patient outcomes. And he also holds multiple degrees and is a frequent speaker and published author. As you could tell, Richard brings a bunch of wealth of information and knowledge and healthcare tech, especially to the show. So you don't want to miss it. Listen up, check it out. Hey rj, welcome to the Guild.

Thanks for having me. I have to live up to that now for sure. The challenge is on. Challenge accepted.

All right, so it seems like AI really is your sweet spot, like at a whole, at a high level. So how did you get into healthcare and then how did you get so passionate, it seems about AI in healthcare?

The thing I always find interesting is people seem to forget that AI has been around for a long time or maybe just gen AI sort of brought it into the common culture. I was fortunate in high school, through the, you know, 80s that I was involved in technology, had the computer, was gonna go to college for that, to get that PhD in artificial intelligence, but then took a job and you know, started earning a living, but always had my hands in IT and developing expert systems over the years and dabbled in multiple fields. Railroad car scheduling was, was intriguing. I used to work for US Steel way way back when in the 90s. But I got to healthcare through some consulting that I was doing pharmaceuticals work for a lot of the pharmaceuticals. I live in the Philadelphia area and it's called the Route 202 Carter. All their headquarters are right there. So it was very easy to work for a bunch of those different companies over the years as a consultant and then trying to figure out where I wanted to go in life and career and make an impact on the world. And healthcare really was it. You know, I'm not a doctor, I don't play one on tv. But I understand technology and that technology is about people. And so I help now. You know, Stenda, my company, which we founded though 20 odd years ago with a good friend, Drew Lewis, we can develop systems that help lots of people. You know, a doctor, nurses, they do incredible things, but It's n of 1. It's one person making a difference. And they do make a difference. But I can help develop those systems then that are impacting more and more people, which is just an amazing feeling when you go to sleep at night.

I love it. So as you're developing software for these type of companies, do you keep quality in mind or testing? Is that part of a, a big part of what you all do or is that how does that work and how does it specific to health care? How maybe it's more critical than maybe other, like a simple application that's just a web website or something.

It testing and quality is at the heart of everything we do. For just what you. The reason you just said that it is about people, it's about their health and wellness. And our typical project is not billing. So it is important to get the billing right. But that's not our typical project. We do focus more on what I think of the innovative patient side of things, dealing with data, particularly as it comes off of a medical device. So blood pressure, blood glucose information, your heart rate, taking that information, marrying it together with the medical record information, your labs and medication to be able to make recommendations. You know, as we work with startups and it's like, oh, we have all this data, let's find patterns and show them to the doctor. Well, you need to do a little bit more than that these days because they don't have a lot of time. So testing and the quality of those applications is paramount. And so Astenda is ISO 13485 certified, which is a standard that says we have a good software development process, that it's well documented, SOP driven, template driven, checklist driven, and you know, as we do code reviews, you know, it's like, let's make sure. We were asking the questions each time to make sure that we are putting out a quality product. And then even beyond the actual software testing, the algorithm testing, then things usually go into a clinical trial. And so we have PhDs on staff that can help design those clinical trials to make sure that they do work in the real world.

So I used to work for a huge healthcare company and I always thought like, this is awful because it's really hard to innovate. I don't know if it's because it was an enterprise or because it's so regulated, like, why even, like how hard is it to then create Software that is innovative and fills everyone's concerns about getting audited and things like that.

And we get audited all the time. We do self audits and we do, we have independent third parties that come in and audit us, customers, audit us. So it always makes it interesting, but that keeps us honest. You know, it's like, okay, we're gonna go through this process, we're gonna do it right. And we do have these conversations with various different, particularly with the startups that are trying to just get into healthcare and they're like, oh wait, this is what we need to do. And it's like, yes, you know, you need to have this documented process. You can't just start writing software and see what happens because there is that level of scrutiny and that it can impact the health of people. So there are different levels of that. Obviously some systems that are, you know, not directly impacting patient health, there's less of a threshold there. But yeah, it does make a difference and it doesn't squash innovation. You just have to be aware of this upfront as you're developing these solutions and thinking about it. The fda, like if you need to go that far and you're going to get, need FDA approval, as you sell your particular product, whatever you're putting on a market, they're your friend, they are there to help you to make sure that you're doing this right, that you're not surprised. So if you do think you're going down this road, you know, talk to the fda, talk to, you know, there's lots of consultants out there obviously that can help you understand that journey, us included. And so just make sure that you're aware of what you need to do. But yeah, it's, I wouldn't say it's squashing innovation by any means. You just have to be aware.

Absolutely. I always think this area is also ripe for disruption, especially with AI. So do you see AI being almost like taking us to a new level, a new era of healthcare maybe where we can get insights more cost effective healthcare? Or is that pie in the sky? It's still bureaucracy and you know, roadblocks.

We are starting to see it already. So healthcare is the last industry to really embrace data. There's a lot of technology in healthcare, the MRIs, the imaging machines, X rays, incredible technology from that perspective. But it's the last industry to really embrace data. And now that we have the medical record systems out there that are starting to capture more and more information and all of the wearables that are on the market that there's just the amount of damage. Yep, here you go. We all have. The amount of information that's available now to the professionals is just opening up the world of possibilities. And in that note of being cautious in healthcare, in the early use cases are around note taking and summarizing content. So low risk. You know, as you go into any project, it, we always, okay, what is the risk of this project? What is going to be the impact of it? And then that drives your, your overall strategy. So, you know, if I'm making, you know, clinical decision algorithms to, you know, treat cancer, that's one thing. Or, you know, if you're just, okay, here's your steps for the day. You know, it's different thresholds of what you have to worry about and think about.

Nice. So are there any specific use cases that you've seen generative AI being applied to healthcare? For example, back in the day, we used to have to do translations into different language. We had an Excel sheet. We'd go to an offshore company that knew the language and say, this is how it should look. And then we go look at the application, make sure it matches. It seems like AI would be very easy to just say, take this ui, translate into this language, and boom, I have the correct translation. Or is that easy? And that's already been done for a while.

So the biggest earliest use case that's getting quickly picked up, I referenced a second ago, but the idea of ambient listening. So doctors, nurses, healthcare professionals don't want to sit there and type on a keyboard and end their notes. It's let the computer technology listen to the conversation in the room to be able to take those notes, to enter the information in the medical record system. So that's quickly becoming, you know, I would think a top use case in medical that's being rolled out faster and faster these days to free up doctors to do what they want to do. There's, I've heard it called pajama time. It's like when you're done seeing patients and you gotta go home after dinner, end of the night, now you're entering all your notes into the system and it's like, oh, you know, go have fun with the family, go out, you know, have fun with friends instead of taking all those notes. Let the power of the technology take that on for you. Low risk again. And then likewise, it's also summary of that information in the emr, which is, which is interesting. You referenced this, the idea of language translation. And I think that is a powerful use case. But as more and more People start looking at what these systems are capable of. You have to think about the training data. And so these systems have gone out and just vacuumed up everything on the Internet. The vast amount of information on the Internet is in English. So there are been various research reports that talk about that language translation into say, Spanish or German. Japanese is not as good as it is in English because of that smaller training dataset. So people are taking this challenge on and creating better and better systems. But that was one of my early excitements. My wife happens to be a nurse manager at a practice and they do bring in language translators. I was like, oh my God, Gen AI, this is going to like solve your problem. You're not, you know, be able to do everything. But that's where you can't rely on it because especially if you don't know Spanish, it's going to sound good to you because you don't understand Spanish. But then that Spanish speaker is going to be like, what are you talking about? Because it's going to lose those nuances unless you're that native speaker or have the ability to translate between the languages. I'm not going to pick him up. I'm not going to know Spanish. I'm like, hey, he's doing a great job. But, you know, might not be. So you do have to be cautious of that. We're in early days still. It's funny to think of, you know, but we are in early days. It's been maybe two years, you know, since Gen AI sort of hit the market, but we're still testing and still trying out and figuring out what it's good at.

How do you avoid over reliance then on AI? Oh, that's all powerful oracle. I'm going to apply it. It's translation beautiful. I'm sure it's an AI, it's an LLM. It's been trained. I'm rolling with.

Is it's. I think of it as an intern. Think of it as, you know, that first year person. You know, it's very capable, but it also sounds very trustworthy, sounds very authoritative. When it just spits out something, it's very easy to be like, oh, okay, it's right. No, you have to, you know, go back and double check it. You know, we do use it a lot. I, you know, huge advocate for it, but it's an 80% solution. Never just sit here and be like, oh yeah, okay. That said X, Y and Z. There's various stories where lawyers have gotten in trouble. You know, different professionals have gotten in trouble because you know, they just took it at face value and didn't cross check the information because you know, even you know, in the case of the lawyer, like made up case references and the lawyer didn't go, you know, check that case reference and to make sure it was real. So you have to be careful of it from that, that perspective. Same thing in the field of medicine. There's story of a couple months ago, it's like how many rocks a day do you eat? Oh, you eat three rocks a day to be healthy.

You and I know, you know, you're not going to eat rocks but you know, insert other food item that's not as good for you, you know, stickers, candy bars and you know, maybe it doesn't give you quite the right advice. So you do have to be cautious of it and it use it because it's taking over the world. When Oprah had a special on it a couple weeks ago, that's when I knew, okay, we hit a flashpoint. Now Oprah's talking about it.

So you make a good point though. Garbage in, garbage out though, like how many rocks. Is there any suggestions you have for testers to make sure that when they are interact with Genai they are using good prompting techniques to make sure they're getting good output?

Yeah, there's a couple things to think about when you're in Genai systems. One, the hallucinations, is it just making stuff up? So you got to be aware of that as you're testing and how do you mitigate against that? Is there bias in the data that was used? So if you start out with a generic large language model and then you fine tune it and train it, add in your specific information material, you know, so you work for major health system, you know, what information do you have available? You're going to feed that into the model to make it more specific to you. So is it accurately representing that information? So you got to be careful of that as well. So it is, it's the challenge here is it's non deterministic so I can answer the ask the same question and five minutes later you can ask the same question again and you're going to get a different answer. So it's not just testing, it's also monitoring over time to make sure that it is performing as you expect.

Nice. So in the pre show you mentioned healthcare is very high level. There's a lot of different verticals within healthcare. So I assume each other vertical has its own standards, its own regulations. Can we use AI to say hey, we tested this, we're in clinical trials. Does this meet all the standards that we have to adhere to? I don't know. Is that, that's probably too vague, but could it help you do something like that?

This is where that bias in the training data and where you mentioned garbage in, garbage out, but it's even how do you code medical information at your institute? So we've seen examples of this and the FDA has even called this out, that the industry has to be somewhat self account accountable for this, that you can't just take an algorithm that was generated in one hospital and move it into another hospital without testing it. What are the specifics of your population? So if, you know, I live in the suburbs of Philadelphia and it's like there's a very specific demographic of people that live here. So now I take that and move it down south and there's a different demographic or I move it out west or Canada or Mexico or Japan, there's different demographics of people. So does the same algorithms that were developed here work for those populations? Probably not. Which is why you need to test and as you're developing them, make sure that you are using data set that is representative of, you know, a wider population.

So do you get pushback about using AI, especially with data because of HIPAA compliance, like leaking real patient information for test data somehow or having these LLMs being trained on it and all of a sudden you could maybe spit out a real patient data when you're prompting it. Who knows?

That is, in my opinion, one of the overblown concerns really.

Okay.

Of that actually happening. I'm not saying it's never gonna happen, but it's an, it's an overblown concern that, you know, if I'm a doctor and I put in some information that somebody else somehow prompts us, and you know, RJ Kaziora, male, in his 50s, you know, comes up with some specific information and then that person knows who I am and can be able to do anything with that, you know, I would not enter that specific information. So you can use the systems, you know, if a physician and you have a specific question, you can, you don't have to put my name in, you know, to be able to do that. You don't have to put an identifier in there to be able to get meaningful feedback from it. With hipaa, you know, if you're a healthcare provider, you need, and you know, we sign business associate agreements all the time, then that makes us accountable for the use of this information as well. The large language models out there, these corporations are now signing the business associate agreements. They weren't early on, but they are signing them now and be able to like hold your data separately. So that's not incorporated into the model to address these concerns. So, you know, as they're rolling out new models, you know, make sure you know, what does your business associate agreement actually cover? Does it cover, you know, the use of, you know, the current model, the model that just came out last week? Probably not, but you do have to be aware of it and think of these. It's all about risk is what it is.

So when you're developing software yourself for all these different clients, do you actually use Genai maybe to create user stories or test scenarios from code or like how do you actually use AI in your day to day type of delivery?

I do, it's even starting as early on. So I've been doing, you know, you think about trying to figure out what your system's going to do, what are the user requirements and you're going to do interviewing, you know, the experts or just the users of your system. And I've been doing these types of interviews for decades and so I can, you know, I, I know I can do them, but by entering the input and my parameters into say ChatGPT or Claude, it's spitting it out a lot faster for me and getting me going, it's that 80%. It's like then I can go and tweak those questions and, and make them different. But it's an efficiency thing. So starting very early on in user interviews, it's like I'm developing a system that's addressing prostate cancer, you know, at a specific institution. If there's a doctor with 30 years of experience, he's written these articles, you can tell it like these journal articles, you know, what questions do I need to ask him to, you know, start getting to, you know, the core of the system that we're going to develop. So doing that writing requirements, it's the same way as you are, you know, record those interviews with all of the different people and then the AI systems are able to extract the meaningful pieces of information from those. Again, it's an efficiency thing, making life a lot easier. So lots of opportunities to use it from that perspective. And then once we do have requirements created, you know, we do a lot of human review, but you can also plug them into the system. And I always love asking the question, what did I forget? What is that other thing that I forget? You know, I referenced our process. It's very SOP Template driven, checklist driven to make sure that we're not forgetting, you know, all the little things that you need to think about when you're developing a system, but use the AIs to help you with that. You know, there's, there's a standard. Was it ISO25000? I think it's 25,000. Where it talks about, you know, look at scalability and usability and stuff like that. So as I'm prompting, writing those prompts, it's like, okay, based on this ISO standard which covers all these things, what am I not remembering to ask kind of thing. And it can put those things together. Well, it's a huge time saver, which is really nice.

Yeah.

But then this, you know, we can continue down the chain. Now I have requirements, now I have, you know, I need to write tests from the software developer perspective, from a QA perspective, or write those tests. I've done an example a couple times now where you have a screenshot, take the screenshot, drop it into one of the, you know, models that has a vision capability, can ingest that image. I was like, write test cases for this. And it, that's where it truly amazed me months ago when I first did this. Like, typically, if you have a data input field and you put a little asterisk next to it, which means, you know, it's a required field. It knew that. And so it was writing test cases that, okay, username is a required field, but then it even helps with security testing, you know, SQL Injection. Oh, you gotta make sure that, you know, SQL Injection isn't a problem. And okay, if you're newer to the field, maybe you don't remember to test SQL Injection or you're not familiar with it. So you can very easily then just be like, oh, wait, what is SQL Injection? How does it work to really help you out from that perspective? And then I'll say probably last is test generation. Particularly in healthcare with the wearables, with the electronic medical record systems, there's so much data that you need to be able to test these systems. It's beautiful for creating data for you.

100% agree. And I love the concept of doing these interviews. It's almost like if you ever heard of behavior driven development, you can create Personas of people based on those interviews and then be able to say, okay, as John, I want to do this, this, and probably come up with some really good ideas so you're not being replaced. It's almost like it's getting you more ideas of like, okay, how can I Make sure I'm creating test cases for all these different things that Persona John is gonna do.

Oh, absolutely. And that's the beauty of it because it does help build that creativity, that brainstorming. It's like, okay, talking to John, who's a surgeon. Generate me 10 questions. Okay, generate me 10 more questions. Generate 10 more questions. Generate 10 More. And it will keep generating more questions for you. You know, there will be, you know, repetition in that, but it just gets the creative juices flowing and reduces your time to get to that point where you have a good set of questions.

And also, I like the point about security testing and things you probably could say, hey, what kind of tests am I missing? And it may say, hey, you're missing security testing. And if you don't know, like you said, you could say, okay, give me some examples of some good security tests that I can run against this specific application. For sure.

Yeah. And it's fascinating. I did, because we do, you know, healthcare and you know, work in the wearable data kind of thing. Yeah, I happen to do triathlon. So much, very much focused on heart rate and different things like that and have been able to say, okay, I, you know, I have a person that's out of shape and 60 years old and you know, generate me a heart rate profile of a person running a mile, you know, this, and it's like, okay. And it generates that data. It's like for every minute, here's, you know, specific it and it starts out, you know, say at 90 beats for a minute and it goes up and, and then, you know, towards the end of the run it comes back down and now I have a nice profile of this person's heart rate over time. And I was like, okay, generate me the same data for a 30 year old athlete who's in really good shape. And it just amazes me because then it drops the heart rate down and it's like it doesn't. Because it doesn't understand heart rate, it doesn't understand exercise parameters or age or anything like that. But it was able to generate a good representative data set that differentiated between an out of shape 60 year old and a, you know, a good, good in shape athlete in their, you know, 20s or 30s. Like it differentiated those and created. So it's very meaningful to be able to help test systems.

No, it's interesting. I know someone that wants to get into running and he had a coach that slowly eased into him with a plan like this. We do this. Day one, day two, day three. Almost sounds like you can Use this as a coach as well for pretty.

Much yes, you had.

Right, Nice. So does it with multimodal coming out, do you find it like sometimes you could feed it like an image of a screen and say, hey, can you help me from a user's perspective? Can you tell me how my user would interact with this or anything? I should test to make sure my user is going to have a good experience. Do you get that broad?

Yeah, we do. Because partially it started out as just this curiosity. It was like, will this actually work? And it does. It's fascinating. You know, it'll be able to differentiate the fields. And I think it's a key thing that if it's not interpreting it correctly, what is that telling you? Is it well designed? Do we have to reconsider how we're using or positioning a particular element? If it didn't recognize it, maybe it is still a good design and the AI just couldn't, you know, interpret it properly. But it should cause you to ask that question like, okay, the AI system can't handle this. What is a human going to do with it?

100%. So RJ, for your company itself, maybe a little background what you all do. So if someone's listening, they're like, I'm trying to develop a healthcare application. Maybe I need a little help here with my AI strategy or development strategy. Is that something you would help them with or could help them with?

Yeah, Astunda is a professional services company. So we don't sell products. We help other companies develop their products, develop strategies and we do, we mer work, work with a mix of like large corporations, government agencies, particularly the R and D department. If we work with like health systems, it's going to be with like the PhD researchers. It's usually very, you know, cutting edge, which is where our, our sweet spot is. We might not quite know what you want to do. We're going to help you develop this system and how it's going to operate. We've had a couple of clients over the years put our name on patents because we've been so much part of that process of creation. Like, okay, yeah, let's put their name on it. We never own anything. You as the client always own everything. But it lets to explore lots of things. And as we have those AI conversations where today a lot of conversations start, one of the first early questions is always around, what's your data strategy? What's your quality strategy like? And some companies, particularly more on the startup sides, they don't have that figured out yet. And you're not going to get very far if you don't have your data strategy. If you don't have a quality strategy in place. As you start thinking about AI, it starts with the data. Garbage in, garbage out. You said it.

Yeah, for sure. So what's cutting edge? You consider cutting edge? Is there anything you see on the horizon, like, yeah, gen AI, I get it, blah, blah, blah. But is there something you're like, okay, people need to be aware of this because this is about to take off, or anything you've been messing around with in the lab.

The big aspects of where all this is going I think is one, the intersection of wearables. Because more and more of these devices are coming out, they're more and more capable, generating more and more data. So it's the intersection of the wearables and the AI to interpret that data and give it back to the person. It's getting care out of the four walls of the hospital. So me as an individual or even as the population ages, there's this concept of care at home, you know, so as my parents are getting elderly, fortunately they're still doing pretty good. But you don't want to have to move them into an assisted living home. They're very comfortable in the home where they live. So how can we apply these tools, the technology, to understand their living environment? That's where a lot of this is going, beyond the four walls of the hospital. So if you know, your mother got out of bed this morning, that she used the facilities, she brushed her teeth, she opened the refrigerator, she turned the coffee on, she turned the TV on. Okay, she's having a good normal day, you know, but. And you're not sitting there with cameras in the room like observing her. And so she doesn't feel like she's being observed or, you know, you're protecting her privacy a little bit more at home where she's comfortable. And then one day it's like, oh, okay, wait a minute, it's nine o', clock, she's usually up at six in the morning. She didn't get out of bed, she didn't turn on the cop, she didn't turn on, you know, let me pick up the phone. They're like, hey, mom, how you doing? Very simple use case of that technology, which is where this is going.

All right, so this might be a little wild, but have you seen the clips of the Tesla robot? And how close are we that to like caregiving where not only can we wear wearables, but also we can use some sort of robotic Assistant to, you know, almost. I don't have kids so I'm betting on or hoping on a these robots to really do that thing in 20, 25 years.

The technology can help you. Yeah, I'm deeply fascinated, particularly by what, you know, Alon did recently kind of thing. It does turn out that there was a little sort of magic behind the scenes in what he was showing and it wasn't as upfront as it looked like on those videos. But to his point, the reason he was doing and the reason he was comfortable, he's like, look at how far we've already come. This is around the corner. So it really is. And Japan as a culture has really embraced this. So they are at the point where they do have like robots that are in the home to help care. You know, they're not totally autonomous kind of thing, just running around like the Jetsons or anything like that by the at this point. But yeah, look to Japan if you're really curious to where this tech is going and what it's, what it's becoming capable of. Japan is really embracing, you know, technology. But yeah, it's coming. It's not quite right around the corner, but it's not that far down the road to having that robot to provide care at home. And you have to, you think about it, you know, self driving cars. It's like okay, a lot of people are self scared or apprehensive of self driving cars. You don't have control anymore. But what's better at driving in the car? You or that computer that's in the car and it's the computer that's better at driving 100%.

Okay, RJ, before we go, is there one piece of actual advice you can give to someone to help them with their AI testing efforts? And what's the best way to find contact or learn more about Stenda?

So to get a hold of us, you know our website ascendegra.com est n d a.com my LinkedIn always, always works. It is RJ kids. The aura love to have these conversations. It just always fascinates me what people are coming up with and thoughts. My one piece of advice is use it and it's that simple. But you need to do a little learning and education and there was this idea around crafting the right prompt and okay, how do I write a better prompt? Well, one of the fascinating things is just ask ChatGPT or Claude or any of these solutions to write me a better prompt. You don't need to become a prompt expert anymore. Ask it to help you, ask it to ask you questions to get to you know a better answer. If you're not sure what to ask it or how to use it, ask it to help you. And it will, which is one of those things that's amazing. And lastly, I would say it's not going to replace you, but people using it are going to replace you. So get to know it.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to testguild.com a521 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. Joe my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences. Together. Let's push the boundaries of what we can achieve.

Contesting Podcast with lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.