Mastering Medical Device Testing with Priyank Soni

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way O the Test Illumination Testing podcast Guiding testers with our nation awesomeness from ancient realms to modern days they lead the way. Oh, the Testicle Ovation Testing podcast.

Hey, want to learn more about testing medical devices? If so, you're in the right place because we have the honor of having Priyank join us. He's an Expert with over 20 years of experience driving innovation in the medical device industry. He's led the development of cutting edge solutions and cardiac mapping, blood recovery systems, lab automation and infusion pumps. Well, he knows his stuff. He's also an expert in embedded systems, AI, ML and software test automation with a strong track record of transforming manual processes into scalable cloud based test automation frameworks. Really excited to have him on the show. We don't have a lot on this topic. You don't want to miss it. Check it out. Hey, Prank, welcome to the show.

Hey, Joe. Hey, good morning. How are you?

Great, great. Great to see you. Great to have you. I guess the first question is, I always ask people, how did you get into software testing?

Software testing, very interesting thing actually happened. I actually started in consumer electronics working on printers. How do you compare, you know, comparison between different PDF files, looking at, you know, software, which is postscript and going into like C, working with, you know, embedded systems. And at one point I was developing, you know, pretty in depth code that was doing a lot of reading and writing, a lot of like threading Mutex and a lot of communication between different modules. And the project manager said, how will you test this? How will you ensure that this always works? And that actually was my starting journey to go deeper into what is actually a requirement. How do you ensure that it meets the expected need of the user? And there we go. The magic happened. It's all about unit testing, making sure when you write it works, right, it works according to the intended use, it works in different environment, it works in different input conditions and it meets what the user is expecting. So that was my starting point. And then I said, okay, if this is I want to do, I would better find a place where there's compliance, there is regulation. And that's how I ventured into medical devices where when you write your end of the day, there's somebody out there whom you're treating. You're making sure that you know the device works. And it could be your friends, it could be one day your family, who knows, right? And that's what is basically, you know, driving me every single Day. How do you produce a software that is going through a lot of compliance, that is going through a lot of checks and balances and ensuring that when it's used, you know, it's always safe and precise?

Well, I love that because a lot of people, I think when they hear a compliance and regulation they're like, I'm out. That's too much pressure. How do you deal with that pressure? I assume there's a lot of regulations you have to deal with. You have to be on top of your game to know like maybe what is expected and not slack. Like how do you keep up your quality with what's expected of you From a compliance and regulatory type of framework, a mindset.

It always goes with the curiosity, like how much curious, you know, to about the processes, how much you're willing to go beyond your regular day to day job and understanding compliance from, you know, different regulatory bodies. How much are you actually reading about what's happening around you, looking into, you know, concepts of software development, life cycles that start always with user needs, how much aware about, you know, how these devices will be used in the field in different settings. So that basically drives a lot of retrospective, right? How actually you started where you are and where you want to be with respect to medical safety. So it all starts with the risk hazard analysis. You ensure that you are trying to develop a software with certain constraints, right? You are actually just not working in silo. You might be working with a lot of different third party tools. You might be working with connected disconnected systems. You might be working with real time system where you're getting a lot of, you know, information and this whole like flow has to follow certain kind of, you know, checks and balances.

All right, so that's another scary thing. You mentioned risk and all these connected systems. How do you identify risk? How do you know you're testing what really is going to cause could cause issues rather than just testing things just because it's low hanging fruit.

It always goes back to the architecture, right? Always go back to the user needs. How well you have defined your user needs, how well you have defined your system and software requirements. And it all goes back to, you know, like how are you building your architecture? Because everything is big from day one in the architecture and then it comes to you know, detailed software design specifications. So if somebody, let's say if somebody is writing a piece of code just for the ui, right? Yes, in the end of the day it's a user interface. But there's a lot of usability engineering skills have to Be applied intuitive. The design is how less confusing the design is. And in this precious system, when we are in actual hospital setting, things are moving really fast. When you're in R and D, oh, you know, I want to click this button, you know, I'll wait, you know, I'll see what's happening, I'll test my requirement. I know exactly the latency and how it's going to work. But in real system, things are very different. Sometimes, you know, you have laurel moving parts, you have all of these connected systems and that's where you know, you have all of these Emotional intelligence has to be applied when you're trying to, you know, even test things, right? How do you trust your team, how you bring innovation and at the same time, you know, processes. Because if you don't have mature process, that means the teams might be making decisions based on their own, you know, understanding. But as a leader, right, you have to really understand what are the regulatory bodies looking for. It depends also on what kind of devices you're working on. There are devices which are class one, class two and class three. So there are different level of risk involved. And when it comes to risk, you have to really understand the software can go wrong. In that case, how do you have risk control measures, what is low, medium and high, and what kind of architecture details are there that can safeguard against these risks. So if you think about any blood, you know, infusion system, or you know, a system that has a pump, there are many things that you have to consider, like you have a speed, you have a certain drug that goes to a certain, you know, level of flow. You need to understand, like air bubbles. So there are so many things that you have to consider as part of risk profile. So you could perform, you know, process risk analysis. You can perform, you know, the cybersecurity risk analysis. You have to also do a lot of these threat modeling to go into this direction of the software that we write nowadays is just not, you know, a piece of library that is just written by you. It interacted, interacts with a lot of different, you know, software of unknown providence. You know, it interacts with a lot of third parties. How are you ensuring to understand, you know, what are the risks associated with these third party software? Have you done your due diligence to either just ask for open bugs that could exist in these external tools if you have the access to the code? Why don't you perform static analysis? Why don't we perform some kind of unit testing and there are a lot of these other kind of things. You can do is you can do more of unit integration testing to ensure that when this package is delivered in a medical device, it's just not testing your own code, but actually going beyond looking into what is the entire package.

Absolutely. I guess you mentioned architecture a lot. I guess maybe I was, I used to do medical software and the software I was working on, I literally, the tech was like 30 years old. And so there was a lot of legacy systems involved. So if you don't have say in the architecture at that point, it is what it is. So how do you work around that then? With the risk?

Yeah, it's a great question. So when you're working in legacy, let's say it was a startup, you kind of was rushing and now, let's say now it's kind of a mature company. Right. But it still goes back the same fundamentals, right. Have you evolved as a company, have you evolved as a team to kind of work, take a step back, look into if do we have robust architecture which involves sequence diagrams, which understands all of this component based, you know, dependencies, like what kind of dependency we have internally? How do you basically have a framework which is written in a way that could be then decomposed into smaller units? Because if you're building a house, right, what do you do first? First is architecture. You basically you have a foundation, right? If you don't have a strong foundation, then how will you build your kitchen? Right? How will you build your basement? So it all comes down to understanding a bigger picture. What is your vertical layer? What is your horizontal layer? What kind of different components will play with each other? Do you have a robust class diagram? Do you have a robust timing sequence diagrams? Because now you're talking about a lot of APIs, a lot of microservices, and now it's not like a huge monolithic code. You're moving all towards more of smaller units. So there are like pros and cons. Like, yes, you don't want a huge monolithic architecture, rather you want piece by piece that can talk to each other. Right? But at the same time, when you have a lot of these interdependencies, how do you ensure your data accuracy is there? Because end of the day, whatever you show on the ui, whatever the user sees, there's a hell of things that goes behind the scene. So that's where as testers we have to just not just look at the ui, you know, look at, you know, overall, you know, the bigger picture, but rather go deeper into the architecture, the design, and then basically that Gives you more mature, more robust test which is kind of like challenging the design and architecture.

Absolutely. So I'll ask this question later, but let's go to this other train of thought. You know, in the pre show you mentioned something about, you know, using ISO 13485 and like all these other kind of standards. How important are these standards and how do you comply with these standards? How do you. Because I tried looking at some of these standards, I'm like, I don't know. I would feed it through ChatGPT now and ask her to summarize it for me. But how do you understand what they're actually asking? Because once again, when I was in medical, a lot of times they went overboard with fda, like we have to do this, this and this and you really didn't. They almost impeded progress because they use that as like you can't do it because we're a medical device. So, so how do you comply with the spirit of these standards but not get almost like shackled or kill your innovation with these standards?

Great question. Again, when you're working with any like innovative mind, right. The main thing they will ask you is I want to get this done right, I know what I'm doing, I know I trust myself, I trust my experience. But end of the day, what we really need to understand is in medical devices, if you have done best of the best job, but if you can't show evidence, if you can't meet the guidelines, if you can't meet the standards, then there are chances that you have to take a step back and redo things. So it's better first to understand the compliance, the processes and talking about, let's talk about IEC 62, 304 which is software development lifecycle. It basically guides not just a testing team or a software development team, but the entire crew who's working hard on developing these innovative products. What is basically the V model. So it all starts with user needs. How do you ensure that the user need is meeting the requirements? You ensure that you have device or software validation which ensures that you're calling your users to basically try your software and medical device in a production level like setting. Then it comes to your detailed design requirements. So these processes that always starts design and development planning. How are you planning your day to day activities? Are you agile, are you waterfall? And then it comes to design inputs. So there is a lot that goes into design inputs, These are interviews that you perform, voice of customers. You try to understand competition, how you can be bettering your design inputs and then it comes is your design implementation. So again, going back to architecture, design implementation within implementation, ensure that you're doing enough unit testing. Because if you don't have these units well tested, you will see bugs down the line. Then it comes to, you know, okay, am I doing my job to do a thorough test of my coding standards? Because now what we are writing is huge. We are also now using generative AI to kind of like feed some of these, you know, inputs or kind of like examples from generative AI and putting into your code that there is a lot of, you know, these different input that comes into your design implementation. So how do you ensure like you have robust, like CI CD pipelines and then finally you have design verification. So that's kind of more of like very formal process where you have a really clean test, you know, design, you have clean approved requirements and everything is looking kind of like, you know, you know, we are now into this formal phase of verification. Once, you know, the test team says, hey, I'm done with verification. I was able to meet all my system and software requirement. But still the job is not done. You still need to go through a robust design validation, which might require like a lot of preclinical studies, which requires people to come in like physicians. You might have end users. It could be also like, you know, your employees working side by side with physician to come in and try the system to ensure that it meets the user needs. And the final end game is design transfer. How do you design, transfer the design to your manufacturing team and then it's a different ball game. You know, when you're launching a product and you need to make sure that there's a co team that's this team working hard to make sure that the launch process is smooth. So it's all about how you're managing your risk from day one until you launch the product.

So if you're using these kind of standards in this kind of environment, can you really do Agile or CICD or. Because sometimes I see companies try to implement what other companies are doing, but it's a totally different context. Are you doing like a version of Agile? And then you have to do verification before it goes in production? Like, how does that work?

Agile is again, like a lot of people come from, you know, I've seen many examples, you know, people coming from consumer electronics or automotive, and then they come into medical devices say, boy, this is so much documentation, so much of process, right? I need to change one line of code and it takes like four days to test. Come on, like you Know I want to push my code, but yep, you have to be careful like when you are treating patient, right. You have to make sure that it's compliant. So agile is a pretty broad term and it all depends to your global compliance and processes within a certain company. I have seen examples where it's hybrid, you basically are working waterfall model, you're working through your requirements and then eventually you will test the code bit later in the game. But what we are seeing now is, is the speed, how speed, you know, how fast you can put your devices into market without compromising the, you know, the security, the safety and the precision of the devices. So the day to day work could be, you know, you are going into agile with certain level of, you know, confidence that you have a pretty robust backlog which is well thought, which is well reviewed to make sure that when a Scrum master is going through these user stories there's actually traceability back to a requirement. And every day there is a lot of testing happening in many different product teams because you need to have a pretty robust cloud infrastructure because otherwise it's kind of like backbone, right. If you don't have healthy virtual machines that are running 24 by 7 and it's able to kick off the bills automatically. Right. So what we am seeing is there's a lot of these interdependencies between manual testing and automation where it goes hands in hand like you need still critical thinking and as a part of Agile you're ensuring that each user story, each bug fix is actually going through a robust layer of testing. It could be a mix of manual testing and automation. End of day it's all about continuous integration. Am I actually pushing my code to the main line continuously to making sure that it's still meeting my high level requirements. So end of the day what happens is like you have spent let's say 20 sprints on testing all of the user stories, but this one cash share. How do you show this all to your users? Because users always come sometimes end life cycle. There are many innovative ways of doing it, right. You can have, you know, you can have your product basically running in a virtual machine as well. Again, just for R and D purposes, just for getting the feedbacks. And you can have like these people connect remotely anywhere, right. And give you the feedback and that goes to the next Sprint itself, right? Okay. I worked on this feature look good in R and D. Okay. Does my physician like or not? Is my end user giving me feedback or not? So that goes into direct, into design change and finally, you know, when all of this, you know, agile Sprint based testing is done, you need to pass which basically means that a certain TV goes into design verification, which is a traditional method. I got my test cases approved, looking good, everything is signed off. I'll spend like a month, let's say you know, running all different manual test cases, ensuring the automation is working and then how do you basically show your traceability? And that's where I'm seeing a lot of innovation, where you know those back in day, you know, doing all of this Excel sheet, figuring out, exporting data from one test management system, looking into some other like JIRA queries. So that's where you know we are seeing a lot of innovation where how to ensure that when a test case is passing it actually trace back to requirement and end of the day like it just not like software requirement, how does it meet the risk hazard analysis, the task analysis. So it's, that's what I'm seeing. A lot of these, you know, like people like intern, they come and they say wow, this is a lot of work. Can I automate this?

Can you automate it? Because like again back in the day it was really tough. We had some software, we had a map to risk and it was really complicated as we had like BDD on the doing the automation. But you had to map it back to this requirement, this risk and tag things and it was really difficult. So how do you handle that? Do you have any special tools? Because also this is a two part question, any tools for that? And also because you're not just doing UI testing, are there any other types of tools that you use that people may not be so familiar with?

So yeah, let's talk about like how do you connect different systems? Yeah, it all goes back to like do we have a pretty robust SDK for that tool? Are they exposing some APIs? Right. How do you get the data in a safe way? Because end of the day there are like these systems where you have your ip, you have a lot of information. So many times we have seen like teams, they come up with their own sort of framework. I've seen teams developing their own backend node, JS application or microservices and understood all of the APIs, whether it's coming from Jira, whether it's coming from Jama. How do you basically connect these dots? Because there's also like lot of this trust within, like how do you trust, you know, some of these cloud based technologies where you know, do you have enough NTA sign? Do you know exactly if it is Oncome data, is it going beyond the boundaries? Right, so these, all of these, you know, limitations actually gives more, you know, insight into can, can we innovate, can we actually develop on our own? And I have seen like many instances where somebody said, you know what, I'll just develop a Python based framework and give it a try, give it to another team that can actually put into a local self hosted cloud where you have a dashboard, you can see data flowing, you can actually create different releases and actually show the whole traceability from a user need to your requirements to the architecture. Because a lot of these systems off the shelf system will not understand.

The entire tool chain set. Because in many times in medical devices there's a lot of things that you do which is not coming through the tool. Like for example, if you're working on a design equivalency, you might be writing a lot of things on your own that's not coming out of tool. You might be doing some critical thinking, writing how did you qualify certain equivalency. So that is all kind of like disconnected offline systems which I'm seeing trend. You can use now a generative AI to summarize things, but end of the day it is still a summary. It's not an in depth information of what went into like point A to point B.

All right, so you know, you didn't mention AI machine learning. You know, you've been doing this for a while. How much has AI machine learning changed what you do day to day? Or is this still in healthcare? Maybe because you have to follow these best practices and standards, you can't really consume AI ML the way you, you would want to.

Maybe again, AI is a complicated topic, especially in medical devices. If you look at fda, right, they're supporting, but they're cautious, they're ensuring that people understand. Is it audit ready? If you have worked on creating, let's say a computer vision model that can understand your screens, it can do object classification. Have you thought about your data really well? What is the source of your data? What is the variance of the data? Right. And then it comes to, you know, I'm giving example of computer vision because that is something, you know, I can, I can share more insight. And then it comes to, you know, how do you do annotations? Are you trying to like put into like human in loop that can actually approve and reject some of the annotations which are actually generated on its own? Do you have audit trail? Do you have a version control of your data set? Because many times when you're trying to train a model or something which is still in proof of concept, but it should be audit ready, it should be transparent. There should be a lot of insight into how did the AI came to a conclusion. Is there a proof? Is this evidence that you can show to these regulatory bodies? And again, yes, maybe people will not understand the heuristic graphs and data analytics that is happening behind the scene. But if the framework can preserve all of this data that is happening behind the scenes, all the data analysis, all. How does the algorithm works? Right? Because these evidence matters a lot. Just showing, okay, I was able to do object classification, I found my object. Good, show me the proof. Right. How did you ensure your data set is version control? If you're using a platform, does it record all of the analysis that was performed, whether it's proof of concept, whether it was training, whether it was validation? And then when it comes to showing the data, how easily a regulator can understand your report, that also matters a lot. How do you ensure that there is an easy flow of information in your reports? Because many times, you know, when these regulators work on these approvals, there might be, you know, looking at some kind of, you know, equivalency between a traditional way of testing, right? Okay, you got the report, you have evidence of your requirements, you have evidence of your traceability. And then now when you feed in something which is tested through AI, does it have the correlation of, you know, how you performed your traditional testing and does it show me the evidence or not?

I think you just gave me an idea for a business. Maybe you've already thought of it, maybe it's already out there. Maybe create an AI, FDA auditor or AI regulator that you can run against your software and see does it match all the regulations. Is there anything like that? Because I've been hearing a lot about how AI is going to replace everyone in 12 to 24 months. But anytime I see a demo with automation, AI tools, it's always like a web browser, right? It's not like a medical device where all these systems you've talked about that aren't really connected or they are connected in weird ways. Where do you see the future of AI and do you see something like an AI, FDA auditor or something like that to help?

I don't know if it is on the horizon, but again, it's a very tricky and goes back to the trust. And what kind of like false positive can happen when you're doing an AI based audit, right? Because the systems that are still ongoing, under development are still evolving. There are a Lot of pieces which are done traditionally. There are nuances, you know, new way of doing things. And when it comes to, you know, auditing things, you know, you have to go back to the basics, have team followed, you know, IEC 6234, you know, IEC standards regarding 60601, which is for the electrical safety. So again, like it depends if you have that kind of system that can give you a basic input to summarize things, which could be a starting point, but it doesn't give you all the intricacies that go behind the scene, which you have to dig into it, right? You have to really kind of gel and understand from. Again, going back to the user need, going back to the architecture and looking at the evidences. It could be a possibility in future. But at this moment I would say it's still early in the phases where there is no common platform, there are no common, I will say, standard that people are following. And until we reach to a point that we have a certain standard that guides automated audits, that could be something that going to guide these platforms and tools where you have a common set of requirements to do these audits. At this point, audit is definitely, you know, much more in depth that you have to do. And you have to also like ensure that, you know, what is the transparency in your system.

All right, kind of an odd question. This is the season where a lot of people are graduating high school and college. Do you still recommend software testing as a viable career? Especially knowing AI machine learning is about to really take off?

Definitely. I think like you need more testers now, much more than you know, what we were thinking before. Because the world is changing at a very fast pace where the traditional methodologies will still be there, there'll be more innovation, there will be more of software coders using generative AI, more of large language models that is giving summarized results explaining the code. And people are actually moving in this direction of taking help, right? Not copy paste, I would say, but taking help from these AI assistant, which puts lot of pressure on testers like how do you ensure a code actually written by human. Plus, you know, some of the AI assistant is actually working. So that's where you need to still expand the critical thinking because you need this, you know, at the moment response that you can understand from the, from the system. So you need to know like if the system is still performing based on our, you know, user needs or it's still performing based on what was expectation, is it still safe, is it still meeting all of the regulatory compliance regulations or not? And then it comes to automation. Of course like automation has been there for like many, many years. But the transformation we are seeing now is something which have to be careful. Instances where you know the reports that are being generated from automation is sometimes could be huge. It could be thousands of test cases that you have to analyze in a short moment of time. Because when you're in agile you might be under pressure. Hey, give me the results by tomorrow. I want to know if I can check in my code now. If you're trying to use a generative AI model to summarize a report, yes it can give you a high level information but it will not tell you exactly what happened into minute details. So you need all set of tools with caution going back to like mature processes. Have you defined your process? Do you have the risk controls? Do you know like how do you produce an audit ready report or not?

Excellent. Okay, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing or testing medical device efforts at all? And the best way to find or.

Contact you, I would say like be, be honest, be lead with empathy, be curious, look around like what's happening. Read a lot right? Understand the mindset of a developer. The more you understand how a developer is trying to understand the requirement. Are you doing your one on ones with your different cross functional groups. It could be quality, it could be regulatory, it could be a system engineering team. So the tester is basically a dynamic individual that is also not just a person who is coding or meeting the requirements, but it's more about the communication, how you communicate well within R and D, within marketing and also many instances where you have to interact with your end customers. So it's a full package. I would say tester is a full package with lot of versatile skills and emotional intelligence. And if you have to reach out to me, I'm on LinkedIn, you can shoot me a message and we'll have.

A link to all this in the comments down below. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to testguild.com a549 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation podcast. Joe, my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good.

Cheers.

Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers throughout the fourth forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres. The bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.