Scientist Stories: Phillipp Mann, AI products for Healthcare

Hi everyone, welcome to the joint Heidelberg AI and Data Science seminar event at DKFZ. And today we're gonna— we're excited to have Philipp Mann here, and he's gonna tell us about how to bring AI solutions to the patient. And I think that's a very interesting topic for this very researchy Heidelberg environment that we usually have here. Because research most of the time or many times ends at the publication. And so we're very excited today to hear about how the path continues and actually to create real life impact. And so Philipp Mann did his PhD at the Heidelberg University for MRI imaging for radiotherapy, and he's still working part-time at DKFZ as a postdoc. Postdoc. So he has research experience, but he also helped this AI-using Berlin-based startup with their product to really bring it into production. So he knows both sides, and that's why we're very interested in hearing what he has to say. So yeah, please welcome Philipp.

So hello, thanks. Paul for this very nice introduction. I would start my screen sharing now. Okay, I hope you can see the start of the presentation. So my, as you already said, Paul, my name is Philipp Mann. I'm working part-time at the DKFZ and also for a company called Medihare. And I just wanted to show you that developing algorithms is a It's a nice thing and it's very important to do, but bringing it to the customer into an application is a totally different story. So starting very shortly, short introduction of the company I'm working in. The company is called Medihare, was founded in 2018. The first prototype of the software we are distributing in the brain was created in April 2018. And then, and it was the first big step and very important step, the CE mark was then available roughly, yeah, like 10 months or 9 months later. And then we had the launch of our second product version another 9 months later and are now on the third product version. So bringing AI solutions to the patient. So you have an idea, you want to create an AI solution, and you want to make it applicable in the clinical routine. The problem is it's not a direct way. It's not like one arrow that you need to address. There are like a couple of things you really need to take into account when you want to successfully launch a product that is then used, accepted, and supported in clinical routine. So those are one of, like, the most important steps that we identified that needs to be solved and to have such a product in the clinical routine, starting with the medical needs, then developing a strategy, how to, how to, like, to analyze the customers, developing marketing, marketing strategy in parallel, and also at the beginning slowly developing are choosing the right algorithms for the problems you want to solve. That's a topic that you widely— that all of you probably know pretty well. The data collection problem that is for AI algorithms. So where do you get the data from? How do you feed your data into the algorithms? Then, and especially for a smaller startup like we are, getting really, really good people It's not that easy because the really good people, they get a lot of offers from big AI firms like Google or like Apple, and they have a salary that we, as a smaller startup, it's hard to pay. So then we have to find different ways of motivating new employees joining our company. Then you have the interface between the algorithm and the final user. So it's the user experience process. Then when this is all analyzed and when you have like a base that covers all the questions, then you have the product, like the minimal valuable product that it's worth showing to the customers, that it's worth testing with the customers. Then you have to prove the product. And for me, coming from a physics department and like more technical perspective, that part is really, really— yeah, it's a lot of work, a lot of documentary where you have to work together with a lot of different institutions to get your product certified. Then once this is realized, you need evidence that your product is actually giving a benefit, that the product is working properly. And then you need to convince the users on one hand, and you need to convince the payers. So larger institutions, they, they usually don't only have the users, they have different so-called stakeholders. That are responsible for— one is responsible for distribution of products, one is responsible for buying the products, and so on. And this is not where it ends. It's basically a feedback loop where you continuously have to improve the product that you offer. So starting with the medical need, so when you think about the following product. So you have an LP, there was a, there was a need, people wanted to listen to music, so you have the LP player, so that worked. Then there was a second development, so they are like a, like a further update basically, and that's the CD, the compact disc. So everybody still uses it, it gets least, or less and less important, but at the time it was launched People used it and people still use it, so that worked. So when you think about the product itself, it became smaller, it became more handy, and the storage increased for each, let's say, product version. So it was an obvious thing to develop the MiniDisc. I don't know who of you had a MiniDisc player. I was 100% sure that this is the way to go. That MiniDisc is the most practical thing and that it's really valuable to have such a thing. But at the end, it turned out it did not work at all. So I think in 2011, they stopped the development of the MiniDisc, and people really did not buy that. It was not a success story. What is the reason? People were not thinking— or one of the reasons is people were not thinking out of the box. So they were continuously thinking, okay, it's better to have like a smaller product with more storage. But then there was the MP3 player coming, and what the MP3 player offered was it was small, it was way more practical to get, to get the music, to get the music into, into the device, into the device you're using, and it became cheaper and cheaper and cheaper. So here the idea was, on one hand, was what's okay to think that a smaller, smaller disk or smaller device it's worth developing, but at the end, the parallel development of the MP3 player kind of managed to make the MIDI disk development stop. So what you always have to do at the beginning, make sure that you develop a problem that is worth solved, that you develop a product that is worth being solved by a software. So what we did at the first step, we evaluated— so we had a lot of background in MRI imaging, and we saw that the amount of images being acquired through the years continuously increasing. And it turned out that the data volume exponentially grows while the ability to analyze such data, which is usually done by the radiologist, is continuous, like it's not increasing that fast. So this kind of information gap is potentially a way or like a hooking point to develop a product that can close that gap. And this is what people did. So for example, we have another company called Zebra Medical Vision What they do, they identify something on images of like flagging images where the image data that is acquired is different from a standard patient. So flagging that there might be something wrong, helping the radiologist to focus on the really important stuff. Then there's another company that's focusing segmentation of stroke and perfusion imaging. That increases better decision-making before surgical interventions. The product we developed or the product we are offering, it's focusing on multiple sclerosis and helping radiologists to improve their efficiency. You can basically say that the MS diagnostics is based on MRI imaging and on these images you see like small white dots that need to be counted, that need to be located. So in what kind of area of the brain are they located? And they also need to be volumetrized, meaning that the size of these little dots also needs to be evaluated, especially in the longitudinal evaluation. So if you have an evaluation on one time point, and then, I don't know, like 6 months later you have another evaluation or another MRI imaging, and then you compare those and you can see how these, how the MS progresses. So medical need is identified, improving the workflows, making diagnostics more reliable, then you need to develop a strategy. So what do you need to take into account? So first of all, You need to make sure that there's a medical need. This is what we already had. Then you need to find out the potential impact of your solution. So what does your, your solution impact have on them, like on the medical or in the financial field for this, for the specific customers you're addressing? Then defining or having the first idea of the minimal valuable product, the MVP, with the, with the essential features. Then the resources you need, you need to collect them. Then data sourcing strategy needs to be developed. Where do you get this data from? Scientific versus commercial product. Do you want to offer it more in a scientific way, or do you really want to make it a scalable product that needs regulatory approval? Market sizing, market targeting. Even if you have a product that is perfectly developed and addresses a special need, if the market size is really, really small or the people you are targeting don't see the value in this product, that doesn't really, that doesn't really, yeah, help you with selling the product. So who is the potential customer? And then the financing strategy to finally develop a business plan that is then being presented. As a next step, you need to find the correct algorithms. So you are the expert in this, in machine learning. I'm not really an expert in that. I know a little bit about this, but you need a team that knows which kind of problems that need to be solved, which kind of machine learning algorithms you use. And for our case, which is basically the detection and the quantification of imaging attributes. We chose a convolutional network-like unit architecture, and if you have questions or more specific questions on that, my colleague Thomas will be available for questions after the presentation regarding this. So the next point, once you acquire the algorithms You need to make sure that you have a data collection. So where do you get the data from and what kind of data you have? Annotated versus non-annotated data. What is the level of annotation? You could have a radiologist that is a senior on the senior level that does the annotation for you. You have a radiologist still in training, so the quality of these annotations differ. So you make sure that you get the best out of that, what you have, and how much data you need to have an algorithm that is working in the environment you are placing the algorithm in. And this is really, really an important point because what I know from research, you have or you develop something that perfectly works well in this environment, you are testing your algorithm. But when you want to scale it, if you have different people that use your algorithm, if you have different images that are being created, that gives an entire another level to the quality of the algorithms you are developing because you have to handle a lot of different kind of datasets and the different images. So the potential sources, what we did, we generated them by our own. We generated them together with our customers. So this is— that turned out to be a nice way to get annotated data, that you also integrate the customer into your data source acquisition process. Then also by scientific collaborations, you can acquire some data, or you get data by sources from commercial providers. People and talents. I, yeah, I should be told you about this. If you have all these huge companies that, that want to acquire the best of the best of the best talents in the field of artificial intelligence, they, they have a lot of resources available and they have a lot of power to convince people to join them. So as a As a smaller startup, you have to give the talents another way of identifying with your company that they are motivated to join you. So what we basically did, we give them a lot of responsibility at the very beginning and a lot of flexibility that they basically can work kind of as like they used to work before in the Vicious environment. But this is, this is, yeah, this is a very, very important point that you have to be aware of once you decide probably maybe to start your own startup, that you need to make sure that you have people that do the work that are well educated to solve the problems you want to address with the product. So, and then a topic that is, I mean, it's not mostly ignored, but people tend to not focus that much on that part, on the user experience progress process. So this is, I mean, that's a really nice visualization of, yeah, what kind of problems you can face if you if you are not focusing that much on the user experience. So on one hand, you have the user that wants to show you how he thinks an interaction with your software is perfectly. Then you have the product leader who maybe understands it in a different way as it displayed here. Then you have a designer, then you have the programmer who worked on the product. And then at the— or not at the end, but in between steps, you have the business consultant that is— who basically sells and describes the product that is wanted by the customer. Then you have on one hand, you have the cost of the development. And at the end, it maybe turns out that the customer really did want to have something completely different. So what I want to say with this is it's extremely important to continuously make sure that the product, how it is being developed, and the product, how it— or the interface between the user and the algorithm itself, that you make sure that you continuously get feedback if you are on the right track. Because if you have different people that are responsible for for making the product happen, you can end up with completely different things if you are not making sure that you have a very sufficient and efficient feedback loop. Also the usability, you could have, for example, here you could develop matches, but if you're not making sure that you have enough space for the fire to not hit your fingers and hit the product, that basically solves the purpose because it creates fire, but at the end no one uses it because it's— I mean, you don't want to burn your hands. Or like this, if you— if, yeah, that's, that's, that's another stupid example. It's often not a bad idea to limit the amount of choices you have. So giving a huge degree of flexibility to to the user, sometimes it's also frustrating. So making sure that only the relevant choices are being made by the users because the user, the UX design and the user interface, that's basically the bridge between the user and the product and it needs to be perfectly designed to make sure that the user wants to continuously use the product. So what we had at the beginning, it was a very, yeah, let's say a very basic sketch of what we developed of the product. And the second version was a little bit more, yeah, a little more details, better visualization, also in combination with that direct interaction of the algorithm itself. And then in the third version, two-pager where we continuously integrated the feedback of our customers. For example, what we didn't have on the screen is that most radiologists, they do the diagnosis and they report in a dark environment. So if they switch to a brighter interaction platform, what we had here, it kind of interrupts their flow. So switching from a bright environment here to the dark mode of our user experience platform, that created a huge benefit for them. They were really, really happy with that. And it was just like switching colors, and that significantly increased the user satisfaction. But To know this and to get these information, you need somehow to continuously question your product and also to get feedback from the customers. So here, there was like a regular survey we perform where we get input of all the functionality the algorithm currently has. How well it works, what they want to have improved, how it is integrated in the daily routine, and so on. And based on this feedback, you could basically continuously improve the product you are designing. So the minimum valuable product, which is, yeah, which will then, based on the previous findings, then be developed. It doesn't make sense to create a product that is completely finished or which has a basic functionality first at the end. So if you have, if you want to create a car, it doesn't make sense to provide the customer wheels, and it doesn't also make sense to have like 2 wheels or 4 wheels if there's no engine in it. So what you basically do is you start from from a really easy product that the customer already can use and is like not 100% happy with, but it gives him an idea of what the product can do and he continuously sees a progress. So he will be continuously satisfied. And the good thing is you're giving the customer the chance to shape the product. So here in that case, And the upper case, you were probably thinking that the customer at the end wants to have a, wants to have a regular car with a top roof. But in the end, maybe he doesn't want to have a, like a, like a regular car. He wants to have a convertible where he can enjoy the sun. And this is only possible when you do a stepwise product, stepwise product development. So I think this is a really nice example. What you see here, so it's an onion sorting machine. It does what it should do. It sorts the onions. It's probably not the most advanced technique, but it serves the purpose. So that when you see that customers are happy with that, you can finally end up with a professional onion sorting machine. Like, like this one here. So start with the minimum and develop close to the customer. And for that, you need to have a good user experience, experience, and a market research strategy. Because if you develop something that nobody wants to have, nobody wants to buy it. And yeah, the little bit drawback for medical products is that the product development itself starts with the documentation. Yeah, like with a thorough documentation right from the very beginning, because a medical environment, you need to have an approved product to being able to scale it or to scale it throughout the customers. So, and the approving process itself, I will not go much deep into that and just give you a rough overview. It took us around 9 months to do that, and we were working closely together with companies that were specialized in getting products certified in Europe, so for the CE mark. And this is an important step. To be aware how much, A, how much work it is, B, it also costs a lot of money. And the earlier you start to get all, or like the earlier you start to get information on how this certification process works, the better it is because if you once have a perfectly working algorithm and then you want to certify it, this is, yeah, this is a lot of work because then you kind of have to have the possibility that you basically have to start not all over again, but that you have to redesign your algorithm that is then conformed with the, yeah, with the approving authorities. And depending on what kind of product you have, the burdens for the certifications differ. So in our case, we are basically providing the customer quantitative information that gives them additional input to do its diagnosis. So the medical product class is— I think it's still Class 1, but once you are also providing info like Once you also have a diagnosis that you put on your— that your algorithm provides, that's a totally different risk class. And you have to be like, you have to make sure that you are aware of what kind of algorithms you develop so that you know in advance how the approval process works and what is required for that. So then you have to approve products, and surely you need to show evidence. In best case, you have already performed some publications of your basic, like, of the algorithms you previously approved. And also make sure that you have publications throughout the, like, once your algorithm is sold, either with corporations, with other scientific collaborations, Or also with customers that are motivated to show how the daily routine changes throughout using your product. What you should avoid is— it's a kind of a nice example in Germany. I don't know if you already know that, but there's a famous commercial where the best research team found out that the best research or that the best toothbrush is the best toothbrush. So providing publications that are— that is only based on your own data, meaning that you use your— like, that you use not publicly available datasets where people can benchmark their own algorithms against your algorithms, and you only have internal research that you show is not the best way of showing evidence in publications. So what you need to do is to identify what kind of questions you want to answer with the publications. So for example, one important question that came up at some point with our algorithm is, okay, if our software really increasing the efficiency. This is definitely worth executing, but you need to find partners, you need to have a good study design, and to finally show that. For that, you need the external partnering, and the best— or like, yeah, for that you need external partnering. Then sometimes people do publications by themselves using a product That is, that is a good case because it's completely independent. And then sometimes people ask you also to, to write reviews of your product or, yeah, like to address more, more general topics in white papers. And for that you need a publication plan. So what kind of corporations you need to, to solve or to, to address these questions, and you need a disciplined execution. So at the very beginning, what we basically did, we used publicly available data that we tested with our algorithm and we benchmarked it with existing algorithms that are currently used in the research environment, which gives you a first impression on how well the algorithm works and where maybe some problems of the algorithm. So then this— the evidence, the approved product, the minimum viable product, the user experience process— this is basically the ground base to convince potential users of the product. So what you need to do is, to convince users, you need to focus on the core messages. So basically providing them with a— like, with the design of your unit architecture doesn't have the radiologist to get the key messages or to convince them. So in our case, the key messages are that it is really simple. It is basically integrated in the current workflow because from a personal experience, every radiologist who needs to change his current workflow, this is like an additional barrier to use a product. It needs to be fast so that he doesn't have to wait forever for the results. And especially in Germany, it needs to be safe. So thinking about cloud computing and storing the data on the evil cloud is a complete no-go. So this needs to be addressed. And surely it needs to be somehow rewarding what you're offering. So those key messages they work in our case, but surely depending on the product you're offering, there may be other key messages that you need to find out to have a high probability that the customer or the user finally uses the product. Then you need to have a multichannel marketing strategy where you use different marketing sources where you use different marketing strategies to address your potential customers, divided in inbound commercial, more or less publications, LinkedIn, websites, or outbound where you go outside and do some public, like do some work in congresses, or you have ads that you present, or you do an email marketing, or yeah, cold calls that are not always That's fun, to be honest, talking to radiologists. And then you have your so-called sales funnel where you start with a lot of, uh, with a lot of people like in the, in the, in the lead section. Then you have some, some customers that are classified as qualified leads when they show first interest. Then they go into the test phase where they test the product, and then at the end You have the customer experience where you also have to make sure that you continuously satisfy the needs of your product, of your customer. And especially in AI, it turns out that for a commercial application, it is extremely important to avoid the black box. You need to have a possibility to explain what you're doing. So the the buzzword explainable AI. I'm pretty sure all of you have heard about it. And what our approach was is basically this. You have from the workflow, you have your images that are being acquired and they are automatically sent to our software tool, which generates reports that are provided to the medical doctor But then if he wants to check what the software has done, there's an internal product structure where you could basically see what the algorithms did. So in our case, this segmentation of these small white dots, like the MS lesions, he has the possibility to see the segmentation in a viewer And if he doesn't agree with the segmentation, he can change the segmentation by removing or adding some structures by one click or two clicks. And this is actually, turns out to be a nice tool because at the same time he could provide us with these modified calculations. Only surely with the patient consent, but by this we also have the possibility to acquire annotated data that is then fed back into the algorithms to continuously improve it. Yeah, so that was the convincing of the users, but this is not the hardest part. Actually, a really, really hard part is to convince the payers and to convince everyone who is part of the decision progress. So for this, you need to early make sure that you know how the decision or the command line basically is in this institution. So for example, for smaller institutions, when they decide to develop or to integrate a new product, that is based on only two radiologists, then for example the initiator, the user, and the decision maker, the approver, could be the same, same person. And gatekeeper is someone who doesn't really want to have the product, or a gatekeeper is someone who, yeah, who basically interrupts the decision progress or is an additional burden for you to have an efficient decision progress. And in larger, larger departments, for example, in larger research institutions or university clinics, those can be completely different people. So you need to develop a strategy, kind of like a convincing strategy for every player, for the initiator, for the user, for the decision maker, the approver. To make sure that in every sub-step everyone is convinced that the product gives them a benefit, the product is not an additional risk to data security. For example, here, like, a very prominent gatekeeper in our case is the IT department since we are, like, our product is basically installed on site. So we are not working with the cloud. That results in the fact that their local IT needs to provide the hardware, the local IT needs to provide the operating system, and in our case, it's Ubuntu what we are using. So some of them actually have never heard about Ubuntu, some of them, they don't have a lot of experience, and that can take— really, that can take a while. That can sometimes take forever. Up to half a year or even longer to make sure that all these different players are at the end convinced of the product you're using. So my advice, if you're planning on doing something, make sure that you know the processes. Make sure that you know how to convince every player for every single step. So plausibility check. About the value proposition you have towards each, each users. You need to make sure that the arguments you're providing is that it, that, that, that with the product it provides the solution to the problem. It also saves time. That's what the software does. It maybe also increases the revenue or the profit depending on the software you are, you are providing. And most importantly, it generates positive emotions for most of the players that are part of that. So once you went through this, once you went through this process, once you don't stop. What you need is an effective feedback loop that you continuously improve all the different sub-steps that I was just presenting you. So going back, for example, to our first minimal viable product. So once you went all, once you went through all these different points, at some point your customers might say, okay, we really need another feature. So you don't only have the small onions, you need the big onions. Then you need to evaluate, okay, do I need new developers? Do I need to rethink the entire product, or is it just an extension of the old product? What does the extension have an influence on the stability of the product? Do we really need this? So is it only one customer or only 10 or 100 customers that need that? So this needs to be evaluated then if you have this input. Or the customer sometimes is not quite sure if he's using the product the right way. So sure, in that case, that doesn't work. So it's a matter of customer education, continuous customer feedback, making sure that everything works properly. Or sometimes you also have these kind of customers that are complaining that the product isn't working, but at the end it turns out they're using it for a completely different purpose that it's not intended to. And to make sure that every, every customer or that every user at the end is happy with the product, you need to continuously evaluate that. So for example, you can do that by customer surveys. You can acquire usage statistics. So how often do you— do they use the product? Is there at some point a decrease in the usage of the product? Having an automatic failure reporting is really important. That you can proactively do modifications on the product before the customer maybe realizes that something went wrong. Like, for example, continuous updates where, like, is the product working itself, having or generating new image data. For example, to increase your— to increase the quality of your product. So at the end, it is really important that when you want to integrate such a product, like commercially integrate such a product, it's always an iterative process where you need to have extensive feedback loops to develop a good product. Personal advice: especially in the field of AI, avoid the black box. Make sure that the human, like there's a human-machine interface on which extent that is happening, that depends on the application, but just providing them with like, with one number or providing them with a suggestion what they will do is at least in my opinion and at the current state, it's not sufficient. And to make your product available or to make sure that the product will have a broad range of applications or a broad range of users throughout different institutions, you need the right strategy and the multichannel communication strategy. To make sure that people, yeah, know the product, know the product well, and see the benefit in that. So with this, I think I was quite in time. I hope you could get a rough overview on what it means to get a product from our first idea or from a first algorithm to a final commercial product. And I'm open for questions.

Thank you very much, Philipp. First, I want to hand over to Lena to address her questions directly to Philipp. Thanks again for your talk. Yeah, so it's really great to see that some of the machine learning work now ends up in the clinics. And obviously AI is quite new from a regulatory perspective. And I was wondering, with all your experience, What do you think should be changed maybe in the German laws or where, you know, at a more broader level to facilitate bringing AI or let's say machine learning to patients? Is this, you know, are there obstacles that you think are necessary in this domain?

So I'm not an expert on that, but, uh, yeah, I have quite a bit of experience. And like, from a technical perspective, um, you're the expert in that, but the benefit of AI is a system that continually learns. And especially when you think about radiologists, if you, uh, if you have 10 radiologists and they should diagnose a case, you have 11 different opinions. So, um The personal tuning of the software to the need of the radiologist is, I mean, I wouldn't say it's necessary, but it's beneficial for wide application. But the personal tuning of an algorithm would imply that the algorithm continuously learns or that the algorithm has a different structure depending on who is using it. Currently, this cannot be certified because then you would basically need a certification process for every single radiologist. And this is, in my opinion, something that needs to be addressed and a strategy needs to be developed on how to, how to handle such situations. Because just rolling out one algorithm for everyone, it's a good strategy, but this is not where it should end. It should go one step further that the algorithms, like the benefit of AI also ends up at the customer side. And it is, yeah, continuously learning based on the needs of the user.

Thanks. So yeah, so the dynamics, I agree that's an important topic. And I'm not sure if you will reply to this question, but I would be curious if you can share some insights on How much money, say in percentage of the whole budget, typically need to be reserved for the data plus annotation part, say data and data curation?

So in our part, we were quite lucky because we have, yeah, like two radiologists, our founders. So we basically get data for free in that case. We had some, yeah, some, scientific collaborations where we got some data. So from our perspective, we didn't spend huge amounts of money to generate annotated data. And since we also had this interaction between the— like, since we also provided the possibility to the customers to give us annotated data, it was easier for us to have access to this kind of data. So basically what they get is they give us the annotated data, surely with the patient content, and then they have to pay less for the software. So in our case, I don't know the numbers, but it's not a lot we spend. But surely if you want to go on a larger scale and you want to have a lot of data, then you need to spend more money. But yeah, That's not what we had.

Thank you very much again.

Yeah, thank you, Doreen. Thank you, Paul, and thank you, Lena, for the invitation. And yeah, I hope you learned something.

Thank you, Philip.

Yeah, have a good one. Bye-bye.