DeepLearning.AI Learner Community Event ft. Jan Jitse Venselaar

Hello everyone. Welcome to DeepLearning AI's online mini event series for AI for medicines, the learner community. My name is Jani Zvensler and I'm very honored and excited to be your guest speaker today. I'll be speaking for the first 10 minutes and then I'll be answering your questions that you posted on Slack for the rest of, for the rest of the time. And yeah, don't forget to post any comments and questions and upvote questions on Slack. Great. So let me share a little presentation I made. Yes, here we go. So that should be now visible. So. So I'll be talking about using machine learning for monitoring ms, notably sclerosis patients. So first a little bit about myself. So I have a background in pure mathematics. I have PhD on non commutative geometry which has absolutely nothing to do with machine learning for AI. But it was very good to do so. I learned machine learning through Andrew's original book the Encore Sierra and doing some projects mostly well for fun. For example, I tried to predict the outcome of the Eurovision Song Festival which was a challenge because well, it didn't take into account any of the quality of the singing and I did a lot of other courses on deep learning of course providers. Then I worked in the semiconductor industry for four years. I was an engineer mostly using, doing semiconductor metrology with machine learning, developing new ways of. Yeah, nanometer shifts of detecting nanometer shifts of computer chips while they're being made. So that was quite interesting. But then I decided to work for, for another type of. Yeah, another type of business. So I was, I'm working currently on machine learning in healthcare. So using machine learning to, to improve healthcare. So this is very relevant I think for the course event and this is what I'll be talking about now. So yeah, just a small disclaimer. Any opinions and statements are my own and not for any of the companies that I amor was affiliated. Okay, so what do we want to do? So we want to monitor the progression of multiple sclerosis. Ms. So what is it? So Ms. Is a chronic disease where nerve cells in the brain and the spinal cord so the central nervous system are damaged by the outer. It's an autoimmune. So the immune system attacks the central nervous system and that leads to deterioration of the control of the muscles. So at first patients will suddenly drop over and they will have more and more trouble walking and at the. Well it's mostly diagnosed, first diagnosed at people between 20 and 15 years old and it's a very long term disease. So people Live decades with it, but progressively becoming worse. And this is where the machine learning comes. So what happens is that. So the disease has different stages, so there's the progressive relapsing, so it generally becomes worse, but there are some relapses, what are called relapses. Then there is a secondary progressive, so there's a few relapses in between. It's steady. And then it becomes worse and worse again and it becomes, at the end, it becomes progressively. There's a primary progressive and relapsing remitting, which is maybe the most famous one of Ms. So you have an attack, you go back most of the same left, but a little bit worse, and then you have an attack and you go back to the same. So these are different types of stages and they can go from one stage to another. Now what happens is that depending on the stage, different medicines, different medications are provided. So maybe you got a walker during a certain stage, maybe you get a different type of medicine which actually are quite effective at suppressing the symptoms. But it depends on which state you're in, hear what medicines you can get. And so you want to know patients, and so I want to know, where am I, in which stage am I, what kind of things should I do, what shouldn't I do, that sort of thing. So that's very. Yeah, it's very interesting for the patients, the doctors and other. So pharmaceutical companies want to know so they can provide new medicines. So you want to know how this is done. And for this, well, you need sort of continuous monitoring of how the patient is progressing. So what can we do there today? So firstly we can. Well, you can do an MRI scan of the brim. It's expensive. Yeah, you can only do it so often. So what you look at is black or white spots on the brain. So these are called lesions or plaques. They're the areas that are damaged by the immune system. So this is expensive and it's quite a burden of the patient, actually. So it's non invasive, but still you sometimes have to put in a contrast liquid. You have to lie still in this MRI machine for quite some time. You have to go to the hospital if you have trouble walking. This is a big step. There's also the option to ask patients to come to a clinic to do some cognitive tests, to do a walking test, see how far you can walk. This is of course, done and you talk to your doctor. But this takes time and it's only a snapshot. It's only a snapshot in time of how you are doing at that time. So people are looking for some continuous monitoring so in advance where I'm based. So the frequency of these visits for Ms. Patients is around one to two times a year. Doctors try to get around this by having a questionnaire about, hey, did you have any relapses in the past months? But people are very. This is typically very subjective. Yeah. So what can we do otherwise? So this is monitoring. So everybody nowadays has a smartphone. You can try to have some smartphone monitoring yourself, just not anywhere here. But the implementation will. So yeah, this allows for continuous monitoring. It's objective, it's very low burden for the patients and it gives you fast feedback. So you can also say try to change your diet and see how that affects your ability to walk or you can try a new medicine and get a feedback of how it's going within one or two weeks without going to the clinic. So that's, this is very interesting. So what I heard from patients is that they like to monitor themselves so they can see for themselves objectively how they're doing. Are they doing step before or are they not progressing? That's very interesting for them to know. Well, the challenge is of course to make this monitoring good enough. So what are the options here? So you can have a questionnaire like you have in the clinics around you feel, are you tired? This is of course very subjective, but if you do it often enough you can try to see a trend. There's a response test which is called the symbol digit modalities test, the SDMT where you basically you have, you see a symbol on the screen and you see a list of symbols with numbers and then you add, well, you press the right number, you say the right number out. Depending on how fast and accurately do that, you can sort of measure the response, your response time and how tight you are. Then there's a new idea is you use the phone on your, on your, or your, the, the camera on your phone to track how tired. This is very, of course a very interesting idea where a lot of machine learning is involved. I'll not be talking about that today because, well, we saw that it was very promising in lab environments. But typically what we saw is that the cameras, the front facing cameras are on the newer phones are good enough, but on the older phones we saw a lot of patients have newer phones or not good enough. And it's very dependent on the light. For example, if you have glasses and there's a little bit of light falling, reflecting off you, you can't track the Isaac. So that's very. Yeah. And then it becomes a burden to the patients that have to take off their glasses. Look, look around. Yeah, it's a lot of things. So what I'll be talking about is seemingly very simple thing. There's so a walking test. So how, how far can you walk in a certain time? So typically this is done certain step time periods like two minutes or six minutes. There's even a 15 second test but that's too short to be very effective now. So yeah, what I worked on was a two minute walking test. So that's measuring how far you can walk in two minutes. Now it's very simple to do this in principle. So just take the GPS locations of your phone and add them all up. Now you can see that in this picture. Sometimes it's simple and sometimes not so simple. So here you see an example of. Well what probably happened is there was some reflections of the GPS signal on some high buildings, high rises. Now this meant that, well we should be, yeah, we can throw away this measurement. So this is not usable because yeah, two minute walking and then there's a jump of around 20, 30 meters. Well typically we see that people walk around 150 meters in two minutes. If you have a barrel of 20 meters it's unusual. Also one of the other option, one of the other problem is that you need to go outside. So if you're a mass patient that, that can be burdens. You can walk 20 to 50 meters in two minutes. Think about it, going outside is quite, it's quite a fatigue inducing thing. Now also with current lockdown you don't want people to go outside willy nilly. So what we did was the following. So we used machine learning for better monitoring. The idea there is that we use the accelerometers and the gyroscope sensors on the phone. Really a classic ML problem. So we have some input data we know is related to the output data, which is well target data, so how far you walked. And you try to match them with the neural network. So the target data is. It's quite fun to see how this was gathered. So you have people walking with four phones, three people abreast and then a walking wheel there measuring okay, how far are we walking in two minutes? And then all these different phones gathering all the data from left to right pocket. Different types of phones, different types of surfaces, different types of people of course walking different heights. Yeah, all this sort of thing. And then, well you train a machine learning model, predict the distance from this on this time series data. So you have quite a lot of data in principle per measurement. So you have two minutes. The phone measures this thing at 100 hertz. So you have, for the gyroscope and the accelerometer you have three dimensions each, so X, Y, Z. So you have six times 12,000 points. Yeah, quite a lot. Quite interesting. Yeah. So let's see. Yeah. So what we did there. Well, in the end, so the network architecture we came up with, so we have the input time series, of course the time series is normalized so the mean subtracted, standardized. And then we had a feature detection layer. So in this case it was. Well, I came up with a 1D contents for this. You could also do short term Fourier transform and then do 2D content on that. Typically we have one or two layers of feature layers. So first 1D content and another one, 1D confnet in my case. And then there was an accumulation layer which was either some simple RNN or LSTM layer or even a normal dense layer would work. And then there was a regression output. So we trained that. So yeah, the machine learning in principle was very simple. You just have to come up with the right network and the right parameters to make this work and of course gather the data, which was actually one of the harder parts to do. And I'll come back to that in a bit. So how did we set up the architecture? Well, mostly so we did everything in TensorFlow because then we could easily put it, all the training also in the cloud. So we use kubeflow for that, which is a product to. Well, it's, it's a, I think a Google project to make, to do easy things easily in the cloud for machine learning. So each of. So this is a typical architecture that you then see. So this is an extract transform mode plus a machine learning, plus a validation step. And each of these is then a separate Docker container where you do this, where there's a small Python script which basically loads in the data, transforms it, does some stuff on it and then writes it out. It takes a few parameters and all these parameters are safe. So this is very important because what I talk about in the end is you have to, You have to validate and you have to show to some medical validation board that what you did is actually that's a very interesting proposition for machine learning because typically, well, we only have to show internally that it works, but not how it works here. You have to show how you came to results and you have to document. So we put everything in Docker containers, put it online, train it there using TensorFlow and then. Well, we, we also use a hyper parameter optimization framework to, to tune hyper parameter. Because if you, especially if you work in the cloud, you can't just all the time upload new Dockers Docker containers. Well, it's easy if you can do that automatically and automatically to your hyperparameter. So I really like it, this Optima framework, there's other parameters, there's. Amazon has a product for that as well, SageMaker something. And also I know there's a hyper parameter optimization Python packages, but these type of things are. Well, they're very simple to set up and they take a lot of the pain out of the tuning of the hyperparameter. You can just run it overnight, of course it'll cost you if you do it in lab, but you can just run it overnight and it comes up with quite a bit reasonable set of hyperparameters. And then for deployment. So the model is putting a Docker container sent to the cloud and then the data from the, from the smartphone app is also sent to the cloud. There the model is applied, the feedback is sent back to the user stored also in the cloud and on, on the phone so that the user, the patient can see directly. And so how far did I walk today? How far did I walk yesterday? Am I progressing? Am I regressing? That sort of thing. And also the general practitioner, the doctor can also see. The doctor typically has a dashboard that they can then look at if they see the patient coming in. So we could do last six months very well. Didn't seem to get any worse. So that's, that's good news. Yeah. So this is the, the general setup. So some of the challenges we faced. Oh, I have to say I didn't do this all by myself, of course. So my, my colleagues and especially I want to call out Amkar Leila. Setting this all up. Actually they did most of the work, I would say. So what are the sort of main challenges? First, from a machine learning point of view, actually even with this setup where we had three people walking at the same time with four phones each, we have their own little training actually. So we, I think in the end we had like 16. Well, no, we had a round 400 individual. So 400 training time series. And we had to put a little quite a bit aside for, for validation for the medical. They set up other rules then for machine learning. So we solved that to using a lot of augmentation. So we had a lot of jitter. So we added some noise on top of the time Series data, we rotated. So we did. So one problem was that on phones, well, you put your phone in one way, you put your phone one way in the pocket and maybe somebody else puts it in the other way. You have to, you can simulate that. You can also simulate all other kinds of rotations. Well, rotation through time sounds a bit amazing, but what I, what I mean here is that. So you, you cut off somewhere you have the, the 120,000 or the 12,000 points, you cut them say midway and then you, so you cut them up midway and you glue the first part to the last one so that, that should give you the same output. So it's a, it's a valid augmentation. And in that way we extended the amount of tr. Actually we also found that it's hard to get the relevant training. So of course when a healthy person is walking for two minutes, typically they walk a lot further than the IMS function. So we had, at some point we realized this and so we started walking slow to get some more data from slow walkers. But the question was how, yeah, how representative is this data for messages? So that's still a bit of a look at the question because it turns out that it's. Well, there are all kinds of legal considerations if you want to gather data from patients. I'll talk about things in the end. So what we did is we rebalanced the training set using oversampling one standard techniques. But yeah, sometimes simple techniques are very effective for this question. So then we looked at, we bucketed the walk, it walked distance into buckets like 0 to 40 meters, 40 to 80 meters, 80 to 120 meters. And then we picked a lot of, well, we over sampled from the shorter distances so they as well represented as the longer distances in the. And then also yeah, there was a lot of non uniformity in the data. So especially if you gather the data on an iPhone or Android phone, that's quite a bit of difference in how everything is measured. So yeah, that's was a bit of a problem. So yeah, we luckily found a library, a software library called Sensing Page, which can be used to at least get it in the same format. And then also what we saw is that we just have to normalize the data so it was exactly, well, 12,000 samples. So we either cut it off or we extend it with zeros. And that was yeah, solved enough. And then yeah, we, we had many different arguments. There's of course an infinite number of network architectures to consider and there was not much literature on this type of problem. So using a time series data for this sort of. For regression. So in the end what we did was we took an iteration from. There's a, there was a paper on the deep sense network which is gauge detection. So how people walking, sitting, running that sort of thing based on this type of data. So it was a classification problem. And then in the end we just, well we took some information from that type of network and then we made a digital aggression problem. And that was also when the hyper tuning. Okay, they very helpful. This deck can also train different types of networks at the same time. So finally, want to talk a bit about what are the challenges if you're working with medical data instead of just well any other data set. So if you're, let's say my example would be semiconductor indices, you're looking at chip data. So it's not. Yeah, medical data is a bit special sometimes because well, I don't have a medical background and you have to hey, what is, what is medically relevant? And I don't know, I can have an idea. But that's not what doctors want to see typically. So you need to work with the clinical experts, with the doctors to say hey we, we can measure this. Is this important to you? We could also, I mean we could have measured the maximum speed that each person would but typically doctors were yeah a bit conservative. So they want to try test method. In this case the two minute walking test was very relevant. You also have to see what is allowed. So you can't just gather all data from all patients and then often will say okay, let's try. Can we detect this? Can we detect that? It's yeah, that, that's a, that's a. Yeah, very typically not allowed. So you're not allowed to go on a phishing application data mining to see what you can do with your data. If the data is from persons, you have to tell them beforehand. I'm going to do this and this with your data and then you're not allowed to do anything else. And it also has to be a usable app. So you have to talk to UI expert experts to see Emma, what I'm proposing is this actually good for their patients. So you can't just tell them to go running patients, you can't tell them to run. That's just dangerous. And they're also not because they have, they have problems controlling the muscles. You can't ask them to do very fine control of the app. So you have to work. So some terms that I, that I learned over the past few months when I was working on this was, Was what is the minimum technical change? So, so what can we detect actually? And, and with what certainty can we detect the change? So for example, if you one day you detect a 50 meter walk and the next day you see that the patient is doing a 14 walk according to the model, is that a significant change? You need to be able to quantify it because otherwise. So the doctor is not going to do that for you. They don't know how your model works, how accurate and how precise your model is. So you should be able to tell that to the patient. To do the doctor, is this actually a significant change? But for that to know, you also need to know what is the minimum clinically relevant change. So when is a doctor getting worked? Because maybe 50 meters, 40 meters the next day. That's just a normal variation of a person. So one day you are there, maybe it's raining, maybe you feel a bit tired, maybe you had some leg problem that day. Extra. It's just a normal change, maybe some. So you should be able to detect that as well. So what is a normal variation? And from what point onwards do I need to start worrying? And of course it would be very. It's actually a requirement that the minimum detectable change should be lower than the minimum clinically relative change, otherwise your model is not going to tell you much. Useful. So if you have a minimum clinically relevant change of let's say 10 meters, but the minimum detectable change is 20 meters, you're only going to see very large changes. And actually you don't want that. And then. So I'm based in Europe, so there's all kinds of rules, CE validation rules. In order to advertise your app for medical purpose, you need to have a whole bunch of rules that you apply. So for example, you need to, you, you. You need to show documentation that, that your app works as intended and is not dangerous. You need to. Yeah, every change you make in your app, for example, if you just change gem1 parameters because you think it will be better, you need to update your documentation. And, and these rules change also. So there was going to be a big rule change on 26 May this year, but luckily that's now postponed because of the crisis. And these rules, they're really. Well, I found them really hard to follow, but luckily there's a lot of experts on that. But these experts don't always understand machine, so they try to. But they're a bit scared of having. Okay, you have a black box model. So, so what is it detecting and how can I trust it? So for that thing, you need typically together a lot of data. Yeah. And then so the data gathering. So patients are typically protected group. So you can't just say, I'm going to find 100 patients and give them this app. That's just not a lot. At least the other. Okay, I see I already talked for quite some time and I say there were some questions. So I see one from Asia Goldschmidt. So this. Well, if I understand correctly, the training set is people walking in a certain. A certain distance and the accelerometer data accumulate. Why not instead use the glitchy GPS data together with the accelerometer to correct the glitches using nml? That's a very good question. I think it's a very worthwhile approach. But. So I worked on this now for the past four months. We had originally a deadline of the 26th of May, and we thought our first approach using just the accelerometer data was the most promising one to get this working in time. I think stitching together GPS and ML data is. I think that accelerometer data is a very promising thing. I think it could work. Yeah. One thing is that the GPS data and the accelerometer data is. Is from a. It has a different time, step, time, frequency. So the GPS data is typically once per second. The accelerometer data is, is. It has a frequency of 100 hertz, so it's 100 times a second. So that's a bit. You have to stitch them together in a proper way. So there's a question from Nefadita by la. I hope I pronounce it correctly, but otherwise. Sorry. So how did you do external validation of your model? What exactly is the regression output? What information is fed back to the patient? So the first one is the. The external validation is we had a group of measurements from people that we didn't consider in the training. So we had a group of test persons that did a lot of walking and we validated the outcome of the model for them with the actual reasons walk. So we were very clear that this was data from test persons that were not seen. And what exactly is the regression out? So the regression output was meters walked in two minutes. So we had a. The target data and this is also the output of the regression model and this is the information that was fed back to the patient and the gp. So we were not in general making statements on how the patient was doing. We were just saying, okay, on this day you walked this amount and by way our position is what we say or minimum detectable change, when is it? The significant change is this amount. But that was all we were saying. So really, it's really important here to, to have a human look at this data and say, okay, I walked very slow that day, but that was because it was raining or I was very tired that day. I walked very fast that day but that was because I was walking with a friend and I had to keep up. So it's very important to have a human in the loop. Yeah. Then there's a question from David. So how did you make the switch from the semiconductor to the medical image analysis? Yeah, that was a very. So they're very far apart in some way. So yeah, I was, I didn't do much. So it was both actually. I see it was both machine learning just in different fields and I was lucky to come into a company which had a lot of expertise on the medical regulations, a lot of expertise on the medical stuff and some machine learning experience. So there I was able to get up to speed relatively quickly by just talking. Okay, what is this medical. What is different in medical compared to. Yeah, I wasn't. Yeah, actually, yeah, it was not much. So I didn't have to do much, just apply to a few jobs and find the right, right fit. I guess I was very lucky at that time. So there's a question from earlier question actually, let's not forget that from Satvik Seracity. So that was. Hi, how can. Exactly. Can I use a neural network? Ah yes, how can I use a neural network? Item code for drug discovery, for example, how can a neural network discover drugs and how does the process work? So pardon me if this question is simple. Well actually I don't think it's a very simple question. So I'm not an expert here, but I think it's. Yeah, so let me try to venture a guess. So there are two things here. What I heard is that. So one thing is. So there's a lot of machine learning and AI going on now to faster see how proteins are folding and see if we can speed that up using AI machine learning. That's a very interesting approach. What I think is on the short term simpler to do and might also be big gates look at a list of already relevant or regulated medicines, so medicines that are known to be safe, at least in certain doses and see if they have other effects that can be used for new diseases. And there's a lot of AI work I think ongoing. So for example, scanning the literature for relevant Effects for example, see all the side effects. Maybe one of the side effects is actually a good one or seeing. Okay, similar medicine here, similar structure here was deemed to be useful and we have this medicine that actually is already. Yeah. Cleared to enter the market for a different time. Well then it's relatively easy to see that medicine test it does. Medicine also works for this. So I think AI and machine learning is very useful in that regard for the, for the protein folding. I think it's more a substitute right now or in addition to brute force computer. And that's one part and I don't know if there's an algorithm code that does this or I'm not an expert here. So there's a question by Lei. So is this kind of like anomaly detection of time series? Well, yes and no. So anomaly detection I would. Is well on time series data. Yes but what we are looking at is, is the whole of the time series data and then attach a number to that. I think typically anomaly detection looks at intervals and then see if those intervals are similar to what you've seen before. Yes or no. So it's more. Yeah. Anomaly detection doesn't look at the whole time series as a whole typically. And this is a Russian problem, not a classification problem. And, and actually anomaly detection is, is I think typically much harder because the problem with anomaly detection is that you, you typically have very little training data for the anomalies hopefully. So that doesn't. There are a lot of extra tricks involved I think in an all detection. But yeah, so processing the time saver data of course is. Yeah, I see that Greg Casey has a very interesting talk about the very interesting link about the testing using to improve or find new drugs. Okay, I think I'll. So it's. Well, I think I'll answer one more. One last question and then. Yeah, so this is my guide Sanka. So thank you for your presentation. What are future improvements in cases model meets a expected time. So possible future improvement that I think. Well of course you want the model to be more precise, ever more precise but at a certain point you're meeting the target so you should look into that I think making the model more flexible. So so now it was really focused on the two minute walk but maybe for another group of patients you want a three minute, six minute walk. You want to go to a type of model that is more fine grains. For example, okay, this was a distance in 2 minute walk but also the top speed was 5 meters per second, that sort of thing. So more output parameters and a more flexible that will be the future movements that will be nicest to have. I hope that that answers your question. I'll be looking at the Slack channel to see if I can do some questions there. Ask yeah, answer them later. And yeah, thank you for joining all today. I hope you enjoyed it. Funny. Very exciting. So yeah, for the questions that I didn't have time to answer, I'll leave comments on Slick. If there are more questions, please ask them. And the recording of today's event will be available on YouTube under the same link if you want to rewatch it later. And yeah, thanks for watching and looking forward to see you at future events. Bye.