deeplearning.ai Learner Community Event ft. Roger Smith

So, good evening, Alice. Thank you for inviting me to share my ideas and the work that we've been doing to use AutoML as an assessment tool for robotic surgeons. I'm not going to talk with my face most of the time. I'm going to switch to a set of slides that I prepared for the presentation and we'll start walking through this here. Let me get this started. There we go. So when I started this project, I worked for Advent Health, which is one of the largest hospital systems here in the state of Florida. And then I finished it when I was working independently. And specifically, what we wanted to do was use deep learning algorithms, neural networks, as an assessment tool for surgical education. So I was working at a center where we train practicing surgeons to become robotic surgeons. And so a surgeon who's a urologist or a gynecologist or a general surgeon usually and is practicing minimally invasive surgery or laparoscopic surgery, often wants to add robotic surgery to their portfolio skills. And so they would come to our training center in order to take a one, two, three or four day course to learn specific ability to use these to use this robotic device. The da Vinci robot is the most widely used robot in the world. There are over a million procedures performed with this device every year. Very often a radical prostatectomy or a hysterectomy and various other surgeries, but several hundred thousands of each of those. In order to learn to use the robot, a practicing surgeon usually goes to a very well equipped and specialized facility like the Nicholson center at Advent Health, where I was working to get those skills from people who are equipped to teach them well. During the course of this course, they get assessed multiple times where an instructor or a visiting surgeon would watch their performance and give them feedback and score them on how well they're performing. And that's a very human intensive process. Currently, when a surgeon pretty much of any kind is assessed, you're usually looking at three ways of doing it. Most recently, there are some simulators that use a computerized environment where you perform the skills in a virtual reality exercise. And that simulator is able to measure your performance as you're going along. It's able to measure how many centimeters you moved your hands, how big of a space you operated in, how many seconds it took you to complete. And then it can give you that data back when you finish the exercise. But if you do that same kind of exercise in what's called the dry lab, you're usually working with some kind of synthetic materials or something that looks like this skills board. That you see in front of you sometimes they're called jungle gyms. And in this case, there's not a computer that can assess how well you do the skills. There's a human who watches you and fills out a Likert scale score sheet from 1 to 5 in a number of different areas to assess how well your holding your instruments steady, how much advantage you're taking of your 3D perception of the space, and how often your instruments collide. And then you can do that exact same kind of assessment by asking a person to suture and cut and tie knots, but do it in a real wet lab environment so they have a piece of tissue in front of them, and they're actually doing it in what's called a wet lab. And this tissue could be a cadaver, or it could be an animal, or it could be grocery store meats that have been placed into a box to look like they're an actual human organ. So assessing the first one is kind of automatic, but assessing the second two is very manual intensive. And it's hard to get experienced instructors and experienced surgeons to spend the time to assess as many people who need to be assessed. So when we started looking at whether a neural network could be used to watch a video and assess a person doing this, we looked at the characteristics of these three different visual images. And we noticed that, of course, that when the image is being generated by a computer simulator, the objects, the colors, the textures, and the lighting are almost always identical. Every time you start that exercise, it looks exactly the same. And then when you're doing the exercise in a dry lab, the objects and colors and texture lighting are usually very similar. And humans have no trouble adjusting to the fact that the objects may be a little closer, the light may be a little dimmer, or the pegs may be a little different position. But a neural network trying to learn this pattern might pick up on the differences that are there and start scoring the differences rather than scoring the movements. And then if you look at a wet lab, it's even more different. The scenery that you see there, the tissue, is almost always different. Every time you watch a video or perform one of these procedures, the tissue looks different and behaves different. And so when you're training a neural network to understand what's happening, it may be distracted by the changes in the tissue as much as it is the differences in the uses of the instruments. So we could see that it was probably be easier to train a neural network to score a video of a simulator scene and then work our way towards Dry labs and wet labs. And it's not that the simulator is the most valuable place to do this scoring. It's the dry lab where the most value exists in current state surgical education. But we're not ready to tackle that until we know it can do it in a simpler environment. So what we did, we have dozens of these simulators that you see here. They're not the actual da Vinci robot. They're like a flight simulator of that robot. And each one of them contains about 65 different exercises. And some of them look like the dry labs you saw on the last slide, and some of the exercises look like the wet labs. And so we picked two of the exercises that every single surgeon does during the course. The first one is called the suture sponge with the yellow box there in the center. And the second one is called the ring and rail, which is the rail that you move a peg down or a ring down on the right. And we collected videos of surgeons doing those all year long. And then those videos, they had the simulator score attached to them. That's natural from the simulator, but we also had experienced surgeons watch all those videos and assign a Likert score to all of them as if they were assigning it to a dry lab exercise. And that's the database that we're trying to train to. So we're going to take the videos that came from the simulator and the scores that came from the humans and see if we can create an AutoML application that watches those videos and outputs scores that are similar to what the human surgeons assigned. Now, we can also do that and have it match the scores that were assigned by the simulator, but that's a different task. And why we're doing that is more difficult to explain, but we are doing that, too. All right, so this is the simplified version of the Likert score sheet that a surgeon would use to assign performance to this exercise. And it doesn't really matter whether that's a real sponge or a sponge that's in a simulator. The process of scoring it is almost the same. And for each one of these areas, depth perception, bimanual dexterity, efficiency, there's an explanation for what a score of one means and what a score of three means and what a score of five means. But I just took that off because it wasn't visible easily on the screen. And doing these kinds of assessments with these Likert scales has been kind of the accepted standard in assessing surgeons for a couple of decades. And so when we're using this and we're explaining how we're teaching a AI program to make the same kind of assessment. The surgeons are right there aligned with us. They're like, we understand why you're doing that. That's how we're doing it. So you're essentially seeing if you can get AI to do the job I usually do and that I usually hate doing because I'm really trying to spend my time doing surgery not watching videos of students who are trying to get better at it. So the question is, can we take all those videos and feed them into a neural network and get it to output its assessment of the quality of performance? We don't want to know in this experiment. We don't want to know what objects are in the scene. We did that project prior to this, we did that back in 2018, 2019. What we want to do, we want it to watch enough of the video so that it can tell whether this surgeon who's performing it is at the expert level, the intermediate level or the novice level. So can a neural network do that for us Now? We chose to use pre programmed cloud services. Prior to this experiment, I had actually taken every single course that's offered by deep learning AI on Coursera. So we learned all the skills for programming it ourselves and for using transfer learning. If we wanted to just train the top layer of something like vgg. Net or Inception or Nasnet, we knew how to do that, but we didn't want to do that. For one, you can make plenty of mistakes in the, in the Python programming process. And we felt like we'd skip over all that work of making mistakes on the programming level if we used one of the cloud services. So we looked at the cloud services that were available from Google, from IBM, from Amazon and from Microsoft, and all of them had services that could look at still images and tell us what work was in the still images. A couple of them had services that could look at a video clip and guess at what was in the video clip. But the only one that offered a service that you could have it watch a video clip and you could train it to identify your own targets at the time was Google and they call their service autoimmune Video Intelligence. The other services are adding things along those lines, but not quite now. We needed cloud storage. Our videos were about three. All of them together were about 3.5 gigabytes worth of video. We needed cloud computation, so we needed CPU, GPU and if provided TPUs, because at the hospital where I was working, we simply didn't have access to all the computational power that we would have needed to do the training on our own. We could wrangle together a couple of computers, but not the GPUs and the TPUs that we would need. And we could get those by the hour from the cloud services from these companies. And then we needed those cloud services to include pre programmed DNN algorithms like TensorFlow. And we really wanted them, or I guess they almost always do, to have transfer learning and to have AutoML. I think I'll explain what AutoML is in just a minute. But the transfer learning means using the bottom 95 layers of the network and only training the top layer to your custom application. And then of course it had to be able to process videos, not just images. So this is one of the exercises, a short clip of one of the exercises that we trained it to identify. We had 254 videos like this. Each of these videos is on average about two and a half minutes long. That's how long it takes a surgeon to do the whole exercise. And you're just seeing a small part of it here. The video being 150 seconds on average is too long for the neural networks to watch the whole thing and assess performance skills on the whole thing. They typically like to watch a segment that's one, two or three seconds long. And we looked at what you can understand in one or two or three seconds and we didn't feel like you could tell whether a surgeon was good or bad in such a short time. So we took those 254 videos and we sliced them into 10 second clips. And when we watch those 10 second clips, they're long enough that you can see the jitteriness of the hand, the mistakes in inserting the needle, the mistakes in extracting the needle, dropping the needle. You can see the things that make the difference between an expert and a novice. And then we down sampled the videos because, because all of them were captured at HD quality or higher. And so we down sampled them to 720p level. And that if you know from our courses, that's even still much higher resolution than it's needed for the neural networks. We could have gone down even smaller. And then we went through lots of data wrangling. It turns out that all of the videos and the data fields that were captured for each video weren't completely consistent across those 254 videos. And so we had to program a lot of macros in both Windows and in Excel to adjust all of the data so that it was first a standardized format for every single entry in every single Video, and then second, that it was in the format that was ingestible by Google's video Intelligence service. And then when you finish this exercise, a simulator will give you an overall score on your performance that varies between 0 and 100 points. And the human, using the Likert scale that I showed you earlier, will give you a summed Likert score of between 5 and 25 points. So we turned all of those continuous scores into three classification levels. And there are thresholds in the simulator and thresholds in the practice of surgery where they assume that some people are at expert level, some people are intermediate level, and some people are at novice level. And there are scores that define where those break thresholds are. And so we converted whatever the score was that they got in their assessment into an expert, intermediate, or novice classification. So we're actually training the neural network to match the classification rather than the scoring number. And I think we could have broken it into scores like 5 to 10, 10 to 15, 15 to 20, 20 to 25. We could have done those as classes as well, but they're less well recognized in the surgical community than the classes of expert, intermediate, novice. So we wanted, in this first introduction to the medical community, we wanted them to see things they were used to looking at. So after we did all of that work, we had 2,333 video clips to work with, each one of them 10 seconds long or a little less in some cases. And the Google AutoML service, when you load that in, you can manually split those into the videos that you want to be part of the training set and the videos that you want to be part of the testing set. Or you can tell Google that it can randomly assign them 80%, 20%. And that's what we did. We let it randomly assign the videos to the training or test set, with 80% in the training set and 20% in the test set. And that way, every time we reran the model, it would generate slightly different results. And so if you ran it multiple times or trained it multiple times, you would get slightly different results. And you could see if there was something funny going on in one of the runs that doesn't appear in the other runs. So in order to figure out how to use Google's cloud services and their machine learning services, and at the time, just their image classification services, I actually had to take an entire specialization on Coursera because I couldn't figure out how to do it just by logging in and reading all their documentation, it was taking so long. So I signed up for their machine learning with TensorFlow on their Google Cloud platform specialization. And I think that's five courses similar to the ones that you all and I are taking right now. And once you finish that course, I knew how to do the actions that I needed to do in Google to make this project work. So here's roughly the inputs and the outputs of this. As I explained, we have all these video clips, 2,300 of them, and they're labeled as being either novice or intermediate or expert. And then they're divided into a training and test set that's randomly done on the Google servers. And you can see in this run that I had 399 videos that were pre labeled as expert and 694 that were labeled as intermediate, and then 640 that were labeled as novice. And then you can see how the AutoML service split those into the ones it would do training with and the ones it would do testing with. Now here's an aside point we ran, we went ahead and ran with this data set. If you've taken some of the earlier courses, you know that it's better if all three of those classes have roughly the same number of instances in them. So I don't have the ability to acquire more samples at the expert level. It's tough enough to get even 399 at the expert level. So I could have chosen to cut 300 out or 250ish out of each of the intermediate novice classes in order to make the class, the three classes balance out better. I didn't do that in the run that we're going through here. But that's a, that's a point you might be interested in later. Okay, so when you turn on Google ML, what output do you get from that? Well, this is a sample of the output you get and it's, this would be the place to explain AutoML. So when you're using AutoML, it automatically runs your data and your labels through dozens of or hundreds of different variations of a neural network architecture. It changes the internal connectivity between the layers and tests it again, it changes the number of layers and the way the layers are connected. It changes the hyper parameters. And so it actually runs your data through 100 different different structures of a neural network and it returns to you the one that got the best performance. Now all of that work is something that we would have had to do manually if we had chosen to do the programming ourselves. But with an AutoML service, it does all that for you. And so the best model that AutoML was able to find for us had an average precision of 0.8. And it had a precision or precision is where you try to minimize your type one error of 72.8% and a recall accuracy of 67.7%. So that's pretty good. I'm going to use the 80.8% as my reference point here. So that that's not too bad. We're really happy with that. And this is a research experiment to even find out if it's possible to find out if the current family of neural networks even have the ability to do this job. And it looks like they can. And there's a lot of adjustments that could be made that I think could improve these outputs. And I'll show you those adjustments on the next slide. So down below that, you see a confusion matrix. And a confusion matrix shows you when your training process or your testing set gave you the answer that your label said it should be. So if the actual label was, this is a novice video, the predicted output was correct as novice 83% of the time. But 14% of the time, it accidentally or mistakenly labeled that novice as intermediate, and 2% of them got labeled as experts. That's not too bad. But then you look at the intermediates and you notice that it only got the intermediate label correct 60% of the time. And it made a mistake and thought 31% of the intermediates were actually novices. That's a big difference. And then you go to the expert, you see roughly the same percentages. 67%. Right. But 25% were thought to be of one step, one level down in expertise. They were just intermediate. So I think there's some adjustment that can be done there to make the model better. So after your model's trained, it's loaded and ready to assess a video that does not have a label associated with it. So this is a picture of a piece of the user Interface on Google AutoML. So you load the video in and it watches this video and tells you what level of skill it thinks is in this video. And this video was not part of the training set and not part of the testing set. This is the first time the algorithm has seen this video. And this video wasn't clipped down to 10 seconds. I fed it a video that was 30 seconds long. And this is the output you get from Google. It watched this video for 30 seconds. And you can see right before 15 seconds, it believed that what it was watching was novice level skill. And then right there in the neighborhood of 20 seconds, it believed that it was seeing intermediate level skill. And then later after between 25 and 30, it believed it was seeing intermediate level skill as well. And this is at its own 50% confidence level. So you ask, well, during the grayed out period, what did it think? Well, what it thought was I can't tell what skill level it is or what classification it should go into with higher than 50% confidence. So it's grayed out because it doesn't know. And I use this example in this presentation because in this case it didn't see always intermediate level performance, it saw novice level performance sometimes in this video. Now I happen to know that this video is an intermediate level performance performance video. The neural network didn't know that, but I knew that. And so I'm happy with the fact that it predominantly thinks it's intermediate, but it's also important to notice that it sometimes thinks it's novice level. I'll show you why I think that's important. So when I think about. So I've run this whole experiment and we've tested the model and we think what else could we do to improve the performance of this model? Well, I could think of four things that we could do. One was adjust the thresholds where we decided where the break point was between novice, intermediate and expert. Especially based on those two 60 and 67% accurate numbers out at the end. It looks to me like we need to move that threshold more towards the novice level, that the threshold is set a little too high towards expert. And that's what's causing some too many of the scores to be identified as the next level down, the novice level or the intermediate level. And so if we move those, I think we would get better performance. Number two is the default for all deep learning projects. Get more videos, just keep getting more and more videos to feed it, so the more videos you can get, the better the network will learn. And then the third one is that we could manually call out some of the videos. So I mentioned earlier that we might call out some of the videos because we didn't have the right balance between the three classes. So that could be one reason we're calling them out. Another reason is something that we saw in that last slide in the example. Sometimes in the video the action that's happening isn't really differentiable. It's so simple that an expert, a novice, an intermediate all do it with exact same level of skill of steadiness of instrument or quickness of movement. And so that video isn't really a good, even though it's labeled as an intermediate. That segment of 10 seconds of video isn't really a good member of the intermediate class. It's not a good member of any class because all classes for that movement look the same. And so if you manually went through and cut those out, then you would have fewer videos that came from an intermediate level performer. But the actions in the video don't reflect that. And then the last one is learn more about how Google AutoML works. I have literally read everything in Google system on how their AutoML works, and I still have lots and lots of questions. This is a brand new service and it's still in beta, as is a lot of Google's work. But there's a lot of questions that there are just no answers to. And if you're not working at Google, I don't know how you get the answer. Like, for example, on this slide here, what is it doing during that gray time? I don't know. And is it when it's training a video that's 10 seconds long, is it really thinking about everything it's seeing in those 10 seconds, or is it heavily weighted towards what it saw in the last second of those 10 seconds? I don't know, it's just not in the documentation. So I think we could improve performance if we knew more about what Google is doing. Okay, so that's what we've done to create an AutoML model that could be used in surgical training. And then as we, as we search on the Internet, we find out we're not the only people that's interested in this. People are interested in this kind of an algorithm so that they can assess videos from sports performances, from artistic and dance performances, from traffic cameras, and from security cameras. So from the traffic camera, is it possible to train the network to constantly watch this traffic camera? And when it sees this incident, should it automatically call an ambulance, or should it automatically call a policeman, or should it do nothing? So, so if it's smart enough, it might be able to do that automatically without somebody having to report it. And then similarly, is this the clerk just tidying up the cash register for the evening, or is this a store that's being robbed? And so should the police be automatically called when it sees this video? So those are some of the other domains that are interested in exactly the same kind of work. All right, so I'm finished. I think based on the work we've done, we can definitely train a neural network to know how to score a simulator video. It's probable that we could do the same with a dry lab, because it's simple enough. Wet lab? I don't think so. I think there's too much variation in the tissue, which is not what's being scored. You're supposed to to be scoring the movement of the instruments and I think the algorithm would get confused. So that's what we've done so far. And I'm going to switch now to the Slack channel and answer any questions that you might pose. Give me a second while I switch. All right, so if you would post some questions, I will answer them if I can. Oh, well, Lalit, where you typed, where do we post questions? That's where you post them. I read it. Good job. So you're posting them in the channel that is May 28th online event. All right, so I have a question from Nilesh. And they say, how do you account for correct sequence of steps if early steps are right and later steps are wrong? How does the training labels account for that? That's a really, really good question. So if the person did the correct movements but did them in the wrong order, I don't think you could assess that because right now when you train video, you have to limit your training to a few seconds. That means that the training set has to be sliced up and somebody smarter than me would have to figure out if there's a way to maintain the temporal sequence between those sliced up segments. In my experiment, there was nothing and so I couldn't. The training set, the trained model wouldn't know the difference between something that should be done first and second. I couldn't do that. Okay. Her second question is for improvement. Is it possible to change the contrast or brightness of the video to create new training samples? So it's possible to change the resolution level? We could do that. So we could. I said we down sampled to 720p. We could go down levels of that? Yes. Could you change the brightness? I think so. Yes. But since the simulator always uses the same level of brightness, you'd be creating a training set that you will never see in the videos that you're assessing. So I don't think that would be appropriate. In a real world setting that might be appropriate. A physical world setting like the dry lab examples, that would probably be appropriate because the lighting would vary from one trainee to another. Oh yeah. Number three, we wrestle with this all the time. So her third question. You're getting a lot of time. Any thoughts on bias overall of such technologies? Because human labeling process can be biased, especially for security use cases. I don't know about security, but human labeling of these videos is Extremely biased. The research that we did at this lab prior to this looked into exactly that, how much variation there was using that very Likert scale. And so we always go through an IRR process where we get the people scoring it to be scoring the same thing at the same levels. But even then, you're right that some of the assessments can be very different from other assessments when they come from humans. And if that's your training set, then you're going to end up training your model to have that same bias in it. So I mentioned in the presentation that there was a reason that we might be training the model to match the simulator. Remember that this bias is one of those reasons, because the simulator is never biased. It's always giving an objective answer. And so we could use that as a means of getting to our three classifications. And we're scoring the simulator scores because we're trying to avoid bias. And that could be a later paper that we're going to do. But we thought it was too hard to explain both those ideas to the medical community at the same time. Okay, let me jump down here. I saw another good one. Why did you not use TensorFlow? And I think you mean TensorFlow directly. We didn't use TensorFlow directly because we thought that would just give us more opportunities to make mistakes. So, as I mentioned, we had taken all of the deep learning AI courses and specialization that are posted on Coursera, so we certainly learned how to do that. Also courses and sequences from IBM and Google. So we know how to do it. But we thought, one, it would give us more opportunity to make programming mistakes and two, it would just take longer. So we thought we could get to a faster answer if we did it with Google AutoML. Second. Okay, and then. Lalette says, why did you say the steadiness of instruments or hands is similar among all the expertise levels? I would think it would be the exact opposite. So when we watch surgeons that are just training, their ability to move the instruments very smoothly and directly to the target is much less precise than expert surgeons. So expert surgeons move kind of like expert tennis players. They go right to the ball and they do the job and move right to the next thing and move right to the next thing. It's very pretty. But surgeons who are novices that are just learning, they kind of tiptoe to where they're going and they might jitter around a bit and pick the needle up from the wrong end and then have to move it back and forth in their hands. So if you're just sitting watching these Videos and you're experienced at it. You can tell the difference between who's done this a thousand times and who's just doing it their first, second or tenth time? Yeah. Are you able to explain the results to others as to why or how the doctors were scored? And I think by that you mean how the training set was assigned scores. Yes, we are. The reason that we were training towards that Likert scale tool that I showed is because that's the standard validated tool that's used in training all robotic surgeons. Right now, that was a project, I believe, out of the University of Southern California or ucla, one of the two of those. But that's become the standard tool. So when we talk about matching that tool, everybody who does robotic surgery knows exactly what that tool does. So that's why we chose that one. Okay. N Albarisin asks, have other cloud providers caught up to match Google AutoML video service since you started, so using Auto ML with video? To my knowledge, no. IBM came out with a service that they call Auto AI just in the last month, and I went and looked into that service to see if it could do video. And as far as I could tell, it could not do video. I haven't recently checked Amazon and Microsoft to see if they could do it or not. So I don't know if they could do it. And there are other smaller providers like Clarify that we didn't look into, but there might be capabilities there that we don't know about. So what are the cost differences doing it automl versus doing it at home on your own computers? And by at home, I mean at work, your business is providing it. We looked at the specs for the computers that Google said that our service was running on, and it was a four processor CPU running a specific Nvidia graphics card. And the price, if we wanted to go out, and they were using 13 of these servers to do this, the price, if we were to go out and buy that hardware, the same hardware that they were running it on, it would have cost us just shy of $40,000 to buy the hardware. And then we would have somebody set it up, renting it by the hour, just for the hours that we needed it. The bill from Google was $1,300. And because we did this as our first project right after signing up for our Google Cloud account, we had credits of almost $1,000. So we only paid actually about $300 out of our pocket. But we've used those credits up. So future products, sorry, future projects would cost the whole. Would have cost us, the whole 1300. Yeah. So Lalit asks, given that surgery affects human lives, what is the level of evaluation scores you will be shooting for in order for this to become deployable or acceptable management or in the hospital business by the administrators or by the societies? So that's a good question. I would say there's. It's a new idea that nobody knows. But I believe if you're walking into there, you're going to have to walk in and preferably in the 90 percentile somewhere somewhere above 90%. I mean, the community might be more accepting of something over 85, but I think much more preferable something in the 90s. And you're going to have to have a discussion about how human assessed videos are not necessarily 100% accurate either. So that it is going to be a challenge to find out where is this an acceptable tool for assessment. How long did data preparation take for getting the data in shape for Google ML as a percentage of time spent on the project? You mentioned that you used Excel macros. Were there any Pandas or other tools used in addition used for data wrangling? Oh, that's a good question. So actually the data collection part where we brought all the data together and experimented with the still images and learned to all of that happened and then we got to the date what I called the data wrangling or the data preparation stage, and I could see how much work it was going to be. And so the project actually stopped right there because nobody wanted to do that work. And I didn't actually do that data wrangling part until COVID lockdown happened. And when COVID lockdown happened and I didn't leave my home for two, I spent about a month, a solid month, preparing the data and cleaning it up and running it through AutoML and then realizing, oops, I made a mistake, go back and fix it again. So about a month of data preparation and it was very tedious and it wasn't pleasant, but because I was on COVID lockdown, it was something to occupy my time. And then did I use Pandas? No, I thought about it because all of the data was already in an Excel spreadsheet. I was like, I can do the same thing in Excel that I would do with Pandas. The same goes for the file names in Windows. Mostly the file names just needed to be rearranged. The file name is like 35 characters long and the right pieces were there. They were just in the wrong order. So usually I just had to slice and dice the file name to get it to work so that they were all consistent. Okay, Well if there's not any other questions, I think we'll wrap it up. I'll give you a just a few seconds to see if there's one more lingering question. There's two people typing. I'll wait until they pop out. Lalith asks, where did you do the video culling? And I think you mean in what tool did I use do the video culling? Well, video culling, video cleaning, I have them here on my computer. Have to escape out of that. So I used two tools. I used one called Bandicut to cut the videos into 10 second segments and I also use something called replay media to down sample them. And Bandicut can act, can also down sample when it cuts. Both of those tools, I wouldn't say are professional quality. They had a lot of little, a lot of little bugs in them that made the job difficult. If you had a professional video quality, you could tell it what you wanted to do to 250 videos and it would go off and do it as a batch job. But both of those tools required a little more manual processing that I was happy with. Again, lucky I was on COVID lockdown. Okay, then the last question here, Justin Gates asks just an idea. When we have for example 30 second videos, is it better to slice it into three videos and then get the average score for three videos so some pre processing and post processing will be used? I think automl is doing something similar. But do you suggest to do this kind of processing since DNN trained for 10 second slices. So I think you're. So what we did, if we had a 30 second video and we cut it into three videos, that 30 second video had been given a score by an experienced surgeon of intermediate. And so we applied the intermediate label to all three of the smaller clips. But as you say, it might be that that first clip is just the surgeon getting ready and it looks the same novice, expert or intermediate. And so that's maybe not a good intermediate clip and I should do something else with it. So that's possible. I don't think that I could decide or I don't think I should decide that that first clip is a novice level because I'm not the experienced surgeon who gave it the original label. So I think I would be compromising the data if I changed it. So here Nilesh says he was impressed that it took one month to pre process the data. He says there's a new visual tool on the horizon called Tableau Prep for data engineering and we might look at that. That's good to know. I know of Tableau, the company and their tools, but I didn't know this new tool that came out. All right, well, we've gone to 50 minutes instead of 30, and I appreciate all your questions. Thank you very much. I think we will wrap it up. So back to Alice, who I think is at the controls.