Wolfram AI in Medical Imaging: Prostate Cancer Detection

Hello, I'm behalf of my colleague Sabra Ansari and myself, Peter Riley. Today we shall present some applications of the Wolfram artificial intelligence platform as employed in medical imaging at Deakin University in Australia. Well, we're using Wolfram due to its ease of use for non programmers, both for staff and students, and for its low overheads. We shall review some work undertaken in two areas. Capture COVID19 detection in chest X rays and prostate cancer detection in magnetic resonance diffusion imaging. We've previously presented our work on COVID 19. This was a pilot study to assess the suitability of undertaking deep neural network training in our particular tertiary environment. This was proposed really on the success of previous work undertaken by one of your students at the 2018 Wolfram Summer Camps where they successfully differentiated viral pneumonia from bacterial pneumonia in chest X rays. So, kudos to Rohit and the summer camps. So we'll just do a quick review of those outcomes for detection and staging of COVID 19 and of the lessons learned therefrom. We'll go into greater detail on the new research project for the detection of prostate cancer and of the current successes and failures of this work in progress. We'll give the clinical results first and then look at the Mathematica details to finish up with special attention to the receiver operating characteristic curve. Well, here you see samples of the X rays to give a differential diagnosis between Covid viral and normal patients. There were 1200 images in each of these classes with we used both Resnet 50 and the Wolfram Image Identify Deep Neural Network. These are typically trained on about 1,000 classes with 1,000 samples in each. Well, here are the results. After submitting the known test images to the trained Network, it shows 100% sensitivity, which is. Which is ideal for a pandemic as we have no false negatives. So we aren't releasing patients with COVID back into the community unknowingly and with a 99% specificity. We have only 1% false positives and that's not too onerous as we're only asking the patient to isolate for a time before re entering the community, albeit unnecessarily. The Wolfram Network gave comparable results with 97% accuracy. Well, this at least proved the pilot study to be a great success and boded well for future endeavours. But we also hoped to apply the technique to staging the disease to identify early stage 1 to 3 days and late 7 to 10 days from onset. From the 1,200 COVID patients, we had only about 400 histories that were available, from which about 205 fell into those two early late categories. An initial network trained on these gave an accuracy of 50%. So basically a toss of a coin. So upon reviewing the images, we realized that there were many ambiguities and artifacts which were compromising the quality assurance of the data set. Upon removal of those, we had about 100 images for training from which we obtained an 80% accuracy. So this is quite reassuring for such a really small training set. So quality assurance issues are a serious hazard in artificial intelligence. It results in data poisoning, which is one of the main lessons we took away from this particular exercise. So the full details of this COVID 19 work are available on the Wolfram Community site. If you wish to review the details, we're moving on to our current work in progress. We're using magnetic resonance diffusion image sets which comprise 16 or more 3 millimeter slices through the prostate volume as represented here. A diffusion image shows differences in water diffusion rates in the tissues, where a brighter signal indicates a blockage or slowing down, which often accompanies a lesion. The question is though, can we differentiate malignant cancer from benign hyperplasia? The latter is really quite prevalent in older patients. It's about 50% in the sixth decade, going to 80% in ninth. This compares to prostatic cancer incidence, which is around 5% in the sixth decade and 9% in the seventh. And beyond. The other question you might be wondering is, how are we going to use a 2D Resnet neural network for 3D volume data? Good question. Okay, so to do so we create a hybrid 2 1/2 dimension image. First of all, we really only require the prostate. So we auto crop each of the image slices to the central 56 square pixels. This does not appear to lose the anterior, posterior or lateral borders of the organ. We then lay these slice tiles into a 4x4 mosaic in this pattern shown to retain some slice contiguity information. We also use only the central 16 slices. So potentially we may lose anterior and posterior margins, but we've not seen any problems to date. So all that pre processing is done automatically in Mathematica. Well, here are some benign and malignant samples. Can you tell which is which? Basically these two are hyperplasia and these two malignant cancer. Difficult to distinguish even for the trained eye. Here's a little bit of Mathematica code, just sort of showing. Here's a stack of. I think it's 21 slices for this particular patient. We determine where to start amongst that lot to get the central 16 slices, we then use image crop which is fantastic. Just to give 56 squared pixel from the center. We then rearrange these images into that S shaped pattern and use image assemble to obtain the mosaic which we can then export. All this can be put into a do loop for massive batch processing through originally for 400 patients. Okay, so after a first naive run with Resnet we only got about 50% accuracy. So once again, once we peruse the images, you can see what the problem is. So we really wish to undertake this testing with as little pre processing as possible. But here you see the variability of image quality in the data set. So here's a good image, but over here we have a really low contrast image and over here we have a really noisy image. Now of course, Mathematica has a terrific toolbox of image processing functions available. Some require sort of manual input. So for example here that's that bad image from over page and some do not. So here's image transform, image transform looks really quite good. Here's the good image as a reference for the bad image and here's the bad image after it's been more or less equalized to represent the good image. So this is a potential way to salvage some of those poor images. However, we were quite keen to demonstrate at least proof of concept that resnet was viable for this type of volume data without going into these types of interventions. So there were 1500 patients available on the prostate cancer public database, out of which about 456 came with good the gold standard biopsy information. The gleason scores of 0 and 7 for 04 benign 04 malignant cancer. So out of these about 100 images were selected by an untrained observer just for similarity in contrast and noise level. So about 50 in each class. We also trained a neural network to do that and it worked really remarkably well. Okay, so here are our results. The outcomes from the trained neural network, a sensitivity and specificity of 80%. So overall accuracy 80%, which is terrific on such a small data set. It was quite surprising and it's quite comparable to the radiologist accuracy at 79%. However, when you look at the radiologist outcomes in more detail, you see a skewed response. They have a 94% sensitivity, so only 6% false negative rate. Well, that of course is their priority. They want to detect malignancy as early as possible so they can send the patient to therapy, thereby improving the patient's chances of survival. But they do so with a specificity of only 63%, 37% false positives. Now that might just look like a number. But what it means is that over a third of all patients who were sent for biopsy come back as negative. The biopsy was unnecessary. There are really quite serious side effects for these biopsies, as you can see from this list, not the least of which is sexual dysfunction. So our results at least reduce this rate to 20%, which would save a lot of needless suffering, but at the expense of missing 20% of the actual prostate cancers. So there's some interesting discussions to be had at both a philosophical and an October level about how you actually play off these different characteristics, but perhaps not just at the moment. However, later on, when we take a look at some of the mathematical detail and we look at the receiver operating characteristic curve, I think I can demonstrate that, in fact, we can also use that same train network to give a 95%, 65% level, which is very comparable to the radiologists. So the hybrid 2 1/2d method is giving quite reasonable results, especially given its very small sample size. So what about a 3D DNN? We basically continued with Resnet based on its previous success with a Covid pilot study. But what else might be available? Well, There is a 3D deep neural network available on the Wolfram repository. Strictly Speaking, it's a 2D plus 1D. This is the MobileNet video deep neural network, which has been trained to recognize 100 different types of human movement and gestures. And fortuitously, it uses a 16 frame video for each class. So it means that we can reformat our 16 tile mosaic into a 16 frame video using the code below. Again, Mathematica anticipates everything. What can I tell you? In medical imaging, we refer to this type of video as a fly through. And here are some examples. I can't run them because they go too fast, but let me sort of see if I can click through them a little bit to give you some sort of idea of what the video might look like. So that's actually hyperplasia. And you see that the brightnesses are quite diffused. Here is the malignant. I think you can see that the brightnesses are more confined and more contiguous. So that is a potential. Characteristic which could differentiate the disease for us. And here you see the results once again, based on the same 100 patients. We have an overall accuracy of 80%. So identical to our previous results. Well, we were also attempting to localize the tumor, but given such small samples, we've only been able to identify sort of row and column of a lesion with 60% accuracy. So that's ongoing work in progress. So the clinical conclusion is that unfortunately we need more images to progress any further. We have recently entered into collaborations with our clinical stakeholders and that should greatly simplify our quality assurance problems. We may be able to rescue some of those older images with pre processing intervention, just so long as it can be done fully automatically. Okay, so that completes the clinical conclusion for the work in progress. I hope you've found something of interest there. If you're interested in the Mathematica details, we'll just quickly go over those in the remaining minutes. Okay, so for simplicity, I have simply bundled all of the benign images into a group and also the malignant cancers into a group. So the first thing we have to do for input to the neural networks is to map each of those images to a particular class. That's a single line. Once we have those, we're going to use the first 40 for a training set and the final 10 for a test set. Likewise, we have to label the malignant cancer images to the G7 classification. Gained first 40 for training, final 10 for testing. We put both of those sets together for training and test. So at this point we can fetch Resnet 50 from the Wolfram library. That takes maybe a minute. We then modify the resnet by dropping just the final two layers and we're going to replace them with the new classification layers. So here's the new classes, just G0 and G7. We're ready to actually start training the new network with the training data. Notice that we set the classifier layer to being on. So basically it's only the classifier layer that's going to be trained. All of the other layers are set to off. So basically they will retain all of the network weightings that were obtained from the much more advanced general image classifier for resnet. Rather than run it. I'm just going to give you a screen grab to show you what you can actually see whilst the network is training. And so over time you'll see this curve being generated over time and you see that the error rate is decreasing quite nicely. So this is indicating that it's moving towards convergence. We could probably extend this a little bit. I haven't seen much difference in terms of outcomes though by doing so. Once we have this network trained, we can export that for safekeeping in the Wolfram format. And now of course, we are going to test it. So we have 20 images in the test set. The first 10 were G0. So it says it gets the first image wrong and it gets the last of the G0 images wrong. And likewise for the G7, it gets this one wrong and it gets this one wrong. So overall it's going to be an accuracy of 80%. What's of interest? I'll just point your attention to the fact that you can get the actual probabilities that the network has used to assign these. So the default probability threshold is 50%. We'll come back to that in just a moment. Also quite useful we see the confusion matrix plot, which shows the true positives, false positives, true negatives, false negatives. You can also directly ask for sensitivity, specificity, but you can also obviously get that from the confusion matrix. But let's take a look at the receiver operating characteristic curve. Now, our network is basically an observer. So all observers have a fixed receiver operating characteristic curve. It can't be changed at the end of the day. For humans it comes down to things like visual acuity and so on. What you can change is where you actually operate along the ROC curve. And that takes training. It can be quite hard work as well. So radiologists, for example, have to undergo very specific training for mammography so that they can sort of shift where needs to be along this ROC curve. So the neural network Mathematica is reporting sort of 80% accuracy. So basically that's the point where the ROC curve intersects this diagonal here. However, in principle, by reevaluating what we take as G0 or G7, by resetting the probability threshold, in principle at least, we could move our observation point, for example, up to this point here. And that would give us a 95% sensitivity and a 65% specificity, which is very comparable to what we saw for the, for the radiologists. That's at the moment we, I say in principle because at the moment we just don't have enough data. As you can see, the curve itself is quite chunky. So we need to smooth that out a little bit further and we need a lot more probabilities to be able to determine what would be an appropriate threshold to reset that that at. Okay, and just finally, here are some of the, the references if you want to follow up any of that material. I hope you found something of interest here and I hope you enjoy the, the rest of the conference. Thank you.