MIT 6.S094: Computer Vision

Today we'll talk about how to make machines see computer vision. And we'll present. Thank you whoever said yes. And today we will present a competition that unlike Deep Traffic, which is designed to explore ideas, teach you about concepts of deep reinforcement learning. Segfuse, the deep dynamic driving scene segmentation competition that I'll present today is at the very cutting edge. Whoever does well in this competition is likely to produce a publication or ideas that would lead the world in the area of perception, perhaps together with the people running this class, perhaps on your own, and I encourage you to do so. Even more cats today. Computer vision today as it stands is deep learning. Majority of the successes in how we interpret form representations, understand images and videos, utilize to a significant degree neural networks, the very ideas we've been talking about. That applies for supervised, unsupervised and reinforcement learning. And for the supervised case. This is the focus of today. The process is the same. The data is essential. There's annotated data where the human provides the labels that serves as the ground truth in the training process. Then the neural network goes through that data learning to map from the raw sensory input to the ground truth labels and then generalize over the testing data set. And the kind of raw sensors we're dealing with are numbers. I'll say this again and again, that for human vision, for us here, we take for granted this particular aspect of our ability is to take in raw sensory information through our eyes and interpret it. But it's just numbers. That's something whether you're an expert computer vision person or new to the field, you have to always go back to and meditate on is what kind of things the machine is given, what is the data that it's tasked to work with in order to perform the task you're asking it to do. Perhaps the data it's given is highly insufficient to do what you want it to do. That's the question that'll come up again and again. Are images enough to understand the world around you? And given these numbers, these set of numbers, sometimes with one channel, sometimes with three RGB where every single pixel have three different colors. The task is to classify or regress producing continuous variable or one of a set of class labels. As before, we must be careful about our intuition of what is hard and what is easy in computer vision. Let's take a step back to the inspiration for neural networks, our own biological neural networks. Because the human vision system and the computer vision system is a little bit more similar in these regards. The structure of the human visual cortex Is is in layers. And as information passes from the eyes to the parts of the brain that make sense of the raw sensory information, higher and higher order representations are formed. This is the inspiration the idea behind using deep neural networks for images. Higher and higher order representations are formed through the layers, the early layers, taking in the very raw sensory information, then extracting edges, connecting those edges, forming those edges to form more complex features and finally into the higher order semantic meaning that we hope to get from these images. In computer vision, deep learning is hard. I'll say this again. The illumination variability is the biggest challenge, or at least one of the one of the biggest challenges in driving for visible light cameras pose variability the objects as I'll also discuss about some of the advances from Jeff Hinton and the capsule networks. The idea with the neural networks as they are currently used for computer vision are not good with representing variable pose. These objects in Images in this 2D plane of color and texture look very different numerically when the object is rotated and the object is mangled and shaped in different ways. The deformable truncated cat intraclass variability for the classification task, which would be an example today throughout to introduce some of the networks over the past decade that have received success and some of the intuition and insight that made those networks work. Classification There is a lot of variability inside the classes and very little variability between the classes. All of these are cats up top, all of those are dogs at bottom. They look very different. And the other, I would say the second biggest problem in driving perception, visible light camera perception, is occlusion when part of the object is occluded due to the three dimensional nature of our world, some objects in front of others and they occlude the background object. And yet we're still tasked with identifying the object when only part of it is visible. And sometimes that part, I told you, there's cats and is very hardly visible. Here we're tasked with classifying a cat when just an ear is visible, just the leg. And on the philosophical level, as we'll talk about the motivation for our competition here, here's a cat dressed as a monkey eating a banana. On a philosophical level, most of us understand what's going on in the scene. In fact, a neural network today successfully classified this image, this video as a cat. But the context, the humor of the situation and the fact that you could argue it's a monkey is missing. And what else is missing is the dynamic information, the temporal dynamics of the scene. That's what's missing in a lot of the perception Work that has been done to date in the autonomous vehicle space in terms of visible light cameras, and we're looking to expand on that. That's what SEG Fuse is all about. Image classification pipeline There's a bin with different categories inside each class. Cat, dog, mug, hat, those bins, there's a lot of examples of each. And you're tasked with when a new example comes along you've never seen before, to put that image in a bin. It's the same as the machine learning task before. And everything relies on the data that's been ground truth that been labeled by human beings. MNIST is a toy data set of handwritten digits often used as examples and coco, cifar, imagenet places and a lot of other incredible data sets, rich Data sets of 100 thousands, millions of images out there represent scenes, people's faces and different objects. Those are all ground truth data for testing algorithms and for competing architectures to be evaluated against each other. Cifar 10, one of the simplest, almost toy data sets of tiny icons with 10 categories of airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck is commonly used to explore some of the basic convolution neural networks we'll discuss. So let's come up with a very trivial classifier to explain the concept of how we could go about it. In fact, this is maybe if you start to think about how to classify an image. If you don't know any of these techniques, this is perhaps the approach you would take is you would subtract images. So in order to know that an image of a cat is different than an image of a dog, you have to compare them. When given those two images, what's the way you compare them? One way you could do it is you just subtract it and then sum all the pixel wise differences in the image. Just subtract the intensity of the image pixel by pixel. Sum it up. If that difference is really high, that means the images are very different. Using that metric, we can look at CIFAR 10 and use it as a classifier saying based on this difference function, I'm going to find one of the 10 bins for a new image that has the lowest difference. Find an image in this data set that is most like the image I have and put it in the same bin as that image is in. So there's 10 classes. If we just flip a coin, the accuracy of our classifier will be 10%. You using our image difference classifier, we can actually do pretty good. Much better than random, much better than 10%. We can do 35, 38% accuracy. That's the classifier. We have our first classifier, K nearest neighbors. Let's take our classifier to a whole new level. Instead of comparing it to just trying to find one image that's the closest in our data set, we tried to find k closest and say what class do the majority of them belong to? And we take that K and increase it from 1 to 2 to 3 to 4 to 5 and see how that changes the problem. With seven nearest neighbors, which is the optimal. Under this approach for CIFAR 10, we achieve 30% accuracy. Human level is 95% accuracy. And with convolutional neural networks, we get very close to 100%. That's where neural networks shine. This very task of binning images, it all starts at this basic computational unit signal. In each of the signals are weighed, summed, bias added, and put in input into a nonlinear activation function that produces an output. The nonlinear activation function is key. All of these put together in more and more hidden layers, form a deep neural network. And that deep neural network is trained, as we've discussed, by taking a forward pass on examples that have ground truth labels, seeing how close those labels are to the real ground truth, and then punishing the weights that resulted in the incorrect decisions and rewarding the weights that resulted in correct decisions. For the case of 10 examples, we the output of the network is 10 different values, the input being handwritten digits from 0 to 9. There's 10 of those. And we wanted our network to classify what is in this image of a handwritten digit. Is it 0, 1, 2, 3 through 9? The way it's often done is there's 10 outputs of the network, and each of the neurons on the output is responsible for getting really excited when its number is called and everybody else is supposed to be not excited. Therefore, the number of classes is the number of outputs. That's how it's commonly done. And you assign a class to the input image based on the highest the neuron, which produces the highest output. But that's for a fully connected network that we've discussed on Monday. There is in deep learning a lot of tricks that make things work, that make training much more efficient. On large class problems, where there's a lot of classes on large data sets, when the representation that the neural network is tasked with learning is extremely complex. And that's where convolutional neural networks step in, the trick they use is spatial invariance. They use the idea that a cat in the top left corner of an image is the same as a cat in the bottom right corner of an image. So we can learn the same features across the image. That's where the convolution operation steps in. Instead of the fully connected networks here there's a third dimension of depth. So the blocks in this neural network as input take 3D volumes and as output produce 3D volumes. They take a slice of the image, a window and slide it across, applying the same exact weights. And we'll go through an example. The same exact weights as in the fully connected network on the edges that are used to map the input to the output here are used to map this slice of an image, this window of an image to the output. And you can make several many of such convolutional filters. Many layers, many different options of what kind of features you look for in an image, what kind of window you slide across in order to extract all kinds of things, all kinds of edges, all kind of higher order patterns in the images. The very important thing is the parameters on each of these filters, these subset of the image, these windows are shared. If the feature that defines a cat is useful in the top left corner, it's useful in the top right corner. It's useful in every aspect of the image. This is the trick that makes convolutional neural networks save a lot of parameters, reduce parameters significantly is the reuse the spatial sharing of features across the space of the image. The depth of these 3D volumes is the number of filters. The stride is the skip of the filter, the step size, how many pixels you skip when you apply the filter to the input. And the padding is the padding, the zero padding on the outside of the input to a convolutional layer. Let's go through an example. So on the left here and the slides are now available online. You can follow them along and I'll step through this example. On the left here is input volume of three channels. The left column is the input, the three squares. There are the three channels and there's numbers inside those channels. And then we have a filter in red, two of them, two channels of filters with a bias. And those filters are three by three. Each one of them is size three by three. And what we do is we take those three by three filters that are to be learned. These are our variables, our weights that we have to learn. And then we slide it across an image to produce the output. On the right, the green. So by applying the filters in the red, there's two of them. And within each one there's one per every input channel. We go from the left to the right, from the input volume on the left to the output volume green on the right. And you can look, you can pull up the slides yourself. Now, if you can't see the numbers on the screen, but the operations are performed on the input to produce the single value that's highlighted there in the green in the output. And we slide this convolutional filter along the image with a stride, in this case of two, skipping, skipping along, they sum to the, to the right, the two channel output in green. That's it. That's the convolutional operation. That's what's called the convolutional layer in neural networks. And the parameters here, besides the bias, are the red values in the middle. That's what we're trying to learn. And there's a lot of interesting tricks discuss today on top of those. But this is at the core, this is the spatially invariant sharing of parameters that make convolutional neural networks able to efficiently learn and find patterns and images. To build your intuition a little bit more about convolution, here's an input image on the left. And on the right, the identity filter produces the output you see on the right. And then there is different ways you can, different kinds of edges you can extract with the activate with the resulting activation map seen on the right. So when applying the filters, those edge detection filters to the image on the left, you produce in white are the parts that activate the convolution, the results of these filters. And so you can do any kind of filter. That's what we're trying to learn. Any kind of edge, any kind of pattern you can move along in this window, in this way that's shown here, you slide around the image and you produce the output you see on the right. And depending on how many filters you have in every level, you have many of such slices that you see on the right, the input on the left, the output on the right. If you have dozens of filters, you have dozens of images on the right, each with different results that show where each of the individual filter patterns were found. And we learned what patterns are useful to look for in order to perform the classification task. That's the task for the neural network to learn these filters. And the filters have higher and higher order of representation, going from the very basic edges to the high semantic meaning that spans entire images. And the ability to span images can be done in several ways, but traditionally has been successfully done through Max pooling through pooling of taking the output of convolutional operation and reducing the resolution of that by by condensing that information by for example, taking the maximum values, the maximum activations, Therefore reducing the spatial resolution, which has detrimental effects as we'll talk about in scene segmentation. But. But it's beneficial for finding higher order representations in the images that bring images together, that bring features together to form an entity that we're trying to identify and classify. Okay, so that forms a convolutional neural network. Such convolutional layers stacked on top of each other is the only addition to a neural network that makes for a convolutional neural network. And then at the end the fully connected layers or any kind of other architectures allow us to apply particular domains. Let's take ImageNet as a case study. In ImageNet, the data set in ImageNet the challenge the task is classification. As I mentioned in the first lecture, ImageNet is a data set, one of the largest in the world of images, with 14 million images, 21,000 categories and a lot of depth to many of the categories. As I mentioned, 1200 Granny Smith apples. These allow to. These allow the neural networks to learn the rich representations in both pose lighting variability and intraclass class variation for the particular things, particular classes like Granny Smith apples. So let's look through the various networks, let's discuss them, let's see the insights. It started with Alexnet, the first really big successful GPU trained neural network on ImageNet that's achieved a significant boost over the previous year and moved on to VGGnet, Googlenet, Agoolenet, Resnet, CU Image and Senet in 2017. Again, the numbers we'll show for the accuracy are based on the top five error rate. We get five guesses and it's a 1 or 0. If you get guess, if one of the five is correct, you get a 1 for that particular guess, otherwise it's a 0. And Human error is 5.1. When a human tries to achieve the same tries to perform the same task as the machinist task of doing, the error is 5.1. The human annotation is performed on the images based on binary classification. Granny Smith, Apple or not, cat or not. The actual task that the machine has to perform and that the human competing has to perform is given an image is provide one of the many classes under that human error is 5.1% which was surpassed in 2015 by Resnet to achieve 4% error. So let's start with Alexnet. I'll zoom in on the later networks. They have some interesting insights, but Alexnet and vggnet both followed a very similar architecture, very uniform throughout its depth. VGGnet in 2014 is convolution, convolution pooling, convolution pooling, convolution pooling and fully connected layers at the end. There's a certain kind of beautiful simplicity, uniformity to these architectures because you can just make it deeper and deeper and makes it very amenable to implementation in a layered stack kind of way in any of the deep learning frameworks. It's clean and beautiful to understand. In the case of VGGnet, 16 or 19 layers with 138 million parameters, not many optimizations on these parameters. Therefore the number of parameters is much higher than the networks that followed it. Despite the layers not being that large, Googlenet introduced the Inception module, starting to do some interesting things with the small modules within these networks which allow for the training to be more efficient and effective. The idea behind the Inception module shown here with the previous layer on bottom and the convolutional layer here with the Inception module on top produced on top, is it used the idea that different size convolutions provide different value for the network. Smaller convolutions are able to capture or propagate forward features that are very local, high resolution in texture. Larger convolutions are better able to represent and capture and catch highly abstracted features, higher order features. So the idea behind the Inception module is to say. Well, as opposed to choosing in a hyperparameter tuning process or architecture design process, choosing which convolution size we want to go with, why not do all of them? Well, several together. In the case of the Google Net model, there's the one by one, three by three and five by five convolutions, with the old trusty friend of Max pooling still left in there as well, which has lost favor more and more over time for the image classification task. And the result is there's fewer parameters that are required. If you pick the placing of these in Inception modules correctly, the number of parameters required to achieve a higher performance is much lower. Resnet, one of the most popular still to date architectures that we'll discuss in scene segmentation as well, came up and use the idea of a residual block, the initial inspiring observation, which doesn't necessarily hold true as it turns out, but that network depth increases representation power. So these residual blocks allow you to have much deeper networks. And I'll explain why in a second here. But the thought Was they work so well because the networks are much deeper. The key thing that makes these blocks so effective is the same idea that's reminiscent of recurrent neural networks that I hope we get a chance to talk about. The training of them is much easier. They take a simple block, repeat it over and over and they pass the input along without transformation, along with the ability to transform it to learn to. To learn the filters, learn the weights. So you're allowed to. You allow every layer to not only take on the processing of previous layers, but to take in the raw and transform data and learn something new. The ability to learn something new allows you to have much deeper networks. And the simplicity of this block allows for more effective training. The state of the Art in 2017. The winner is squeeze and excitation networks that unlike the previous year with CU Image, which simply took ensemble methods and combined a lot of successful approaches to take a marginal improvement. Scnet got a significant improvement, at least in percentages. I think it was a 25% reduction in error from 4% to 3%. Something like that. By using a very simple idea that I think is important to mention simple insight. It added a parameter to each channel in the convolutional layer in the convolutional block. So the network can now adjust the weighting on each channel based for each feature map based on the content based on the input to the network. This is kind of a takeaway to think about about any of the networks you talk about. Any of the architectures is a lot of times recurrent neural networks and convolutional neural networks have tricks that significantly reduce the number of parameters, the bulk, the sort of low hanging fruit. They use spatial invariance or temporal invariance to reduce the number of parameters to represent the input data. But they also leave certain things not parameterized. They don't allow the network to learn it, allowing in this case the network to learn the weighting on each of the individual channels. So each of the individual filters is something that you learn along with the filters, makes a huge boost. The cool thing about this is it's applicable to any architecture. This kind of block, the squeeze and excitation block is applicable to any architecture because obviously it just simply parameterizes the ability to choose which filter you go with based on the content. It's a subtle but crucial thing. I think it's pretty cool. And for future research it inspires to think about what else can be parameterized in neural networks, what else can be controlled as part of the learning process, including Higher and higher order hyper parameters which, which aspects of the training and the architecture of the network can be part of the learning. This is what this network conspires. Another network has been in development since the 90s ideas by Jeff Hinton but really received, has been published on and received significant attention in 2017 that I won't go into detail here. We are going to release an online only video about capsule networks. It's a little bit too technical, but they inspire A very important point that we should always think about with deep learning, whenever it's successful is to think about what, as I mentioned with the cat eating a banana on a philosophical and the mathematical level have to consider what assumptions these networks make and what through those assumptions they throw away. So neural networks due to the spatial with convolutional neural networks due to their spatial invariance, throw away information about the relationship between the hierarchies, between the simple and the complex objects. So the face on the left and the face on the right looks the same to a convolution neural network. The presence of eyes and nose and mouth is the essential aspect of what makes the classification task work for convolutional network where it will fire and say this is definitely a face. But the spatial relationship is lost, is ignored. Which means there's a lot of implications to this. But for things like pose variation, that information is lost. We're throwing that away completely and hoping that the pooling operation that's performed in these networks is able to sort of mesh everything together to come up with the features that are firing of the different parts of the face to then come up with the total classification that it's a face without representing really the relationship between these features at the low level and the high level at the low level of the hierarchy at the simple and the complex level. This is a super exciting field. Now that's hopefully will spark developments of how we design neural networks that are able to learn the rotational, the orientation invariance as well. Okay, so as I mentioned, you take these convolutional neural networks, chop off the final layer in order to apply it to a particular domain. And that is what we'll do with fully convolutional neural networks, the ones that we task to segment the image at a pixel level. As a reminder, these networks through the convolutional process are really producing a heat map. Different parts of the network are getting excited based on the different aspects of the image. And so it can be used to do the localization of detecting not Just classifying the image, but localizing the object and it could do so at a pixel level. So the convolutional layers are doing the encoding process. They're taking the rich raw sensory information in the image and encoding them into an interpretable set of features representation that can then be used for classification. But we can also then use a decoder up sample that information and produce a map like this. Fully convolutional neural network segmentation, semantic scene segmentation, image segmentation. The goal is to as opposed to classify the entire image, you classify every single pixel is pixel level segmentation. You color every single pixel with what that pixel, what object that pixel belongs to in this 2D space of the image. The 2D projection in the image of a three dimensional world. So the thing is, there's been a lot of advancement in the last three years, but it's still an incredibly difficult problem. If you think about the amount of data that's used for training and the task of pixel level of megapixels here of millions of pixels that are tasked with having assigned a single label, it's an extremely difficult problem. Why is this interesting important problem to try to solve as opposed to bounding boxes around cats? Well, it's whenever precise boundaries of objects are important. Certainly medical applications when looking at imaging and detecting particular for example detecting tumors in medical imaging of different organs and in driving in robotics, when objects are involved it's a dense scene involved with vehicles, pedestrians, cyclists with we need to be able to not just have a loose estimate of where objects are, we need to be able to have the exact boundaries. And then potentially through data fusion fusing sensors together, fusing this rich textual information about pedestrian cyclists and vehicles to lidar data that's providing us the three dimensional map of the world where we have both the semantic meaning of the different objects and and their exact three dimensional location. A lot of this work successfully, a lot of the work in the semantic segmentation started with fully convolutional networks for semantic segmentation paper FCN. That's where the name FCN came from in November 2014. Now go through a few papers here to give you some intuition where the field is gone and how that takes us to seg fuse the segmentation competition. So FCN repurposed the imagenet pre trained nets, the nets that were trained to classify what's in an image entire image and chopped off the fully connected layers and then added decoder parts that up sample the image to produce a heat map here shown with a tabby cat A heat map of where the cat is in the image. It's a much slower, much coarser resolution than the input image 1, 8 at best. Skip connections to improve coarseness of upsampling There's a few tricks. If you do the most naive approach, the up sampling is going to be extremely coarse because that's the whole point of the neural network. The encoding part is you throw away all the useless data to the most essential aspects that represent that image. So you're throwing away a lot of information that's necessary to then form a high resolution image. So there's a few tricks where you skip a few of the final pooling operations to go in similar way and this is a residual block to go to go to the output produce higher and higher resolution heat map at the end. Segnet in 2015 applied this to the driving context and really taking it to kitty data set and have shown a lot of interesting results and really explored the encoder decoder formulation of the problem. Really solidifying this the place of the encoder decoder framework for the segmentation task. Dilated convolution I'm taking you through a few components which are critical here to the state of the art dilated convolutions. So the convolution operation as the pooling operation reduces resolution significantly and dilated convolution has a certain kind of gridding as visualized there that maintains the local high resolution textures while still capturing the spatial window necessary. It's called dilated convolutional layer and that's in a 2015 paper proved to be much better at upsampling high resolution image. D lab with a B v1, v2, now v3 added conditional random fields which is the final piece of the of the state of the art puzzle here. A lot of the successful networks today that do segmentation, not all do post process using CRFs conditional random fields. And what they do is they smooth the segmentation. The up sampled segmentation that results from the FCN by looking at the underlying image intens. So that's the key aspects of the successful approaches today. You have the encoder decoder framework of a fully convolutional neural network. It replaces the fully connected layers with the convolutional layers de deconvolutional layers. And as the years progress from 2014 to today, as usual, the underlying networks from AlexNet to VGG Net and to now Resnet have been one of the big reasons for the improvements of these networks to be able to perform the segmentation. So naturally they mirrored the ImageNet Challenge performance in adapting these networks. So the state of the art uses RESNET or similar networks conditional random fields for smoothing based on the input image intensities and the dilated convolution that maintains the computational cost but increases the resolution of the upsampling throughout the intermediate feature maps. And that takes us to the state of the art that we used to produce the images to produce the images for the competition. RESNET DUC for Dance up sampling convolution instead of bilinear upsampling you make the upsampling learnable. You learn the upscaling filters that's on the bottom. That's really the key part that made it work. There should be a theme here. Sometimes the biggest addition that could be done is parameterizing one of the aspects of the network that you've taken for granted, letting the network learn that aspect and the other. Not sure how important it is to the success, but it's a. It's a cool little addition. Is a hybrid dilated convolution. As I showed that visualization where the convolution is spread apart a little bit in the input from the input to the output. The steps of that dilated convolution filter, when they're changed, it produces a smoother result because when it's kept the same the there certain input pixels get a lot more attention than others. So losing that favoritism is what's achieved by using a variable different dilation rate. Those are the two tricks, but really the biggest one is the parameterization of the upscaling filters. Okay, so that's what we. That's what we used to generate that data. And that's what we provide you the code with if you're interested in competing in segfuse. The other aspect here that everything we talked about from the classification to the segmentation to making sense of images is there the information about time, the temporal dynamics of the scene is thrown away. And for the driving context, for the robotics context and what we'd like to do with seqfuse for the segmentation dynamic scene segmentation context of when you try to interpret what's going on in the scene over time and use that information, time is essential. The movement of pixels is essential through time. That understanding how those objects move in a 3D space through the 2D projection of an image is fascinating and there's a lot of set of open problems there. So flow is what's very helpful to as a starting point to help us understand how these Pixels move flow optical flow, dense optical flows. The computation our best, our best approximation of where each pixel in image one and moved in the in the temporally following image. After that there's two images in 30 frames a second. There's one image at times zero. The other is 33.3 milliseconds later. And the dense optical flow is our best estimate of how each pixel in the input image moved to in the output image. The optical flow for every pixel produces a direction of where we think that pixel moved and the magnitude of how far moved. That allows us to take information that we detected about the first frame and try to propagate it forward. This is the competition is to try to segment an image and propagate that information forward for manual annotation of an image. So this kind of coloring book annotation where you color every single pixel in the state of the art data set for driving cityscapes that it takes 1.5 hours 90 minutes to do that coloring. That's 90 minutes per image. That's extremely long time. That's why there doesn't exist today a data set. And in this class we're going to create one of segmentation of these images through time, through video. So long videos where every single frame is fully segmented. That's still an open problem that we need to solve. Flow is a piece of that. And we also provide you the this computed state of the art flow using flow net 2.0. So flow net 1.0 in May 2015 used neural networks to learn the optical flow, the dense optical flow. And it did so with two two kinds of architectures flown at S, flown at simple and flown at core flow net C. The simple one is simply taking the two images. So what's, what's the task here? There's two images and you, you want to produce from those two images. They follow each other in time 33.3 milliseconds apart. And your task is the output to produce the dense optical flow. So for the simple architecture you just stack them together. Each are RGB so it produces a six channel input to the network. There's a lot of convolution. And finally it's the the same kind of process as the fully convolution neural networks to produce the optical flow. Then there is flow net correlation architecture where you perform some convolutions separately before using a correlation layer to combine the feature maps. Both are effective in different data sets and different applications. So flow net 2.0 in December 2016 is one of the state of the art frameworks code bases that we use to generate the data I'll show combines the flow net S and flow net C and improves over the initial flow net, producing a smoother flow field. Preserves the fine motion detail along the edges of the objects and it runs extremely efficiently. Depending on the architecture, there's a few variants, either 8 to 140 frames a second. And the process there is essentially one that's common across various applications. Deep learning is stacking these networks together. The very interesting aspect here that we're still exploring and again applicable in all of deep learning. In this case, it seemed that there was a strong effect in taking sparse, small multiple data set and doing the training. The order of which those data sets were used for the training process mattered a lot. That's very interesting. So, using Flow Net 2.0, here's the data set we're making available for PsychFuse, the competition cars. MIT Edu PsychFuse first, the original video US driving in high definition 1080p and A8K 360 video original video driving around Cambridge. Then we're providing the ground truth for a training set. For that training set, for every single frame, 30 frames a second, we're providing the segmentation frame to frame to frame, segmented on mechanical Turk. We're also providing the output of the network that I mentioned, the state of the art segmentation network. That's pretty damn close to the ground truth, but still not. And our task is, this is the interesting thing is our task is to take the output of this network. Well, there's two options. One is to take the output of this network and use other networks to help you propagate the information better. So what this segmentation, the output of this network does is it only takes a frame by frame by frame. It's not using the temporal information at all. So the question is, can we figure out a way? Can we figure out tricks? Can we to use temporal information to improve this segmentation so it looks more like this segmentation. And we're also providing the optical flow from frame to frame to frame. So the optical flow based on flow net 2.0 of how each of the pixels moved. Okay. And that forms a segfuse competition. 10,000 images. And the task is to submit code, we have starter code in python and on GitHub to take in the original video, take in for the training set, the ground truth, the segmentation from the state of the art segmentation network, the optical flow from the state of the art optical flow network and Taking that together to improve the stuff on the bottom left, the segmentation to try to achieve the ground truth on the top right. Okay with that I'd like to thank you. Tomorrow at 1pm is Waymo in Stata 32123. The next lecture next week will be on deep learning for a sense in the human understanding the human. And we will release online only lecture on capsule networks and Gans general adversarial networks. Thank you very much. Added conditional random fields, which is the final piece of the state of the art puzzle here. A lot of the successful networks today that do segmentation, not all do post process using CRFS conditional random fields. And what they do is they smooth the segmentation, the up sampled segmentation that results from the FCN by looking at the underlying image intensities. So that's the key aspects of the successful approaches today. You have the encoder decoder framework of a fully convolutional neural network. It replaces the fully connected layers with the convolutional layers, deconvolution layers. And as the years progress from 2014 to today, as usual, the underlying networks from Alexnet to VGG Net and to now resnet have been one of the big reasons for the improvements of these networks to be able to perform the segmentation. So naturally they mirrored the ImageNet Challenge performance in adapting these networks. So the state of the art uses RESNET or similar networks conditional random fields for smoothing based on the input image intensities and the dilated convolution that maintains the computational cost but increases the resolution of the upsampling throughout the intermediate feature maps. And that takes us to the state of the art that we used to produce the images to produce the images for the competition RESNET DUC for dance upsampling convolution instead of bilinear up sampling, you make the upsampling learnable. You learn the upscaling filters that's on the bottom. That's really the key part that made it work. There should be a theme here. Sometimes the biggest addition that could be done is parameterizing one of the aspects of the network that you've taken for granted, letting the network learn that aspect and the other. Not sure how important it is to the success, but it's a cool little addition is a hybrid dilated convolution. As I showed that visualization where the convolution is spread apart a little bit in the input, from the input to the output. The steps of that dilated convolution filter, when they're changed, it produces a smoother result because when it's kept the same there, certain input pixels get a lot more attention than others. So losing that favoritism is what's achieved by using a variable different dilation rate. Those are the two tricks, but really the biggest one is the parameterization of the upscaling filters. Okay, so that's what we're, that's what we used to generate that data and that's what we provide you the code with if you're interested in competing in SEC Fuse. The other aspect here that everything we talked about from the classification to the segmentation to making sense of images is there. The information about time, the temporal dynamics of the scene is thrown away. And for the driving context for the robotics contest and what we'd like to do is SEGFUSE for the segmentation dynamic scene segmentation context of when you try to interpret what's going on in the scene over time and use that information. Time is essential. The movement of pixels is essential through time. That understanding how those objects move in a 3D space through the 2D projection of an image is fascinating. And there's a lot of set of open problems there. So flow is what's very helpful to as a starting point to help us understand how these pixels move. Flow optical flow, dense optical flows. The computation our best, our best approximation of where each pixel in image 1 and moved in the in a temporally following image after that there's two images in 30 frames a second. There's one image at time zero. The other is 33.3 milliseconds later. And the dense optical flow is our best estimate of how each pixel in the input image moved to in the output image. The optical flow for every pixel produces a direction of where we think that pixel moved and the magnitude of how far moved. That allows us to take information that we detected about the first frame and try to propagate it forward. This is the competition is to try to segment an image and propagate that information forward for manual annotation of an image. So this kind of coloring book annotation where you color every single pixel in the state of the art data set for driving cityscapes that it takes 1.5 hours 90 minutes to do that coloring. That's 90 minutes per image. That's extremely long time. That's why there doesn't exist today a data set. And in this class we're going to create one of segmentation of these images through time through video. So long videos where every single frame is fully segmented. That's still an open problem that we need to solve. Flow is a piece of that, and we also provide you the this computed state of the art flow using flow net 2.0. So flow net 1.0 in May 2015 used neural networks to learn the optical flow, the dense optical flow. And it did so with two two kinds of architectures flown at S, flown at simple and flown at core flown at C. The simple one is simply taking the two images. So what's, what's the task here? There's two images and you, you want to produce from those two images, they follow each other in time 33.3 million.