Efficient Computing for Deep Learning, Robotics, and AI (Vivienne Sze) | MIT Deep Learning Series

Happy to have Vivian Si here with us. She's a professor here at MIT working in the very important and exciting space of developing energy efficient and high performance systems for machine learning, computer vision and other multimedia applications. This involves joint design of algorithms, architectures, circuits, systems to enable optimal trade offs between power, speed and quality of result. One of the important differences between the human brain and AI systems is the energy efficiency of the brain. Vivian is a world class researcher at the forefront of discovering how we can close that gap. Please give her a warm welcome.

I'm really happy to be here to share some of the research and an overview of this area of fission computing. So actually, what I'm going to be talking about today is going to be a little bit broader than just deep learning. We'll start with deep learning, but we'll also move to, you know, how we might apply this to robotics and other AI tasks and why it's really important to have efficient computing to enable a lot of these exciting applications. Also, I just want to mention that a lot of the work I'm going to present today is not done by myself, but in collaboration with a lot of folks at MIT over here. And of course, if you want access to the slides, they're available on our website. So given that this is a deep learning lecture series, I want to first start on talking up a little bit about deep neural nets. So we know that deep neural nets has, you know, generated a lot of, a lot of interest, has a very many very compelling applications. But one of the things that has, you know, come into light over the past few years is the increasing need of compute. OpenAI actually showed over the past few years that there's been a significant increase in the amount of compute that is required to perform deep learning applications and to do the training for deep learning over the past few years. So it's actually grown exponentially over the past few years. It's grown, in fact, by over 300,000 times in terms of the amount of compute we need to drive and increase the accuracy of a lot of the tasks that we're trying to achieve. At the same time, if we start looking at basically the environmental implications of all of this processing, it can be quite severe. So if we look at, for example, the carbon footprint of training neural nets, if you think of, you know, the carbon footprint of flying across North America from New York to San Francisco, or the carbon footprint of an average human life, you can see that neural networks are orders of magnitude greater than that. So the environmental or carbon footprint implications of computing for deep Neural nets can be quite severe as well. Now this is a lot having to do with compute in the cloud. Another important area where we want to do compute is actually moving the compute from the cloud in to the edge itself into the device where a lot of the data is being collected. So why would we want to do that? So there's a couple of reasons. First of all, communication. So in a lot of places around the world, and just even a lot of just places in general, you might not have a very strong communication infrastructure. Right. So you don't want to necessarily have to rely on a communication network in order to do a lot of these applications. So again, removing your tethering from the cloud is important. Um, another reason is a lot of the times we, you know, apply deep learning on a lot of applications where the data is very sensitive. So you can think about things like healthcare where you're collecting very sensitive data. And so privacy and security, again, is really critical. And you would, rather than sending the data to the cloud, you'd like to bring the compute to the data itself. Finally, another compelling reason for, you know, bringing the compute into the device or into the robot is latency. So this is particularly true for interactive applications. So you can think of things like autonomous navigation, robotics, or self driving vehicles where you need to interact with the real world. You can imagine if you're driving very quickly down the highway and you detect an obstacle, you might not have enough time to send the data to the cloud, wait for it to be processed and send the instruction back in. So again, you want to move the compute into the robot or into the vehicle itself. Okay, so hopefully this is establishing why we want to move the compute into the edge. But one of the big challenges of doing processing in the robot or in the device actually has to do with power consumption itself. So if we take the self driving car as an example, it's been reported that it consumes over 2000 watts of power just for the computation itself, just to process all the sensor data that it's collecting. Right. And this actually generates a lot of heat. It takes up a lot of space. You can see in this prototype that's being placed, all the compute aspects are being placed in the trunk generates a lot of heat. It generates and often needs water cooling. So this can be a big cost and logistical challenges for self driving vehicles. Now you can imagine that this is going to be much more challenging if we shrink down the form factor of the device itself to something that is perhaps portable in your hands. You can think about smaller robots or Something like your smartphone or cell phone. In these particular cases, when you think about portable devices, you actually have very limited energy capacities. And this is based on the fact that the battery itself is, is limited in terms of the size, weight and cost. Right? So you can't have very large amount of energy on these particular devices itself. Secondly, when you take a look at the embedded platforms that are currently used for embedded processing for these particular applications, they tend to consume over 10 watts, which is an order of magnitude higher than the power consumption that you typically would allow for for these particular handheld devices. So in these handheld devices, typically you're limited to under a watt due to the heat dissipation. For example, you don't want your cell phone to get super hot. Okay, so in the past, you know, decade or so or decades, what we would do to address this challenge is that we would wait for transistors to become smaller, faster and more efficient. However, this has become a challenge over the past few years. So transistors are not getting more efficient. So for example, Moore's Law, which typically makes transistors smaller and faster, has been slowing down. And Dennard Scaling, which has made transistors more efficient, has also slowed down or ended. So you can see here, over the past 10 years, this trend has really flattened out. Okay, so this is a particular challenge because we want more and more compute to drive deep neural network applications, but the transistors are not becoming more efficient. Right. Um, so what we have to turn to in order to address this is we need to turn towards specialized hardware to achieve the significant speed and energy throughputs that we require for our particular application. When we talk about designing specialized hardware, this is really about thinking about how we can redesign the hardware from the ground up, particularly targeted at these AI, deep learning and robotic tasks that we're really excited about. Okay, so this notion is not new. In fact, it's become extremely popular to do this. Over the past few years. There's been a large number of startups and companies that have focused on building specialized hardware for deep learning. So in fact, New York Times reported, I guess it's two years ago, that there's a record number of startups looking at building specialized hardware for AI and for deep learning. Okay, so we'll talk a little bit about what specialized hardware looks like for these particular applications. Now, if you really care about energy and power efficiency, the first question you should ask is, where is the power actually going for these applications? And so as it turns out, power is dominated by data movement, so it's Actually not the computations themselves that are expensive, but moving the data to the computation computation engine, that's expensive. So, for example, shown here in blue is, you know, a range of power consumption, energy consumption for a variety of types of computations, for example, multiplications and additions at various different precisions. So you have, for example, floating point to fixed point and 8 bit integer, and same with additions. And you can see, as it makes sense as you scale down the precision, the energy consumption of each of these operations regardless. But what's really surprising here is that if you look lower at the energy consumption of data movement, right, again, this is delivering the input data to do the multiplication and then moving the output of the multiplication somewhere into memory, it can be very expensive. So for example, if you look at the energy consumption of a 32 bit read from an SRAM memory, this is an 8 kilobyte SRAM. So it's a very small memory that you would have on the processor or on the chip itself. This is already going to consume five picojoules of energy, so equivalent or even more than a 32 bit floating point multiply. And this is from a very small memory. If you need to read this data from off chip, so outside the processor, for example, in dram, it's going to be even more expensive. So in this particular case, we're showing 640 picojoules in terms of energy. And so you can notice here on the horizontal axis that this is basically the, um, this is an exponential axis. So it can, you're talking about orders of magnitude increase in energy in terms of data movement compared to the compute itself. Right? So this is a key takeaway here. So if we really want to address the energy consumption of these particular types of processing, we really want to look at reducing data movement. Okay, but what's the challenge here? So if we take a look at a popular AI robotics type of application like autonomous navigation, the real challenge here though is, is that these applications use a lot of data, right? So for example, one of the things you need to do in autonomous navigation is what we call semantic understanding. So you need to be able to identify, you know, which pixel belongs to what. So for example, in this scene, you need to know that this pixel represents the ground, this pixel represents the sky, this pixel represents, you know, a person itself. Okay, so this is an important type of processing. Often if you're traveling quickly, you want to be able to do this at a very high frame rate. Um, you might need to have large resolution. So for example, typically if you want HD images, you're talking about 2 million pixels per frame. Um, and then often, if you also want to be able to detect objects at different scales or see objects that are far away, you need to do what we call data expansion. For example, build a pyramid for this. And this would increase the amount of pixels or amount of data you need to process by, you know, one to two orders of magnitude. So that's a huge amount of data that you have to process right off the bat there. Um, another type of processing or understanding that you want to do for autonomous navigation is what we call geometric understanding. And that's when you're kind of navigating. You want to build a 3D map of the world that's around you. And you can imagine the longer you travel for the larger the map you're going to build. And again, that's going to be more data that you're going to have to process and compute on. Okay, so this is a significant challenge for autonomous navigation in terms of amount of data. Um, other aspects of autonomous navigations, also other applications like AR VR and so on, is understanding your environment. Right? So a typical thing you might need to do is to do depth estimation. So, for example, if I give you an image, can you estimate the distance of how far a given pixel is from you? And also semantic segmentation, we just talked about that before. So these are important types of ways to understand your environment when you're trying to navigate. And it should be no surprise to you that in order to do these types of processing, the state of the art approaches utilize deep neural nets. Right? Um, but the challenge here is that these deep neural nets often require several hundred millions of operations and weights to do the computation. So when you try and compare it to something like you would all have on your phone, for example, video compression, you're talking about, you know, two to three orders of magnitude increase in computational complexity. So this is a significant challenge because if we'd like to have, you know, deep neural networks be as ubiquitous as something like video compression, we really have to figure out how to address this computational complexity. We also know that deep neural networks are not just used for understanding the environment or autonomous navigation, but it's really become the cornerstone of many AI applications, from computer vision, speech recognition, gameplay, and even medical applications. And I'm sure a lot of these have been covered through this course. Um, so briefly, I'm just gonna give a quick overview of some of the key components in deep neural nets. Not because, you know, I'm sure all of you understand it, but because since this area is very popular, the terminology can vary from discipline to discipline. So I'll just do a brief overview to align ourselves on the terminology itself. Uh, so what are deep neural nets? Uh, basically you can view it as a way of, for example, understanding the environment. It's a chain of, of different layers of processing where you can imagine for an input image at the low level or the earlier parts of the neural net, you're trying to learn different low level features, such as edges of an image. Um, and as you get deeper into the network, as you chain more of these kind of computational layers together, you start being able to detect higher and higher level features until you can, you know, recognize a vehicle, for example. And you know, the difference of this particular approach compared to more traditional ways of doing computer vision is that how we extract these features are learned from the data itself, as opposed to having an expert come in and say, hey, look for the edges, look for, you know, the wheels and so on. The fact that it recognizes these features is a, is a learned approach. Okay, what is it doing at each of these layers? Well, it's actually doing a very simple computation. Um, this is looking at the inference side of things basically. Effectively what it's doing is a weighted sum, right? So you have the input values, um, and we'll color code the inputs as blue here and try and stay consistent with that throughout the talk. Um, we apply certain weights to them and these weights are learned from the training data and then they would generate an output which is typically red here. And it's basically a weighted sum, as we can see. Uh, we then pass this weighted sum through some form of non linearity. So, you know, traditionally used to be sigmoids. More recently we used things like relus, which basically set, you know, non zero values or negative values to zero. Um, but the key takeaway here is if you look at this computational kernel, the key operation to a lot of these neural networks is performing this multiply and accumulate to compute the weighted sum. And this accounts for over 90% of the computation. So if we really want to focus on, you know, accelerating neural nets or making them more efficient, we really want to focus on minimizing the cost of this multiple and accumulate itself. Um, there are also various popular types of deep neural network layer layers used for deep neural networks. Um, they also often vary in terms of, you know, how you connect up the different layers. So for example, you can have feed forward layers where the inputs are always connected to the outputs. Uh, you can have feedback where the outputs are connected back into the inputs. Um, you can have fully connected in inputs where basically all the outputs are connected to all the inputs or sparsely connected. And you might be familiar with some of these layers. So for example, fully connected layers, just like what we talked about, all inputs and all outputs are connected. They tend to be feed forward. When you put them together, they're typically referred to as a multi layer perceptron. You have convolutional layers which are also feed forward, but then you have sparsely connected weight sharing connections. And when you put them together, they often refer to as convolutional networks. And, and they're typically used for image based processing. You have reoccurrent layers where we have this feedback connection, so the output is fed back to the input. When we combine recurrent layers, they're referred to as recurrent neural nets. And these are typically used to process sequential data. So speech or language based processing. And then most recently, which has become really popular, it is the tension layers or tension based mechanisms, they often involve matrix multiply, which is again multiply and accumulate. Um, and they're, when you combine, these are often referred to as transformers. Okay, so let's first kind of get an idea as to why, you know, convolutional or deep learning is much more computationally more complex than other types of processing. So we'll focus on, you know, convolutional neural nets as an example. Although many of these principles apply to other types of neural nets. And the first thing to kind of take a look as to why it's complicated is to look at the computational kernel. So how does it actually perform convolution itself? So let's say you have this 2D input image. If it's at the input of the neural net, would be an image. If it's deeper in the neural net, would be the input feature map. And it's gonna be composed of activations. Or you can think from an image, it's gonna be composed of pixels. And we convolve it with, let's say a 2D filter, which is composed of weights, right? So typical convolution, what you would do is you would do an element wise multiplication of the filter weights with the input feature map activations. You would sum them all together to generate one output value. Now we would refer to that as the output activation, right? And then because it's convolution, we would basically slide the filter across this input feature map and generate all the other output feature map activations. And so this kind, this kind of 2D convolution is pretty standard in image processing. We've been doing this for decades, right? Um, what makes convolutional neural nets much more challenging is the increase in dimensionality. So, first of all, rather than doing just this 2D convolution, we often stack multiple channels. So there's this third dimension called channels. And then what we're doing here is that we need to do a 2D convolution on each of the channels and then add it all together, right? And you can think of these channels for an image. These channels would be kind of the red, green and blue components, for example. And as you get deeper into the feature map, the number of channels could potentially increase. So if you look at Alexnet, which is a popular neural net, the number of channels ranges from 3 to 192. Okay? So that already increases the dimensionality one dimension of the neural neural net itself in terms of processing. Another dimension that we increase is we actually apply multiple filters to the same input feature map. Okay? So, for example, you might apply M filters to the same input feature map, and then you would generate an output feature map of M channels, right? So in the previous slide, we showed that, you know, convolving this 3D filter generates one output channel in the output feature map. If we apply M input M filters, we're going to generate m output channels in the output feature map. And again, just to give you an idea, in terms of the scale of this, when you talk about things like Alexnet, we're talking about between 96 to 384 filters. And of course, this is increasing to, you know, thousands for other advanced or more modern neural nets itself. And then finally, often you want to process more than one image at a given time, right? So if you want to actually do that, we can actually extend it. So N input images become n output images, or n input feature maps becomes n output feature maps. And a typical. We typically refer to this as a batch size, like the number of images you're processing at the same time. And this can range from 1 to 256. Okay? So these are all the various different dimensions of the neural net. And so really, what someone does when they're trying to define what we call the network architecture of the neural net itself is that they're gonna select the different or define the shape of the neural network for each of the different layers. So it's gonna, you know, define all these different dimensions of the neural net itself. And these shapes can vary across the different layers, just to give you an Idea. If you look at MobileNet as an example, this is a very popular neural network cells. You can see that the filter sizes, right? So the height and width of the filters and the number of filters and number of channels will vary across the different blocks or layers itself. Um, the other thing I just want to mention is that when we look towards popular DNN models, we can also see important trends. So shown here are the various different models that have been developed over the years that are quite popular. A couple of interesting trends. One is that the networks tend to become deeper. So you can see in the convolutional layers, they're getting deeper and deeper. And then also the number of weights that they're using and the number of Macs are also increasing as well. So this is an important trend. The DNN models are getting larger and deeper. And so again, they're becoming much more computationally demanding. And so we need more sophisticated hardware to be able to process. All right, so that's kind of a quick intro or overview into the deep neural network space. I hope we're all aligned. So the first thing I'm going to talk about is how can we actually build hardware to make the processing of these neural networks more efficient and to run faster? And often we refer to this as hardware acceleration. All right, so we know these neural networks are very large. There's a lot of compute. But are there types of properties that we can leverage to make computing or processing of these networks more efficient? Uh, so the first thing that's really friendly is that they actually exhibit a lot of parallelism. So all these multiplies and accumulates, you can actually do them all in parallel. Right. So that's great. So that what that means is high throughput or high speed is actually possible because I can do a lot of these processing in parallel. What is difficult and what should not be a surprise to you now is that the memory access is the bottleneck. So delivering the data to the multiply and accumulate engine is what's really challenging. So I'll give you an insight as to why this is the case. So let's take, say we take this multiply and accumulate engine, what we call a Mac. It takes in three inputs for every Mac. So you have the filter weight, you have the input image pixel, or if you're deeper in the network, it would be input feature Mac activation. And it also takes the partial sum, which is like the partially accumulated value from the previous multiply that it did, and then it would generate an updated partial sum. So for every computation that you do. For every Mac that you're doing, you need to have four memory accesses. So it's a 4 to 1 ratio in terms of memory accesses versus compute. The other challenge that you have is, as we mentioned, moving data is gonna be very expensive. So in the absolute worst case, and you would always try to avoid this, if you read the data from DRAM off chip memory, every time you access data from dram, it's going to be two orders of magnitude more expensive than the computation of performing a Mac itself. Okay, so that's really, really bad. So if you can imagine, again, if we look at Alexnet, which has 700 million Macs, we're talking about 3 billion DRAM accesses to do that computation. Okay. But again, all is not lost. There are some things that we can explore to help us along with this problem. So one is what we call input data reuse opportunities. Which means that a lot of the data that we're reading, we're using to perform these multiplies and accumulates. They're actually used for many multiplies and accumulates. So if we read the data once, we can reuse it multiple times for many operations. Right. So I'll show you some examples of that. First is what we call convolutional reuse. So again, if you remember, we're taking a filter and we're sliding it across this input image. And so as a result, the, the activations from the feature map and the weights from the filter are gonna be reused in different combinations to compute the different multiply and accumulate values, or different max itself. So there's a lot of what we call convolutional reuse opportunities there. Another example is that we're actually, if you recall, gonna apply multiple filters on the same input feature map. So that means that each activation in that input feature map can be reused multiple times across the different filters. Finally, if we're gonna process many images at the same time or many feature maps, a given weight in the filter itself can be reused multiple times across these input feature maps. So that's what we called filter reuse. Okay, so there's a lot of these great filter reuse opportunities in the neural network itself. And so what can we do to exploit this reuse opportunities? Well, what we can do is we can build what we call a memory hierarchy that contains very low cost memories that allow us to reduce the overall cost of moving this data. So what do we mean here? We mean that if I have, if I build a multiply and accumulate engine, I'm gonna have a very small memory right beside the multiply and accumulate engine. And by small, I mean something on the order of under a kilobyte of memory locally beside that multiply and accumulate engine. Why do I want that? Because accessing that very small memory can be very cheap. Uh, so for example, if to perform a multiply and accumulate with an ALU, it's 1x reading from this very small memory beside the multiply and accumulate engine is also going to be the same amount of energy. I could also allow these processing elements, and a processing element is going to be this multiply and accumulate, plus the small memory. I can also allow the different processing elements to also share data. Okay? And so reading from a neighboring processing element is going to be 2x the energy. And then finally, you can have a shared larger memory called a global buffer. And that's going to be able to be shared across all different processing elements. This tends to be larger, between 100 and 500k bytes. And that's going to be more expensive, about 6x the energy itself. And of course, if you go off chip to DRAM, that's going to be the most expensive at 200x the energy. And so the big issue here is the way that you can think about this is what you would ideally like to do is to access all of the data from this very small local memory. But the challenge here is that this very small local memory is only 1k byte. But we're talking about neural networks that are millions of weights in terms of size, right? So how do we go about doing that? So there's many challenges of doing that. Just as an analogy for you guys to kind of think through how this is related, you can imagine that accessing something from like, let's say your backpack is going to be much cheaper than accessing something from your neighbor or, you know, going back to, let's say, your office here somewhere on campus to get the data versus going back all the way home, right? So ideally you'd like to access all of your data from your backpack, but if you have a lot of work to do, you might not be able to fill it in your backpack. So the question is, how can I break up my large piece of work into smaller chunks so that I can access them all from this small memory itself? And that's the big challenge that you have. And so there's been a lot of research in this area in terms of what's the best way to break up the data and what should I store in this very small local memory? So one approach is what we call weight stationary. And the idea here is I'm going to store the weight information of the neural net into this small local memory, okay? And so as a result it really minimize the weight energy. But the challenge here is that the other types of data that you have in your system, so for example, your input activations shown the blue and then the partial sums that are shown in the red, now those still have to move through the rest of the system itself, so through the network and from the global buffer. Okay. Typical types of work that are popular that use this type of kind of data flow or weight stationary data flow, which is what we call it because the weight remains stationary are, are things like the TPU from Google and the NVDLA accelerator from Nvidia. Another approach that people take are they, well, they say, well, so the weight, I only ever have to read it, but the partial sums, I have to read it and write it because the partial sum I'm gonna read accumulate, like update it and then write it back to them. So there's two memory accesses for that partial sum data type. So what, maybe I should put that partial sum locally into that small memory itself. So this is what we call output stationary. Because the accumulation of the output is gonna be local within that one processing element that's not gonna move. Uh, the trade off, of course, that the activations awaits now have to move through the network. Um, and then there's various different works called, like for example, some work from KU Leuven and some work from the Chinese Academy of Sciences that are using this approach. Um, another piece of work is saying, well, you know, forget about the inputs and the, or so the outputs and the waste themselves. Let's keep the input stationary within this small membrane. It's called input stationary. And some of the work, again from some research work from Nvidia has examined this. But all of these different types of work really focus on not moving. One piece of type of data either focuses on minimizing weight energy or partial sum energy or input energy. I think what's important to think about is that maybe you want to reduce the data movement of all different data types, all, all types of energy, right? So another approach, this is something that we've developed within our own group, is looking at what we call the row stationary data flow. And within each of the processing elements you're gonna do one row of convolution. Um, and this row is a mixture of all the different data types, right? So you have filter information. So the weights of the Filter, you have the activations of your input feature map, and then you also have your partial sum information. So you're really trying to balance the data movement of all the different data types, not just one particular data type. Um, this is just performing a one row. But you. We just talked about the fact that the neural network is much more than a 1D convolution. So you can imagine expanding this to higher dimensions. So this is just showing how you might expand this 1D convolution into a 2D convolution. And then there's other, you know, higher dimensionality that you can map onto this architecture as well. I won't get to the details of this, but the key takeaway here is that you might not want to focus on one particular data type. You want to actually optimized for all the amount, all the different types of data that you're moving around in your system. Okay. And this can just show you some results in terms of how these different data types or these type of different types of data flows would work. So, for example, in the weight stationary case, as expected, the weight energy, the energy required to move the weights shown in green is going to be the lowest. But then the red portion, which is the energy of the partial sums, and the green or so the blue part, which is the input feature map or input pixels, the. That's going to be very high. Output stationary is another approach. As we talked about, you're trying to reduce the data movement of the partial sums shown here in red. So the red part is really minimized. But you can see that the green part, which is the weight stationary data movement or weight movement, is going to be increased, and the blue is the input is going to be increased. There's another approach called no local reuse. We don't have time to talk about that. But you can see that row stationary, for example, really aims to balance the data movement of all the different data types. Right. So the big takeaway here is that, you know, when you're trying to optimize, you know, a given piece of hardware, you don't want to just optimize one, you know, for one particular type of data. You want to optimize overall for all the movement in the hardware itself. Okay. Another thing that you can also exploit to save a bit of power is the fact that, you know, some of the data could be zero. So we know that anything multiplied by zero is gonna be zero, right? So if you know that one of the inputs to your multiply and accumulate is gonna be zero, you might as well skip that multiplication. In fact, you might as well skip, you know, accessing data or accessing the other input to that multiply and accumulate engine. So by doing that you can actually reduce the power consumption by almost 50%. Another thing that you can do is that if you have a bunch of zeros, um, you can also compress the data. Uh, for example, you can use things like run length encoding which where basically a run of zeros is gonna be represented rather than you know, 00,000. You can just say I have a run of five zeros. And this can actually reduce the amount of data movement by up to 2x in your system itself. And in fact in you know, neural nets there's a large way, you know, possibilities of actually generating zeros. First of all, if you remember that reloop, it's setting negative values to zero, so naturally generates zeros. And then there's other techniques. For example, we call pruning, which is setting some of the weights of the neural net to zero as well. And so this can exploit all of that. Okay, so you know, what is the impact of all these types of things? So we actually looked at building hardware in particular a customized chip that we called Iris to demonstrate these particular approaches. In particular the row stationary data flow and exploiting sparsity in the activation data. So this iris chip has 14 by 12 so 168 processing elements. You can see that there's a shared buffer that's 100 kilobytes and it has some compression decompression before it goes to off chip dram. And again that's because accessing DRAM is the most expensive. Shown here on the right hand side is a die photo of the fabricated chip itself. Right. And this is 4 millimeters by 4 millimeters in terms of size. And so using that you know, row stationary data flow, it exploits a lot of data reuse. So it actually reduces the uh, the number of times we access this global buffer by 100x. And it also reduces the amount of times we access the off chip memory by over 1000x. This is all because you know, each of these processing elements has you know, a local memory that is trying to read most of its status from and it's also sharing with the other processing. So overall when you compare it to a mobile gpu, you're talking about an order of magnitude reduction in energy consumption. If you'd like to learn a little bit more about that, I invite you to visit the Iris project website. Okay, so this is great. We can build Custom hardware. But what does this actually mean in terms of building a system that can efficiently compute neural nets? So let's say we take a step back. Let's say we don't care anything about the hardware and we're, you know, a systems provider. We want to build, you know, an overall system and what we really care about is the trade off between energy and accuracy. Right? That's, that, that's the key thing that we care about. Um, so shown here is a plot and let's say this is for an object detection task, right? So accuracy is on the X axis and it's listed in terms of average precision, which is a metric that we use for object detection. Um, it's on a linear scale and higher the better. Vertically we have energy consumption. This is the energy that's being consumed per pixel. So you kind of average it. You can imagine a higher resolution image can consume more energy. It's going to be an exponential scale. So let's first start on the accuracy axis. And so if you think before neural nets had its resurgence in around 2011, 2012, actually state of the art approaches used features called histogram of oriented gradients. Right. This is a very popular approach to be very efficient in terms of or quite accurate in terms of object detection and we refer to as hog. The reason why, you know, neural nets really took, took off is because they really improve the accuracy. So you can imagine Alex set here almost doubled the accuracy and then vgg, you know, further increased the accuracy. So it's super exciting there. But then we want to look also on the vertical axis which is the energy consumption. And I should mention basically you'll see these dots. We have the energy consumption for each of these different approaches. These approaches are actually measured or these energy numbers are measured on specialized hardware already that's been designed for that particular task. So we have a chip here that's built in 65 nanometer CMOS process. So they use the same transistors around the same size that does object detection using the HOG features. And then here's the Iris chip that we just talked about. I should also note that both of these chips are built. In my group, the students who built these chips started designing the chips at the same time and taped out at the same time. So it's somewhat of a controlled experiment in terms of optimization. What does this tell us? When we look on the energy axis, we can see that histogram of oriented gradients or HOG features are actually very efficient from an energy point of view. In fact, if we compare it to something like video compression, again, something that you all have in your phone. Hog features are actually more efficient than video compression, meaning for the same energy that you would spend compressing a pixel, you could actually understand that pixel. So that's pretty impressive. Um, but if we start looking at AlexNet or VGG, we can see that the energy increases by two to three orders of magnitude, which is quite significant. Uh, I'll give you an example. So if I told you on your cell phone, I'm going to double the accuracy of its recognition, but your phone would die 300 times faster. Who here would be interested in that technology? Right, so. Exactly. So nobody. Right? So in the sense that battery life is so critical to how we actually use these types of technologies, so we should not just look at the accuracy, which is the x axis point of view, we should really also consider the energy consumption. And we really don't want the energy to be so high. And we can see that even with specialized hardware, we're still quite far away from making neural nets as efficient as something like video compression that you all have on your phones. So we really have to think of how we can further push the energy consumption down without sacrificing accuracy, of course. Okay, so actually there's been a huge amount of research in this space because we know neural nets are popular and we know that they have a wide range of applications, but energy is really a big challenge. So people have looked at how can we design new hardware that can be more efficient, or how can we design algorithms that are more efficient to enable energy efficient processing of DNNs? And so, in fact, within our own research group, we spend quite a bit of time surveying the area and understanding what are the various different types of developments that people have been looking at. So if you're interested in this topic, we actually generated various tutorials on this material as well as overview papers. This is an overview paper that's about 30 pages and we're currently expanding it into a book. So if you're interested in this topic, I would encourage you to visit these resources. But the main thing that we learned about as we were doing this kind of survey of the area is that we actually identified various limitations in terms of how people are approaching or how the research is approaching this problem. So first, let's look on the algorithm side. So again, there's a wide range of approaches that people are using to try and make the DNN algorithms or models more efficient. So for example, we've kind of mentioned the idea of pruning the idea Here is you're going to set some of the weights to become zero, and again, anything times zero is zero, so you can skip those operations. So there's a wide range of research there. Um, there's also looking at efficient network architectures. Meaning rather than making my neural networks very large, these high three dimensional convolutions, can I decompose them into smaller filters? Right. So rather than this 3D filter, can I make it a 2D filter and kind of, you know, also 2D, but you know, one by one and into the screen itself. Another very popular thing is reduced precision. So rather than using the default of 32 bit float, can I reduce the number of bits down to eight bits or even binary? We saw before that as we reduce the precision of these operations, you also get energy savings and you also reduce data movement as well because you have to move less data. A lot of this work really focuses on reducing the number of Macs and then number of weights and those primarily because those are easy to count. But the question that we should be asking if we care about the system is does this actually translate into energy savings and reduce latency? Because from a system's point of view, those are the things that we care about, right? We don't really, you know, when you're thinking about something running on your phone, you don't care about the number of max and waves, you care about how much energy it's consuming because that's going to affect the battery life or how quickly it might react, right? That's, that's basically a measure of latency. And again, hopefully you haven't forgotten, but basically data movement is expensive, right? So you really depends on, you know, how you move the data through the system. So the key takeaway from this slide is that if you remember where the energy come from comes from, which is the data movement, it's not because of how many weights or how many max you have, but really it depends on where the weight comes from. If it comes from this small, you know, small memory register file that's nearby, it's gonna be super cheap as opposed to coming from off chip dram. So all weights are basically not created equal, all Macs are not created equal. It really depends on the memory hierarchy and the data flow of the hardware itself. Okay. So we can't just look at the number of weights and the number of Macs and estimate how much energy is gonna be consumed. So this is quite a difficult challenge. So within our group we've actually looked at developing different tools that allow us to estimate the energy consumption of, of the neural network itself. So, for example, in this particular tool, which is available on this website, we basically take in, you know, the DNN weights and the input data, including its sparsity. We know the different shapes of the different neural, of the different layers of the neural net and we run an optimization that figures out, you know, the memory access, how much, you know, the energy consumed by the data movement, and then the energy consumed by the multiply and accumulate computations. And then the output is going to be a breakdown of the energy for the different layers of the, the neural network. And once you have this, you can kind of figure out, well, where is the energy going? So I can target my design to minimize that energy consumption. Okay. And so by doing this, when we take a look, it should be no surprise. One of the key observations for this exercise is that the weights alone are not a good metric for energy consumption. If you take a look at Googlenet, for example, this is running on kind of the IRIS architecture, you can see that the weights are only account for 22% of the overall energy. In fact, a lot of the energy goes into moving the input feature maps and the output feature maps as well. Right. And also computation. So in general, this is the same message as before. We shouldn't just look at the data movement, one particular data type, we should look at the energy consumption of all the different data types to give us an overall view of where the energy is actually going. Okay, and so once we actually know where the energy is going, how can we factor that into the design of the neural networks to make them more efficient? So we talked about the concept of pruning. Right? So again, pruning was setting some of the weights of the neural net to zero, or you can think of it as removing some of the weights. And so what we want to do here is that now we know that we know where the energy is going, why don't we incorporate the energy into design of the algorithm, for example, to guide us to figure out where we should actually remove the weights from? You know, so for example, let's say here, this is on Alexnet for the same accuracy across the different approaches. Traditionally what happens is that people tend to remove the weights that are small. And we call this magnitude based pruning. And you can see that you get about a 2x reduction in terms of energy consumption. However, we know that like the size of the weight has nothing to do with, or the value of the weight has nothing to do with the energy consumption. Ideally, what you'd like to do is remove the weights that consume the most energy. In particular, we also know that the more weights that we remove, the accuracy is going to go down. So to get the biggest bang for your buck, you want to remove the weights that consume the most energy first. One way you can do this is you can take your neural network, figure out the energy consumption of each of the layers of the neural network. You can sort, then sort the layers in terms of high energy layer to low energy layers, and then you prune the high energy layers first. So this is what we call energy aware pruning. And then by doing this, you actually now get a 3.7x reduction in energy consumption compared to 2x for the same accuracy. And again, this is because we factor in energy consumption into the design of the neural network itself. All right, and the prune models are all available on the IRIS website. Another important thing that we care about from performance point of view is latency. So, for example, latency has to do with how long it takes when I, you know, give it an image, how long will I get the result back? People are very sensitive to latency. Um, but the challenge here is that latency again is not directly correlated to things like number of multiplies and accumulates. And so this is some data that was released by Google's mobile vision team and they're showing here on the X axis, the number of multiplies and accumulates. You can do so going towards the left, you're increasing. And then on the Y axis, this is the latency. So this is actually the measured latency or delay it takes to get a result. And what they're showing here is that the number of Macs is not really a good approximation of latency. So in fact, for example, given, you know, layers, neural networks that have the same number of Macs, there can be a 2x range or 2x swing in terms of latency, or looking at it in a different way, giving, you know, layer neural nets of the same latency. They can have a 3x swing swing in terms of number of max. Right? So the key takeaway here is that you can't just count the number of max and say, oh, this is how quickly it's gonna run. It's actually much more challenging than that. And so what we wanna ask is, is there a way that we can take latency and use that again to design the neural net directly? So rather than looking at max, use latency. And so, together with Google's mobile vision team, we developed this approach called NetAdapt and this is really a way that you can tailor your particular neural network for a given mobile platform for a latency or an energy budget. Right. So it automatically adapts the neural net for that platform itself. And really what's driving the design is empirical measurements, so measurements of how that particular network perform on that platform. So measurements for things like latency and energy. And the reason why we want to use empirical measurements is that you can't often generate models for all the different types of hardware out there. In the case of Google, what they want is that if they have a new phone, you can automatically tune the network for that particular phone. You don't want to have to model the phone as well. Okay, and so how does this work? I'll walk you through it. So you'll start off with a pre trained network. So this is a network that's, let's say, trained in the cloud for very high accuracy. Great, start off with that. But it tends to be very large, let's say. And so what you're going to do is you're going to take that into the NetAdapt algorithm. You're going to take a budget. So a budget will tell you like, oh, I can afford only this type of latency or this amount of latency, this amount of energy. Um, what NetAdapt will do is gonna generate a bunch of proposals, so different options of how it might modify the network in terms of its dimensions. It's gonna measure these proposals on that plat target platform that you care about. Uh, and then based on these empirical measurements, NetAdapt is gonna then generate a new set of proposals and it'll just iterate across this until it gets an adapted network as an output. Okay. And again, all of this is on the NetAdapt website. Um, just to give you a quick example of how this might work. So let's say you start off with as your input a neural network, you know, that has the accuracy that you want, but the latency is 100 milliseconds and you would like for it to be 80 milliseconds. You want it to be faster. So what it's gonna do is it's gonna generate a bunch of proposals and what the proposals could involve doing is taking one layer of the neural net and reducing the number of channels until it he hits the latency budget of 80 milliseconds. And it can do that for all the different layers. Then it's going to tune these different layers and measure the accuracy. Right. So let's say, oh, this one where I just shortened the Number of channels in layer one maintains accuracy at 60%. So that means I'm going to pick that and that's going to be the input or the output of this particular design. So the output at 80 milliseconds, hitting an accuracy of 60%. And it's going to be the input to the next iteration. And then I'm going to tighten the budget. Okay. Again, if you're interested, I just invite you to go take a look at the NetAdapt paper. But what are the, what is the impact of this particular approach? Well, it gives you actually a very much improved trade off between latency and accuracy. Right. So if you look at this plot again, on the X axis is the latency, right? So to the left is better. So it's lower latency. And then on the X axis or Y axis it's gonna be the accuracy. So higher, better. So here you want to higher to the left is good. And so we have first shown in blue and green various kind of handcrafted neural network based approaches. And you can see netap, which generates the red dots as it's iterating through its optimization. And you can see that it achieves for the same accuracy, it can be up to 1.7x faster than a manually designed approach. This approach is also under the umbrella of basically network architecture. Search is kind of also in that kind of flavor. But in general, the takeaway here is that if you're gonna design neural networks or efficient neural networks that you wanna run quickly or you wanna be energy efficient, you should really take, you know, put hardware into the design loop and take in, you know, the accurate energy or latency measurements into the design itself of the neural network. Um, this particular, you know, example here is shown for an image classification task, meaning I give you an image and you can classify it to the right. Um, you can say what's in the image itself. Um, you can imagine that that type of approach is kind of like reducing information right from a 2D image. You reduce it down to a label. This is very commonly used. Um, but we actually want to see if we can still apply this approach to a more difficult task of something like depth estimation. In this case, you know, I give you a 2D image and the output is also a 2D image where each pixel shows the depth of each or, you know, the output or the picture is basically showing the depth of each pixel at the input. Um, this is often what we refer to as, you know, monocular depth. So I give you just a 2D, you know, depth image input you can estimate the depth itself. The reason why you want to do this is, you know, 2D cameras or regular cameras are pretty cheap, right? So it'd be ideal to be able to do this. Um, you can imagine like the way that we would do this is to use an autoencoder. So the front half of the neural net is still looking like a, what we call an encoder. It's a reduction element. So this is very similar to what you would do for a classification. But then the backend of the autoencoder is a decoder, so it's going to expand the information back up. Right. And so as I mentioned again, this is going to be much more difficult than just classification because now my output has to be also very dense as well. Um, and so we want to see if we could make this really fast with approaches that we just talked about. For example, NetAdapt. So indeed you can make it pretty fast. So if you apply NetAdapt plus the, you know, compact network design and then do some depthwise decomposition, you can actually increase the frame rate by an order of magnitude. So again, here I'm gonna show the plot on the x axis. Here is the frame rate on a Jetson TX2 GPU. This is measured with a batch size of one with 32 bit float. Um, and on the vertical axis it's the accuracy, the depth estimation in terms of the Delta 1 metric, which means the percentage of pixels that are within 25% of the correct depth. So higher the better. And so you can see the various different approaches out there. This star, red star is the approach using a fast depth, using all the different efficient network design techniques that we talked about. And you can see you can get an order of magnitude over a 10x speed up while maintaining accuracy. And the models and all the code to do this is available on the Fast Step website. We presented this at icra, which is a robotics conference in the middle of last year and we want to show some live footage there. So at icra we actually captured some footage on an iPhone and showed the real time depth estimation on an iPhone itself. And you can do achieve about 40 frames per second on an iPhone using fast depth. So again, if you're interested in this particular type of application or efficient networks for depth estimation, invite you to visit the website for that. Okay, so that's the algorithmic side of things, but let's return to the hardware, building specialized hardware that are efficient for neural network processing. So again we saw that there's many different ways of making the neural Network efficient from network pruning to efficient network architectures to reduce precision. The challenge for the hardware designer, though, is that there's no guarantee as to which type of approach someone might apply to the algorithm that they're gonna run on the hardware, right? So if you only own the hardware, you don't know what kind of algorithm someone's going to run on your hardware unless you own the whole stack. So as a result, you really, really need to have flexible hardware so it can support all of these different approaches and translate these approaches to improvements in energy efficiency and latency. Now, the challenge is a lot of the specialized DNN hardware that exists out there often rely on certain properties of the DNN in order to achieve high efficiency. So a very typical structure that you might see is that you might have an array of multiply and accumulate units or a Mac array, and it's going to reduce memory access by amortizing reads across arrays. What do I, what do I mean by that? So if I read a weight once from the memory, weight memory once, I'm going to reuse it multiple times across the array, send it across the array. So one read and it can be used multiple times by multiple engines for multiple max. Similarly, activation memory, I'll read it input activation once and reuse it multiple times. Okay. Uh, the issue here is that the amount of reuse and the array utilization depends on the number of channels you have on your neural net, the size of the feature map, and the batch size, right? So this is again just showing two different variations of, you know, you're gonna reuse based on the number of filters, number of input channels, uh, feature map, batch size. And the problem now is that when we start looking at these efficient neural network models, they're not gonna have as much reuse, right? Particularly for the compact cases. So, for example, a very typical approach is to use what we call depthwise layers. We saw you took that 3D filter and then decomposed it into a 2D filter and a one by one, right? And so as a result, you only have one channel, so you're not gonna have much reuse across the input channel. And so rather than, you know, filling this array with a lot of computation that you can process, you're only gonna be able to utilize a very small subset, which I've highlighted here in green, of the array itself for computation. So even though you throw down, you know, 1,000 multiplies, 10,000 multiplies accumulate engine, only a very small subset of them can actually do work. And that's not great. So this is also an issue because as I scale up the array size, it's gonna become less efficient. Ideally what you would like is that if I put more, you know, cores or processing elements down, the system should run faster, right? I'm paying for more thing, more cores. But it doesn't because it can't, the data can't reach or be reused by all of these different cores. And it's also made difficult to exploit sparsity. So what you need here are two things. One is a very flexible data flow, meaning that there's many different ways for the data to move through this array. Right. And so you can imagine row stationary is a very flexible way that we can basically map the neural network onto the array itself. You can see here in the iris or rose stationary case that you know, a lot of the processing elements can be used. Another thing is how do you actually deliver the data for this varying degree of reuse? Um, so here's like the spectrum of, you know, on chip networks in terms of basically how can I deliver data from that global buffer to all those parallel processing engines. Right. Um, one use case is when I use these huge neural nets that have a lot of reuse. I, what I want to do is multicast, meaning I read once from the global buffer and then I reuse that data multiple times in all of my processing elements. You can think of that as like broadcasting information out. And a type of network that you would do for that is shown here on the right hand side. So this is low bandwidth, so I'm only reading very little data. But high spatial reuse, many, many engines are using it. On the other extreme, when I design these very efficient neural networks, I'm not gonna have very much reuse. And so what I want is unicast, meaning I'm gonna, I don't wanna spend, send out unique information to each of the processing elements, right? So that they can all, you know, work. Um, so that's gonna be as shown here on the left hand side, a case where you have very high bandwidth because there's a lot of unique information going out and low spatial reuse. So not sharing data. Now it's very challenging to go across this entire spectrum. One solution would be what we call an all to all network that satisfies all of this. So all things are connect, all inputs are connected, all outputs that's gonna be very expensive and not scalable. Um, one solution that we have to, this is what we call a hierarchical mesh. So you can break this problem into two steps at the lowest level you can use an all to all connection, right? And then at the higher level you can use a mesh connection. And so the mesh will allow you to scale up. Um, but the all to all allows you to achieve a lot of different types of reuse. And with this type of network on chip, you can basically support a lot of different delivery mechanisms to deliver data from the global buffer to all the processing elements, so that all your cores, all your computes can be happening at the same time. At its core, this is one of the key things that enable the second version of Iris to be both flexible and efficient. So this is some results from the second version of Iris. It supports a wide range of filter states, both the very large shapes as well as very compact, including convolutional fully connected depthwise layers. So, so you can see here in this plot, you know, depending on the shape, you can get up to an order of magnitude speed up. It also supports a wide range of sparsities, both dense and sparse. So this is really important because some networks can be very sparse because you've done a lot of pruning, but some are not. And so you want to efficiently support all of those. You also want to be scalable. So as you increase the number of processing elements, you know, the throughput also speeds up. Um, and as a result of this particular type of design, you get an order of magnitudes improvement in both speed and energy efficiency. All right, so this is great, and this is one way that you can, you know, speed up and make neural networks more efficient. Um, but it's also important to take a step back and look beyond just, you know, building specialized hardware, the accelerator itself, both in terms of algorithms and the hardware. So can we look beyond the DNN accelerator for acceleration? Um, and so one good place to show this, an example is the task of super resolution. So how many of you are familiar with the task of super resolution? All right, so for those of you who aren't, the idea is as follows. So I wanna basically generate a high resolution image from a small resolution image. And why do you wanna do that? Well, there are a couple of reasons. One is that it can allow you to basically reduce the transmit bandwidth. So for example, if you have limited communication, I'm gonna send a low res version of a video, let's say, or image to your phone, and then your phone, you can make it high res. Okay, that's one way. Um, another reason is that, you know, screens in general are getting larger and larger. So every year at CES they announce like A higher resolution screen. But you know, if you think about the movies that we watch, they're all, a lot of them are still 1080p, for example, or fixed resolution. So again, you want to generate a high resolution representation of that, you know, low resolution input. Um, and the idea here is that your high resolution is not just interpolation because it can be very blurry, but there's ways that like kind of hallucinate a high resolution version of the video or image itself. Um, and that's basically called super resolution. Um, but the, one of the challenges for super resolution is that it's computationally very expensive. So again, you can imagine that the state of the art approaches for super res use deep neural nets. Um, a lot of the examples we just talked about, neural nets are talking about input images of like 200 by 200 pixels. Now imagine if you extend that to like an HD image, it's gonna be very, very expensive. So what we wanna do is think of different ways that we can speed up the super resolution process. Not just by making DNNs faster, but kind of looking around the other components of the system and seeing if we can make it faster as well. So one of the approaches we took is this framework called fast, where we're looking at accelerating any super resolution algorithm by an order of magnitude. And this is operated on a compressed version video. So you know, before I was a faculty here, I worked a lot on video compression. And if you think about the video compression community, they look at very video very differently than people who process super resolution. So typically when you're thinking about image processing or super resolution, when I give you a compressed video, what you basically think of it is as a stack of pixels, right? A bunch of different images together. But if you asked a video compression person, you know, what is a compressed video look like? It actually compressed video is a very structured representation of the redundancy in the video itself. So why is it that we can compress videos is because things like different frames look very, you know, consecutive frames look very similar. So it's telling you, you know, which pixels in frame one is related to which pixel or looks like which pixel in frame two. And so as a result, you don't have to send the pixels in frame two and that's where you get the compression from. So actually what a compressed video looks like is a description of the structure of the video itself. Okay. And so you can use this representation to accelerate super resolution. So for example, rather than applying super resolution to every single low res frame, which is the Typical approach that you would apply super resolution to each low res frame and you would generate a bunch of high res frame outputs. What you can actually do is apply super resolution to one of the small low resolution frames and then you can use that free information you get in the compressed video that tells you the structure of the video to generate or transfer and generate all those high resolution videos from that. And so it only needs to run on a subset of frames. And then the complexity to reconstruct all those high resolution frames once you have that structured image is going to be very low. So for example, if I'm going to transfer to N frames, I'm going to get an end frame and X speed up. So to evaluate this, we showcase this on a range of videos. This range of videos is the data set that we use to develop video standards. So it's quite broad. And you can see first on the left hand side is that if I transfer to four different frames, you can get a four acceleration and then the PSNR which indicates the quality doesn't change. So it's the same quality but 4x faster. If I do transfer to 16 frames or 16 acceleration, there's a slight drop in quality, but still you get a, basically a 16x acceleration. So the key idea here is again you want to look beyond, you know, the processing of the neural network itself to around it to see if you can speed it up. Um, usually with psnr you can't really tell too much about the quality. So another way to look at it is actually look at the video itself or subjective quality. So on the left hand side here, this is if I applied super resolution on every single frame. So this is a traditional way of doing it. Um, on the right hand side here, this is if I just did interpolation on every single frame. And so where you can tell the difference is by looking at things like the text. You can see that the text is much sharper on the left video than the right video. Now fast plus src9 using fast is somewhere in between. So fast actually has the same quality as the video on the left hand side, but it's just as efficient in terms of processing speed and as the approach on the right hand side. So it kind of has the best of both worlds. Um, and so the key takeaway for this is that if you want to accelerate DNNs for a given process, it's good to look beyond, you know, the hardware for the acceleration, but you can look at things like the structure of the data that's entering the neural network, accelerator There might be opportunities there. For example, here, temporal correlation that allows you to further accelerate the processing. Um, again, if you're interested in this, all the code is on the website. So to end this lecture, I just want to talk about things that are actually beyond deep neural nets. I also, I know neural nets are great, they're useful for many applications. But I think there's a lot of exciting problems outside the space of neural nets as well, which also require efficient computing. Um, so the first thing is what we call visual inertial localization or visual odometry. This is something that's widely used for robots to kind of figure out where they are in the real world. So you can imagine for autonomous navigation before, before you navigate the world, you have to know where you actually are in the world. So that's localization. This is also widely used for things like AR and VR as well, because you can know where you're actually looking in AR&VR. What does this actually mean? It means that you can basically take in a sequence of images. So you can imagine like a camera that's mounted on the robot or the person as well as an imu. So it has accelerometer and gyroscope information. And then visual inertial odometry, which is a subset of slam, basically fuses this information together and the outcome of visual understoodometry is the localization. So you can see here, basically you're trying to estimate where you are in the 3D space and the pose based on in this case the camera feed. But you can also measure IMU information there as well. And if you're in an unknown environment, you could also generate a map of that environment. So you know, one of these is a very key task in navigation. And the key thing is can you do it in a very energy efficient way. So we've looked at kind of building specialized hardware to do, do localization. This is actually the first chip that performs like complete visual inertial odometry. On chip we call it navion. This is done in collaboration with Certesh Kerman. So you can see here, here's the chip itself. It's 4 millimeters by 5 millimeters. You can see that it's smaller than a quarter. And imagine mounting it on a small robot at the front end. It does basically processing of the camera information. It does things like feature detection, tracking, outlier elimination. It also processes, does pre integration on the IMU and then on the backend it fuses this information together using a factor graph. Okay. And so when you compare you know, this particular design, this Navion chip design, compared to mobile or desktop CPUs, you're talking about two to three orders of magnitude reduction in energy consumption because you have this specialized chip to do it. So what, you know, what is the key, key component of this chip that enables us to do it? Well, again sticking with the theme, the key thing is reduction in data movement. In particular, we reduce the amount of data that needs to be moved on and off chip, so all of the processing is located on the chip itself. And then furthermore, because we want to reduce the size of the chip and the size of the memories, we do things like apply low cost compression on the frames and then also exploit sparsity, which means number of zeros in the factor graph itself. So all of the compression and exploring sparsity can actually reduce the storage cost down to under a megabyte of storage on chip to do this processing. And that allows us to achieve this really low power consumption of 25 milli of below 25 milliwatts. Um, another thing that really matters for autonomous navigation is once you know where you are, where are you gonna go next? So this is kind of a planning and mapping problem. Um, and so in the context of things like robot exploration, where you wanna basically explore, explore an unknown area, you can do this by doing what we call Shannon's computing, Shannon's mutual information. Basically you want to figure out where should I go next where I will discover the most amount of new information compared to what I already know. Right. So you can imagine. So shown here is like an occupancy map. So this is basically the light colors show the place where it's free space, it's empty, nothing's occupied. The dark gray area is unknown. And then the black lines are occupied. Things like walls, for example. And the question is, if I know that this is my current occupancy map, where should I go and scan, let's say with the depth sensor to figure out, you know, more information about the map itself. So what you can do is you can compute what we call the mutual information of the map itself based on what you already know. And then you go to the location with the most information and you scan it and then you get an updated map. So shown here below is a miniature race car that's doing exactly that. Right. So over here, oops, over here is the mutual information that's being computed. So it's trying to go to those light, light areas of the yellow areas that has the most information. So you can see that it's going to try and back up and come and scan this region to cover or figure out more information about that. Okay, so that's great. It's a very principled way of doing this. The problem of this kind of computation, the reason why it's been challenging is again, the computation in particular the data movement. So you can imagine at any given position, you're gonna do like a kind of a 3D scanning with your lidar across a wide range of neighboring regions with your beams. You can imagine each of these beams with your LiDAR scan can be processed with different cores, so they can all be processed in parallel. So parallelism again here, just like the deep learning case, is very easily available. The challenge is data delivery. Right? So what happens is that you're actually storing your occupancy map all in one memory. But now you have multiple cores that are gonna try and process the scans on this occupancy map. Right. And so you only actually, typically for these types of memories, you're limited to two pores. But if you wanna have, you know, n core, 16 cores, 30 cores, it's gonna be a challenge in terms of how to read data from this occupancy map and deliver to the cores themselves. Um, if we take a closer look at, you know, the memory access pattern, you can see here that as you scan it out, the numbers indicate which cycle you would use to read, you know, each of the locations on the map itself. Okay. And you can see it's kind of a diagonal pattern. So the question is, can I break this map into smaller memories, right. And then access these smaller memories in parallel? And the question is, if I can break it into smaller memories, how should I decide what part of the map should go into which of these memories? So show here on the right hand side, um, in the different, different colors basically indicate different memories or different banks of the memory. So they store different parts of the map. And again, if you think of the numbers as, you know, the cycle with which each location is accessed, what you'll notice is that for any given color, at most two numbers are the same. Meaning that I'm only gonna access two pieces of the location for any given bank or memory. So there's gonna be no conflict. So I can process all of these beams in parallel. Okay. And so by doing this, this allows you to compute the mutual information of the entire map. Um, and by entire map can be a very large map, let's say 200 meters by 200 meters at 0.1 meter resolution, um, in under a second this is very different from before, where, you know, you can only compute the mutual information of a subset of locations and just try and pick the best one. Now you can compute on the entire map so you can know the absolute best location to go to to get the most information. Um, this is 100x speed up compared to a CPU at a tenth of the power, right, on an fpga. So that's another important example of how data movement is really critical in order to allow you to process things very, very quickly. And having, having specialized hardware can enable that. All right, so one last thing is looking at, you know, so we talked about robotics, talked about deep learning. But actually what's really important is there's a lot of important applications that, where you can apply efficient processing that can help a lot of people around the world. In particular, looking at monitoring neurodegenerative disease disorders, we know things like dementia, things like Alzheimer's, Parkinson's, affects tens of millions of people around the world and continues to grow. This is a very severe disease. The challenge for this disease is that one of the many challenges, but one of the challenges that the neurological assessments for these disease can be very time consuming and required a trained specialist. So normally if you are suffering from one of these diseases or you might have this disease, what you need to do is you need to go see a specialist and they'll ask you a series of questions, right? They'll do like a mini mental example, like what year is it? Where are you now? Can you count backwards? And so on. Or you might be familiar with, like people are asked to draw the clock, these tests. And so you can imagine going to a specialist to do these type of things can be costly and time consuming. So you don't go very frequently. So as a result, the data that's collected is very sparse. Also, it's very qualitative. If you go to different specialists, they might come up with a different assessment. Repeatability is also very much an issue. What's been super exciting is it's been shown in literature that there's actually a quantitative way of measuring or quantitative evaluating these types of diseases, potentially using eye movements. Eye movements can be used by a quantitative way to evaluate the severity of or progression or regression of these particular type of diseases. You can imagine doing things like if you're taking a certain drug, is your disease getting better or worse? Eye movement can give a quantitative evaluation for that. But the challenge is that to do these eye movement evaluations, you still need to go into that first you need a very high speed Camera that can be very expensive. Often you need to have substantial head support so your head doesn't move, so you can really detect the eye movement. You might even need IR illumination so you're more clear, can more clearly see the eye. Again, this still has the challenge that for clinical measurements of what we call saccade latency or eye movement latency or eye reaction time, they're done in very constrained environments. You still have to go see the special itself. And they use very specialized and costly equipment. In the vein of enabling efficient computing and bringing compute to various devices. Our question is, can we actually do these eye measurement measurements on a phone itself that we all have? And so indeed you can develop various algorithms that can detect your eye reaction time on a consumer grade camera like your phone or an iPad. And we've shown that you can actually replicate the quality of results as you could with a phantom camera. So shown here in the red are basically eye reaction times that are measured on a subject on an iPhone 6, which is obviously under $1,000 way cheaper. Now compared to a phantom camera shown here in blue, you can see that the distributions of the reaction times are about the same. Why is this exciting? Because it enables us to do low cost in home measurements. What you can imagine is a patient could do these measurements at home for many days, not just the day they go in. Then they can bring in this information and this can give the physician or the specialist additional information to make the assessment as well. This can be complementary, but it gives a much more rich system set of information to do the diagnosis and evaluation. So we're talking about computing, but there's also other parts of the system that burn power as well. In particular, when we're talking about things like depth estimation, using time of flight. Time of flight is kind of very similar to lidar. Basically what you're doing is you're sending a pulse and waiting for it to come back. And how long it takes to come back indicates the depth of whatever object you're trying to detect. The challenge with depth estimation, with time of flight sensors that can be very extreme expensive, right? You're emitting a pulse, waiting for it to come back. So talking about up to tens of watts of power, the question is, can we also reduce the sensor power if we can do efficient computing? So for example, can I reduce how often I emit the depth sensor and kind of recover the other information just using a monocular based camera? So for example, typically you have a pair of a depth sensor and an RGB camera. If at time zero, I Turn both of them on. And time one and two I turn them off, but I still keep my RGB camera on. Can I estimate the depth for at time two and time three? Okay, and then the key thing here is to make sure that the algorithms that you're running to estimate the depth without turning on the depth sensor itself is super cheap. So we actually have algorithms that can run on VGA at 30 frames per second on a Cortex A7 which is a super low cost embedded processor. And just to give you an idea of how it looks like. So let's see, here's the left is the RGB image. In the middle is the depth map or the ground truth. If I always have the depth sensor on, that's what it would look like. And then on the right hand side is the estimated depth map. In this particular case, we're only turning on the sensor only 11% of the time. So every, every ninth frame and your mean at relative error is only about 0.7%. So the accuracy, air quality is pretty aligned. Okay. Um, so you know, at a high level, what are the key takeaways I want you guys to get from today's lecture? Uh, first is efficient computing is really important. It can extend the reach of AI beyond the cloud itself because it can reduce communication, networking costs, enable privacy and provide low latency. And so we can use AI for a wide range of applications ranging from things like robotics to healthcare. Um, and in order to achieve this, energy efficient computing, it really requires cross layer design. So not just focusing on the hardware, but specialized hardware plays an important role, but also the algorithms itself. And this is gonna be really key to enabling AI for the next decade or so or beyond. Okay. And we also covered a lot of points in the lecture, so the slides are all available on our website. Um, also, just because it's deep learning seminar series, I just want to point some other resources that you might be interested if you want to learn more about efficient processing of neural nets. So again, I want to point you first to this survey paper that we've developed. This is with my collaborator Joel Emmer. It really kind of covers what is, you know, what are the different techniques that people are looking at and give some insights of the key design principles. We also have a book coming soon, it's going to be within the next few weeks. We also have slides from various tutorials that we've given on this particular topic. In fact, we also teach a course on this here at MIT. 6, 825. If you're interested in updates on all these types of materials, I invite you to join the mailing list or the Twitter feed. The other thing is, if you're not an MIT student but you want to take a two day course on this particular topic, I also invite you to take a look at the MIT professional education option. We run short courses on MIT campus over the summer. So you can come for two days and we can talk about the various different approaches that people use to build efficient deep learning systems. Uh, and then finally, just if you're interested in just video and tutorial videos on this talk, I actually at the end of November during Neurips, I gave like a 90 minute tutorial that goes really in depth in terms of how to build efficient deep learning systems. So I invite you to visit that. And we also have some talks at like the Mars Conference on Efficient Robotics and we have a YouTube channel where this is all located. Um, and then finally, I'd be remiss if I didn't acknowledge a lot of the work here is done by the students. So all the students in our group as well as my collaborators, Joel Emmer, Certesh Kerman and Thomas Helt, and then all of our sponsors that make this research possible. So that concludes my talk. Thank you very much. Sa.