CS231n Winter 2016: Lecture1: Introduction and Historical Context

There's more seats on this side. Uh, people are walking in late. So, uh, just to make sure, uh, you're in CS231N, the deep learning or neural network, uh, class for visual recognition. Anybody in the wrong class? Okay. Good. All right, so welcome and happy New Year. Happy first day of winter break. So this class, CS231N, this is the second offering of this class when we have literally doubled our enrollment from 180 people last time we offered to about 350 of you signed up. Just a couple of words to make us all legally covered. We are video recording this class, so, you know, if you're uncomfortable about this for today, just go behind the camera or go to a corner that the camera is not going to turn. But we are going to send out forms for you to fill out. In terms of allowing video recording. So that's just one bit of housekeeping. So, all right. My name is Fei-Fei Li. I'm a professor at the computer science department. So this class I'm co-teaching with two senior graduate students, and one of them is here, is Andrej Karpathy. Andrej, can you just say hi to everybody? We have, well, I don't think Andrej needs too much introduction. A lot of you probably know his work follow his blog, his Twitter follower. Andrej has way more followers than I do. So he's very popular. And also Justin Johnson, who is still traveling internationally but will be back in a few days. So Andrej and Justin will be picking up the bulk of the lecture teaching. And today I'll be giving the first lecture, but as you probably can see that I'm expecting a newborn very soon. Speaking of weeks, so you'll see more of Andrej Karpathy and Justin Huang in lecture time. We will also introduce a whole team of TAs towards the end of this lecture. Again, people who are looking for seats, if you go out of that door and come back, there is a whole bunch of seats on this side. Okay. So let's, uh, so this, for this lecture, we're going to give an introduction of the class, the kind of problems we work on, and the tools we'll be learning. So again, welcome to CS231N. This is a vision class. It's based on a very specific modeling architecture called neural network, and even more specifically, mostly on convolutional neural network. And a lot of you hear this term Maybe through a popular press article we, or coverage, we tend to call this the deep learning network. Vision is one of the fastest growing field of artificial intelligence. In fact, Cisco has estimated, and we are on day 4 of this, by 2016, which we already have arrived, more than 85% of the internet cyberspace data is in the form of pixels, or what they call multimedia. So we basically have entered an age of vision, of images and videos. And why is this so? Well, partially, and to a large extent, is because of the explosion of both the internet as a carrier of data, as well as sensors. We have more sensors than the number of people on Earth these days. Every one of you is carrying some kind of smartphones, digital cameras, and cars are running on the street with cameras, so the sensors have really enabled the explosion of visual data on the internet. But visual data or pixel data is also the hardest data to harness. So if you have heard my previous talks and some other talks by computer vision professors, we call this the dark matter of the internet. Why is this the dark matter? Just like the universe is consisted of 85% dark matter, dark energy, is these matters of energy that is very hard to observe. We can infer it by mathematical models in the universe. On the internet, these are the matters, pixel data are the data that we don't know, we have a hard time grasping the contents. Here's one very, very simple aspect for you to consider. So today, YouTube servers, for every 60 seconds, we have more than 150 hours of videos uploaded onto YouTube servers. For every 60 seconds. Think about the amount of data. There's no way that human eyes can sift through this massive amount of data and make annotations labeling it and describe the contents. So think from the perspective of the YouTube team or Google company. If they want to help us to search, index, manage, and of course for their purpose put advertisement or whatever, manipulate the content of the data, We're at a loss because nobody can hand-annotate this. The only hope we can do this is through vision technology to be able to label the objects, find the scenes, find the frames, locate where that basketball video where Kobe Bryant is making that awesome shot. And so these are the problems that we are facing today, the massive amount of data and the challenges of the dark matter. So computer vision is a field that touches upon many other fields of studies. So I'm sure that even sitting here, many of you come from computer science, but many of you come from biology, psychology, are specializing in natural language processing or graphics or robotics or medical imaging and so on. So as a field, computer vision is really a truly interdisciplinary field. The problems we work on, the models we use touches on engineering, physics, biology, psychology, computer science, and mathematics. So just a little bit of a more personal touch, I am the director of the Computer Vision Lab at Stanford. Stanford. In our lab, we— I work with graduate students and postdocs and even undergraduate students on a number of topics, and most dear to our own research, who some of them, you know, Andrej Karpathy, Justin Huang come from my lab, a number of TAs come from my lab. We work on machine learning, which is part of a superset of deep learning. We work a lot on cognitive science and neuroscience as well as the intersection between NLP and speech. So that's, that's the kind of landscape of computer vision research that my lab works in. So also to put things in a little more perspective, what are the computer vision classes that we offer here at Stanford? Stanford through the computer science department. Clearly you're in this class, CS21N, and so you— some of you who have never taken computer vision probably have heard of computer vision for the first time, probably should have already done CS131. That's an intro class of previous quarter we offered, and then And then next quarter, which normally is offered this quarter, but this year is a little shifted, there's an important graduate-level computer vision class called CS231A offered by Professor Silvio Savarese, who works in robotic and 3D vision. And a lot of you ask us the question that, are these, you know, do these replace each other? This class CS231N versus CS231A. And the answer is no. And if you're interested in a broader coverage of tools and topics of computer vision as well as some of the fundamental fundamental topics that comes— that relates you to 3D vision, robotic vision, and visual recognition, you should consider taking 231A. That is the more general class. 231N, which we'll go into starting today more deeply, focuses on a specific angle of both problem and model. The model is neural network, and the end goal is visual recognition mostly. But of course they have a little bit of overlap, but that's the major difference. And next, next quarter we also have possibly a couple of, a couple of advanced seminar level class, but that's still in the formation stage, so you just have to check the syllabus. So that's the kind of computer vision curriculum we offer this year at Stanford. Any questions so far? Yes. Is 131 a strict requirement for this class? 131 is not a strict requirement for this class, but you'll soon see that if you've never heard of computer vision for the first time, I suggest you find a way to catch up because this class assumes a basic level of understanding of computer vision. You can browse the notes and so on. All right. Okay. So the rest of today is that I will give a very brief broad stroke history of computer vision, and then we'll talk about 231N a little bit in terms of the organization of the class. I actually really care about sharing with you this brief history of computer vision because, you know, you might be here primarily because of your interest in this really interesting tool called deep learning, and this is the purpose of this class. We're offering you an in-depth look and just journey through what this deep learning model is. But without understanding the problem domain, without thinking deeply about what this problem is, it's very hard for you to go on to be an inventor of the next model that really solves a big problem in vision or to be developing developing, making impactful work in solving a hard problem. And also, in general, problem domain and model, the modeling tools themselves are never, never fully decoupled. They inform each other. And you'll see through the history of deep learning a little bit that the convolutional neural network architecture come from the need to solve a vision problem. And then vision problem helps the deep learning algorithm to evolve and back and forth. So it's really important to— I want you to finish this course and feel proud that you're a student of computer vision and of deep learning. So you have this— both the toolset and the in-depth understanding of how to use the toolset to tackle important problems. So it's a brief history, but doesn't mean it's a short history. So we're gonna go all the way back to 200— sorry, 540 million years ago. So why did I pick this? You know, on the scale of the Earth's history, this is a fairly specific range of years. Well, so I don't know if you have heard of this, but this is a very, very curious period of the Earth's history. Biologists call this the Big Bang of Evolution. Before 503, 4, 540 million years ago, the Earth is a very peaceful pot of water. I mean, it's a pretty big pot of water. So, We have very simple organisms. These are like animals that just float in the water, and the way they eat and hang out on the daily basis is, you know, they just float, and if some kind of food comes by near their mouth or whatever, they just open the mouth and grab it. And, uh, we don't have too many different types of animals. But something really strange happened around 540 million years. Suddenly, from the fossils we study, there's a huge explosion of species that biologists call speciation. It's like suddenly, for some reason, something hit the Earth that animals start to diversify and they get really complex and they start to, you know, you start to have predators and preys and then they have all kinds of tools to survive. And what was the triggering force of this was a huge question because people were saying, oh, did another, whatever, meteoroid hit the Earth or, you know, did environment change? It turned out one of the most convincing theory is by this guy called Andrew Parker. He's a modern zoologist in Australia, from Australia. He, he studied a lot the fossils, and his theory is that it was the onset of the ice. So, one, one of the first trilobites developed an eye, a really, really simple eye, it's almost like a pinhole camera that just catches light and makes some projections and registers some information from the environment, suddenly life is no longer so mellow. Because once you have the eye, the first thing you can do is you can go catch food. You actually know where food is. You're not just like blind and floating in the water. And once you can go catch food, guess what? The food better develop eyes to run away from you. Otherwise, they'll be gone. So the first animal who had eyes were like in an unlimited buffet. It's like working at Google. It's just like it has the best time eating everything they can. But because of this onset of the ice, what the zoologists realize is the biological arms race began. Every single animal needs to learn to develop things to survive or to, you know, you suddenly have preys and predators and all this. And the speciation began. So that's when vision began, 540 million years. And not only vision begun, vision was one of the major driving force of the speciation or the, the Big Bang of evolution. All right, so, so we're not gonna follow evolution for, with too much detail. Another big important work that focus on engineering of vision happened around the Renaissance, and of course it's attributed to this amazing guy, Leonardo da Vinci. So before Renaissance, you know, throughout human civilization from Asia to Europe to India to Arabic world, we have seen models of cameras. So Aristotle has proposed the camera through the leaves, Chinese philosopher Mozi have proposed the camera through a box with a hole. But if you look at the first documentation of really a modern-looking camera, it's called camera obscura, obscura, and that is documented by Leonardo da Vinci. I'm not gonna get into the details, but this is, you know, you get the idea that there is some kind of lens, or at least a hole to capture light reflected from the real world, and then there is some kind of projection to capture the information of the real-world image. So that's the beginning of the modern, you know, engineering of vision. It started with wanting to copy the world and wanting to make a copy of the visual world. It hasn't got anywhere close to wanting to engineer the understanding of the visual world. Right now we're just talking about duplicating the visual world. So that's one important work to remember, and of course After camera obscura, then we start to see a whole series of successful, you know, some film gets developed, you know, like Kodak was one of the first companies developing commercial cameras, and then we start to have camcorders and all this. Another very important, important piece of work that I want you to be aware of as a vision student is actually not an engineering work but a science, piece of science work that's starting to ask the question is, how does vision work in our biological brain? You know, we now know that it took 540 million years of evolution to get a really fantastic visual system in mammals and in humans, but what did evolution do during this time? What kind of architecture did it develop from that simple trilobite eye to today, yours and mine? Well, a very important piece of work happened at Harvard by two, at that time, young, two very young very young, ambitious postdoc, Hubel and Wiesel. What they did is that they used awake but anesthetized cats, and then there was enough technology to build this little needle called electrode to push the electrode through into the, the, the, the, well, the skull is open, into the brain of the cat, into an area what we already know called primary visual cortex. Primary visual cortex is an area that neurons do a lot of things for, for visual processing. But before Hubel and Wiesel, we don't really know what primary visual cortex is doing. We just know it's one of the earliest stage— other than your eyes, of course— but earliest stage for visual processing, and there's tons and tons of neurons working on vision, and we really ought to know what this is because that's the beginning of vision, visual process in the brain. So they put this electrode into the primary visual cortex, and interestingly, this is another interesting fact, if I don't drop all my stuff I'll show you, primary visual cortex, the The first stage, or second, depending on where you come from, I'm being very, very rough here. First stage of your cortical visual processing stage is in the back of your brain, not near your eye. Okay? It's very interesting because your olfactory cortical processing is right behind your nose, your auditory is right behind your ear, but your primary visual cortex is the furthest from your eye. And another very interesting fact, in fact, not only the primary, there's a huge area working on vision. Almost 50% of your brain is involved in vision. Vision is the hardest and most important sensory perceptual cognitive system in the brain. You know, I'm not saying Anything else is not useful, clearly. But it take nature this long to develop this sensory system, and it takes nature this much real estate space to be used for this system. Why? Because it's so important, and it's so damn hard. That's why we need to use this much space. OK, back to human visual. They were really ambitious. They want to know what primary visual cortex is doing because this is the beginning of our knowledge for deep learning neural network. So they were showing cats. So they put the cats in this room, and they were recording neural activities. And when I say recording neural activity, they're basically trying to see, if I put the neural electrode here, like, do the neurons fire when they see something? So for example, if they show cat, their idea is, if I show this cat a fish— apparently at that time cats eat fish rather than these beans— would the cat's neuron get happy and start sending spikes? And the funny thing here is a story of scientific discovery. A scientific discovery takes both luck and care and thoughtfulness. They were showing this cat fish, whatever, mouse, flower. It just doesn't work. The cat's neuron in the primary visual cortex was silent. There was no spiking or very little spiking, and they were really frustrated. But the good news is that there was no computer at that time. So what they have to do when they show these cats these stimuli is they have to use a slide projector. So they put a, put a slide of a fish and then wait till the neurons spike. If the neuron doesn't spike, they take the slide out and put in another slide. And then they notice every time they change slide, —like this, like, you know, this squarish film. I don't even remember if they used glass or film, but whatever. The neuron spikes. That's weird. You know, like, the actual mouse and fish and flower didn't drive the neuron, excite the neuron, but the movement of taking the slide out or putting a slide in did excite the neuron. It can't be the cat is thinking, oh, finally they're changing the new You know, new object for me. So it turned out there's an edge that's created by this slide that they're changing, right? The slide, whatever, it's a square, rectangular plate. And that moving edge drove or excited the neurons. So they really chased after that observation. You know, if they were too frustrated or too careless, they would have missed that, but they were not. They really chased after that and realized neurons in the primary visual cortex are organized in columns, and for every column of the neurons, they like to see a specific orientation of the stimuli. Simple oriented bars rather than the fish or a mouse. You know, I'm making this a little bit of a simple story because there are still neurons in primary visual cortex we don't know what they like. They don't like simple oriented bars, but by and large, Hubel and Wiesel found that the beginning of visual processing is not a holistic fish or mouse. The beginning of visual processing is simple structures of the world, edges, oriented edges. And this has a very deep, deep implication to both neurophysiology and neuroscience as well as engineering modeling. It's— if later when we visualize our deep neural network features, we'll see that simple a simple edge-like structure emerging from our model. And even though the discovery was in the late '50s, early '60s, they won a Nobel Medical Prize for this work in 1981. So that was another very important piece of work related to vision and visual processing. And so when did computer vision begin? That's another interesting— that's another interesting story, his history. The precursor of computer vision as a modern field was this particular dissertation by Larry Roberts in 1963. It's called Block World. He, uh, just as Hubel and Wiesel were discovering that the visual world in our brain is organized by simple edge-like structures, Larry Roberts, as an early computer science PhD student, were trying to extract these edge-like structures in images. And as a piece of engineering work. And in this particular case, his goal is that, you know, both you and I as humans can recognize blocks no matter how it's turned, right? Like, we know it's the same block. These two are the same block even though the lighting changed and the orientation changed. And his conjuncture is that just like Hubel and Wiesel told us, it's the edges that define this structure. The edges define the shape, and they don't change, rather than all these interior things. So Larry Roberts wrote a PhD dissertation to just extract these edges. It's, you know, if you work as a PhD student in computer vision, this is like undergraduate computer vision, wouldn't have been a PhD thesis, but that was the first precursor computer vision PhD thesis. Larry Roberts is interesting. He kind of gave up his work in computer vision afterwards and went to DARPA and was one of the inventors of the internet. You know, he didn't do too badly by giving up computer vision. But we always like to say that the birthday of computer vision as a modern field is in the summer of 1966. In the summer of 1966, MIT Artificial Intelligence Lab was established. Before that, actually for one piece of history you should feel proud as a Stanford student, there are two pioneering artificial intelligence labs established in the world in the early 1960s, one by Marvin Minsky at MIT, one by John McCarthy at Stanford. At Stanford, the artificial intelligence lab was established before the computer science department. Professor John McCarthy, who founded AI Lab, is the one who is responsible for the term artificial intelligence. So that's a little bit of a proud Stanford history. But anyway, we have to give MIT this credit for starting the field of computer vision because in the summer of 1966, a professor at MIT AI Lab decided it's time to solve vision. AI was established. We start to understand first-order logic and all this, and I think Lisp was probably invented at that time. But anyway, vision is so easy. You open your eyes, you see the world. How hard can this be? Let's solve it in one summer. So— especially MIT students are smart, right? So the Summer Vision Project is an attempt to use our summer workers effectively in the construction of a significant part of a visual system. This was the proposal for last summer. And maybe they didn't use their summer workers effectively, but in any case, computer vision was not solved in that summer. Since then, it become the fastest growing field field of computer vision and AI. If you go to today's premium computer vision conferences called CVPR or ICCV, we have like 2,000 to 2,500 researchers worldwide attending this conference. And a very practical note for students, if you are a good computer vision machine learning students, you will not worry about jobs in Silicon Valley or anywhere else. So it's actually one of the most exciting fields. But that was the birthday of computer vision, which means this year is the 50th anniversary of computer vision. That's a very exciting year in computer vision. And we have come a long, long way. Okay, so continue on the history of computer vision. This is a person to remember, David Marr. He was also at MIT at that time working with a number of very influential computer vision scientists, Shimon Ullman, Tommy Poggio, and David Marr himself died early in the '70s, and he wrote a very influential book called Vision. It's a very thin book, and David Marr's thinking about vision, he took a lot of insights from neuroscience. We already said that Hubel and Wiesel give us the concept of simple structure. Vision starts with simple structure. It didn't start with a holistic fish or holistic mouse. David Marr gave us the next important insight, and these two insights together is the beginning of deep learning architecture, is that vision is hierarchical. So Hubel and Wiesel said, okay, we start simple, but Hubel and Wiesel didn't say we end simple. This visual world is extremely complex. In fact, I take a picture, a regular picture today with my iPhone. There is, I don't know my iPhone's resolution. Let's suppose it's like 10 mega, megapixels. The potential combination of pixels to form a picture in that is bigger than the total number of atoms in the universe. That's how complex vision can be. It's, it's really, really complex. So Hubel and Wiesel told us start simple. David Mark told us build a hierarchical model. Of course, David Mark didn't tell us to build it in a convolutional neural network, which we will cover for the rest of the quarter, but his idea is this: to represent or to think about an image, we think about it in several layers. The first one, he thinks we should think about the edge image, which is clearly an inspiration, took the inspiration from Hubel and Wiesel, and he personally called this the primal sketch. It's, you know, the name is self-explanatory. And then you think about 2.5 This is where you start to reconcile your 2D image with the 3D world. You recognize there is layers, right? You know, I look at you right now, I don't think half of you only has a head and a neck, even though that's all I see. But there is— I know you're occluded by the row in front of you, and this is the fundamental challenge of vision, we have an ill-posed problem to solve. Nature had an ill-posed problem to solve because the world is 3D, but the imagery on our retina is 2D. Nature solved it by first a hardware trick, which is two eyes. It didn't use one eye, but then there's going to be a whole bunch of software trick to merge the information of the two eyes and all this. So the same thing with computer vision. We have to solve that 2.5D problem. And then eventually we have to put everything together so that we actually have a good 3D model of the world. Why do we have to have a 3D model of the world? Because we have to survive, navigate, manipulate the world. When I shake your hand, I really need to know how to extend out my hand and I can grab your hand in the right way, that is a 3D modeling of the world. Otherwise, I won't be able to grab your hand in the right way. When I pick up a mug, the same thing. So that's David Marr's architecture for vision. It's a very high-level abstract architecture. It doesn't really inform us exactly what kind of mathematical modeling we should use. It doesn't inform us of the learning procedure, and it really doesn't inform us of the inference procedure, which we will get into through the deep learning network architecture. But that's the, that's the high-level view, and it's an important, it's an important concept to learn in vision, and we call this the representation. A couple of really important work, and this is a little bit Stanford-centric to just show you, as soon as David Marr laid out this important way of thinking about vision, the first wave of visual recognition algorithms went after that 3D model because that's the goal, right? Like no matter how you represent The goal here is to reconstruct that 3D model so that we can recognize object. And this is really sensible because that's what we go to the world and do. So both of these two influential work comes from Palo Alto. One is from Stanford, one is from SRI. So Tom Binford was a professor at Stanford AI Lab, and he and his Student Rodney Brooks proposed one, one of the first so-called generalized cylinder model. I'm not going to get into the details, but the idea is that the world is composed of simple shapes like cylinders, blocks, and then any real-world object is just a combination of these simple shapes given a particular viewing angle. And that was a very influential visual recognition model in the '70s. And Rodney Brooke went on to become the director of MIT's AI lab, and he was also a founding member of the iRobot company and Roomba and all this. So he continued very influential AI work. Another interesting model coming from local Stanford Research Institute, I think, SRI, is across the street from El Camino, is this pictorial structure model. It's very similar. It focused— it has less of a 3D flavor but more of a probabilistic flavor, is that the objects are made of still simple parts, like a person's head is made of eyes and nose and mouth, and the parts were connected by springs allowing for some deformation. So this is getting a sense of, okay, we recognize the world, not every one of you have exactly the same eyes and the distance between the eyes, we allow for some kind of vari— variability. So this concept of variability start to get introduced reproduced in a model like this. And using models like this, the reason I want to show you this is to see how simple the work was in the '80s. This is one of the most influential models in the '80s recognizing real-world objects, and the entire paper of real-world objects is these shaving razors. But using the edges and simple shapes formed by the edges to recognize this by David Lowe, another Stanford graduate. So that's kind of the ancient world of computer vision where we have been seeing black and white or even synthetic images. Starting the '90s, we finally start to move into colorful images of real world. What a big change. Again, a very, very influential work here. It's not particularly about recognizing an object. It's about how do we carve out an image into sensible parts, right? So if you When you enter this room, there's no way your visual system is telling you, oh my God, I see so many pixels. Right? You immediately have grouped things. You see heads, heads, heads, chair, chair, chair, a stage platform, piece of furniture, and all this. This is called perceptual grouping. Perceptual grouping is one of the most important problem in vision, biological or artificial. If we don't know how to solve the perceptual grouping problem, we're going to have a really hard time to deeply understand the visual world. And you will learn towards the end of this class, this course, a problem as fundamental as this is still not solved in computer vision. Even though we have made a lot of progress, before deep learning and after deep learning, we're still grasping the final solution of a problem like this. So this is again why I want to give you this introduction for you to be aware of the deep problems in vision and also the current state and the challenges in vision. We did not solve all the problem in vision despite whatever the news says. We're far from developing Terminators who can do everything yet. So this piece of work is called Normalized Cut. It is one of the first computer vision work that takes real-world images and tries to solve a very fundamental difficult problem. Jitendra Malik is the senior computer vision researcher, now professor at Berkeley, also Stanford graduate. And you can see the results are not that great. Are we gonna cover any segmentation in this class? We might, right? You'll see we are making progress, but this is the beginning of that. Another very influential work that I want to bring out and pay tribute So for— even though these work, we're not covering them in the rest of the course, but I think it, as a vision student, it's really important for you to be aware of this because not only introduces the important problem we want to solve, it also gives you a perspective on the development of the field. This work is called Vila-Jones face detector, and it's very dear to my heart because As a graduate student, fresh graduate student at Caltech, it's one of the first papers I read as a graduate student when I entered the lab and I didn't know anything, and my advisor said, "Read this amazing piece of work that we're all trying to understand." And then by the time I graduated from Caltech, this very work is transferred to the first the first smart digital camera by Fujifilm in 2006. As the first digital camera that has a face detector. So from a transfer point, technology transfer point of view, it was extremely fast and it was one of the first successful high-level visual recognition algorithm that's being used by consumer product. So this work just learns to detect faces and faces in the wild. It's no longer simulation data or very contrived data. These are any pictures. And again, even though it didn't use a deep learning network, it has a lot of the deep learning flavor. The features were learned. The algorithm learns to find features, simple features like these black and white filter features that can give us the best localization of faces. So this is a very influential piece of work. It's also one of the first computer vision work that is deployed on a computer and can run real time. Before that, computer vision algorithms were very slow. The paper actually is called Real-Time Face Detection. It was granted Pentium II chips. I don't know if anybody remember that kind of chip, but it was on a slow chip, but nevertheless it run real time. So that was another very important piece of work. And also one more thing to point out. Around this time, this is not the only work, but this is a really good representation. Around this time, computer— the focus of computer vision is shifting. Remember that David Marr and the early Stanford work was trying to model the 3D shape of the object. Now we're shifting to recognizing what the object is. We lost a little bit about can we really reconstruct these faces or not. There is a whole branch of computer vision graphics that continue to work on that. But a big part of computer vision is, at this time around the turn of the century, is focusing on recognition. That's bringing computer vision back to AI. And today, the most important part of the computer vision work is focused on these cognitive questions like recognition and AI questions. Another very important piece of work is starting to focus on features. So around the time of face recognition people start to realize it's really, really hard to recognize an object by describing the whole thing. Like I just said, you know, I see you guys heavily occluded. I don't see the rest of your torso. I really don't see any of your legs other than the first row. But I recognize you, and I can infer you as an object. So people start to realize, gee, it's not necessarily that global shape that we have to go after in order to recognize an object. Maybe it's the features. If we recognize the important features on an object, we can go a long way, and it makes a lot of sense. Think about evolution, right? If you're out hunting, you don't need to recognize that tiger's full body and shape to decide you need to run away. You know, just a few patches of the fur of the tiger through the leaves probably can alarm you enough. So we need to— vision is quick. Decision-making based on vision is really quick. A lot of this happens on important features. So this work called SIFT by David Lowe, again, you saw that name again, is about learning important important features on an object. And once you learn these important features, just a few of them on the object, you can actually recognize this object in a totally different angle on a totally cluttered scene. So up to deep learning's resurrection in 2010 or 2012, for about 10 years The entire field of computer vision was focusing on using these features to build models to recognize objects and things, and we've done a great job, we've gone a long way. One of the reasons deep learning network became more and more convincing to a lot of people is we will see that the features that a deep learning network learns is very similar to these engineered features by brilliant engineers. So it kind of confirmed, even though we needed David Lowe to first tell us these features work, and then we start to develop better mathematical models to learn these features by itself, but they confirmed each other. So the historical importance of this work should not be diminished This work is the intellectual foundation for us, one of the intellectual foundations for us to realize that how critical or how useful these deep learning features are when we learn them. I'm going to skip this work and just briefly say because of the features that David Lowe and many other researchers taught us, we can use that to to learn scene recognition. And around that time, the machine learning tools we use mostly is either graphical models or support vector machine, and this is one influential work on using support vector machine and kernel models to recognize a scene, but I'll be brief here. And then One, almost one last model before deep learning model is this feature or feature-based model called deformable part model, is where we learn parts of an object like parts of a person and we learn how they configure each other, they configure in space and use a support vector machine kind of model to recognize objects like humans and bottles. Around this time, that's 2009, 2010, the field of computer vision is matured enough that we're working on these important and hard problems like recognizing pedestrians and recognizing cars. They're no longer contrived problems. Something else was is needed is benchmarking because as a field advancing enough, if we don't have good benchmark, then everybody's just publishing papers on a few set of images and it's really hard to really set global standard. So one of the most important benchmark is called Pascal VOC Object Recognition Benchmark. It's by a European— it's a European effort. That researchers put together tens of thousands of images from 20 classes of objects. These are one example per object like cats, cows, maybe no cats, dogs, cows, airplanes, bottles, you know, horses, trains, and all this. And then we used And then annually our computer vision researchers and labs come to compete on the object recognition task for Pascal Object Recognition Challenge. And over the past, you know, like through the years, the performance just keeps increasing. And that was when we start to feel excited about the progress of the field. At that time, here is a little bit of a more closer story close to us is that my lab and my students were thinking, you know, the real world is not about 20 objects. The real world is a little more than 20 objects. So following the work of Pascal, Visual Object Recognition Challenge, we put together this massive, massive project called ImageNet. Some of you might have heard of ImageNet. In this class, you will be using a tiny portion of ImageNet in some of your assignments. That ImageNet is a dataset of 50 million images, all cleaned by hand and annotated over 20,000 object classes. Don't worry, it's not graduate student who cleaned it. It's— that would be very scary. It's Amazon Mechanical Turk platform, the crowdsourcing platform. Having said that, graduate student also suffered from, you know, putting together this platform, but it's a very exciting dataset, and we started We started to put together competitions annually called ImageNet competition for object recognition. And for example, a standard competition of image classification by ImageNet is 1,000 object classes over almost 1.5 million images, and algorithms compete on the performance. So actually, I just I heard somebody was on the social media was referring ImageNet Challenge as the Olympics of computer vision. That was very flattering. But here is something that— here is bringing us close to the history-making of deep learning. So the ImageNet Challenge started in 2010. That's actually around the time Pascal, you know, our colleagues, they told us they're going to start to phase out their challenge of 20 objects. So we phased in the 1,000-object ImageNet challenge. And y-axis is error rate. And we started with fairly significant error. And of course, every year the error decreased. But there is a particular year. The error really decreased. It was cut in half, or almost, is 2012. 2012 is the year that the winning architecture of ImageNet challenge was a convolutional neural network model. And we'll talk about it. Convolutional neural network was not invented in 2012. Despite how all the news make it sound like it's the newest thing around the block, it's not. It was invented back in the '70s and '80s. But having a convergence of things we'll talk about, convolutional neural network showed its massive power as a high-capacity end-to-end training architecture and won the ImageNet challenge by a huge margin. And that was quite a historical moment. From a mathematical point of view, it wasn't that new. But from an engineering and solving real-world point of view, this was a historical moment. That piece of work was covered by New York Times and all this. This is the onset. This is the beginning of the deep learning revolution, if you call it, and this is the premise of this class. So at this point I'm gonna switch. So we went through a brief history of computer vision for 540 million years, and now I'm gonna switch to the overview of this class. Is there any other questions? Okay, all right, so We've talked about, even though it was kind of overwhelming, we talked a lot about many different tasks in computer vision. CS231n is going to focus on the visual recognition problem. Also, by and large, especially through most of the foundation lecture, we're going to talk about the image classification problem. But now you know everything we talk about is going to be based on that ImageNet classification setup. We will, we will get into other visual recognition scenarios, but the image classification problem is the main problem we will focus on in this class, which means please keep in mind visual recognition is not just image classification, right? There was 3D modeling, there was perceptual grouping and segmentation and all this. But that's what we'll focus on. And I don't need to convince you that just even application-wise, image classification is an extremely useful problem. From big commercial internet companies' app point of view to startup ideas, you want to recognize objects, you want to recognize food, you want to do online shop— mobile shopping, you want to sort your albums, So image classification is, is, is, can be a bread and butter task for many, many important problems. There is a lot of problem that's related to image classification, and today I don't expect you to understand the differences, but I want you to hear that throughout this class we'll make sure you learn to understand the nuance and the details of different flavors of visual recognition. What is image classification? What's object detection? What's image captioning? And these have different flavors. For example, while image classification might focus on the whole big image, object detection might tell you where things exactly are, like where the car is, the pedestrian, or the hammer, and where the relationship between objects, and so on. So there are nuances and details that you will be learning about in this class. And I already said CNN, or Convolutional Neural Network, is one type of deep learning architecture, but it's the the overwhelmingly successful deep learning architecture, and this is the architecture we will be focusing on. And to just go back to the ImageNet challenge, so I said the historical year is 2012. This is the year that Alex Krizhevsky and his advisor Geoff Hinton proposed this, convolutional neural network. I think it's a 7-layer convolutional neural network to win the ImageNet challenge model. Before this year, it was a SIFT feature plus support vector machine architecture. It's still hierarchical, but it doesn't have that flavor of end-to-end learning. And fast forward to 2015, 2015, the winning architecture is still a convolutional neural network. It's 151 layers by Microsoft Asia researchers, and it's called the ResidualNet, right? Is that called the ResidualNet? So I'm not so sure if we're going to cover that, Definitely don't expect to know every single layer, what they do. Actually, they repeat, so it's not that hard. But every year since 2012, the winning architecture of ImageNet Challenge is a deep learning-based architecture. So like I said, I also want you to respect history. CNN is not the first deep learning architecture. Was not invented overnight. There is a lot of influential players today, but there are a lot of people who build the foundation. Actually, I don't have the slides. One important name to remember is Kunihiko Fukushima. Kunihiko Fukushima was a Japanese computer scientist who built a model called NeoCognitron, and that was is the beginning of the neural network architecture. And Yann LeCun is also a very influential person, and he's really— the groundbreaking work, in my opinion, of Yann LeCun was published in the 1990s. So that's when mathematicians, which Geoff Hinton, Yann LeCun's PhD advisor, was involved, worked out the backpropagation learning strategy, which if this word didn't mean anything, Andrej Karpathy will tell you in a couple of weeks. But the mathematical model was worked out in the '80s and the '90s. And this was a— Yaliku was working for Bell Labs at AT&T, which is an amazing place at that time. There's no Bell Labs today anymore. That they were working on really ambitious projects, and he needed to recognize digits because eventually that product was shipped to banks and the US Post Office to recognize zip codes and checks. And he constructed this convolutional neural network, and this is where he's inspired by Hubel and Wiesel. He starts by looking at simple edge-like structures of an image. It's not like the whole letter A. It's really these just edges. And then layer by layer, he, you know, he filters these edges, pull them together, filters, pull, and then they build this architecture. 2012, when Alex Kruszewski and Geoff Hinton used almost exactly the same architecture to participate in the ImageNet challenge, almost there's very few changes, but that become the winning architecture of this. So Andrej Karpathy will tell you more about the detail changes. The capacity, the model did grow a little bit because Moore's Law helped us. There's also a very a very detailed function that changed a little bit of a shape from a sigmoid to a, to a more rectified linear shape, but whatever. There's a couple of small changes, but really by and large nothing had changed mathematically. But two important things did change and that drove the deep learning architecture back into into its renaissance. One is, like I said, Moore's Law. And hardware made a huge difference because these are extremely high-capacity models. When Yaliku was doing this, it's just painfully slow. Because of the bottleneck of computation, he couldn't build this model too big. And once you cannot build it too big, it cannot fully really realize its potential. From machine learning standpoint, there's overfitting and all these problems you cannot solve. But now we have much faster and bigger transistors— not transistors, bigger capacity microchips and GPUs from NVIDIA. NVIDIA made a huge difference in deep learning history that we can now train these models in a reasonable amount of time, even if they're huge. Another thing I think we do need to take credit for is data, the availability of data. That was the big data. Data itself is just, you know, it doesn't mean anything if you don't know how to use it, but in this deep learning architecture, data become the driving force for high-capacity model to enable the end-to-end training to help avoid overfitting when you have enough data. So if you look at the number of pixels that machine learning people had in 2012 versus Yann LeCun had in 1998, it's a huge difference, orders of magnitude. So that's So this is the focus of 231N, but we'll also go— it's also important, one last time I'm gonna drill in this idea that visual intelligence does go beyond object recognition. I don't want any of you coming out of this course thinking we've done everything, you know, we've solved vision and ImageNet challenge defined the entire space of visual of recognition. It's not true. There are still a lot of cool problems to solve. For example, dense labeling of an entire scene with perceptual grouping so I know where every single pixel belong to. That's still an ongoing problem. Combining recognition with 3D is a really— there's a lot of excitement happening at the intersection of vision and robotics. And this is definitely one area of that. And then anything to do with motion affordance, and this is another big open area of research. There is a— I put this here because Justin is heavily involved in this work. You know, beyond just putting labels on a scene, you actually want to deeply understand the picture, what people are doing, what are the relationship between objects, and then we start getting into the relation between objects, and this is an ongoing project called Visual Genome in my lab that Justin and a number of my students are involved, and this goes far beyond image classification we talk about. And what is one of our holy grails? Well, one of the holy grails of computer vision is to be able to tell a story of a scene, right? So think about you as a human. You open your eye. The moment you open your eye, you're able to describe what you see. And in fact, in psychology experiments, we find that even if you show people this picture for only 500 milliseconds? That's literally half of a second. People can write essays about it. We paid them $10 an hour, so they didn't— it wasn't that long, but I figure if we pay them longer, more money, they probably can write longer essays. But the point is that our visual system is extremely powerful. We can tell stories. And I would dream, and this is my challenge to Andrej Karpathy's dissertation, that can we give you, give a computer one picture and out comes a description like this? You know, and we're getting there. You'll see work that you give the computer one picture, it gives you one sentence. Or you give the computer one picture, it gives you a bunch of sentences. Bunch of short sentences. But we're not here yet. But that's one of the holy grail. And another holy grail is continuing this, continuing on this, I think is summarized really well by Andrej Karpathy's blog, is, you know, take a picture like this, right? There's, the stories are so refined. There's so much nuance in this picture that you get to enjoy. Not only you recognize the global scene, it will be very boring if all computer can tell you is man, man, man, room, you know, room, scale, mirror, whatever, cabinet, locker, that's it. You know, here you recognize who they are, you recognize the trick Obama is doing, you recognize the kind of interaction, you recognize the humor, you recognize, There's just so much nuance that this is what visual world is about. We use our ability of visual understanding to not only survive, navigate, manipulate, but we use it to socialize, to entertain, to understand, to learn the world, and this is where vision, the grand goals of vision is. And I don't need to convince you that computer vision technology will make our world a better place, despite some scary talks out there. Even going on today in the industry as well as research world, we're using computer vision to build better robots, to save lives, to go deep exploring, and all this. Now, okay, so I have like what, 2 minutes, 3 minutes, 5 minutes left. Great, time. Let me introduce the team. And Andrej Karpathy and Justin Liad are the co-instructors with me. TAs, please stand up and just say hi to everybody. Can you like just say your name quickly and your like what year and just don't give a speech, but Yeah, start with you. Your name? I'm Tuasis. I'm a 5th year. 5th year what? PhD, School of Screen and Television. Okay. I'm Song. I'm a 4th year PhD student in EE. I'm Bart. I'm a 4th year PhD student in computer science. I'm Albert. I'm a 2nd year master's student in computer science. I'm Serena. I'm in the 3rd year of my PhD with Fei-Fei. I'm Yuma. I'm a 4th year PhD student in computer science. Cool, so these are the heroes behind the scene. And so please stay in touch with us. There are two really the best way, and almost, I almost wanted to say the only way, and I'll tell you what's the exception, is stay in touch through Piazza as well as the staff mailing list. Anything course related, please, please do not send any of us personal email because I'm just gonna say this, if you don't hear replies or your issue is not taken care of because you send a personal email, I'm really sorry because this is a 300+ people class. This mailing list actually tags our email and, and, and help us to process. The only time I respect you to send a personal email mostly to me and Andrej Karpathy and just reason is confidential personal issues, you know, and I understand if you don't want that to be broadcasted to a team of 10 TAs, that's okay, but that should be really, really minimal, uh, the, the only time that you send us an email, and also, you know, just again, I'm going on maternity leave for a few weeks starting the end of January, so, so Please, if you decide you just want to send an email to me and it's my due day for my baby, I'm unlikely going to reply you promptly. Sorry about that. Priorities. So a couple of words about our philosophy. This is— we're not going to get into the details. We really want this to be a very hands-on project. And this is really— I give a lot of credit to Justin and Andrej Karpathy. They are extremely good at walking through these hands-on details with you so that when you come out of this class, you not only have a high-level understanding, but you have a thorough— you have a really good ability to build your own deep learning code. We want you to be exposed to state-of-the-art material. You're gonna be learning things really that's as fresh as 2015. And it'll be fun. You'll get to do things like this. Not all the time, but, you know, like turn a picture into Van Gogh or this weird-looking thing. So it'll be a fun class in addition to all the important tasks. You learn. We do have grading policies. These are all on our website. I'm not gonna iterate this. Again, one thing I want to be very clear, actually two things. One is late policy. You are grown-ups, we treat you like grown-ups. We do not take anything at the end of the course and say, "Oh, my professors want me to go to this conference and I have to have like 3 more late days." No, you are responsible for using your total late days. You have 7 late days. You can use them in whatever way you want with zero penalty. Beyond those, you have to take a penalty. Again, if there's like really, really exceptional medical family emergency, talk to us on an individual basis, but anything else, conference deadline, other final exams, like missing cat or whatever. We budgeted that into the 7 days. Another thing is honor code. This is one thing I have to say with a really straight face. You are in such a privileged institution. You are grown-ups. I want you to be responsible for honor code. Every single Stanford student taking this class should know the honor code. If you don't, there's no excuse, you should go back. We take collaboration extremely seriously. I almost hate to say that statistically, given a class this big, we're gonna have a few cases. But I also want you to be an exceptional class. Even with a size this big, we do not wanna see anything that infringes on academic honor code. So read the collaboration policies and respect that. This is really respecting yourself. I think I'm done with, you know, these prereqs, you can read it. I'm done with anything I want to say. Is there any burning questions that you feel it's worth asking? Yes. What would the midterm exam look like? Like what's going to be more mathematical, more coding, more content? Good question. Andrej, do you have midterm rules? A bit of everything. A bit of everything, which means they haven't figured it out. Yeah, we will give you sample midterms. OK? All right. Thank you. Welcome to the class.