Stanford CS230 | Autumn 2025 | Lecture 2: Supervised, Self-Supervised, & Weakly Supervised Learning

I'm Kian Katon Furush and I am the co creator and co lecturer with Andrew for this class CS230.

And I will teach about half of the in person lectures this quarter. Outside of Stanford, I work in industry. I lead a company called Workera which uses AI to measure skills. Um, and with the history of CS230 students that have started AI startups and companies, what I try to do usually is to bring a lot of examples from industry. So what you should expect from these in class lectures is not as much of the academic side of things, which we learn anyway in the online videos, but also the industry specific input. And some of the topics that we'll cover this year together include decision making in AI projects, which we're gonna see today. You know, I want you to come out of today's lecture feeling like you had some fun. It was interactive. And also you have a better way to make decisions in AI projects because you've seen how deep learning researchers, engineers and scientists make their decisions solving problems in industry. Um, other topics later in quarter for the in classroom time include things like, you know, adversarial attacks and defenses. We might have some time to cover it today, deep reinforcement learning, which is really hot in the market right now and I think it's very important to know about it. And then all the stuff that is very practical like retrieval, augmented generation, AI agents, multi agent system. As we go deeper into the class and you get the baggage of neural networks, we'll be able to cover even more fun topics. Okay, so today's lecture is gonna be structured in three parts, uh, maybe four depending on whether we have time. We'll start with a little recap of the week, um, what you've learned online about neurons and layers and deep neural networks. Then we get into a set of supervised learning projects, um, including a day and night simple, you know, vanilla classification, the trigger word detection, which, which is actually a project you're gonna build at the end of the class yourself. Um, and then face verification, which we'll see also variation of how face verification algorithms work. In the third section we'll focus on self supervised learning and weekly supervised learning. Don't worry if you don't know these terms, we're gonna learn them together. Um, and we talk a lot about embeddings because embeddings are the connective AI systems online today and it's important to know about them. And finally, if we have time, we'll also talk about adversarial attacks and defenses with more and more AI Systems in the wild, knowing how to defend them is very important, and knowing how to attack them can also teach you how to defend them. So we'll cover that as well. Sounds good. Please interrupt me as we go through the lecture. We want this to be very, um, conversational as much as possible. So, recap of the week. Um, is the, the core way that AI learns from data in a traditional supervised learning setup. You can think of it as an input, such as this little image of the confused cat, and an output, in this case a number between 0 and 1 that represents the chance that there might be a cat on the picture one or there is no cat on the picture zero. What the model is. And oftentimes you'll see me refer to the model as two things. There's an architecture which is essentially the blueprint of the model, the skeleton and parameters. It might be a few parameters, it might be billions of parameters like the models that OpenAI, DeepMind and others work on.

Um, outside of that.

And so when you think about AI models being deployed in the wild, like when you think about what's happening with ChatGPT, you know, you can, you can really come down to there's two files somewhere on the cloud. One that describes the architecture of the model, one that describes the parameters that are part of this architecture. And you keep calling that to those two files and you get your inference or your outputs. That's really what's happening behind the scene. Much more complicated than that, obviously, but those are the two critical components of a neural network architecture and its parameters that are trained.

How does the model learn is through a gradient descent optimization, meaning I send the picture of the cat through the model and the model at the beginning is not trained. So it's probably wrong. It tells me, I think there is no cat, I think it's zero. And then I use something, something called the loss function to compare the ground truth. There is a cat on the picture with the prediction from the model. At this point in time, those two numbers are far from each other. That should be a penalty which the loss function describes. And then in order to give feedback to the parameters, we use this gradient descent update. We do that many, many times. What it means is that we take our parameters and we tell them, hey, you should go a little bit more to the right or a little bit more to the left. Um, until that number, that is the prediction for the cat is closer to the ground truth. We do that with batches of data, millions of images of cats, and images of anything else, and we give that feedback repetitively to the model until the parameters are calibrated and the model is in fact finding the cat on this picture. Nothing new here. You've seen it in the videos. Any question on that learning setup? No. Okay, easy. So far there's many things that can change in this setup and you'll see in the class. First thing is the input. The input does not have to be an image. It can be text, like when you chat, you know, with ChatGPT, it can be audio, it can be video, it can be structured data, it can be spreadsheets and numbers. Those will see a variety of examples in the class and how it influences the architecture. Um, the output again doesn't have to be 0 and 1. This is an example of a classification. Um, you could turn this problem into a regression. Uh, for example, if I was asking you what's the age of the cat? Estimate the age of the cat. That would be a regression task, not a classification task anymore. Later in the class we'll also see generative task. In fact, lecture four is gonna focus on diffusion models, generative adversarial networks where the output actually is much bigger than the input typically, you know, so you can have a low resolution of a cat as input and the output is a high resolution of the same cats. The, the output is bigger than the inputs, which can be counterintuitive to people. Um, other things that can change include the architecture. Um, you've learned about the vanilla multi layer perceptron or the fully connected neural network. That's what we're learning right now online together. Um, by the end of the class, you'll have many architectures that you'll be familiar with from RNNs and convolutional neural networks, transformer models. All of these at the end of the day use the basis neural network that you're learning right now. They're just stacked on top of each other differently. Uh, the loss function is actually a big focus of today's class and of the class in general. The loss function, which is what gives the feedback to the model. Um, you were right or you were wrong. And what to do about it is an art designing good loss functions. You know, great deep learning researchers are very creative when it comes to designing loss functions. And in fact when will built the algorithm called yolo, it's, it is called yolo for you only loop once, not you only live once. But YOLO has a very, you know, difficult to understand at first loss function. And there's a reason why the loss function was Designed like that. So by the end of this class, you'll also have a better intuition on how do we design great loss functions. Uh, other things I'm not gonna cover right now. The activation functions in your neural network, the optimizer that you use for your gradient descent loop, and then the hyperparameters that might come in when you train your algorithms. Okay, nothing new here. Um, this is the basic setup you've also learned this week about neurons. Uh, the easiest way to think about a neuron is the classic logistic regression algorithm where I'm taking the image of the cat. So an image in computer science, with the way the machine reads it, is three channels RGB for the three colors. Red, green, blue. And we take all these numbers, we put them in a vector. The vector is then fed into a neuron, and the neuron has two components. The linear part W transpose X plus B, W being the weights and B being the bias. And then an activation function, in this case the sigmoid function, which is very handy because it takes any number and it puts it between 0 and 1 so that the output can look like a probability. Classic setup. And here the probability is 0.73, which is above 0.5, which tells me the model thinks there's a cat on the picture because one is a cat, zero is no cat.

So, question for you to get started. How would you modify this binary classification that detects cats in an algorithm that would be able to detect multiple animals, such as a cat, a dog, and a giraffe. What do you need to change about this neural network? Yeah.

Okay, so you would change the output layer to match to the number of animals you want to detect. Yeah. Correct. Anyone wants to add anything else? Yeah, the data that goes in there. How would you change the data?

Okay, very good. Yeah, you need. You need data from dogs and giraffes, and also maybe nature in general. What else do we need? Not to forget?

Yeah, maybe you could add a neuron.

Yeah. Okay. Add one neuron per animal. Those neurons will be independent from each other, and each neuron would focus on one animal. Yeah, good point. It's actually what we're going to do. So, yes, I think your suggestions were the right one. We could multiply this output layer to have three neurons instead of one. All of them. Because it's a fully connected neural net. See the entire pixels flattened in the vector, and then each of them will be focused on an animal. The number one mistake that we see in projects is that people add more data but forget to adjust the labels. So how do the labels need to be adjusted here? It's not anymore 0 and 1. Right. What type of LA labels do we need.

To train this?

Yeah.

Yeah, okay. You know how we call that?

Okay, so. Yeah, yeah.

Yeah. Vectors. Yeah, I think you're seeing the same thing, but yeah. So here you would, you would use a one hot vector or a multi hot. You know, one hot means we'll have.

You'Ll have a vector of size three and if there is a cat on the picture, the label is gonna be 010 because the second neuron is gonna be responsible to detect cats. Um, in fact, that would be called the one hot vector. Uh, oftentimes you'll have multiple animals on the picture because cats and dogs can appear together. Cats and giraffes less so. And dogs and giraffes. I've never seen one in the same picture. But anyway, um, you'll have a multi hot vector. If you have a cat and a dog on the picture, you'll probably label it as 1, 1, 0. And the reason I'm ment It may sound silly, but in a lot of projects people change their data, they forget to change their labels and then they wonder why it doesn't work.

Okay, cool. Now, in the class we use a specific notation with superscript and subscript. So when you'll see me refer to something like a 11. The superscript in square brackets indicates the layer that you're in. So you're in the first layer. The subscript refers to the, the index of the neuron. Okay. And A is for activation. So a subscript 3 square bracket 1 is A is the NE, the output of the third neuron of the first layer. Okay, again, if I continue, second layer would be written like that and then you'll get your probability. The deeper the network is, the more capacity it has. This is the word we use, capacity. What it means is that if you send a million pictures of cats and a million pictures of non cats to a shallow network, it might not have the capacity to learn what's in the dataset. It's just not flexible enough. The deeper the network, the more capacity it has. So in fact, a network that is super deep. Imagine a billion parameters transformer model with 1 million pictures of cat and non cat. It will just overfit to those pictures, meaning it's not going to learn what a cat is, it's just going to learn by heart those million pictures because its capacity is way bigger than the data set it's fed. So it's very Important to understand the amount of data you're going to feed and the complexity diversity of that data will probably dictate the capacity of the models you need to use.

Okay, now just to give you a little bit more intuition on what happens inside those neural networks, we take this relatively shallow network, but, you know, call it three layers, and we train it on a data set of facial images. Uh, ignore the task in, in, in face data sets, there's a lot of tasks you could do. You could do a face verification, you could do a face recognition, you could do, um, you know, face clustering, things like that. We, we'll talk about that later. But let's say it's been trained really well on understanding faces. If you now unpack this network and you sort of query each neuron and look what's going on inside, what you'll notice is that the first layers are going to be better at encoding low complexity features, while the deeper networks are going to be better at encoding higher complexity features. So here's how it goes. Nothing too complicated for now. Um, the neuron in the first layers, they're looking at pixels because you're giving them directly the pixels. So they're going to be good at stitching those pixels together. And maybe the first neuron will be good at detecting a diagonal edge, the second neuron will be good at vertical edges, and the third one at horizontal edges because they're just looking at pixels and trying to make sense out of them. Now you go one layer deeper in the middle of the network, those are not seeing pixels, they're seeing the output of the first layer, which is already slightly more complex. So what you can expect the layers in the middle of the network to give you or to activate for is higher level features like an eye or a nose or an ear. Because it turns out if you have a few edges, you can start detecting circles. And so you would see a neuron that is really good at detecting circles. Eyes. The deeper you go in the network, the more you get closer to the task itself, which in this case facial analysis, let's say you would see the last few neurons detect larger features of the face because again, they're seeing higher complexity information. Does that make sense? This concept, you know, we call it encoding. We'll also talk about embeddings. It's very important because when you train neural networks, you want to make sure first that they're understanding what they're doing. And that's one way to see it. We'll have an entire lecture on interpreting and visualizing neural networks later this quarter. Um, and on top of that, um, you probably can make use of some of those encodings and embeddings. We, we'll see that later. But why? In the vector space, the distances between those concepts are important. You can already imagine that for, for tasks like search, searching in a database, having a neural network able to encode information in a very meaningful way can allow you to find concepts that are close to each other and associate them with each other and concepts that are far from each other and dissociate them from each other.

Okay, so this was the warm up for today. How much time we spent? Okay, 15 minutes on the warmup. That's good. So we learned a few new words. Model architecture parameters. I didn't talk about it, but feature engineering versus feature learning, that's the core concept in, in deep learning is feature engineering is what we used to do before deep learning, which is uh, you might actually build an algorithm that is good at detecting eyes. It's just uh, good at scanning eyes. You manually build it, you know, and then you build another one that is good at detecting, uh, mouth. A mouth. And then you put them together to feel and detect faces. We don't do that anymore. We do end to end learning. Meaning we let the data speak for itself and train the model. This is called feature learning. It's automatic and that's how the neural network actually learns those features without you needing to tell it. Eyes are important to detect faces. You don't need to do that.

Um, encoding and embedding. The, the, the real difference is encoding is any, any vector representation. And um, an embedding though, sorry, it should have been uh, an embedding is when an encoding has meaning. Meaning the distance between two encodings has, has meaning. You know, they might be close or far from each other and it tells you something. There is a logic to it.

And then we talked about one hot and multi hot vectors. Okay, so end of the recap. Now let's go into supervised learning projects and we're gonna make decisions together and walk through it. The first case study is day and night classification. So here's my problem for you. Given an image that I give you, classify it as whether it's the day or whether it's the night. Okay? Open ended problem. Ignore that foundation models exist. You don't have access to ChatGPT or Cloud or whatever. The point is to get under the hood and have those discussions because obviously it's a toy example. So what do you do? Like what data do you want to collect to solve this problem.

Yes.

So tell me more. Check how pixels in the same row are.

What does it tell you? That if you look at a row of pixels and the row next to it, if they're very different, what does it tell you?

Okay, so you say if the delta between pixels that are closed geographically to each other is high, then there's probably colors, Color changes. So it's day most likely. Is that it? Okay, so that's an example of feature engineering. It's like you're going for it and grades like you're going for it and you're trying to understand what's a pattern that tells me that a picture is day or night. What else can you do.

In the world of neural networks?

Your other ideas. Yeah.

Okay, good. Yeah. So, yeah, I agree. I said 10,000 images. But how do you even determine how many pictures you need.

To get? Started with this project.

Feed some data. So you start with like 10 pictures and then you continue to go. Yeah, you could do that, might, you know, take some time. I think the question is, how easy is it to collect that data? Like, how would you collect that data?

Probably for the same location, you get pictures per day.

Okay. Yeah, we could put our phone out there and then record the day and the night and have a stream of pictures and add it to the data at same location, but different lightings. Yeah.

Andrew mentioned that this was a very hard problem because in my opinion, also it depends on how, like, what kind of model you want to have. Right. For example, if you were building something for one location to sense weather, day and night, then that one location would be really nice. But then if you, if you were trying to look at like anywhere in the world, like in any kind of climate, whether it's day and night, then you would have to have like an extremely diverse set of images. And because the problem is so broad, you would also need like so many, like, huge amount of data.

That's a great point. So just to repeat for the people online, you have to define the task first because the task can be easy. You can be in a park in a very specific location and say, just detect if it's there or night. Or you can have a problem which is your camera can be anywhere, and that makes it more complicated. And also the amount of data you'll need is probably much more for that second problem. That's what you said right now actually, that's a great thread. Tell me about cases where this problem would be really hard to solve.

Yeah.

Yeah, Indoor. Indoor pictures. Actually. Can you how could you tell if you took a picture of me with the screen here, you know, what time it was? You couldn't. You actually can. Yeah, there's a clock here. Um, so that, that's very interesting because, because if that was part of the task, then our problem would be hard and 10,000 images is not going to cut it to understand time, you know, and so it's very important to define the task very well. What else can be hard other than indoor pictures?

Also, also, the clock can be a.m. or p.m. but you can probably take additional information, which is how people are dressed, and think that it might be warmer outside than colder. You could. Again, it can be very complicated at the end of the day. But a human would say, someone is teaching, students are in class. It's probably not.

12:00Am, you know, or so, I mean, like, it gets complicated. What else can be hard?

Okay, sunny, cloudy. Yeah.

Great point. If you're in the north of Norway right now, or Sweden, even the clock can tell you probably if it's day or night. Dawn and dusk. Yeah, exactly. Those are great examples. Actually, this is a good semantic one because you need also to define exactly what's in the definition of day and night. So, long story short, the problem can seem easy at first. It can be very complicated. And trust me, if you wanted to do this really well, even the foundation models today couldn't do it in certain cases.

Um, okay, let's say we have 10,000 images. And you talked about the split of images earlier, and I agree with you. You want a mix of different situations in order to be able to cover all of them. Um, and going back to our discussion on model capacity, if it's you, if it's just a simple problem, you probably need just a small capacity model if you want to add all these edge cases, you probably are looking for bigger capacity models and more data. Uh, what's the input to our model? I think someone said it already. So let's say a picture of a picture of a, you know, day or night or whatever. What's the resolution we're going to work with? How do you determine resolution when you build a data set? And why does it matter?

Yes.

Okay, so you want to choose a resolution that's going to be the same across the board. You're going to vectorize everything that's going to fit in the same size matrix. Yeah. Why is it important to have the right resolution? Let's say.

I think it's important to have a similar resolution.

If the picture is the same.

Okay, so you're saying we want homogeneity in the data set in terms of resolution, I would say that could be a thing. But you know, today you can actually write scripts that downsize high resolution images before it gets in the model. And so it would probably solve. Upsizing is slightly harder. You would need another algorithm. But you know, it's okay to have different resolutions, but you still want to know what's the target resolution that you're looking for.

If you have too low resolution, you lose some of your features. But if you have too high resolution, you have too much data. A lot of ways to optimize.

Exactly, exactly. Low resolution. We lack information, so we might get things wrong. For example, the clock might not appear in the picture if it's too low resolution. High resolution means more compute needed if you have big pictures. And, uh, it's gonna be heavier to train your model and you're gonna pay more. And also your cycles of iterations are gonna be longer in an AI project. That's why resolution matter a lot. Uh, so my question is, what resolution do we go for?

Is there a way to just get to a number really quickly in the next 10 minutes? Let's say if we were doing this project.

Yeah.

Yeah, exactly. I think that's a great idea. You're saying use the human as a proxy. Yeah. So actually this is how we did it back in the days. We would print pictures in different resolutions and run it through our friends and say, can you tell if it's the day or the night? And it turns out that below a certain resolution they can't anymore. It's just like, I don't have the information I need. And you know, where we ended is somewhere around 64 by 64 by 3. And I stress the three because later this quarter we'll see that OpenAI and DeepMind. DeepMind as well in reinforcement learning. You know, it turns out for one of the famous algorithm we learned together for reinforcement learning discovered that you can actually remove colors and the model is not impacted. But your training is way simpler. You know, in this case, I think colors matter because of the blue sky, because it does have an inherent information about whether it's day or night. And it turns out the task, if you give it to humans is much harder without colors than it is with colors. Yeah. So those type of insights we could get over the next 10 minutes, literally by using humans as a proxy. This is a toy example. But I want you to think about that in your AI project. You're gonna be at points where you wanna validate a hypothesis and the best proxy you'll have is the human. Yeah.

Okay. What's the output for this model?

Day or night?

Yeah, day or night. Zero or one? Yeah.

Is there a relationship on how many input you have and how many?

Yeah, good question. So does the size of the image impact the input layer? The size of the input layer, is that it? Yeah, it doesn't. You can make your decisions. You can, you can have three neurons and you send a massive picture inside it. Um, oftentimes. And that's why I, I often say deep learning is an engineering field. You have to try it or you have to know the hacks. Um, uh, typically the, the network. If it's, you know this in, in, in a binary classification, you're going from a high dimensional input to a very low dimensional output. So by nature you'd imagine your network is going to be like this, probably meaning you need more edge detection at the beginning and then higher feature and higher feature until at the end it detects the phase. So it's gonna be like that. We're gonna see examples in week four where the output is bigger than the inputs and your network is probably like this, you know, so I. You'll build intuition during the class to know when you want to downsize your input layer or upsize your input layer. Yeah, and by the way, these hacks are valuable. Like why does meta go out and give crazy offers to a few? Because they know that stuff. They know those hacks literally. Okay.

Um, okay, so again, someone said it. The output is 0 or 1. The last activation is going to be a sigmoid. Yes, question.

Yeah. Instead of forcing yourself to pick a specific resolution, you come up with a more general model by taking an.

Yeah, you can do that. You can probably say in the data set, I don't care of the sizes because I'm scraping data from everywhere. I'm collecting data on my phone, I'm putting it all in a database, I write a script that down samples or up samples everything to the same resolution and then I send those lower res images in the network. Because what I care about is the training of my network. I want it to be fast and efficient. You could do that? Yeah, for sure. And there are networks we're gonna see in the YOLO is an example of a network where in the early, in the later versions of yolo, it does that automatically. It can take any resolution and it just samples it accordingly. Um, okay, last activation is gonna be sigmoid. We wanted the output to be between 0 and 1 and the architecture, most likely a shallow network. We don't need too much capacity unless the task is highly complex and a convolution should do the work really well. You don't know what a convolution is yet. You're gonna know in a few weeks. But those are known to be good for sequences and for images. And finally the loss function. What loss function would you use?

Huh? Which one?

Sigmoid.

Sigmoid? No, sigmoid is the activation. What's the loss function?

Logistic loss. Yeah, logistic loss, also called binary cross entropy loss, which is the one you've seen in the videos this week.

Yeah.

When you're like selecting your images and your resolution, how much is like designing that around the hardware implementation matter? Like if you have certain memory bandwidth, adder sizes, graphing, is that this or do you just let that.

So repeating the question how much the amount of hardware and the quality of the hardware you have at your disposal influences those decisions? The answer is a lot.

But you know, in fact, I'm assuming you're training it on your laptop right now, which will work, but you need to make those trade offs and I think that's why those skills matter. Now if the task is more complicated, uh, you look at the hardware you have available, you'll make a quick back of the envelope calculation. The calculation is really about how fast our iteration cycles can be. It's not about necessarily the performance of the model. In the end, you want to be able to iterate super quickly. And if your model takes one year to train, you're not going to be able to iterate. You know, you need to reduce it somehow.

So there are situations that we're going from like 64 by 64 to 65 by 64. Double your compute time depending on.

Yeah, yeah, for sure. Yeah.

Are the features of the model like every scalar in the 64x64x3 array?

What do you mean?

The, the features is every single scalar value.

So the 64 by 64 by 3 is like literally you look at a picture and you look at one pixel. These have three values that describe the color. Right. And then you just flatten it. And then those numbers are given to the input layer of your neural network.

Yeah. Um, okay, let's move on. Just for the sake of time. This was just the easy warmup. The two things I just want you to remember from that project is we're gonna build a lot of proxy projects and it's important to remember them for your own project. So you remember, oh, I Remember this experience experiment we did on humans? Oh, I remember how many images we needed for that project. And then that can, that can help you make decisions faster. And then the other thing that is worth highlighting is the human experiments. We're gonna do a lot of human experiments, starting with the next example. And those are usually helpful to make quick decisions in your project when you're in the industry. So second project, trigger word detection. Let me give some context on this. The general problem, okay, you're familiar with like Alexa and all these, let's say Siri and things like that that you might have in your kitchen listening to you, everybody knows. So the way these network typically work, these models is it's not a single model, it's a cascade of models for energy and efficiency purposes. So for example, if you have a virtual assistant in your kitchen, the first model is activity detection. It just detects if there is any volume. You know, because you don't want to listen with the heavy model at all time. It just uses a lot of energy. Right. You want a very lightweight model that understands when volume is playing. And so let's say this network detects volume. It calls another network that is focused on the activation word, the trigger word. Alexa. See? Hey, Siri. Okay, Google, that's usually the second layer. Um, and that one is only listening for a specific keyword. If the keyword comes in, it would typically call a better model that's slightly slower, that might be heavier and more energy consumption and that might understand your, what you're trying to do, you know. And then I'm not gonna go into the details, but you have architectures that get very complicated. Back in the day, some of these companies were doing one model to set up a timer, one model to buy something online. One model was very complicated. Today it's slightly simpler and more end to end. But I just want you to know the cascade of models that are being called, because this case study is about the second model is about the trigger word. So here's my problem for you. Given a 10 second audio speech detect the word activate, how would you build that off the shelf like that? Starting from zero, what data would you collect?

Yep.

So we are looking at frequencies that like might receive and for example, frequency. We want to figure out what was the word. And we also want to look at, okay, how long once is usually take to say the word. And then we will run some kind of algorithms that will translate our frequency.

To like a Fourier transform or okay, what you described is a Fourier transform or like plus the pre processing you're right. So you're saying audio is a bunch of frequencies with, you know, values. And we want to first pre process that, then to give it to an algorithm. And the length of the sequence matters as well, because if you want to detect the word activate, you know that the length needs to be. There's a minimum length. You can't say activate in less than 10 milliseconds. Right. So the length matter as well. Okay, that, that's good insight. What else? What data? How would you collect that data?

Yes. Microphone. Like with your phone. Okay. Like you would go around campus and record people, how would you, what would you ask them to say?

Okay, you would ask you would you record a bunch of people saying the word activate? You would ask them anything else?

Other words.

Other words?

Yeah, Say a sentence.

Say a sentence. Yeah. It turns out you have website that are random generators and you just say say this and say this and you record everything.

Yeah, say the word.

Deactivate.

Ah, good one. So you're saying you want to find negative words that are close to the positive word just to make sure that the model learns that. That's a great one. Yeah. Actually it turns out activate is a really bad word to choose. The reason Alexa. And actually there was a lot of discussions at Amazon back in the days around what would the word be? And turns out it's very important what you choose because you want to choose a word that is not used in common language, otherwise your assistant is always turning on. Right. And Alexa is not ideal either. It's not bad, but it's not ideal either. Um, okay, so let me just narrow down the problem. Let's say we've gone around campus and we've collected a bunch of 10 second audio clips. Okay.

Do we need to think about the distribution of the data? Like why does it matter? Like why? Campus only might be limited, let's say. Yeah.

Okay. Accents. Turns out the first version of this model that I built with Andrew, my German friends could not make it work. None of my German friends would make it work. And we had to collect more data from. And I had two German roommates at the time, so we had to collect more data from them because there's just a certain way that people would say words. Okay, good insight. What else other than accents? Yes.

Good point. Average age on campus is probably younger than if you actually cross campus and go to Palo Alto. And in fact the frequencies are going to be different that younger people use. That's correct.

Yeah.

How fast? Some people speak fast. Some people. And it has to do with the language of origin, Some people just speak faster. Correct. And look at this. Like when you hear someone who speaks fast versus someone who speaks slow, it doesn't make a big difference to you as a human, but if you actually just had access to the numbers and the frequencies, it would look completely different. So the model actually struggles a lot with that problem.

Yeah, I don't know if this is.

Very important for the ratio of male to female.

Yeah, for sure. Ratio of male to female. Anything that would modify the frequencies and on average, yes, there's different frequencies or distribution. Male to female. Yeah.

Background noise, very important. Turns out on Stanford campus you don't hear the Metro.

So it's very likely that your algorithm will not work for people in New York that are taking the subway all the time because of the background noise behind it. Yeah. Okay. I think we get a sense of like the, again, the complexity of the tasks ahead of us. Let's say the Input is a 10 second audio clip. I'm gonna call X. And this audio clip has a few things that are special to it. So one of the things is negative words which are in purple, positive words which are in green, and then the background is in orange. Okay, so this is for example, someone saying, hi, activate yourself, whatever, you know. You see what I mean? Activate is the positive word. Uh, what's the resolution we'd want? Okay, I'm not gonna ask you this question because we don't have speech expert. A speech expert would know it. Um, what you can do though, without being a speech expert is to go on GitHub and find another speech project. And you actually search for the hyperparameters they're using. And if you're using human audio, you'll find that the same numbers will work for your project. Okay, so you do that little search and you'll find that there is just a certain sample rate that works well with human voice.

Uh, what's the output?

0 or 1. Okay, let's try something. So let's say the output is 0 or 1. 0 meaning there is no positive word. The word activate is not there one meaning there is a positive word in that 10 second audio clip. So we're gonna do a little human experiment. I've selected three.

Let me turn the volume on. I've selected three.

Audio samples of around 10 seconds. Okay. Um, I'm not gonna tell you what the language is, okay, because the model doesn't know language when we start training it. So you're acting like the model. That's the experiment. I'm Just telling you that the first and the third sample have the word that we're looking for. And I'm not telling you what the word is again because the model doesn't know what the word is at the beginning of training. Okay? So now up to you to guess what the word is. And I hope one of you will save us and find the word. Okay, let's try it.

Too loud. Second sample. Wait, let me see if I can put the microphone here.

Anybody has it or no?

Impossible. Third one.

Okay, who has the word.

Mojo? Okay, maybe that's not it, but maybe. Yeah, in the back.

You're Italian. You speak. Okay.

Yeah. It's funny. Nobody finds it usually in the first try, but you do find you. You did find it. Yeah, that's correct. Um, okay, let's try again and do it with a different labeling scheme this time. Okay, let's try again. I'm gonna play it again, but the labeling scheme has changed. Okay.

It's the.

Third one.

Okay, what's the word? Someone who's not Italian. Speaker.

I'm not sure people heard, so I want to try someone else. Yeah, you've heard it.

Something like promo.

Okay. Not for promodigio. It's pomerigio. But you're close, you know, um, was it easier the second time or the first time? Way more. Way easier. So if it's easier for you, it's easier for the model, basically. Um, and so what is the question I'm posing here is we could go with the first labeling scheme, which is easier for us to label, frankly. Right. You don't need to indicate the location of the word. Um, but how much more data do you think we'll need in order for the model to figure it out? It's probably 1000x data. Right. And so the question is, is the second labeling scheme a thousand times harder for us to label than the first one? And the answer is no. Um, so the answer is very clear. You would rather have the second labeling scheme and your model is going to learn way faster than in the first case.

So that's the type of human experiment you can do. Now we're going to use that labeling scheme. Yeah. Question.

Yeah, we'll talk about it. Question, is it manual labeling or not? Yes, right now it's manual, but I'll explain some tricks we can use. Yeah.

Yeah. You, you, you. How do you pick the trade off between the different labeling strategies and not in fact, you, you. You know, today you could have, you could have a pre training and a post training that have different Labeling schemes. The, the question is for pre training you want the model to get really good and you don't want to, you want to avoid a cold start problem. The problem of the cold start is with the first labeling scheme, maybe even with a thousand data points that you detect that you collect and label manually, it's not even gonna understand anything. So you need, the second labeling scheme can be a great way to work around the cold starts. You might need less data, but it will start understanding what you mean. And then the rest of the data might be labeled differently essentially. Um, okay, let me talk a little bit about the labeling scheme. We're, we're actually gonna use a slightly different labeling scheme. And the reason is the first, the, the one with one, one and only zeros. Um, the risk is you can actually have a model that performs 99.9% accurate. That is all zeros, just predict zero all the time. It's very accurate. But the thing is there's just one, one to find and it's very hard to find it. And so the network is going to lean toward zeros, meaning the data is the labels are so skewed towards zero that it's going to be hard to find any signal in it. And so that deep learning researchers use generally is can we actually do a little more balanced between the positive and the negative labels? Just a pure engineering hack, not much science behind it. Uh, the last activation, we're gonna use a sigmoid, but because it's a sequential problem, we're gonna use sigmoid in sequence. At every time step, there's gonna be a sigmoid activation. And then the architecture, we're gonna learn it later in the class. You don't need to worry. This would be likely an rnn. You'll learn later in the class what that means. And then the loss function. Can someone guess what loss function we would be using? Yes. Yeah, binary cross entropy. But we're gonna use it sequentially, meaning at every step we're gonna compute that with the sigmoid output.

So on the output, so you just label the activate word as multiple ones.

Correct. You take your, your input 10 second inputs, you look at where the positive word activate is and you put ones in there when it plays and you force the model to predict hey, activate has been played. There's a lot of nuances you see because you actually are going to build this project with like, do we want the ones to start exactly when the words start, do we want to have a delay? There's a lot of technical questions around that. Yeah, you had a question, I was.

Wondering, like, what do you mean by steps?

What I mean by steps is audio is a sequence data. Um, and so it's time, step, by time step. And so what I mean is that at every time step, whatever your sample rate is, you're gonna have a prediction at every step. Okay. Um, so what, what is critical to the success of this project is really the labeling strategy. Here is how we did it. Um, uh, and you know, it's a. You just have to know it or not know it. It actually takes a long time to figure something like that out. And so thankfully, when I was a grad student, one of my senior PhDs helped me with these methods and he was able to guide me and save me probably months because I would not have figured out myself this probably. So here's what we did. We took three databases, we created three databases. One that would have positive words. So as you were saying earlier, we record people for the word activate. One that would have negative words, other words, including deactivate. But also kitchen and lion and dog and whatever. And then we have background noise. Um, turns out background noise in audio data is almost free. There's just a ton of background noise online. You can go on many platforms online video platforms, you can just take the audio. It will be background noise. So background noise is free, you don't need to go and collect it. Most likely. Uh, the other two are though, are harder. Yeah. Question.

Did you record the first two?

Yeah.

How many?

So I'll tell you, we, I'll tell you how we do it in a second. But yes, we recorded everything manually. What we did is we took, we went online, we scraped free license data. Actually it's a skill to know the licensing models. You're going to learn that in the project mentorship with the TAs. What licensing allows you to do what with data? It's good to know forever. You learn it once and then you know, what is CC by, what is CC by? Na, what is the MIT license, what is the Apache license, et cetera. And so we take 10 second audio clips, we clip the background noise and we went around campus and we literally recorded people saying activate in other words. And we cut it when they said it. So the word was contained exactly to the amount that it was needed. Then we created a python script that randomly inserts words randomly non overlapping words. So for example, this would be created synthetically. I would have 10 second of background noise, I would insert two negative words and I would insert one positive word. Okay. The trick Is that because the Python script did that? The Python script knows where Activate was put, so it can label automatically. And so it turns out that we went around campus, we used an app even to get help. So we hired a couple of people to come with us. Each brought their phone, and we went all around campus recording people to say, activate, in other words, and we created those data sets within three hours. We had millions of data points. Because think about it. Let's say you have 1,000 activates across campus, 10,000 other words, infinite background noise. Imagine how much data you can create with that when you actually write the Python script. You can also add some data augmentation, you can reduce some frequencies, you can augment some frequencies, you can accelerate it, you can decelerate it. So you can actually create a pretty meaningful data set for this problem in three hours.

Now, I'm talking about training sets, because you don't want to use that data for test sets. For test sets, you want data to be as real as possible. You're familiar with the concept of training test set, Right. So for the test set, we had to manually label data, but it was a much smaller set than the training sets. Right. So it's much more convenient. Okay. The second, um, important part was architecture search. I'm not gonna talk about it too much here, but there are architectures that just work better for these types of problems. And, um, this is an example of an architecture on the right that works way better. Um, and my learning from that project was just go to the expert and ask them what they've tried and try to learn from their mistakes. And in fact, I remember Oni Hanoun, which was in the first, first level at the Gates Computer Science building, and he knew that this architecture was gonna fail, and he knew why, because he's done so many speech projects, and he just knows what works and what doesn't work. So that's what your TAs are here for, actually in your project.

So you should give them a call and say, hey, is what I'm doing good? Or give me a pointer for what I might spend my next week doing. Uh, okay, so learnings from this section, this case study data collection strategy is extremely important, including the data labeling strategy, using human experiments matters as well, and then referring to expert advice. That's the type of thing you want to do in a project.

Okay. Do you understand conceptually how such project is built now at a high level? Okay, good. Yeah.

Question at the application layer, how often do you have to create your own architecture?

So how often do you need to do an architecture search nowadays less often than before. But this class is all about understanding what's going on under the hood. So we're going to walk you through that in practice. Um, it depends on your problem. Like I'll give you, in the industry, you might have a. You might be building a company that requires the model to be running on the browser. And so you have additional constraints that push you to create your own architecture, collect your own data, fine tune the model the way you want. For many startups out there and companies out there, you're gonna start from a foundation model. You're gonna start from a foundation model, and then you might actually quantize it or prune it or modify it to meet your needs in terms of latency, in terms of memory, in terms of hardware capacity. And knowing what's going on like we did, is important in those cases. Yeah.

Okay. Uh, super. So we're one hour in, we are about halfway through the class, a little bit ahead, and I have a few more case studies to cover with you. Okay. By the end, we'll have a full set of proxy projects to work with. Okay, so this one is cool. Its face verification.

A school wants to use face verification to validate student IDs in facilities like dining halls, gym, and pool. So let me explain what it is. It's like you arrive at the Arriaga gym, and instead of just, you know, normally you would swipe your student id, right, and your picture will show up on the screen and there's someone sitting there. It's going to compare the picture they're seeing on the screen to the picture they're seeing with their eyes. And if it's the same, they're gonna say, you can go ahead. Right. Be slightly different here.

You're gonna. You're gonna be verified by a camera. So there's actually a camera ahead and you're walking in and you're swiping your card. And we're gonna compare the picture in the database to the picture that is being taken by the camera to make sure that it's the same person.

Okay, how do you get started?

Okay, so everybody who got admitted uploaded pictures at least one. So we have that already in the database, correct? Yeah. What else?

When you produce cameras that are going.

To be used for.

You're saying the camera matters? Yeah, for sure the camera matters. In fact, we talked about resolution. Do you think the resolution is lower or higher than the Day and Night project?

Probably higher. In fact, it's going to be higher. And again, I would Go back to literally doing a human experiment and showing pictures of twins and asking people if they can differentiate the twins, and you'll see that the resolution matters, actually. Okay, yeah. You wanted to add something?

Oh, yeah.

Like, you also need, like, noise, like other people that are not used to it.

Okay. Also data from outside the university.

Which features human features.

Understand the person's human feature, but they're already in the picture. Right. So the picture would have the feature, or you would add anything on top of that.

So you typically, you would not actually want to get the feature level. You would just want to say, we need to make sure it's in the data. If it's in the data, the neural network will learn it.

Absolutely. Same person multiple times because the angle matters, the time matters, the, you know, et cetera. Yeah, okay. Same thing. Okay.

Crop the image so that the.

Okay, so you're saying we might do some pre processing to crop all the images so that it's centered, or at least that the image that we're training the model with looks like the image that the camera is gonna take because the model will run on the camera. Okay, all these are good. So let's say our data set is picture of every student labeled with their name. So this is one of my friends, Bertrand, and he has his picture, which is this picture from the student id. Um, and then the input is, okay, he shows up in front of the building. He's a little bit confused, but he showed up and a picture was taken. The resolution, we talked about it, what we use here is 4:12, 412 by 3. It's much higher than before. Okay. As we were expecting, because we need small details, even eye color is identifiable. Right. So these things we cannot find without a higher resolution. In fact, if you actually go through airport security and you use some of these fast tracks which take a picture of you, trust me, the resolution is going to be even higher than that. Much higher than that, because they're actually getting into the iris at that level. Right. So what's the output? The output is 0 or 1. Yeah, it's Bertrand or it's not Bertrand. Okay, we're. We're good so far. Uh, the architecture. Um, let me actually, Let me actually ask you, how would you do this comparison without neural networks? Let's say a very basic way. If you had to start with the first method. Yeah.

So you would. Feature engineer, you would say, like, for example, you would define 10 features that are good for identifying people, and you would have a filter for each of them, and you would Run it on the picture and say, yes. Do we have this feature or not? Essentially. Okay, yeah, it's a good one. Even more basic than that would be pixel comparison. You just compare the two pixels. What's the problem with doing a pixel comparison? So the idea is I take the two pictures, I compare them, and if they're close enough in pixel wise comparison, then it's the same person. If they're far, it's not the same person. What can go wrong?

Difference in what?

Lighting.

Lighting. Yeah. Actually what's interesting with the lighting is if you look here. So in this one, you take the top left pixel right here. Okay. It's bright. You take the top left pixel. On this one it's dark, or at least dark green. The difference between these two pixels is massive. It's close to 255, yet the pixel doesn't even matter. So why would you use that? It would penalize the comparison without actually mattering at all. So that's a good point. Yeah.

Yeah, absolutely. Background difference, translation invariance. Like imagine the same picture but the person is like 3 pixels to the right. The co comparison will be completely different because it's a pixel comparison rather than a semantically meaningful comparison. Okay. What are other things that can go wrong?

The distance from the camera.

Distance from the camera again, you know, rotation, invariance, translation, invariance, scale invariance. All of these matter. What, what are other things that are not geometric modifications? Yeah. Wearing glasses or hats. What else?

Hairstyle. Yeah, the beard. You know, and in fact, you look here, he has much more beard than on the picture. And he was much younger, by the way, on the first picture. And oftentimes we look, we're younger on our student IDs than we actually are in person. And that may make a difference. Okay, so these issues we all talked about, I think people get it. Uh, good. So, um, our solution is to use the concept of encoding. You remember what we talked about earlier with the face example? It turns out a network that has been trained to understand faces should have meaningful information in those layers that you can use as comparison that is more meaningful than a pixel comparison. So this is how it goes. We have Bertrand picture from the student id, we run it through a deep neural network and we get a vector.

What is this vector? It's a vector that we grab in the middle of the network somewhere. Remember what I said? If the, if the, if the vector is taken earlier in the network, we're gonna get lower level features. If you go deeper, it's gonna get More facial features. So you probably go slightly deeper. You go much deeper actually, for this example. And then, same thing, you, you run the exact same network on the picture from the camera. And normally, if the network was trained well, those two vectors should be close to each other.

Distance is 0.4. You set the threshold, you might do a little study to see what's the right threshold. And of course, this threshold is gonna determine the number of true positives that you're gonna get versus false positives versus false negatives. You know, the higher the threshold, the more likely you'll make a mistake, the more relaxed you are. Right. So here, let's say I set a threshold of 0.5. Hey, I'm confident enough that this is Bertrand. And again, airport security might have a much higher, lower threshold than the dining hall at Stanford, obviously, you know. Okay, so this is the general idea behind what we want to do. But I still haven't talked to you about how the network is trained. The network right now is not trained. We haven't learned how to train it. Yeah. Question.

Is it like just a regular picture classification network that has a binary output or is it.

What's the network? We'll learn it. Actually, we're going to see it together. Yeah, I'll describe it, but I don't want to get into the architectural nitty gritty. I want to focus on the general training scheme that we're going to use. And then you're actually going to build that at some point in the quarter. Yeah. Other questions.

Dimensional vector. What exactly hidden features are there?

Yeah, what's. What are the hidden features in the 128 dimensional vector? We don't know. That's the point of deep learning is you have to create a loss function that will modify your parameters in a way that forces it to learn features. But I can tell you that dimension number three is for eyes and dimension number six is for. I can make a study. We'll study it later this quarter and tell you that this neuron is actually good at detecting certain types of features. But right now I can't tell you unless I do that study.

Okay, so question for you. How would you build a training and a loss function to make that possible to train that network? You have ideas?

It's not an easy question.

Try.

Where to start.

Yes.

So mean squared error between what? You're right. Actually, two vectors. Mean squared error. Because it's a, you know. Yeah, but.

The loss function, probably.

So are you saying we would take pairs of pictures? We will run it through the network we will then take the two vectors that we get and do apply the loss function. Some distance, L1, distance, L2, distance. And then trace it back and say, these were the same people. You should have been closer. That's what you mean? Yeah. It's a good idea. Yeah. Someone else wanted to say something.

Yeah, that's another one. Cosine similarity. That could also be our loss function.

Okay, like what?

Great idea. Data augmentation. So you say I can take the picture of Bertrand and probably mirror it, flip it, rotate it, crop it. And I would use more data that way. Yeah, absolutely. That. That would help a lot, actually. Okay, so all of these are good ones. Um, if I summarize your point though, because that's really the key to the designing a good loss function. What we really want is that similar picture of the same person end up with similar vectors and picture of different people end up with different vectors. If we rephrase it in plain English. So what we'll do is that we'll build a dataset of triplets. The triplets includes a picture that is the anchor, a picture that is the positive. The reason it's called positive is because it's the same person as the anchor, and a picture that is called the negative because it's a different person than the anchor, and by definition also a different person than the positive.

What happens if you only have one picture?

One picture of a person? Great question. If you have one picture of a person, then you can't do that. We'll actually see another method that would allow us to do it even with one picture of a person.

You can rotate it. That's true. You could actually do some data augmentation as he was mentioning, and build a data set starting with one picture. But this approach will not be the best one. We'll see another approach right after. That would work better. Yeah.

Why are we comparing a vector from the model with the vector from another model instead of just comparing the output?

Oh, good question. Why do we compare a vector rather than the output of the model? Right.

So what's the output of the model? We actually haven't talked about the architecture, but I'm assuming you're saying it's a binary number in between 0 and 1. Because it's a single dimension, it cannot hold meaningful information. So you probably want to have a vector that is big enough where you believe it has enough flexibility to hold information that can allow to us to verify if the same person is on the picture, essentially. Okay, I'm gonna move on. And. And if There's. So what we want is to minimize the encoding distance between the anchor and the positive. And we wanna maximize the encoding distance between the anchor and the negative. So, question for you. What I'm going to ask you is to take 10, 15 seconds, look at the slide, and you're going to start voting for A, B or C.

By the way, ENC is encoding. It's just how I call the the vector that we get out of the network.

A is the anchor, N is the negative, and P is the positive.

So A is the anchor picture. Uh, N is the negative picture, which is different from the anchor, the different person. And P is the positive picture, which is the same as the anchor.

And ank of A is when you run A through the network, you get the vector anc of A.

Okay, let's look at the results.

A.

47 for A, 23 for B, 3 for C. So someone who said good job first. That is correct.

Someone who selected A wants to tell us why. Yeah.

Coding distance between the anchor and the positive, you just minimize the distance right there. But the maximizing distance between the anchor and the negative, this is like minimizing the negative.

Correct. Correct. Great. So actually the keyword here is minimize. If I had said maximize, the answer, indeed, as you say, would have been different. Because here we're looking at minimizing the distance between the anchor and the positive. And in fact minimizing this or maximizing the opposite of it. That's why the answer is A. Okay, good stuff. Let's keep going. Um, so going back to the initial setup, um, we had a cat and we were predicting a binary number here. Instead, we have three pictures going through the network in parallel. So you can imagine it's batch processing. It's like the three are going in the same network at the same time. And then you're getting three vectors. You're computing the loss function. Okay, you're doing this loss function we talked about. I'm not gonna talk here about the alpha number, but you're gonna learn when you build it why the alpha number matters.

Hint is maybe zero would have been a correct answer if you didn't have the alpha number. So it would have created instability in the model. Uh, but you do that many, many times. You push the parameters to the right or the left. And because of the way you created your loss function and your data labeling your. The way you structured your data and the loss function, essentially the model is gonna learn by itself to create similar encoding for pictures that are of the same person and separate encodings for pictures that are not from the same person. And you didn't need to do feature engineering. You didn't need to talk about eyes and ears and whatever, because it will figure it out. You know that you created the learning environment to allow that to happen.

So congratulations. You designed your first loss function, and we're going to design many more in this course. Um, this, by the way, is from FaceNet. It's a. It's a paper from 2015 from Short et al. And you'll see in the slides I used, I always put the, the reference to the papers in case you want to go back and study the actual paper. Many students do it for their projects. This is a great one. Great, great paper to, to look. Lot of citations as well. Um, let me make it slightly complicated. More complicated. We learned face verification, and now we want to do face identification. How is that different? Identification is a school wants to use to recognize students in facilities. So imagine face verification is you swiped your card and then that picture was compared to the picture of the camera. That's verification. The two. Are they the same or not? Identification is. You have this picture in the database somewhere. The person enters immediately. You can identify them, you know, so the difference for, for those of you who fly in the US Is when you, when you go through Global Entry, many people don't even need to put their passport or anything. They just watch, look at the camera, and they move on. That's identification. Okay, but actually, when you're in Europe, for example, you put your passport in, then you walk in, then it takes a picture. That's verification. Is it a difference or no? Yeah.

The negative or. How do you create those triplets, essentially?

No. In real time. That's a great question. I didn't talk about it. So at train time, you, you have databases and you, you create the triplets automatically. Like you pick pictures from the same person, or you use data augmentation and you add a random picture from someone else. You create millions of triplets like that, or billions of triplets. At test time, you only take the picture from the camera, run it. You don't use the negative. You just take the picture from the camera. You run it through the network. The person swipes, you take the picture from the swipe, run it through the network. You do the comparison. You let them in or not. So there's no more negative at test time. In practice, it's just a trick to train the model. Okay, so how would you do face identification? Using what we learned for face verification. Is there any small Tweak you can make that would make this network work for identification?

Yes.

Correct. Correct. What is it called in machine learning?

There's a machine learning algorithm that we can stack on top of what we just did.

He said you can compare. So because we don't have two pictures anymore, we just have one from the camera. You just compare the vector of the. You run this by the network, you get the vector and then you compare it to the database.

No, but good try. You have a database of all the student pictures. You run everything through the network. Instead of storing the image, you store the vectors. And then someone shows up and you're looking in the database. Is there any vector that is super close to this one that's identification? What is this algorithm called in machine learning?

Pretty simple algorithm.

No.

Okay, I'm gonna make it easier. What if instead of having one picture of a student in the database, you had three of each student, you have three vectors for each person and then you're trying to find the nearest vectors in the database from the one that the camera takes.

I used a keyword. No.

Yeah, K nearest neighbors. That's a K nearest neighbor algorithm. It's, it's essentially you, you want to explain what you, what you meant. Why is it K nearest neighbor?

Well, I was just thinking about like, what's the nearest vector? Yeah, closest to the vector.

Yeah, it's K nearest neighbor in high. For high dimensional vectors. So here is a simple example of K nearest neighbor for two dimensions. In Pract, it's 128 dimensions, so I can't put it on a slide, of course, but let's say in green you have the query point. The query point is the camera picture. Okay. And then you, you run a nearest neighbor algorithm and you say, are there three vectors in the database that are close to this vector? And you can add additional checks. Are these three vectors from the same person? If they are, then it's very likely the person is correct because you just prove that the three closest vectors in the database are from three the same person three times. So it's higher likelihood. You could even do it for 10 nearest neighbor. If you want to be really secure, let's say you go to the airport every time and every time they take a picture of you. And now they can do a 10 nearest neighbor on, on that search. Does that make sense? Now, slightly more complicated. You want to do phase clustering. So you know in your phone sometimes it says, um, it put automatically all the pictures from your mom in one folder and from your dad in Another folder. Right. Uh, how does it do it? How could you make a tweak to again, what we created or encoding network? How can you use that to create that?

K means. Yeah, exactly. K means algorithm, which is an unsupervised learning algorithm, clustering. So you have a bunch of pictures, you have vectorized all of them with the network you trained. And normally the vectors that are from the same person should be clustered around the same place. And that's very simply how big companies do it on your phone.

Yes.

Yeah. So if, if the person is not in the database, then you shouldn't find any vector that is close to the vector you're taking picture of. The closest vector might be above your threshold in terms of distance, and you wouldn't let that person in.

I think that's where they make you sign up for global entry.

Yeah, for sure. Yeah, they make you sign up. So it's interesting because companies know that they need to build these algorithms and then some like the admission process, the signup process might include certain data points. And now you're starting to understand how it's used in the background. Right. Okay, let's move on. Yeah, one question.

Oh, good question. So are you comparing each new picture? So I take a picture of my mom with my phone. What's going to happen? This picture is going to likely, if you're doing clustering, is going to be compared to the centroid of my mom. So the phone keeps probably a centroid of my mom. And if it's close enough to this centroid over another centroid, it's going to probably put it in that folder, essentially. No. Yeah. One more question and then we move on.

Yeah.

What algorithm do you use to determine how many centroids you want?

Ah, yeah, there is. Okay. How many. How do you figure out how many centroids you want? There is an algorithm I. You. You'll study it in CS229, not in CS230, but you know.

Okay. Uh, okay, what we learned here is what an encoder network is. We learned about positive anchor, negative for the triplet loss. The loss I showed you is called the triplet loss because of the triplets. And then we learned the different variations of phase verification, identification and clustering. Now we're gonna get to an interesting section which is brand new, around self supervised learning. So note that everything we did so far, the day and night classification, the trigger word detection and the triplet loss were supervised learning. We had labels essentially. Day and night is very classic supervised learning. You label data with 0 and 1, same for trigger word detection. Face verification, you can debate, it can be different. But anyway, we, we focused on supervised learning. Now we're gonna talk about self supervised learning. And my question for you is the following. Um, labeling is expensive. We know that. So how would you redo what we did? With a different approach that does not require labels. Meaning, you remember, even in face verification, we sort of had the name of the students in the database with their face and we might have multiple pictures of them. Let's say you don't even have that. You just have faces in the wild, unlabeled.

How would you do things differently?

Any idea? Yeah.

Okay. Let the neural network find the pictures that are close to each other. But how would you train that network? Like you're starting with a network that doesn't do anything and you give it an image. It gives you a random vector at first. So how would you train it?

Yeah, there's some kind of cluster. And then use those centroids as like.

Your do some clustering offline. But again, my question is the clustering algorithm, how is it trained? How do you cluster if you don't have any encoder network? Because the clustering came after we trained the encoder network. The clustering only worked because we had a good encoder network.

But if you have like for example, a bunch of pictures of the same person and then you run that through the network.

Right, but you don't know if it's the same person. That's what I'm saying is like vectors would be similar. No, because that's the network you're training. The vectors are not similar because that's, that's the network we want to train. Right now you, I gave you a network. It's completely random. You give it my picture on Saturday and on Sunday, the vectors are completely off. So how do you start? Yeah. Yeah.

Okay, tell me more.

You just train the model to the image first creating a blade representation of that, then.

Another model that gives.

Okay. Okay. Yeah, you're ahead. But we'll study that in two weeks, actually. So we'll do autoencoders two weeks from now in class.

Anyone else has an idea? Yeah.

Okay. It's actually similar to what he was saying to reconstruct the original image. Yeah, that's what we learned. It's, you know, there's a lot of methods, diffusion models work like that, autoencoders. We learn about that in two weeks when we focus on generative AI, Generative modeling. Um, here I want to present also sort of a Generative method. But it's really interesting because it will be your foray into self supervised learning. Here is the idea. If we have pictures in the wild, going with the methods you had mentioned around data augmentation, you can actually force the network to learn from the data itself. So let me give you an example. I take you look at the picture of this dog and you rotate it by 90 degrees. It's still the same dog, right? A human would say it's the same dog. What are we using in our brain? We're using the ability to understand rotation invariance and to understand the semantics of the dog. And so technically, if you gave those two images to the network.

You could create a loss function that compares those two pairs and has to have vectors that are close to each other. Does that make sense? Other thing you can do, you can do a patch. You can literally take an image of a face and put a patch on half of the image, you know, and then you, you say um, and then you do the same thing on the other image, you put the other patch the other half and now you tell the network these two should have the same vector, pretty much. So you use your data augmentation stream scheme on massive data sets online to force the network to learn from the data itself.

Okay? No need to forge triplets per se. You just take a picture, you make a variance of it with noise, with rotation, with cropping, with translation, with whatever you want. And then you put these two in the dataset and you say these are two the same person. It should have the same vector. That makes sense. That's why it's called self supervised learning, because you don't have labels. You just create a learning environment that makes the, the network learn from the patterns of the data directly itself.

Okay, so this is an example called Sinclair. Again the paper is, is right there. And this shift from supervised triplets, FaceNet, which was a paper from 2015, to self supervised pairs. That is why modern models are trained on billions of unlabeled images. Okay. That's how we create. It's much simpler when you think about it. You can literally write a script and scrape and it will label, auto label the images and put them in pairs, do variations and then you'll end up with a very powerful pre trained model, much simpler than people think. It's not that hard, you know, at the end of the day, most of the complexities are gonna come from compute. Right? Um, okay, so this method is called contrastive learning. Okay, we're gonna talk about it a little More in two weeks. Um, self supervision is not only an image thing, it's also used in other modalities. For example in text. The, the, the principle is the same. You predict what belongs together and you push away what doesn't. That's for images. And here what we're. What, what, what? The core of GPT, some of you probably have heard of that, is a method called next token prediction. We're gonna learn later about tokens in the class, but today forget about tokens. Just think the words. We're trying to look at a sentence and predict the next word. Why is this self. Self supervised learning? Because you don't label data. You just literally grab data from online and you create a scheme that forces the model to learn from the patterns of the data using self supervised learning. But the self comes from the fact that you didn't label it manually yourself.

So let's do a few examples. And the, the reason I want to do the examples is because we want to talk about emerging emergent behaviors that stem from the tasks we defined. So give me the word you're thinking of. I poured myself a cup of.

Some people said tea, coffee. Anybody said anything else? Water, cup of water. Healthy people said what? Okay, yeah. So what's the emergent behavior that you can expect the model is going to learn based on just that example?

Yeah, good point. Because you know that whatever is here first fits in a cup so it understands that uh, the second reason is porg, so it's a liquid. So just this sentence without even labeling is going to generate emergent behaviors that we've never trained the model for. For. That's what's interesting about modern AI, if you will. It's, you don't need to define the tasks. You know.

The. Frankly the same way think about face verification. Back in the days we used to do what I showed you, where we would create triplets and we would be very specific about. This is for face verification. You could actually scrape all the images online and do the contrastive learning that I showed you next. And it will still be good at detecting faces without you having even defined that task in the first place just by doing the contrastive prediction. Uh, second. Okay, first example also. Again, I was trying to predict, but people usually say different things. I think the majority of people think coffee. It's very cultural. You go in another country, it's gonna be tea for sure. Uh, and that forces the model, um, to really think about everyday co occurrence pattern like them, you know, being liquid, being of a certain size occurring together. So for example, there's probably a lot of sentences online that says pouring a cup of tea and there's a lot saying pouring a cup of coffee. Because of that, the model should understand that these two things are probably close to each other because their context is similar. Uh, second example, uh, the capital of France is.

Nobody. Huh?

Paris. Okay, what's the emergent behavior? You can expect the model to learn.

Yeah, learn about facts. Exactly. So this is really predicting the next token forces the model to encode real world facts, such as Paris being the capital of France. Oops, sorry. Uh, what about the third example? She unlocked her phone using her.

Body parts. I don't know what your phone, type of phone you have. But wait, what would you say? Face, password, fingerprint. Yeah, all of them are possible. So again, the network we learned probably that password, fingerprint and face can be used to unlock stuff. Yeah, yeah. And in fact here probably fingerprint or face might nowadays be the more common because of how the world has evolved. Uh, but back in the days it would be password for sure. Um, and so this forces a semantic understanding that these things are probably all meant to unlock information. Um, the next one, car the cat chased, the dog or mouse or ball. And again, uh, the model will learn probabilistic reasoning. Meaning, because in the dataset it will find variations of the sentence with different, um, actually conclusion, it would say that there's a lot of things that the cat can chase, you know, and so that's probabilistic reasoning. What about the last one? If it's raining, I should bring an umbrella. Uh, what's the emergent behavior? It's reasoning and inference is, um, the model we learn to connect conditions. So for example, raining requires you to be protecting yourself from the rain with an umbrella. That's reasoning. Okay, so long story short, emergent behaviors are unexpected capabilities that arise from simple training objectives at scale without being explicitly taught or labeled.

Later in this class we're gonna have a full lecture on deep reinforcement learning where we're gonna talk about emergent behaviors in robotics or in gaming, where turns out the agent you're training learns to do certain strategies that you didn't expect they would do. AlphaGo is a good example. If you've watched the documentary.

Okay.

Self supervision is not just about text and images. We've seen the next token prediction for GPT and we've also seen construction passive learning for images. My question here is the following. What other examples of modalities can you think of?

And tell me the task that you would define audio. So for audio, what would you do?

Audio. How would you do a self supervision in audio?

Exactly. Mask out portion. So mask out 20 time steps. And because you know what the data was, you have a label, you knew what the truth was. You can do a self supervision task. It would work great. Again, the only limitation is compute and scale. What other modalities?

Self driving.

So self driving is a good example. It's very multimodal. There's a lot of different things happening in self driving. We'll talk about it in a future lecture. What, what else? What other modalities can you think of? Videos. What would you do? Video.

Take frames out. You can take some frames out. Same principle as audio biology. Some people work in healthcare biology here. Yeah, a couple. Well, you know about amino acids and protein structure. You can actually mask portion of the inputs such as a protein structure or DNA and then complete it and it will force the model to understand those patterns.

So great stuff in there. But the world is very multimodal. We experience words, images, sounds and actions together. How can we connect them? When you think about multimodality, you want to connect texts and images. Let's say, what do you need to do those? You actually need labeled data. You need image captions. So for example, you have a bunch of pictures online of the cat is looking at the camera. So there's a picture and there's a label underneath. Just like on Instagram. Let's say people put captions. Right. And the reason you can connect those modalities is because of that data set, because you have a lot of that. Now this is not typically called supervised learning. It will be called weekly supervised learning because you're not actually labeling images with captions. You are benefiting from naturally occurring pairings in the world. There is naturally occurring pairings of images and texts. Okay, so now what I want you to do is to find other examples that are not just images captioned, but naturally occurring examples of different modalities that appear in the wild together that we could use to connect modalities. The whole point of connecting modalities is that our vectors now can represent.

Different modalities close to each other in space. Okay, so think about that.

Please continue. I'm going to read some of the answers, but we keep going.

All right, so stock price sequence is a, is a single modality. You would, you would look at stock price and you can mask and then predict. But I think maybe what you mean is you would put additional data points in there as well. Uh, let me see if I have something audio Paired with video. Audio and video is a great one. Audio and video is naturally paired. You know, you take a YouTube video, it has the audio and the video. And so when a dog is barking, uh, you have the audio of the dog barking and the video of the dog barking. And so you can create a pairing between those two modalities. Uh, transcription. So a lot of movies have subtitles. And so by definition a video stream or stream of images will be naturally connected to text which would also be naturally connected to audio.

Music and song title. Again, that's a great one. Audio and text are connected.

Genotype and phenotype. Good one as well.

Medical imaging with ultrasound. That's a great one. Naturally occurring. You usually, if you google an ultrasound, you'll have the different types of images that occur together naturally.

Game footage and keyboard action. Again, another great one. So you know, price scenario code. Good one. The. Yeah, great, great examples. Facial expression. So TLDR is.

We have ways to connect modalities. Um, oftentimes some modalities are gonna connect very naturally. Most things connect to text. So that's what you wanna use as your shared space typically. But here this is an example of a paper called ImageBind. And the interesting thing about ImageBind is it says that most.

Most things connect through a single modality. So for example, thermal data connects to imaging, imaging connects to text. So text is gonna connect through images with thermal data. And what's the consequence of that? If I may show you a little example, is that you can, this is a demo from Meta called Image Bind. Uh, it's a cool one. You can actually see things occurring together. So for example, you put a text drums and of course you can get an audio of a drum.

But you can also see what the closest image in the vector space is to that concept. So all the spaces are now bound together. Um, you can also do um, audio and image. So you, you, you give it a dog barking.

And a picture. What can you expect a dog on the beach? That's the multimodal embedding, the connecting tissue between those different modalities. And that's probably one of the biggest innovation of the last few years, connecting those shared spaces. Okay, I'm not gonna cover the full paper, but the core insight is that there are shared spaces. There are spaces like text and image that connect to most modalities that can allow us to connect those modalities this together. Okay. We learned a lot of things here. Embeddings, self supervised learning, contrastive learning, data augmentation. Next, token prediction, weekly supervised learning. And then the shared embedding stays with the central pivot usually being text. Okay, uh, that's all for today. We're not gonna have time to cover the adversarial example, but we're gonna cover it in two weeks together. Uh, you're gonna have more neural network.