Stanford CS230 | Autumn 2025 | Lecture 3: Full Cycle of a DL project

So, um, what I'd like to do today is chat with you about the full cycle of a deep learning project. And as I promised in the first lecture, rather than me talking at you for an hour and 20 minutes or whatever, um, we'd love for this to be much more interactive. And so what I'm going to do is illustrate this an example, but also asked you a bunch of questions along the way about what you would do if you are the one working on the project I'm gonna use as an illustrative example. Okay, so plan for this to be quite interactive and please interrupt anytime and ask questions, since I think that's why, you know, I want to do this in person here at Stanford. Um, so one of the reasons why developing machine learning or deep learning or other types of AI projects, including AI projects using large language models or agent guide workflows or whatever, is that AI projects are different than traditional software engineering projects. So one of the biggest difference between AI projects of the many different flavors, supervised learning on base, you know, generative, AI based, is that in traditional software projects you write code and you control your code, right? Write whatever code you want, compile it, your code does what you tell it to. But AI projects.

You know, involve both code as well as data that you train your algorithm on. And you almost never know what strange and wonderful things there are in your data. For example, if you're working on a face recognition application, that's a running example I'm going to use today. Then as you're just getting started on the project, you actually, it's really difficult to know in advance what you find in the data. Is the lighting of the face is good. Do you struggle with people with very long hair or people with short hair? Do people making weird facial expressions make your system struggle? Do people wearing glasses make your system struggle? So data is so rich that a lot of the time you can't predict in advance what your AI system is going to do. Not because you don't control the code. You can write whatever neural network or whatever code you want, but you don't know what's in the data. And this is why, unlike traditional software engineering.

Machine learning development is a much more iterative process where you just have to build something, see how it works, and then through a process discover, almost discover what is in the data and therefore what you should be doing to change your code to make your overall system perform. And this is also true not just for. This is true not just for deep learning based systems. This is also true for modern large Language models. If there's been a lot of hype Buzz about how LLMs, large language models are hard to control, I think there's a lot of excessive hype about that, you know, kind of fear mongering, there's a little bit of that. But one of the reasons why none of us know in advance what LLMs do is because it was trained on a lot of data, more data than any human could possibly look at. And we just don't know, you know, with precision what is in all that data that the ARM was trained on. And so because we don't know the data, we can't really look at all the massive tens of trillions of tokens of data it was trained on. It's hard to know exactly how a large language model will perform, which is why building agentic AI applications or building large language models based applications is also very empirical or very experimental. Meaning you just have to build something, then see where it goes well, see where it goes poorly, and then use that to fix problems. And that's how you drive progress. That makes sense. So because you control your code, but you don't really know, it's hard to control the data. And this is true both for the data you have stored in your hard disk. You can have, I don't know, terabytes of data stored on a hard disk. I don't really know what's in my hard disk. Um, and the thing you really don't control is what data the world will give you in the future. So we deploy a system in the world, say face recognition, which I, we should talk about where people wear a thick, heavy scarf that covers part of their face when it's winter. You just don't know there are all these things in data that will surprise you. And, and if you don't even control your past data that's already in your hard disk, you certainly can't control your future data.

So it turns out that a lot of machine learning classes talk about building models, right? And you learn a lot from the online videos about how to build powerful deep learning models. But it turns out that building the overall machine learning system or a deep learning system.

Has a lot more work than just training models. But, um, if you look at a lot of causes, there's actually a very strong focus on modeling because I think that's what academia has focus on, right? We can train models, evaluate models, publish papers on models. And so you find that a lot of causes for focus on, you know, training a good deep learning model, and that is absolutely important. Um, but because we know how to evaluate models, different research groups can benchmark different models. There's a lot of academic research work on that that's reflected in our courses. But this is just a small part of what you need to do if you want to build an effective deep learning system. And what I want to do today is um, go outside the small box of models to give you a broader view of what it feels like to develop a deep learning or AI or machine learning system. And this is what it often.

This is what building a deep learning system would be like, which is first.

You know.

Right. Specify the problem, figure out what we actually are working on. The running example I want to use today will be to build a face recognition system or face rec system for security for deciding when to unlock a door. And I know Kian talked about some face recognition which I believe face rec, face rec architectures last week as well. But specific application I'm going to talk about is something I've worked on actually I built one of the commercial systems that.

For this is.

If this is a door.

And you know, this is, well, you or a friend or maybe someone that you don't want to let in approaching if you have a low camera, sorry, bad drawing, take a picture of whoever's approaching the door and decide whether or not to unlock the door. Right. So face recognition to decide who's authorized, enter like a restricted location, like a, you know, corporate office building or your house or whatever. Um, and actually one, one, one common use case that one of my teams built was swipe key cards. So sometimes key cards get stolen. And so one of the systems we deployed, you know, fairly large office complexes was if you swipe a key card, we also just take a picture, there's a student, discard it to make sure that the key card is held by the person, you know, whose face is shown on the key card. So it makes it harder if someone to steal a key card to gain unauthorized access to like an office complex. Right. So I'm going to use this as a, the motivating example for today. And after.

Specifying a problem.

Typical process is then, right.

Sometimes the open source model you can download. But let's say for today that the open source models aren't good enough. You want to train your own model. But typical process, we get data.

You know, design a model.

Train a model, um, and then we will iterate through these steps a bunch of times until the model looks like it's performing well enough. Um, and then after that we have to deploy it.

And Monitor and maintain the model. Okay, so I'm actually going to talk a bunch about multiple of these steps today because I want you to come away with a feel for when you're working on a real machine learning application. What, what, what the important steps you will face are. Right? Um, and so, as I alluded, machine learning development is very iterative process. So for these three steps, we often drive a, um, rapid development loop where we.

You know.

Oops, sorry.

Design the model.

Train it.

Analyze the results, and then maybe design or update the model, the data or something, and iterate around this loop many times before you are satisfied. And. All right, just one more detail. Um, it turns out that for face recognition, the very common architecture which you learn in detail about later in the online videos is a neural network called a Siamese network. And what that does is a neural network that takes as input two pictures.

And two pictures get fed to a neural network or a deep learning algorithm. And it's a job of the deep learning algorithm to tell us, are these two pictures the same person or different persons? Right? Because if you're trying to set this system, say, for your house, and maybe you have, I don't know, a few family members or roommates you want to let in, then it's, I don't know, it'd be quite annoying if you had to retrain a neural network for every single hole. So the most common way to do face recognition is of a neural network that inputs two pictures. And the job of the neural network is, tell me, are these two pictures of the same person? And then the way you set up the system is input a few registration pictures. So take a picture of yourself, take a picture of your roommate, take a picture of, you know, any family members you want to have access. So then when someone comes, you can quickly check if the person that just showed up is one of the people that's authorized, and then let them in. And then the corporate keycard swipe. Example, if someone swipes a card and, you know, my card says, this is Andrew's card, then I'll quickly pull up my registration picture to double check if I am, if I seem to be the same person as the, you know, Andrew Ng that was registered. Right? So this is how.

This is a typical neural network architecture.

All right, so I have a question for you. One thing, one, one. One thing I'd like to do today is walk you through a number of scenarios and invite you to think about what decision you would make. If you are the CTO of a startup building these technologies, right? So, so my question for you is.

If you are the CTO of a startup building the next face recognition system and if your lawyers have said you aren't allowed to download data from the Internet. Right. So let's not download data from the Internet for this application, how would you go about getting data to train the system? So what you need is a bunch of pictures of people, right, to train a neural network to say, you know, are they the same person or not? So maybe take, take a few minutes to think about it and then I'll see if people raise hands and give some answers. And one, one specific question I have as well is how long would you take to collect data before you start training a model? Right. So I think, uh, well, you need to get data. We've designed a model, I guess, and then you need to train model. So what does the timeline look for you? We specify the problem, you know, we've designed the model, how many hours or days or weeks we just spent to get data and how before you start running, you know, gradient descent. Right. I see a handout. Oh sure, go ahead.

Are we laying like everyone around the world or are we like specifically looking at like Bay Area America scenario?

Let me, let me leave it a bit more open ended. Say you just graduated from Stanford and you're CTO of a three person startup building this thing that eventually, hopefully you sell all around the world. But your goal is to just get started with the three of you working out of, you know, Palo Alto, California. Right. And do it as your real self. Right. With the resources that you have.

All right, anyone want to venture an answer? So how do you, how do you get data to train the neural network?

Go for it.

Some company.

And collect data. So agree to share.

Yeah, cool. Video streaming service. By video streaming service. You're thinking like, you know, Netflix and Hulu or are you thinking like security videos? Oh, they zoom. Oh, you see, that's cool.

I see. Cool. Cool. Great. Cool. All right. Within streaming service, how long do you think it'll take to do that?

Yeah. Cool. All right, cool. Thank you. All right, creative idea. Any other ideas? How would you go about and get data?

You start at work for a three person startup. If I had a larger company out the camera. We plan to have the camera. Ultimately we have swipe and just take action points if they would like to participate in data collection and take.

Which might have a general license, but.

Yeah. Cool. Awesome. Right? Stick a camera there, let people opt in. Right. To get the picture taken. Cool. I think I saw a hand up Go for it.

Oh, sorry.

That's cool. How would you get users?

That's cool, right? Yeah. By your own people, you like, grab some friends and also they'll give you that LinkedIn photo. Give you some pictures from your camera, girl. Yeah. Cool. I like that. That's fast. I like that. Any other ideas? Oh, go.

I see. Interesting. Yeah. Ask Stanford to send an email. Cool. But I love Stanford. Stanford's a wonderful institution. Awesome. That's gonna take a while.

With a creative idea. I like. I like the creative ideas. So let me show you one guiding principle for how I would encourage you to approach this problem of data collection. I appreciate all the creative ideas, actually. Um, one of the frameworks I often use to decide how I collect data is speed. Because I find that actually especially you're building a startup, right? One of the, in my opinion, one of the strongest predictors for whether a startup will succeed and also whether a small innovative project in a large corporation can be a giant corporation. But if a team of three working on a small innovator project in a big, big company, I find that one of the biggest predictors for the chance of success is just the sphere sheet speed of execution, the sheer speed of getting stuff done. And so when I'm sitting on a team and brainstorming different tactics, I will gravitate toward the tactics that let me get a data set very quickly. And quickly usually means, you know, like one or two days, um, even if it ends up with a inferior, smaller, lower quality dataset or whatever. Um, because I don't really know what problems I'll see in my data. And the quicker I can, you know, get the data set, train a model, see where it goes wrong, the quicker I can then discover what's wrong with my data and fix it. Um, true story. Chatting with a CEO that told me he once actually, once, um, she, she actually had spent, um, I think it was over $100 million, definitely more than tens of million dollars. Spent a lot of money buying a company for his data. Um, and then he actually said, hey, Andrew, I spent all this money to get all this data. You know, can you help me figure out how to monetize this, how to make money off of this? Right? And I kind of look at it and go, boy, I kind of wish I hadn't done that.

Right?

Uh, um. And what I find is that the value of data, it's just so difficult to know in advance, right? What's important and what's not important about data. So for a lot of these tactics, um, for Example, I think student ID is an interesting one. But you know, are student ID photos weird in some way? Right? Or are they too expressionist or people smiling too much on student IDs? Actually had no idea actually my, my Stanford ID, I look really weird in my Stanford ID.

Or, and, and I think you and.

Actually like the idea of seeking a camera and just letting people, you know, come up and opt in and take a picture if you can do it quickly. And I think in a big company or even in a small startup, if you know it's important to respect user privacy or individual privacy. But if you can stick a camera in some place, you know, that doesn't kind of. What's the word? Invade people's privacy. They don't want to be any part of this. But let people opt in, take the pictures of permission. If you can do that in days, I find that to be valuable. One of the things I found for quite a few of my projects found that, you know, Stanford students, our community here is pretty cosmopolitan with people from all around the world. We're not fully representative of the world's, you know, distribution of people, but we actually do have people from all around the world. A lot of Stanford people are actually very nice. So one thing I've done multiple times is when I need to collect data, we'll go to places on campus with high foot traffic. Turns out cafeterias have very high foot traffic. Or just ask people, hey, work on the project. Can I get a sample of your voice? They're going to take a picture of you kind of with, you know, really informed constraint, tell people, is it okay if I do this? I've been delighted at how collaborative, right. Stanford students are.

But and, and I find that one thing I've often done is gone to my teams and said we have two days to collect data. It's like whatever. It's 11:52am Now. Let's figure out what we can do by what day is it? Tuesday. Let's figure what we do by Thursday, 11:52am so let's give us the 48 hours. Let's brainstorm how can we collect data. And it's fine that the data is, isn't all there, find that the data is low quality. But that velocity lets us more quickly train a model, figure out what's wrong with the data, and then jigger or retweak how we collect data.

Right.

So I find that there's some teams that ask, um, how can we collect the data we need and how long? And then ask how long Will that take? That usually leads to much slower execution. I instead tend to go to my teams and say we're two days or one day or maybe a week. Right. Some short time span like that and say what's the most creative, you know, respectful, responsible but creative way you can use to collect data in just short time span? And one of the ways to think about that too is training a model takes, I don't know, let's say it takes two days. Maybe I should take more like one day. Right. If you can train a model in a couple days, then I would not spend, you know, like whatever, right. Two months to find data to train a model. Because then this becomes a huge bottleneck because you can train a model relatively quickly. Let's take a commensurate amount of time to design a model or design the data as train a model. And depending on how long it takes to train a model, sometimes training model needs to run overnight. Um, I actually see teams sometimes take one day iteration loops around this. The fast moving teams I work with, we often go around this loop once per day for the smaller models. If we're training a large AI foundation model, sometimes training a model takes weeks or even months, then the process can be different. If, if your, if your model run is going to be, you know, two months, then yeah, maybe makes sense to spend, you know, a couple months to get the data really. Right. But for face recognition you could train that overnight or actually in a couple hours quite easily. So it makes sense not to spend massive amounts of time before you go in to train the model. Does that make sense? And because.

Oh, the, the word empirical means experimental, right? Sorry, I've used the word a few times today and I think.

We, we, we say that machine learning is very empirical process, meaning it's very experimental process. You have to do it, see what happens inside, what to do next. I know that sometimes others criticize our field as like we never know what we're doing, we just try stuff and see what works. And, and you know, there's a little bit of truth to that. Um, uh, I think understanding neural networks, hyperparameters, architecture, that is really valuable. So we don't just try stuff at random, but because we don't know what's in the data, we do try a lot of things and then discipline process, understand what works and what doesn't and then use that to navigate forward. Does that make sense? And then I'll just say there's one exception to the advice I'm giving here, which is if you're working on a project that you have worked on many times before. So, you know, whatever. I've built a bunch of facial recognition systems. So I kind of have a sense from previous experience and from reading research papers that certain things I know are just not going to work if I have, you know, 100 images, right? So there's some face recognition systems that I know I probably need at least 50,000 images before it'll have any hope of working. So because of that prior experience, having gone through that loop a lot, I now have a basis to say, okay, I do need 50,000 images, then I might invest upfront and put more effort up front to get those 50,000 images. But I think for most applications you work on for the first time, if you don't have a academic literature to justify certain larger data investments, or if you don't have the prior experience yourself, then I would focus on the speed of iterating around this loop. Make sense?

Um.

All right. And to relate this to large language models based applications as well, a lot of us, a lot of you are probably, well, you may be building applications by prompting OMS and calling an API, right? Like an OpenAI oranthropic or Gemini or Metalama or whatever, API to get things back from the OM. One of the reasons that too is a very experimental, a very iterative empirical process is because when you write an OM prompt, you don't really know in advance how well it's going to do because it was trained on data that none of us have really looked at. And that too is why instead of theorizing for a long time about what prompt to use an omn, you know, just try it out and then it's by doing that you then see the problems and then your focus can be on fixing the problems. Makes sense. And in fact there's a lot of discussion on responsible AI. How do you make sure AI systems are safe or they don't have kind of unforeseen circumstances? And because a lot of the theorizing, you know, you can only theorize so much, I find that if you want to build safe, responsible AI systems, one of the best ways to do that is to just build it and then experiment with it in a sandbox environment. Just don't let it out into the world until you've tested rigorously. But just go build something and then, you know, test it, probe it in the safety of your own laptop. Right? Don't let it innocent users and have some weird impact on innocent third parties. But it's only by building it and then probing it that you can figure out, where can it go wrong? Where will it say inappropriate things? Where would it, you know, respond inappropriately to certain user queries? And then that tells you where the problems are that you may work on addressing. Okay, all right, yeah, question.

How do you use.

Results to guide that?

Oh, sorry, say that again.

So like, when you come up, like when you analyze.

How do you use that to kind of figure out what's wrong?

Yeah, right. Um, great question. So when you analyze something's wrong with your model, how do you use that to take the next step forward? Right. Um, that's a big topic that we'll talk at length multiple times in some of the videos and also in some future lessons. But maybe, long story short, um, one of the things you could do is change your neural network architecture. You may realize that maybe the SIMEON sample isn't working or read the literature. Change architecture. The other thing you can often do is change your data. Um, so data centric AI is the discipline of systematically entering your data to build a successful AI system. And it turns out that you build a face recognition system, let's say by I, I, I, I like your hat. Snap hat. That's cool. But you may, let's say hypothetically that you find that the system has really struggles recognizing people that are wearing hats. Then you may say, all right, I need to get more data with people wearing hats. Right. And, and so it's often that looking at what goes wrong, which we call error analysis, that then gives you the insight to say, oh, it works well on these types of data or these types of users, but it struggles with these type of users in terms of data. So can I fix my data or get more data just on the subset of cases it struggles with? And that's a very common motion, that's a very common process for then driving the performance of your system. And, and this is also why.

Blindly going out to grab more data, Willy Lilly, that's often not a good strategy because it's just too much data you get. Do I want more your data with people with, you know, long hair or short hair or people with facial hair, sorry, too much hair, too many hair examples. Or, you know, do I want people with.

Wearing a scarf covering part of the face? Or people that wear glasses where they don't wear glasses, or people that, you know, are slightly turned away or people. So there's just so many different types of data. You could invest effort to get more of that until you build a system, see where it goes. Well, see where it goes poorly. It's really difficult to decide where to get more data and just get more data of everything under the sun. That's very slow and expensive. And I think part of the hype about the value of data has led people to have sometimes overly simplistic view of data. Right. You know, yes, of course I want more data, but just grabbing more data of all types of data is a very inefficient, very expensive way to improve my system. And even if you look at the way that frontier models are trained right now, it's not a game of just grabbing more data of anything under the sun. It is identifying the subcategories where it's valuable invest to get high quality data. Which is why if you look at it turns out in AI there are two clear buckets of value in oms. Right? There's the general answering people's questions. I think you know, OpenAI, ChatGPT is doing really well there. Gemini Anthropic have momentum but there's answering general questions. And then one of the verticals is really valuable is AI for coding assistant. So I think, you know, Claude has been ahead for a while, but OpenAI, you know, I don't know. Gemini 2.5 Pro. Some of the models are making really good progress in coding as well. And if you look at the team, look at the work that the frontier teams are doing to improve coding, building iterative agentic workflows is part of it. But also finding, finding clever ways to come up with coding related data is also part of it. And if you want your alarm to do better in coding, you don't grab data willingly with low quality random Internet chat, social media, whatever, data. Right. But instead having high quality coding related data is how you can have a focus effort to improve your om's ability to code. So a lot of these things are actually uh, yeah, yeah, I, I, I, I feel like the, the like more data is better. That is absolutely true. But it's also overly simplistic. Data is not monolithic. There's subcategories of data and having a view on what piece of data to really invest in. Getting a lot more of that's really important to um, being efficient in how you improve the system performance. That makes sense. All right, well we'll talk more about error analysis in this data. Any other questions?

Cool.

Yeah, go for it.

Yeah, yeah. Right. So as you collect data, the quality of data matters a lot too. Um, which is why and, and, and I think data quality is, is tricky. I think um, when building an application for the first time, I would still focus on speed and collecting some data quickly. But as you then analyze where a system is going well and what's doing poorly, you often find that um, the quality of data really matters. Um, so.

Chat is notable. This now, later, um.

I'll come back. So, so let's see what's good. All right, let me, let me talk about LM1. Um, there's a lot of low quality random chat on the Internet that's not that useful for training LLMs. But I think, you know, this is actually now well known that if you, if you can legally access very high quality written authored articles of books, they're highly edited, very insightful, that's very high quality data for training alarms. Um, and I, I'll give an example later as well. For face recognition, it turns out that um, you know, blurry images, right. Would be a, a lower quality than sharp in focus images, assuming that's representative of how you want to recognize people's faces. So that, that really matters as well. All right. Anything else?

Cool.

All right. Oh yeah, go for it.

Question about the data, how close they need to be to as we want them to do. For example, in space recognition where you're trying to convert two picture.

Model just to convert objects, people, space.

It'S much faster, but we're moving away from the ideal task. We want the big. So what is the balance here?

Yeah, so, right. Just repeat for Mike. How important is it that the data you collect is similar to the distribution of things you want to work on? I think it's important, but not as important as most people might think.

So it turns out that, uh, it turns out one of the reasons neural networks have been so effective is because when you build a very large neural network, you can throw all sorts of data into it, including data that is, you know, not perfectly tuned to your test set distribution. And it often doesn't hurt so long as the network is big enough. So, so you raise the example. What if we have it identified two identical objects, right. If it's generic objects like, I don't know, water bottles and markers, maybe that's too far ahead for you. But if we were to use, let's say, you know, simulated cartoonish characters, right, that look really different than real humans, my guess is it probably won't hurt at all and may even help a little bit. And so throwing in a lot of data. If we train a large neural network with a lot of capacity to absorb even some slightly relevant data, um, it usually doesn't hurt. And might even help a little bit. But how much it helps is another empirical question that will be problem dependent and we often just need to try it out. But um, I think maybe, you know, like in the past, people used to have an obsession that the data you train on has come from exactly the same distribution as the distribution you test on. That used to be how machine learning was done. I don't know, like 10, 15 years ago. That's really not true today anymore anymore. I think we're very comfortable. And I think when neural networks were much smaller, when you could train very small models, there was a sense that you didn't want to distract the neural network with irrelevant data. Right. Because computer was expensive with few parameters. If you're distracted on irrelevant stuff, maybe it gets less good at the core task you really care about. But if you can train a bigger than neural network, which is getting easier and easier these days, um, then it's become much more okay to toss in some data that, that hopefully isn't incorrect data. I think incorrect data is a problem, but just irrelevant examples hurts much less because big neural networks is like a human brain. Right. You know, I don't know, the fact that whatever, the fact that I learned to play the piano, you know, probably doesn't make me worse at AI, right? Because, because hopefully my brain is big enough to learn to play the piano and learn some stuff about AI. Um, uh, and I think as neural networks get big enough, they can learn some other irrelevant things and also do well the core tasks you really care about. But this was less true when, you know, if, if my brain was really small, then, um, I, I, I don't know if any are fans of Sherlock Holmes, but I think Sherlock Holmes had an attic theory that your brain has only so much capacity, so you've got to forget some stuff to learn new stuff. But when you train very large neural networks, that's, that's much less true.

Cool.

All right, thank you for all the questions. Um, so what I want to do is just keep going through this flow. Right? So we talked about your data design model trainer model. We'll talk a lot more about error analysis later this quarter of how to figure out where your algorithm is still subpar and where to focus efforts to improve it. But what I want to do is give you a sense of deployment. Right. So.

All right.

So when you have trained a model, it's often a bunch of software engineering work to then maybe take your model hosted in the cloud or local server and have it run inference. Right. So very common Architecture for deploying machine learning model will be that you have a neural network as a piece of software. You deploy it maybe on the cloud hosted service so that your software can accept a picture or accept two pictures and reply back, you know, do I unlock the door or not? Or else this is the same person or not. So that's kind of software engineering work that needs to be done. But in practical deployment settings.

I'll actually tell you, it turns out that if you're building a practical face recognition system, you probably find that if you are trying to unlock door, you know, to a corporate campus is too expensive or too slow to stream video 24,7 to Cloud to classify every frame at 30 frames per second to see if you know there's a person there you should unlock the door for. So in practical face recognition systems, what we end up doing actually, right. So. So actually let's take the example of um, someone walk up to the door to your home, to your door at home. Um, uh, and you want to see if there's someone there that you should unlock the door for.

Right.

So a lot of systems will actually have image from the camera and then uh, try to build a system. Um, all right, right. So what we have so far.

Right Is a system that takes this input an image and maybe a reference image. Neural network says do I unlock the door or not? But it turns out that if streaming video is too expensive, classify every frame is too expensive. We'll often end up with a system like this.

Um.

I'm gonna.

Well, when VAD sends a visual activity detection.

And what this does is a usually low cost, low power, inexpensive compute job to run to just very quickly maybe try to figure out is there a human face there? Because it turns out actually if you're building something unlocked your front door, you know, to you and your friends, if you look up my front door in my house, like it's pretty boring. You know this, there's a, there's a, there's a wall, we see a little bit of a street, but nothing moves most of the time. So this very obviously almost all of the time it's very obvious that there's no one outside my front door trying to be let in. And it'd be very wasteful to stream all that video to the Internet for classification, right. So visual activity detection is usually a low cost, low pass system to just very quickly decide should I do the work.

Of sending this to the larger face recognition system neural network that may be hosted on the cloud to have it do the much more computationally expensive work to Decide, you know, 0 or 1, do I unlock the door or not? Okay, so this type of optimization in order to make the system computationally feasible to deploy is fairly common. And I'm going to give you two options for how to implement VAD and I'm gonna ask you to reflect on them and then tell me which one you would pick to get started. Okay, so option one.

Is a non machine learning based method which is.

See if the number of pixels changed.

Is greater than some threshold epsilon. Right. So if the camera is stationary, maybe looking at a wall, most of the time the pixels barely change because you know, it's just a wall. And so you can write a little bit of code using, you know, some image library like Pill Python imaging library or some simple relatively lines of code to just say has the number of pixels whose RGB values have changed more than some threshold, has more than, you know, 10% of the pixels change compared to what it looked like a second ago. In order to see if there's anything in front of your camera to even decide to pass this to a more high powered neural network.

Option 2.

Would be to train a small neural network.

Right. So face recognition is a pretty complex task. You have to look at multiple cues in the face, look at the eyes, look at the mouth, look at the, you know, so that teaches a relatively large neural network, but just taking a quick glance to see is there even a human in front of my door? That's a much simpler task. So option two would be training a very small, very low power, very lightweight neural network to just very quickly tell you do you think there's a human there?

And then use this trained model to decide is it worth passing on to a much more powerful neural network running in the cloud with a lot more resources to make the final determination. Okay, so if you are the again CTO or three person starter building this, how would you, what, what would you, how would you start? We'll give everyone a few second reflection on this. Now get people's opinion, get people's thoughts.

All right, go for it.

You might live in a street where people walk by.

So what type of problem you have. And you can also go with an option.

Yeah, right, cool. All right, just, just repeating my mic. So problem dependent depends on whether you're on the street with a lot of people walking past. Maybe consider other algorithms, you know, they're even cheaper than small neural net. Cool.

We could probably just use our, we run in this while we collect more data and then break that one with.

The second yeah, yeah. Cool, great. So use option one in the short term and then maybe replace that with option two or we have more data. Yeah, cool. It's very, very same.

I was thinking, why not using two options? Like, because the.

Very simple, only if.

Z.

Oh, oh, sorry. So you, oh, use both options or put both in the pipeline. Oh, that's interesting. I see. All right, so see if option one, let option one see if anything change, if something's changed, then process on your network. Oh, okay. That's cool. That could work.

See?

Yeah, cool. Yeah, that could work. So it could cascade at multiple steps. Yeah.

If it's cheap to run the neural network.

Because then you don't have to worry about someone coming to the door and nothing happens. If it's more expensive, then I would probably invest into options.

Yeah, right. Yeah. So how expensive is it to run this new network in the cloud? So actually, so I would usually want to design both of these to run at the edge, meaning on the device.

Oh, this one.

So it turns out that streaming video is fairly expensive. Um, so I think.

Yeah, I feel like, boy, I don't have numbers at tip and fingertips, but I think running this 247 is not feasible. So we definitely need something to filter it down. But sending, you know, I don't know, few images every minute is probably not a problem.

Yeah, okay.

Oh, sorry.

All right. Sample certain times, Sample certain frames of the video in order to get all of them. Yes, I guess so. Yes. Although, yeah, although I, I think if you have a video of my front door, um, you need a way to sample other than random, I guess. Right. Because lot of time nothing happens, uh, unless you take one frame per minute, I guess, which would be okay, but then we don't want someone waiting there for a full minute before we finally get around to sampling it and then sending it, so. Yeah, go ahead.

Communities that train neural network on option two. Or could that be bias?

Yeah. Could we use the same images for training this network, to train this, the option two network? Maybe. I, I, I think it's actually one of those things. I would, I, I would say we have to try it before we know if it works. Might be doable. Depends a lot on what data we collected to train the brain network. Yeah. Cool. Anything else? All right, let's take, take a couple last minutes. Go ahead.

Cool. Right. Yeah. How about use option one, keep a really low threshold so we send very few images? Is that what you mean? Yeah, I think that'd be reasonable to try.

Well, right. So that we don't miss Too many people, but yeah. Right, let's take one last comment. Share. Share a perspective on how.

Right, yes, yes. So one of the weaknesses of option one is if there's a tree swaying in the background, or if you see a road and there's a car that drives by, or if there's a busy street, the car's driving by all the time, or when the neighbor's cat comes and visits, those will trigger this, Right? So those are downsides. So let me show you a perspective of how I would approach this and what we actually do. So if you are the CTO of a startup implementing this, to me it again comes down to speed, right? And so what I would ask is, how long does it take to implement option one and how long does it take to implement option two? And I think everyone made good points about the pros and cons of these two options. And if you're actually doing this, the approach I would take is, you know, just ask what can I do really quickly so I can then deploy it and see whether it works? And I'll fix it if it doesn't work. Because I think these insights, like, were trees swaying in the background affected? You know, it turns out that actually does probably, but maybe not if the wind isn't that strong or if. Right, but. But it's very difficult to know answers to questions like these because you don't really know until you stick on a bunch of cameras in front of doors and see the data. Then you can get more confident in how well option one works, like how, how often does the neighbor's cat actually come by? And if there's a car that drives past, are the cars usually so far away that it doesn't matter because the number of pixels per car is Right. So all of these are very empirical questions. It's really difficult to answer. And the fastest way to get answers is just implement something quick and dirty and then run it, see what goes wrong and then fix it. So what happens on. And tell you for what, what the experience of a lot of phase recognition teams is. Um, because option one is so quick to implement, this is like, I don't know, five lines of Python or something, right? Um, and get an hour to write it for you. Just get an hour to write the five lines of Python for you. Uh, so you can implement this in, you know, 20 minutes, maybe even less. Whereas this, I think, will take you at least many hours, maybe longer to implement. So my instinct would be the start of this. And it turns out having built many face recognition systems, I'LL tell you, this actually doesn't work well enough for all the problems people raise. And so we wind up going to this. But implementing this also gives you insights into where it's going wrong. And I'll give you one example of one insight that, you know, we and many other face recognition teams have, which is it turns out that.

When you're approaching a camera that's trying to recognize your face, there are some frames that are going to be really clear and in focus. And a lot of frames are really blurry. Right. If you just look at video, someone walking toward the camera, sometimes just, you know, when I'm stepping, sometimes the velocity of my face is higher and sometimes it's lower. Right. That's just what it is. And so sometimes my face is in focus, sometimes it's more blurry. And it turns out that if you can select out the high resolution frames and feed that to face recognition, you get much higher quality results. Right. So this is the kind of stuff that I assume, you know, most people would not know about until you work on a system like this. But it turns out that when I was working on a system like that, having a system to not just do vad, but also capture the video and then deliberately select not just one, but maybe five frames that are high resolution and in focus of the person, that actually gave a significant boost to the accuracy of our face recognition system. And I mentioned this as an example of the sort of discovery that you have only when you implement one of these systems, maybe even implement this system, but you implement this system, you see, boy, we're trying to recognize a lot of pictures from blurry faces. Maybe we need to do something about that. Right. And this is driving the empirical process that may then lead you to train a neural network both to see if there's a face, but also to see if the picture of the face is in focus to select out that frame for downstream processing. Does that make sense? And this is why you find that for some meaningful fraction of questions I asked in, in, in, in class, the, the, the thing that's the speediest to do is often the right answer. Because that, that obsession with speed really lets you go in and figure out what's in your data and improve your system more efficiently. Right.

So.

Right.

So, yeah. So what happens in practice starts with this. Discover how it doesn't work, because it doesn't actually work and tell you that. And then figure out. But use those learnings to do this. Yeah.

Problems like this individual.

Yeah. So is there a, is there a sense of what is a good classification accuracy problems like these, it's actually really difficult. Um, one of the common benchmarks you, you learn more about this in the third module on the online videos as well is building machine learning systems is easier if we have a reference level of performance like an aspirational target accuracy, which is often human level performance. Um, so, and, and so.

It turns out that the way you diagnose bias and variance, which talk about later in this course, is easier if you know what's an achievable level of accuracy. And very often we'll use what a human expert could do as achievable level of accuracy. Um, and then in the case of face recognition, definitely under controlled environments were better than humans. So definitely actually the AI systems we build, they're way better than I am at recognizing human faces. And I think AI systems are better than I want to say probably the vast majority of humans, maybe all humans at this point that really distinguishing of two pictures are the same. And then, and then, then it gets really difficult then actually gets more difficult once you're even better than humans. But those are. Yeah, but, but at least, but until you're as good as humans on, on a certain task, using a human level performance is often a good benchmark. And then if you're doing something that even humans can't do well, like it turns out, you know, recommending online books or movies or whatever, humans aren't actually that good at that. I think most of us have a hard time recommending good movies even to our closest friends. AI actually probably does it even better than many of us do as humans. Then those things is harder to establish a baseline.

Cool.

All right, now one.

Final aspect I want to touch on is after deploying a model to monitor and to maintain the model.

One thing that often happens is you train. A machine learning model works great, you know, does well in your training set, does well in your test set, works great and then you deploy it and then annoyingly the world changes and your system no longer works right. So we sometimes call this data drift or concept drift where the distribution of data the world gives you is different than what you had in your training set or concept drift, which is with the input output mapping your X and Y changes in the world compared to your training set. But to, to, to ground this, if you are training a face recognition system now in, you know, I don't know, when it's not too cold here in California, you get faces of a certain distribution. But as we approach winter, if it starts to rain More people are wearing scarves, rain jackets, you know, people look different, right? Or maybe, maybe we approach next summer, more people are wearing sunglasses. Then the data distribution changes. Or if you train the system based on data here in California, but we then decide to deploy it, you know, in a different country where people dress differently or where their appearance is different, the world keeps on giving us different data.

And so one of the jobs of, I think us as machine learning engineers is to put in place systems that monitor this type of concept drift or data drift, and fix problems as they arise. When you're out building machine learning systems, I have seen a segment of AI engineers that think their job is to do well on a test set. I've been in a bunch of these conversations where the machine learning person will say, look, I did well on the test set. My job is done. But then a product owner or business owner will say, no, your system doesn't work. Look at all the ways it does not work. And then if the machine learning person says, well, that's not my problem, I do it on the test set. I think that's not a constructive way to move it forward. So I encourage you to think of yourselves if you're building a machine learning system. I think of my job as building something that works, and that can be different than building something that works on the test set. So if ever you work on a product and someone says, you know, I know you did well on the test set, but your system doesn't work, I would encourage you not to respond, but my job is to do well on the test set. I encourage you to think about why doing well on the test set doesn't translate to doing well on the application that people actually care about. And then they'll lean in and go and fix that. Right? And one of the common problems for why doing one on the test set doesn't translate to doing well in real life is if the data distribution changes, in which case you may need to update the distribution of data you're training on in order to capture what it has changed in the world. And just a few other examples, I feel like, you know, the world gives us new data all the time. So if you're building a web search engine, sometimes, sometimes, sometimes a new politician is elected or there's a new movement, or some new video goes viral or, I don't know, Taylor Swift releases a new album, I don't know, whatever. And then people are suddenly searching for a brand. I thought someone. I thought I'd get a lot from that. No. All right. Never mind. No, no sifties here. All right.

But then suddenly a lot of people are searching for a brand new thing. And so the distribution of data you get is different because the world has changed. Or I've done a lot of work in factories. There's actually a reasonable chance a cell phone you have in your pocket may have been inspected by software that my teams wrote. But you know, sometimes the materials change or there's a new machine installed in the manufacturing line and this machine makes a new type of scratch on the cell phone. So data changes in inspection lines as well. Or one, one thing that actually surprised me when I work on self driving cars, one of the teams I was working with, we trained a lot in data in California. And then when we took the cost of Texas, you and I as people, we can drive just fine in California or in Texas. But it turns out the traffic lights look really different in Texas and California. Right? So traffic lights, horizontal, vertical. I think part of it is there are very high winds in some parts of Texas. So while traffic lights tend to be strung up differently to be robust to high winds.

Right.

And so traffic lights actually look pretty different in Texas. So the models we train in California, they don't work on Texas. We're going to get new data, refresh the data. So a lot of that is a process of monitoring and maintaining the model even when something in the world changes. And.

Before going on to monitoring the model performance and maintaining it. One interesting difference in performance between this and this is option one is a very simple model, right? It, it, it basically has one parameter which is epsilon, the fraction of pixels that change. And so this, because it's so simple is actually very robust to changes in distribution. For example, say it's a hot summer and a lot of people are wearing sunglasses, you know, well, the fraction of pixels that change, right, Even when people are wearing sunglasses doesn't change that much. Um, maybe if it's Halloween and people wearing crazy large costumes, maybe that would change. But this is actually a very robust algorithm because it's so simple. In contrast, if you train on data with no one wearing sunglasses, uh, because you know, it's not, sun's not that hot these days, right? Then with everyone starts to wear sunglasses. This is actually less robust. So one of the advantages of very simple non machine learning based systems is they may be less susceptible to data drift because maybe I tuned this parameter epsilon, you know, to limited data set. But even when the data changes, this tuning can quite robust. But if I train a neural network with say thousands or tens of thousands of parameters, then I'm more likely to have overfit to people without sunglasses, so people start wearing sunglasses. You're more likely to have to update this model.

And if you are building a.

Sorry.

Cool.

If you're building a system.

It turns out to be incredibly helpful if you can get user permission to stream a little bit of data back to your cloud hosted service so that respecting user privacy, being careful, opt in transparent privacy practices. I think that privacy is really important. Right. So do be transparent, do the right thing for users and if you're able to do that and get a load the data back to cloud to plot dashboards.

And maybe one. One practice I've seen is when building a high stakes application.

One good practice would be to gather your team together and get a diverse set of opinions on all the things you could change, all the things that could go wrong. And by I've built quite a few machine learning systems and I've found that when we sit down and brainstorm all the things that could go wrong, including the data distribution changes, I don't think I have ever seen something go wrong in real life that we did not identify as a possible problem. I might be wrong, but at least right now when we sat down, we really brainstormed all things that go wrong. I think I have yet to see something go wrong that was not on the list of stuff that we brainstormed. Right. And so if you brainstorm and, and, and this is true for safety critical applications as well. Right. It turns out creative teams, you can actually think of all sorts of things that could change the data or things that go wrong and that then lets you try to design a set of dashboards or metrics to put in place to monitor whether or not any of these things that you think might go wrong actually do go wrong. So we may put in place dashboards like um, how, how often it turns out re authentication is a, is a common thing. How often does the user need to authenticate twice before they let through? That's a, that's actually a sign of user frustration. Right. So build a dashboard to do that. How often does the user have to try twice? It probably means something went wrong the first time. Um, how often do you accept versus reject the user? Um, and how. What is the latency of the system? And I find that just as it's difficult to know in advance what's in your data, it's actually difficult to know in advance what dashboard will be the most useful. And so the best practice I tend to recommend is brainstorm a lot of things that go wrong, brainstorm a lot of metrics and then just, you know, create very rich dashboards where this is time latency.

Time reauthentication.

Right?

Time, the, the number of zeros versus ones, and then draw plots over time.

To see how these rates trend over time. And if you're able to have a lot of dashboards and sample and just, you know, look at some of the data for where you suspect you may be making a mistake, that often is then a good way for you to have a higher chance of spotting when there might be a problem. And in the times that built large dashboards with, you know, 20, 30 metrics, I found that it's surprisingly difficult to know in advance which dashboards would turn out to be useful. Um, I think in, in exploratory data analysis in data science, again, because we often don't know what's in the data or what the data would give us in the future. We just, you know, frankly plot a lot of stuff and then go and figure out from there what is interesting. Right. So the cost of plotting something in the Jupyter notebook is fairly low. So let's just plot a lot of stuff, have a lot of dashboards, and if you end up with, you know, 30, 50, 100 dashboards tracking these metrics over time, then hopefully in a few days or a few weeks, you figure out that a lot of them are really boring. So we figure out that, well, latency is just another thing because when cloud hosts a deployment, it's just very constant. So, well, let's get rid of that because that's just a very boring plot. I'm just not worried about that. And so often plot a lot of things and then prune back to then have a smaller number of metrics that we track and monitor for the long term. And then eventually, when you get a sense of.

You know, just being the normal range for some metric, you can then also put in place upper and lower alarms so that if it ever goes above or below certain bounds, they'll trigger an alarm. Like go page someone to take a look to figure out if something's gone wrong. That makes sense. And so unfortunately.

Just because you train a model a lot of time in a real world production setting, it doesn't mean you're done because you deploy it. And then the world will give you surprising data. And having a plan to monitor what happens as well as to maintain the model, meaning get, get new data, update the system to fix problems as they arise. That is often an important part of the practicalities of deploying a machine learning system as.