Stanford CS230 | Autumn 2025 | Lecture 6: AI Project Strategy

So what I want to do today is continue our discussion on AI project strategy. So if you're building a deep learning system for some tasks and today, for the first part of today, I'm going to use a speech recognition, voice activated device example and then for the second half we're going to use a kind of AI deep researcher example. But what I want to do is walk you through a couple concrete examples of, you know, projects you might work on and let you understand what it feels like to be in the thick of building an AI system, making day to day decisions on what to do next. I find that, I think, as you've heard me say before, understanding the algorithms is important. So in this class you learn a lot, you know, from the online videos about the deep learning algorithms, how to build pipelines. But even beyond understanding how the algorithms work, what really drives performance is a team's ability to have an efficient development process. How do you tune how the parameters, how do you collect data? When you try something and it doesn't work the first time, which it often doesn't, what do you do next? Your skill in making those decisions is what often makes a massive, literally 10x difference in productivity. And as I was reflecting, was preparing for what to say to you today, I was reflecting on quite a few projects where that 10x difference in productivity is really not an exaggeration, right? Maybe more than 10x. But I've literally seen many teams in many well known companies with good brands spend a year work on a project that I will see a more skilled team execute in like a month, right? So these differences in skill are real. And one of the challenges finding people learning this is if you work for some company, maybe you work on a different project every year or two, right? But so it takes you like you know, two years, two years or a year of your life to gain experience on one more project. And then after, I don't know, 10 years, you've finally seen 10 projects and are pretty experienced. But what I want to do is in today's class, walk you through a few concrete examples of projects that similar to one, the kind of simplified versions of stuff that I've seen myself to try to accelerate your hands on experience, looking at these projects and thinking through if you are the one in the hot seat building a system and it works or doesn't work or there's problem, whatever, making those decisions for what you would do. So try to get you through that today with a couple examples rather than you having to spend years and years of your life to finally see a small number of examples of how these projects can be driven.

So the first multi, the first of two motivating examples I'm going to use today is building a voice activated device.

In my house. I have Amazon Echo where I have a lot of them actually and I think it's a delightful experience. But those devices like that Amazon Echo or Google Home or the Apple 3 HomePod, they require quite a bit of setup. They require some setup connect to the WI fi figure a way to connect through your phone. So you know, actually for a long time, even though I, I built smart speakers for a long time in my house, I had one light bulb connected to my home WI fi Internet. You know, because it's just so much of a hassle to set things up, I, I think now I have two light bulbs connected in my house. I guess I should, I should connect more stuff. But so for this example I want to talk about if you are part of a startup building a new product that makes it much easier to get these voice control devices without the user needing to do this home WI fi setup process. So.

Not very good at drawing. But if you could go to some store and buy a jest lamp and the desk lamp already has a name, let's call this lamp Robert and you could just take it, plug in its electricity, plunk on your desk and then say Robert, turn on. And then it turns on. Say Robert, turn off. It turns off without needing to be connected to the Internet. Is it cloud access and all that then that would give users an easier setup experience. There's a project that my friends and I actually discussed a few years ago that we thought would actually be a decent startup idea. We decided not to do it because there are too many other ideas that we're even more excited about. But we felt that actually a reasonable, reasonable startup idea to build a little IC circuit, little integrated circuit to sell to say lamp manufacturers and other device manufacturers to make it really easy. If say some company sells lamps to build little things so they can very quickly make their devices voice enabled and if you have a few pre built names, maybe give the users a choice. You can call your lamp Robert or Laner or I don't know or Johnny or I don't know, Alice or whatever, have a little switch, then users could just buy a lamp, put it down and then immediately have it be voice controlled without needing to worry about how do you get this onto my WI FI network and if my Internet is down then my whole house stays dark as well. Dissolve things like that.

I Actually did once have an. I was building a lot of voice assistants. I actually set up my office to have a lot of voice control devices so had different names of different lamps. Standing desk is his name as well. So I'd say I forget what my desk name was, like Jonathan, go higher or whatever. And then my standing desk would go up and down. It's actually pretty cool.

So what I want you. So.

Just for today's illustration, probably we should give the different devices different names because you can't have every device in your house called Robert. Otherwise you say Robert, turn on. The whole house illuminated and Robert turned off. Whole house is plunged into darkness. So we found out you do need different devices to have different names. Uh, but just for today I'm going to use as illustration the task of training a neural network or building a system that detects when someone says Robert, turn on. And you need kind of Robert, turn on. Robert, turn off. If you have a choice that needs to be Lena, turn on. Lena, turn off. A variety of handful of options of names, but just the simplicity. I'm going to worry only about detecting the phrase.

Robert, turn on.

And what do we do for this? You can rinse and repeat to give the turn off command and a handful of other names to make it user selectable what name they want to give this thing.

So.

This is something that needs to run on device small IC circuits and my. And I'm going to ask you a question, right?

When you've graduated from CS230 or maybe when you graduate from Stanford, if you are the CTO of a startup responsible for building this.

What would you do? Right? So call out. So imagine you just graduated from CS230 or just graduated from Stanford and you're the CTO of a startup and you want to build this lamp that can turn on when anyone says Robert, turn on. How would you approach this problem? And I know this is an incredibly open ended question and it turns out life is incredibly open ended, right? You graduate from CSU 30, you have to decide what to do. So if this is what you're doing, feel free to raise your hand and call out. If you want to build this product, what's the first thing you do? How would you think about it?

Go for it.

Yeah, cool. Yeah, right. So get some like open source speech text model or something and then see.

If you can run.

Yeah, that'll be a good start on anything else. Yeah, go ahead.

Increasing complexity. The first one would detect audio.

Just going on here. The second one would detect the word Robin and The third one would try to parse the phrase.

Oh, wait, sorry. Three models. One that detects robit and then one.

That the first one detects.

The second one detects the word Robin. The third one tries to understand.

I see. Cool. Right. Okay, right. So a few models to detect Robin understand rest sentence or detecting the sound. Cool. Go for it.

Let's say you want to explain and have different surgery. Then you have to repeat like the same process for gain balance. So instead I think what you could do is take a model that takes given like those streams autofill and another autofill and see like 32 identify 12 policy.

Okay. Oh, interesting. Oh, like a. It's like a Siamese network. We actually teach a Siamese network later in school where some people inputs two audio files and decides they're saying the same words. So you can more easily generalize the new words than Robin. Cool. Now that's actually pretty interesting. Yeah, go for it.

I don't know how to make this device more, but I think that maybe people need to just have some plugin and like have a program on my mobile. So kind of like Siri. So I can go like image that I activated on.

Turn on.

Oh, sorry, you mean just connect up your device to Siri.

Signal to turn.

I see. Okay, cool. Yeah, right. Yeah, that sounds interesting. It sounds like a different product than this if we need to connect to your cell phone and all that though. Yeah, cool. Right. So let me just. Lots of interesting ideas. Let me just make some observations. So I find that when building software products.

There are actually lots of good ideas, lots of reasonable things you could try. But, you know, as you heard me mention a few weeks ago, I think one of the strongest predictors for the odds of you building something compelling is speed. So I find that of all of these ideas, I think some are better than others, but it doesn't. But whether the idea is a bit better, a little bit worse, it is important, but it's actually secondary to how quickly you can just get something built. So if you're actually the CTO of a startup like this, I would encourage you to look and say, all right, what can we build today? Or what can we build maybe in a week and try any of these architecture choices and build it and see what happens. Because even if what you build is a little bit less good, you can find out all in two days. They can course correct very quickly.

I've actually built a lot of smart speakers, so I have maybe firsthand experience of this, and I just share some things That I happen to know that there's no reason. But it turns out that, let's see, at least today, general purpose speech recognition is still a little bit heavyweight. Takes quite a lot of processing power, is a bit expensive to run on an edge device if you want to make this just a few dollars. But it turns out that if you look at the smart speakers, there's usually.

If you want to train a neural network just to test detect one phrase, be it a phrase like, you know, okay, Google or Hey Siri or Alexa or whatever, any of the smart speaker trigger words that can be done with a fairly small neural network. Although to your point, if we want to do different neural networks, different words, we'll need to swap out different neural networks or rinse, repeat that, that but if you have only a small handful of names, phrases 1 to 10, I think, I think that'll be okay. And then one other piece of advice I would give if you're embarking on this is the first thing I would do actually, if I was working on this for the first time, is actually a literature search. And it turns out that.

I think we're fortunate that the AI world has a ton of open source software and a ton of open research papers. Somewhat surprisingly, despite smart speakers having been around for a long time, there still to this day isn't like a single architecture that everyone's agreed on on the best way to do this. If you look at the literature, there's still actually a diversity of opinions on how to do this type of wake word or trigger word.

So when you say something like okay Google or Hey Siri or Alexa, that's sometimes called a wake word because a wakes up the device or trigger word triggers the device check activity. So somewhat surprisingly, even though we're smart speakers for a long time now, for more than a decade.

There still isn't a single agreed on unified architecture that the community has agreed on and what's the best algorithm to do this. But I feel like if you are embarking on this for the first time.

The number one boost in your speed of learning, it could be implementing something. But I would say doing a literature search and trying open source software would be the even faster accelerator. And I want to give you a few tips for that. Right. Um, real quick. So you know, today if, if this is, there are a lot of research articles and blog posts and GitHub repos on many topics, certainly wake word detection. And what I find is that this is research paper one, research paper two, research paper three, research paper four. I find that sometimes people will spend a lot of time reading research paper one until you're done just 0% completeness, 100% complete, they spend a lot of time reading research paper two, spend a lot of time reading research paper three and I just recommend you not do that. Instead, when I'm doing a literature search, what it often feels like like is do a few web searches for a handful of resources, skim all of them 0% complete, 100% complete, right? Based on your initial reading, you may decide to go back to paper three and spend more time to really read and understand that. But that helps you find additional references. You can skim and maybe you find a paper seven that's really seminal, spend a lot of effort. But this is what doing a very broad survey of the literature will, where you really put in the time to finish only a very small number of resources, but spend a lot more time skipping around and getting a cursory level understanding of a broader set of papers, which can also.

Point you to the more useful resources to focus attention on. And just one thing I've seen among Stanford students.

There'S one other thing that I find people tend to underuse, which is trying to talk to experts. So if you are actually a CTF or startup trying to build this for the first time, I feel like we all want to do our own work and not bother other people, which is good. But I just encourage you to consider if after you've done your own work, if reach out to an expert can really accelerate your learning. So something I've often done is I will do my own work. I don't have to call random experts to try to boggle them before I've at least done my work. But if you've sometimes I'm reading a paper, I'm really struggling to understand it. And I find that instead of me struggling for another four hours, if I send the authors a respectful email and say, hey, I read your paper, try to understand this. I'm still confused. Can you help me out? Can you explain this to me? A lot of the time, including when I was less well known, but I think a lot of people, if they see that you're doing your work and not just reaching out to them before you've even done anything, not everyone, but a lot of research authors would actually be quite understanding and try to help you out. And so I find that for a lot of projects I did, finding that one expert, sometimes, sometimes a standard professor, actually we have a lot of speech faculty and the Stanford faculty as well. But when I had problems with my speech recognition system. You know, sometimes I call like, you know, Dan Juroski, whatever, and then like a half hour conversation or even a 10 minute conversation really accelerates what I've been able to do. So I just encourage you to. It takes you like 10 minutes to send an email and maybe there's a 50% chance of respond, I don't know. Right. Not a high spend chance, but sometimes that tends to be really high roi. Right. And then what I think you find is that if you do a literature search, you will likely discover that most of the robust enterprise grade smart speaker systems all have a specialized system trained to detect the wake word or the trigger word. Right. And again, there's a variety of architectures. You probably come up with some good neural network architectures for this. Um, and, and, and oh, it turns out that there is no data set on the Internet with lots of people saying Robert turned on. Right. That's just not a thing.

Right.

So if you decide that the names are Robert and Lenor and you know, I don't know, Jeremy and Alicia and whatever, sorry, if there are people with those real names. It was actually kind of weird when you chose people's names, but we just did that for some reason. But it turns out that there is no large data set on the Internet of lots of people saying Robert turn on. So if you want to train a neural network to detect if someone has said this phrase, you need to collect that data yourself. Right. Um, so let me then ask you another question. Um, say you've done a literature search, found some open source code you can try, but that has led you say, to conclude that you need data sets of people saying Robert turn on. Um, you, you need a data set to train to distinguish between someone saying Robert Ternon versus not someone saying Robert turn on. Where would you, how, how would you approach getting a data set like that? So yeah, go for it.

Text to speech.

Yeah, cool. Yeah, yep. You use text to speech as one method. I'll come back to that later. So you need ideas.

Oh, sorry.

Yep, cool, yep. I like that. Yeah. So walk around and ask people, tell them what you're doing, get the permission to record and use the voice. And if they're okay giving permission, just record their voices. Yeah, I like that. Anything else?

Cool. Yeah.

Oh, sorry.

Where?

Yep, cool.

Yeah, yep, cool, yep. Right, so samples of people saying not Robert. Yeah, yep. Yeah, cool.

Robber.

I see. Cool. Actually, actually. Right, so the, the data, what people are saying, rob it. But you don't want that to be Mistaken. That's a good point. I'll come back to that later. Yes. Did I say something? Oh, same thing. Oh, interesting. Cool. Awesome. Great. All right, so one of the. Yes. So I really like the idea of just going around and asking people for permission to record their voices. And by the way, Internet's word privacy is important. Consent is important. So, you know, don't do anything sneaky or weird. Right? Just tell people clearly what you're doing, ask them for permission, make sure any permission is freely given, and if they don't give permission, it's fine, move on. I really respect people's privacy. That. That is really important. Having said that, I find that a lot of people in the world are very nice. Not everyone, but the vast majority of people in the world seem very nice. And you ask them nicely.

I know these because I've done this myself. They will give permission for you to provide their data.

And then just one of the framing I encourage you to think about, as always, is the speed. So how long will it take you to wander around campus or wander around San Francisco and collect a sample of what? And I think you actually get a lot done. You'll get many dozens of samples, maybe hundreds of samples easily in a day. So I think those are good tactics. And then let me share with you some things. And then it turns out synthetic data using text to speech is an interesting tactic. I would usually not use that as the first thing I do, mainly because it turns out.

I just. It's hard to know.

How accurate synthetic data is compared to true, you know, natural data. And maybe just to show you one thing I've run into, I've actually, you know, done a lot of synthetic data for speech recognition. Right. One thing you have to watch out for is if you go to a lot of the synthetic, you know, sources of data, how many voices, how many different voices does it provide? And is it going to be a pain to get enough diversity in different people's voices? So it turns out, I don't know. For example, I often talk to OpenAI Voice on my phone instead of typing, but the number of voices there is limited. And if you go to TTS provider, I guess there are now some services with a larger number of voices, but these are things you end up worrying about and it's all solvable is synthetic data does work for speech recognition, right? But it turns out that there are often enough details associated with fiddling with the synthetic data generation process that that ends up taking longer. So for a lot of machine learning applications, you know, using Synthetic data is a good idea and eventually you might get around to it. But using synthetic data is usually not the first step type of data I would collect because usually there are just too many hyperparameters and too many knobs you have to worry about. And then at the back of your head you're wondering what is there something weird about the synthetic data that I not thought of before? And once you collect natural data, collect real data, it's just one less thing to worry about.

Maybe just tell one more story. Not his speech, but self driving cars, right? So you're building self driving cars, you want to detect other cars. Where do you get pictures of cars? It turns out that a lot of people will have the idea of, oh, there are lots of video games with cars driving around while we use video games to get pictures of cars out of the video game.

But it turns out a problem with a lot of video games games is there could be like 20 different cars in the entire video game. Depends on the video game. But it turns out that to have a realistic video game, you don't need a thousand different cars and there are tons of different car designs on the road, but you need a very narrow set of cars. And so to a human seeing the same 20 cars over and over, looks at the road, looks fine, the video game plays fine. But for a lot of video games, the data just isn't rich enough to capture anywhere near the richness of the real world. Whereas in contrast, if you've got real pictures, there's just one less thing you have to worry about. Right? So I find it for synthetic data, it works. It's very valuable too. Use it a lot. But I usually get to it only later in the process. Makes sense.

So, and then in the interest of building these things quickly, let me share with you the types of things that my teams have done to collect data for this. I feel like when you read research papers or you take courses often you get a very clean view of data. I'm going to tell you about one of the weird random hacks that one of my teams has used to build a very serious, working really well commercial system, right? At least as part of the journey to do so, which is, let's see. So.

Right.

So, collected.

100 training audio clips.

And 25 development sets to tune to and zero tests, right. One thing I'll often do if my main goal is to just build a system that works as opposed to publish research paper, is to not have a test set, we just have a dev set that we will tune the parameters to. And if I want to publish a research paper, I probably need a clean, unbiased test set. But if my goal is to just build something, you know, and have it work and ship it, sometimes I just say I'm not going to bother to collect any test test data, just a training set and a dev set. And I just unapologetically. Unapologetically to my system, to the dev set. Right? And one thing you could do is.

Let's see.

So audio clips. So audio is. Let's see. So you may have seen the audio waveform, which are just time. Your X axis is time. And audio waveforms kind of are these wiggly time series, right? And sound is very rapid vibrations in the air that your ear perceives as sound. And what the microphone does is it records these very rapid changes in air pressure. Right? So that's why you see all the waveforms that look like this. And so one thing that one of my teams was once did was collect 100 samples of, you know, audio waveforms where somewhere in the middle of it is someone saying Robert, turn on. So, you know, hey, how are you going? Oh, yep. And let's say Robert turned on. And we could not talk about some other things. Right? And so the phrase Robert turn on takes about one second to say. And one way to collect a constructive training set would be be to take a three second clip. So this is where Robert turn on was said. This is where they just finished saying Robert turn on. And so if you collect 100 audio clips like this, one way to turn this into a bigger training set is to take these long audio clips and then.

This becomes a training example with a label one, because that's an utterance where the end of the utterances corresponds to when someone just finished saying Robert turn on.

And in contrast, this would be an example, you know, would be a negative example. Um, and this too is a negative example, you know, and just two as negative example. Does it make sense? So given a say 10 second clip, given the 10 seconds of audio, we will have cut out a phrase where you get a positive label if that three second audio corresponds to someone just finishing saying Robert Turner, which is when you should turn on the lamp and anything else is labeled zero. Right? And so this is a way to take say 100 training examples and turn that into 3,000 binary examples. So 100. 100 audio clips. Sorry, I should say 100 audio clips.

Right. And if we take 30 windows out of this, they can turn this into.

Say 3,000 training examples, each with a binary zero, one label that labels is this moment in time when someone just finished saying the phrase Robert, turn on. Right?

So it turns out that if you do this, and we did do this, then we wound up, um, let's see.

We wound up with a system, so we ran this and tested this on the dev set, um, and we wound up with a system that when trained to predict binary classification, was 97% accurate.

Right?

But it turns out it did this by outputting zero all the time that had zero detections. Right. And so with this training set, we basically trained a very large neural network where I would have gotten exactly the same result with that one line of Python code. Right?

So this is the kind of stuff that happens in real life, Right. And, and by the way, I'm sharing these stories not, you know, just to entertain you, though hopefully you're entertained, but because I think it is by living these experiences that you, you know, go, oh, I could see this problem. This is what I do. I could see this. So my question to you is, if you collected this data, train the system, 97% accuracy, isn't that fantastic? But you realize is that you just implemented via a huge neural network, a print 0 statement or the equivalent of print 0, that is never finding the phrase Robert, turn on. Like what, what, what do you do next? What's going on? What do you.

Want?

Yeah, cool, awesome. Right, so, so increase the number of robbery turn ons in the training example. This is a very unbalanced training example and it's did great by just saying, I never hear this phrase. Right. How would you go about that? You or anyone else, how would you go about increasing the number of positive examples?

Go for it.

Oh, but how, how do you do that?

Oh, audio editors. I see, Wow. I see, Cool. Yeah, interesting. Okay, yes, that would work. That gets the synthetic data again, right. Which actually works. And it also has all the complexities with synthetic data, but it turns out it does work. So I know it works because I've done that too. But. Yeah, wait, I see, Go ahead.

Yeah, yep, that works too. So one thing you could do is take your positive examples, the examples of Robertson on, and just duplicate those examples. Or, or maybe just give those examples more weight to the training objective. Any other ideas?

Increase the noise.

Oh, increase the noise as an example. Yeah, yeah, cool. Yeah, that, that, that, yeah. So few variations of syntax. Yeah, yep. So I think the, the, the easy things to try one would be to take the examples and duplicate it. Right. So you know, mathematically this is equivalent to taking your positive examples and just making multiple copies of that in your training set, it turns out that won't work. The logic solve it. And by the way, unbalanced data sets is a common issue in training neural networks. I'll tell you the rule of thumb I use is many neural networks are pretty good at handling up to a 1 to 10 ratio of unbalanced data sets. So people often worry, what if an unbalanced data set, in this example, we have a one to 30 ratio and your mileage may vary. Sometimes it works, sometimes it won't. In this case, it didn't work. But I find that if you have a, you know, like a one to two ratio, just usually not that worried about it. Neural network is fine. Training like that, Maybe up to 1 to 10 is when I start to worry about it being a little bit too unbalanced for, you know, standard training procedures and when I might do something to make it a little bit more balanced. But duplicating these samples would be one tactic. Okay, go ahead.

What if you also penalize false negatives?

Yes. So penalize false negatives. Yes. That will work too. So I think there are a few ways to train the. To change the cost function. You can give the positive examples more weight or. Yes. Or penalizing false negative would be another way to change the cost function. That will work. Yes, I want to. Okay.

Yeah, go ahead.

Yeah. Yes. So you can also decrease the number of negative examples. That will work too. The one downside of that is like, it's okay. I think it's fine to do what you said. The one slight downside is if you are reducing the diversity of the negative examples, then the neural network has just a little bit less information to learn from. I think this is small. What I just said is a small difference though. I think that will also work. It's a quick thing to try.

Yeah.

And just tell you another. I'll tell you what my team actually did, and I'm telling you this not because it's a brilliant technique I'm proud of, but because I just want to tell you the examples of the types of hacks that actual commercial machine learning teams do that actually work. Right. So I'll tell you what we did. It was very close to duplicating the positive examples, but we had just a slightly different variation, which was we said that instead of the positive example being the one window where Robert turned on Just finished. So, so here we're having a sequence of labels. Right. Where zero corresponds to.

Is that moment in time when someone just finished Robert Turnon. And in the architecture I described, we are detecting a very, very narrow window in time where someone just finished Robert Erronon and they got a turn on. But the hack that we used was actually just extend this out a little bit. So instead of saying just someone just finished saying Robert, turn on in the last, you know, 100 milliseconds, we extended that out to half a second or a second. And so if someone finished saying Robert, turn on anytime in the last half second, let's turn on the light. Right. And so this actually generates a few more training examples. Um, the reason we did that again it's a small difference, is this actually creates a little bit more diversity in the positive examples than just duplicating it. Because now this red rectangle is a positive example and so is this one, and so is this one, and so is this one. So it just creates a little bit more diversity in the positive examples. Um, I expect this will make a very small difference in the training, in the learning algorithms performance. But it's just, you know, maybe just slightly better to have slightly, um, yeah, covers the, the space of positive attempts. Just a loop, just very, very little bit better sense.

So again, just to keep going with the story.

Let'S say you do this and now.

You still do well.

On the training set.

95% accuracy on training set, but.

50% accuracy. So not good enough on your dev set. Right. So you fix one problem and another one comes up. And so what do you do next? Go for it.

Yeah, cool. Awesome. Yep. Overfitting is use regularization. Go ahead.

Is it so, right? Is it that training and dev set distributions are not the same in this example because we collected the training and the dev sets. If we randomly shuffle between training and depth, then they would be the same distribution. Actually, yeah. But sometimes there's one other thing I do see, which is if you. It turns out early in the history of machine learning there was always this assumption of obsession with making the training set and the dev set and the test set have the same distribution. I think it's because if the training set and the test set the same distribution, it's easy to prove theorems, easy to give guarantees, whatever. So you kind of academic machine learning. There was always this theoretical assumption which makes the theory work way better. Your training set and your test set come from the same distribution. Right. It's just like from a publishing papers point of view that makes life much better. From a practical point of view, what I see is a lot of the time your training set distribution is just different than your test set distribution. That's just life because, you know, it's hard to demand data of a certain sort. And so, for example, one common thing is if you do use synthetic data generation, you can come up with really clever ways to generate synthetic training data, like using TTS or editing audio, whatever. There. These techniques let you generate a massive training set, but the price of that is, you know, your synthetic data. I mean, that's not how users talk, right? Users don't speak synthetic data, they speak raw data. And so to make sure that your test metric truly reflects how users will perceive your product, you know, I would put true data in your test set. And so you end up with a lot of systems where your training set distribution is synthetic data or other things you fit it with. And then your test set is whatever the world actually cares more about. And so two distributions is very different. Um, and then I think the question you just asked is, is the training set distribution too different from the synthetic data? That is a problem. Those would be good questions to ask, right?

All right, so.

If you see this, this is overfitting because we're doing freight on training set, not so well on dev. So one thing to try, first thing to try is very regularization. So that'd be a good thing to try. And then beyond using regularization, I find that for a lot of speech problems, if you're overfitting, getting more data is nice. And so just to share with you, I know suggestion of synthetic data earlier, I'll share with you one thing for synthetic data that does work, which we eventually wound up using, which is it turns out that, sorry, this is audio stream, right? It turns out that there are a lot of, if you can get audio clips of background noise. So for example, in this room, if I'm quiet, we can hear a little bit of air conditioning noise. So a lot of rooms have little background noise. Or if you're near a highway, there's car in the background. So there are actually most people where they may use the lamp, there's a little bit of background noise. So if you're able to gets audio of background noise and then additionally records some very clean audio clips of Robert turn on. Uh, it turns out that if you take two audio waveforms and sum them together, you end up with an audio waveform that sounds like someone's saying this in the presence of the background noise. Right? So it's the, I think it's called the superposition property of sound, the basic sound ads, which is why if you take two audio Clips and you just add the audio waveforms together. Then you end up with an audio recording that sounds like both sounds going on at the same time. So we can do is take background noise.

There's actually a lot of audio clips of background noise on YouTube as well and so on that check the licensing terms before you use stuff like that. But they're actually quite a lot of openly licensed audio clips of, you know, just someone like a quiet, like a coffee shop noise or someone sitting in a house studying or whatever. And so if you can take some background noise audio clips and then take a clean voice, someone saying robert, turn on. Right. And if you add these two together, they end up with background noise, Robert turn On, and more background noise. And this becomes a positive example using the process that we talked about just now. And if you have a handful of clips of Robert turn on and a lot of clips of background noise, so you can synthesize the phrase Robert turn on against a lot of different types of background noise and create, you know, a pretty large training set. Um, it turns out one problem with what I just described is if you do exactly what I just said, you won't actually find a Robert turn on detector. You end up with a voice activity detection detector because you have a lot of background noise. And anytime anyone says anything, the only thing people say is robert, turn on. It's much easier to just decide, is there a loud sound of someone talking? Does it actually recognize things? So the other thing we should do is instead adding Robert, turn on. Pick a dictionary or find other types of audio. I think that's where you have the comment just now of make sure don't confuse other things. And, I don't know, get someone to say the word cardinal or whatever other words. So you have a bunch of examples of people saying words other than Robert, turn on. And then also a bunch of examples of people saying robert, turn on. And by having a handful of clean recordings of Robert Ternon and a handful of clean recordings of people saying other stuff and synthesizing a data set like this, you can get a data set with many thousands of examples of Robert Ternon pretty efficiently. And if you train a neural network on this, you get like a decent wake word or trigger word detector that makes sense.

So.

Yeah, right.

So with a process like this.

So this is what working on a machine learning project feels like, right? You try something doesn't work. Like you find out that you have a skewed data set. Be creative in how you collect data. You may find a skew data set it just doesn't work. You may find this overfitting, and I find that it is the ability to drive iterations on a system that determines how quickly you can get something to work.

How would you sample differentiate text?

So, for the Robert Turnon example.

So if I found that there are a lot of users that are listening to music, then I would probably synthesize data with music in the background and then have someone say voucher. So if you think a lot of users listen to music and there's our desk lamp, we want to make sure we catch them saying Robert, turn on.

Then I would probably synthesize more training data to include loudish music in the background.

So one of the genre beats music.

Or deep metal, which you don't know, like what the user would listen to. And when you play two different music, it's extremely different.

Oh, yeah. So it turns out that if you have a. I. I feel like as a rule of thumb, if you can have a more diverse set of training data, it's usually better, Right? So, you know, so maybe I'm not sure what music my user base likes to listen to the most. Is it classical? Is it rock? Is it edm? Is it whatever? If you could collect a very rich training set that includes all of the above and even more, then usually that'll do better than going too narrow and risking picking wrong. Yeah. And the one asterisk to what I said is, so long as you can train a neural network that's big enough, usually more diverse data. More data. More diverse data is better. If your neural network is too small, then it may lack the capacity or intelligence to memorize all this stuff, which could be a problem. You need a very small network that runs at the edge. But usually, um, usually if you have the capacity to get more rich, more diverse training data.

That ends up delivering better results. Make sense? Cool.

Cool.

All right, now I want to share with you.

So it turns out when you're building a machine learning system, very common experience is you try something and it doesn't work. And then you have to figure out all the wonderful ways it could be not working and then go and fix whatever is not working. Right? And even in this example we went through, is the training data to skew? Is it overfitting? So, so you need either regularization or more data, or is it like something else? Or disinfectant data distribution not matching real data distribution? It turns out that when you're training a neural network, when you're training, when you're building an AI system, it's really difficult to know in advance what's going to go wrong next. I find that in software development, if I'm writing traditional software, you kind of control all the code and so it's more okay to write a spec and then you just build it and then it kind of works. You still need to debug it, but the bugs are my own bugs, right. In contrast, when you're building a machine learning system, it's much more like, I don't know what's going to happen next, maybe because I don't know what the data will give me, how to predict how the algorithm will perform. And so the workflow of machine learning feels much more like debugging than development. And by debugging I mean it's this iterative process where you build a system and you're just repeatedly trying to find how it doesn't work and fix it. And if you're trying to do a task that humans can do, then the bugs or the gaps in performance is often whatever a human can clearly do but the AI system is unable to do. So for many machine learning teams that I've led and worked in, if you can get the team to healthy rhythm of this debugging cycle, you can make really rapid progress. So for example, it turns out, let me give one example. I've been on teams where we would do the following morning, afternoon, evening.

Night. So we run our trading jobs at night, right?

And for example, it turns out for some of these models we're training for speech recognition. It takes, let's say it takes four hours, right? So the training job takes about four hours to train and doing it well. So training night, in the morning, we look at the results.

You look at the results, do error analysis, try to figure out what's wrong.

In the afternoon.

You know, we write code, right, to figure out how we're going to fix whatever thing we discovered the day before. And in the evening, right before we go home, we launch the training job.

And then it runs overnight and the next morning we do it again and again and again. And it turns out that if you can fix one problem a day, that's actually pretty good and within, you know, like, huh. And then you, you, you saw me walk through three, three or four problems when, when I built this for real, there are slightly more problems than that. Right. But I find if you can get a team inside cadence, where you train stuff, look at results, figure out is this bias variance, data mismatch. Oh, actually this recode and your data.

Fix it, you know, and in the evening, before you go home, launch the job. Maybe if you have time in the evening, just baby the trading job a little bit, make sure it's still running, but then come back in the next morning to see how it did. Then you just do this over and over and you can make really rapid progress. And I find there's often this discipline, right, that, that lets you make progress. In contrast, the teams that kind of wake up and they go, huh, what do we, we do next? Oh, let's call a meeting this afternoon. Wait, where's the data? Okay, I guess we'll meet tomorrow to look at the data. And then. And then like, oh, our infra is down. And teams like that move much slower than the really disciplined teams that just keep on rolling forward. And one more observation. It turns out that iteration cycle, sometimes it is driven by this. So I've worked on machine learning jobs where it took us about four hours to train a model. And four hours is long enough. Don't want to wait for it, but, you know, running overnight is just fine. I've also worked on teams where a typical training job took about three weeks on average. Right. And so if it takes three weeks to train a neural network for a particular type of model we're working on, then the pacing is very different. So how long it takes to train a neural network really drives the pacing of this. Where some, sometimes we would, you know, launch a training job and then like, hope for the best, we would monitor it during those three weeks. It's not that we do nothing for those three weeks, but you just go launch a trading job and then frankly, you know, hope that it works three weeks later. But we take three weeks and after that, tons of analysis and debugging and whatever work that would take us one to two weeks to set us up for the next training job, they're going to take us another three weeks for good result. And then we're also parallelizing while launching this job, we're analyzing from a previous job and we launch a few jobs asynchronously. But I find that how long it takes, this is a huge driver for how you do this. The other end of the spectrum is if it takes 10 minutes to train a neural network, then that's wonderful. You just train it, get a result, train it, get a coffee, come back, you know, then do the analysis. And then the bottleneck is how quickly can you analyze the data and get more data. So this really drives the design of this process. And one of the things that happens is for a lot of Projects, you start off with a small thing that takes 10 minutes to train, but then as you get more and more data, you know, then like you're bigger than. But now it takes four hours to train, it goes in bigger, then it takes two weeks to train. It's like, oh, now this thing takes a month and a half to train. So I've experienced that as well. Where unfortunately as a performance climb, we decided to started, we had to train bigger and bigger models with more and more data. And then it went from this really fantastic 10 minute iterations to these like months long iterations. But they just had to be done because we were training bigger and bigger networks. Right? Yeah.

Um, we probably didn't stop the model that much. Uh, so sometimes you can look at checkpoints, um, and if you see that for some reason, um, so actually if you're training like a three week job, you kind of expect the performance to be at a certain level after a couple days or after a week. And if it's way off, then before burning another two weeks, you may ask, is my learning rate clearly wrong? Or did. Or maybe this, this new data set we tried is clearly wrong. So you can actually start to run some analyses in the checkpoints and sometimes we would yank the job and just kill it. I would say that happened very rarely.

Yeah.

Yeah. Yes. So what you learn about in the online videos is transfer learning. We train a large data and then maybe fine tune on just a much smaller data set. And then if you process the fine tuning, a much smaller data set takes, you know, half an hour. Right. Then that's great. Then you can also drive much faster iterations in that based on that half hour is a limiting factor for the training time.

Cool. All right.

One reason I obsess about speed is because.

It turns out that if the x axis, imagine you're building a startup to launch this product, right.

I find that.

If a team takes, you know, twice as long to do it, they're just much less competitive in the marketplace. Right. So here's what I mean. So if this is error and this is your months, right. There's some machine learning systems that we work on for months to keep on trying to improve it. And if this is you.

And if a competitor, say, takes twice as long to reach the same level, so instead of taking this long to get here, they take this long to get here. Instead of taking this long to get here, they take twice as long to get here. So the competitor kind of does that. Right? It just takes twice as long. You know, they take Two days instead of one day to do something. But it always takes two days to do what you would take one day to do. Then the performance over time looks like this.

And what the customer cares about at a certain moment in time is you are so much better than them. Right. So these two extreme differences in speed really translates into a massive difference in the performance of your system versus someone else's system in the marketplace. And I find that the fast moving teams are just so much more effective. And you might think, yeah, I took two days, they took one day, what's the big deal? The big deal is not that your day is slower, the big deal is you're two times slower. And.

Just in the marketplace is just not competitive if you are. If you take twice as long for some applications. Okay.

All right. There's one other example I want to cover. Let me try to do that quickly.

So what I've talked about so far is speech recognition. Your wake words is detection, which is more of an end to end deep learning system where you know, your input audio goes to a neural network and in this outputs, you know, is this rto Robert, turn on? Right. So, so the entire system you're trying to build is just a single neural network. For a lot of applications you build, you end up building pipelines or sometimes call them cascades. I'm going to use the term pipelines. We saw one example of this last time where to detect people coming up to unlock a door with face recognition. We had video that fed to visual activity detection to see if anyone is even in front of it. And then that fed to neural network to recognize is this an authorized person? And then 01 to say is this a person that we should unlock the door for? Right. What I want to do is use a different example of a pipeline of a AIT researcher.

So it turns out all, all of the, I don't know, I think all of the leading lm well, almost all of the leading LL providers have deep researchers where you can ask a query or go and see, search the Internet, look at a lot of web pages and come back and synthesize a very thoughtful report. Right. And so yeah, but I actually use the OpenAid researcher quite a lot. I think that I know some of the team that built really well, but I think some others also did a good job. But so you know, you input a query like.

This example I'm taking from the agentic AI course online, but input the query like show me the latest research on black holes. Right? Some, some query like that and then.

One Thing you might do is take a query like whatever, right? Black hole research.

And then use an LLM use a lot language model to generate search terms to feed to a website search engine.

So may generate a few terms like, you know, black hole research, latest in, I don't know, astronomy and black holes or whatever. So generate a handful of search terms. This then call out web search engine, be it serpent, Google data, go Bing. I don't know, I used to really quite a lot but actually, actually more and more word search engines designed for AI rather than for human, which I think it pretty neat. So this can.

Call the web search engine.

And then Fetch the top URLs.

Oh, sorry, I mean fetch the top pages other than the top URLs. Server web search engine returns 10 pages. You may not want to fetch all 10 of them. You can read the snippets, decide which ones are most relevant. Again maybe with an LLM, with a large language model to decide what are the pages you want to download. So just like a human, you know, I'll do a web search, I won't click every single link. I'll take a glance and then decide which ones I want to click. So identify and Fetch the top URLs and then feed all that into.

Writing and this gives the final output. Right. So this is a. By the way, I say this is a more traditional deep researcher article. Sorry, deep researcher architecture. The more modern deep researcher article architectures let the system decide when to do more web search, when to fetch more web pages and are more, you know, autonomous, more agentic. But this is what the early deep researcher architectures like apply this more of a linear pipeline. There are more modern architectures where system will fetch some pages and autonomously decide do I need to go back and do more research and some what topics and iterate a few times. But this would be like a, you know, pretty decent basic deep researcher, right?

And it turns out that if you actually build this.

There one is a in the speech recognition system. Robert, turn on LAMP example. We talk a lot about how to improve one component which is a neural network. If you have a pipeline like this, the other thing you need to do which is critically important is to decide of all the different components in the pipeline, which one do you want to focus your attention on. I find that one thing that makes a huge difference in a team's performance is again be able to drive a disciplined evaluation and error analysis process to decide what to work on. I find that a lot of machine learning is not wildly doing things to see what works. It's actually a very thoughtful, very disciplined process where the system maybe it's not doing as well as we wish, so we should look at it to decide in terms of lots of things to be wrong. Is it generating search terms that aren't quite right? Or maybe we're using a web search engine that is returning results that aren't that good? So for example.

Is my web search engine comprehensive enough? Or is it just, you know, maybe I picked a lower cost web search service or something that's just not returning the latest materials. It turns out for the for some Internet articles, many web search engines will do really well. But if you ever want to fetch news, really fresh news, there's actually a lot of difference in the performance. Some web search engines are much better at having really fresh content. Some just don't update as frequently. So do I need to switch web search engines or am I successfully identifying the best web pages to fetch? So if I have black hole science, I think NASA.gov is pretty very very authoritative web page. But if a web search engine returns.

I don't know, Bob's backdropastronomyblog.com that's less authoritative than NASA.gov Am I correctly fetching the most authoritative scientific articles versus whatever is hyped up? Or maybe like.

Am I fetching a lot of random TikTok videos rather than really scientific authoritative things? And then lastly, given all this information, am I writing a thoughtful article with an LLM from the final research output? So it turns out that with a pipeline like this, there are lots of steps that could go wrong and your ability to decide what is the components you should focus your effort on, that's a massive driver of your productivity in improving a system like this. And the good news is that there's actually one other thing I've seen.

Sometimes I've seen a few experienced machine learning people. So actually one thing I've seen sometimes a team will build a system like this and if it's a less experienced team, they will system it's not quite working trying to decide what to do next. One thing I've seen many times is if you get a few senior machine learning people around at Stanford, I've seen this quite a few times. Sometimes a student's done a project, maybe a system like this. If you get a few experienced professors to look at the system, you find that our opinions on what to do next, there's remarkably little variance.

So you find that that many experienced AI people look at it and will go well based on what it seems we think this is the problem or this is wrong, we should try this. It's not that we always agree with each other 100%, but the variance is much less than you might think. And to me, I think there's a methodology behind how to approach these problems, whereas the variance in what to try next among less experienced engineers is much larger. So, so I, so to me, I think there's actually, if you have a systematic way of doing error analysis to figure out where it's actually underperforming. You know, experienced people all kind of. Doesn't mean experienced people are always right. There's just no variance. Right. But there really is a methodology to figure out what to do next. And one of the key ideas is error analysis. And I talk about this on online videos as well. But this is so important. We just go through.

Right? And one thing to do is to look at the output of each of these intermediate steps. And it's one of these things. I know, I'll say it. You probably even agree it's a good idea. But the percentage of people that actually do it when the time comes are found to be far lower than 100%. Right. But there's just, you know, take a handful of queries. Be it, is it latest in black hole science or should I rent or buy an apartment or you know, what's the, what are. Help me make fun plans for a weekend in Santa Cruz, whatever, some, some handful of queries for a deep researcher. Um, and then look at what search terms it generates and see if it makes sense. Look at the web pages of Fetches, see if they look good. Look at the top pages in fetch and see if the top web pages selected is similar or if it's materially different than what you would pick. Like my example, what if this decides to fetch Bob's Backyard Astronomy page.com, rather than NASA.gov right? And you go, okay, maybe I need to change how to do that. And then also look at the final writing to see, given the source articles, is it writing the appropriate article. And what I find is for a process like this.

You know, it turns out error analysis is often a very manual process because when the system isn't performing error analysis or gap analysis is often a manual process of figuring out what a human would do that is better than what the AI system would do. And we're trying to inject knowledge from the human into the AI system. So because AI doesn't have this knowledge yet, usually you do need a human time to do this. And I know you know, and people are talking about automating some of this and maybe there'll be problems there. But so far I find error analysis, it just takes human to look at it and have the insights into where AI system is doing something different than an expert human would be. And it's that insight that then points you is in the direction for how to improve it. And so for and, and so often I would build a spreadsheet like this where I have a query that is you know, black hole size.

And then another one. Do I rent versus buy, you know, in whatever San Francisco or you know, weekend activities in Santa Cruz. Right, whatever. Just have, I don't know, maybe up to 100. I find that I often have patients look at up to 100 examples beyond 100, sometimes more than 100, but it's somewhere between 10 and 100. Have a list of queries and have a list of steps. And so the steps would be right. Search terms.

Was that web search.

Fetch pages.

So this is what it feels like to do error analysis. It is a labor intensive process because there's a process of identifying where a human outperforms AI to try to close the gap. Just do this over and over. But so what I would do is I'll sit down often with a spreadsheet in front of me open like this and I will run, you know, black hole science through the whole system and then just read are the search terms satisfactory? And if I'm not explaining black hole science, I'm going to need to get an exhibit or do do a bit of work to figure that out. But if I find that the black the search terms to send to a web search engine are completely satisfactory that you I'll show just say okay, this looks good. Then given those search terms, I'll look at the web search results and say huh, did the web search engine retrieve reasonable things? And it looks good. Then great. And then I look at did my system maybe an LLM, did it make a good choice in the top web pages to actually go and fetch? And maybe I found this retrieving it, chose to retrieve Bob's backyard astronomy blob, but skipped NASA.gov, then I'll say okay, there's a problem there, right? And then often I make a note, you know, Bob instead of NASA just to kind of just take notes on what you're seeing. And then I'll say given these sources, this is writing okay, maybe it's okay. And then the process is to take a handful somewhere between 10, 20 and 100 of articles where the performance is subpar. And it's called error analysis because I want to focus on where it's underperforming. Right. So there's some articles where it's doing a great job on. I would tend to pay less attention to those. But our focus on finding anywhere from 10, 20 to 100 queries where it's clearly underperforming what I think a human should be doing or what I hope this will do. And then going through to just try to get a sense of how often the hotspots are in different parts of the pipeline. And so maybe for rent versus buy, I might say, boy, there's really authoritative blogger that talks about this. But somehow web search misses. I don't know, maybe I'm using the wrong web search engine. Weekend in Santa Cruz. Yes, buy into some hyped up tourist things rather than actually finding locally interesting web pages and so on. And then by doing this for 20, 30, 50 queries, you can then start to get a sense of where the hotspots are. Right. So again, all of these are subpar results. I'm focusing on queries where the performance is not good enough. And then if I find that, I don't know, 40% of the time is failing to fetch the world top web pages only 5% of the time. Right. It's failing to do that actually. Sorry, let me choose. They find that 70% of the time I'm really not happy with this. 5% of the time web search is not good enough. You know, maybe 20% of the time short terms are wrong and maybe 20% of the time the writing is not good enough. These don't have to add it to 100%. Sometimes you have problems in more than one column, but if this is what it turns out to be, then we'll go. Well, clearly a lot of the my dissatisfaction results is because it's just not choosing good pages from the web search to return. So let me go focus on that. And the thing about a lot of machine learning systems is you just don't know if you don't do this error analysis ahead of time in terms of what component to focus on. And so I find that teams that know how to drive this process systematically and it sometimes takes us hours, that daily schedule thing, sometimes it takes us a few hours to go through this process to reach conclusions. But then the benefit of spending like whatever, you know, three, four hours on this is that it can save you weeks of otherwise heading in the wrong direction. So teams can drive this evals and error analysis process in a very methodical away. You're much better at picking what direction to work in. And that allows your team to go way faster. And there's way more than a 2x difference. I've actually literally visited teams that have been working on something for six months. I go like, gee, I could have told you six months ago this wasn't going to cut it. Imagine if this was the problem. We're not writing the right web pages. But for some reason reason the team kept on trying out different web search engines. Maybe there's someone trying to sell a new web search service to you. So you spend a lot of time with the sales team, do a lot of integration with new website. There's actually more web search services than most people know. Right. And I actually swap between them whenever I feel like it. So imagine you spend all your time trying to come up with a better web search service for six months. It's entirely possible. Then you could just. It just won't move the needle for the overall performance. Which is why the cycle error analysis process is so important. Right. I walked through this with the example of one pipeline. The online videos go through other examples as well. But both are building deep learning pipelines and for kind of AI agentic pipelines. I think this is a very important concept to master.