Andrew Ng recommends building a simple model first, then using error analysis to find what matters most. Train early even if it's rough, set up test sets and metrics, then iterate on the biggest problems. Skip this only if you have deep experience or strong academic precedent on the exact problem.
If you're working on a brand new machine learning application, one of the pieces of advice I often give people is that I think you should build your first system quickly and then iterate. Let me show you what I mean. I've worked on speech recognition for many years and if you're thinking of building a new speech recognition system, there are actually a lot of directions you could go in, a lot of things you could prioritize. For example, there are specific techniques for making speech recognition systems more robust to noisy background. And noisy background could mean cafe noise, like a lot of people talking in the background, or car noise, the sounds of cars in highways, or other types of noise. There are ways to make a speech recognition system more robust to accented speech. There are specific problems associated with speakers that are far from the microphone. This is called far field speech recognition. Young children's speech poses special challenges both in terms of how they pronounce individual words as well as their choice of words in the vocabulary they tend to use. And if sometimes your speaker stutters or if they use nonsensical phrases like, you know, there are different choices and different techniques for making the transcript that you output still read more fluently. So there are these and many other things you could do to improve a speech recognition system. And more generally for almost any machine learning application there is, There could be 50 different directions you could go in and each of these directions is reasonable and would make your system better. But the challenge is how do you pick which of these to focus on? And even though I've worked in speech recognition for many years, if I'm building a new system for a new application domain, I would still find it maybe a little bit difficult to pick without spending some time thinking about the problem. So what I would recommend you do if you're starting on building a brand new machine learning application, is to build your first system quickly and then iterate. What I mean by that is I recommend that you first quickly set up a dev and test set in metric. So this is really, you know, deciding where to place your target and if you get it wrong, you can always move it later. But just set up a target somewhere and then I recommend you build an initial machine learning system quickly find a training set, train it and see, start to see and understand how well you're doing against your dev and test set in the valuation metric. When you build your initial system, you then be able to use bias variance analysis, which we talked about earlier, as well as error analysis, which we talked about just in the last several videos to Prioritize the next steps. In particular, if error analysis causes you to realize that a lot of the errors are from the speaker being very far from the microphone, which causes special challenges in speech recognition, then that would give you a good reason to focus on techniques to address. This is called far field speech recognition, which basically means handling when the speaker is very far from the microphone. A lot of the value of building this initial system can be a quick and dirty implementation. You know, don't overthink it. But a lot of the value of your initial system is having some learned system. Having some trained system allows you to look at this bias and variance to try to prioritize what to do next, allows you to do error analysis, look at some mistakes, to figure out all the different directions you could go in, which ones are actually the most worthwhile. So to recap, what I recommend you do is build your first system quickly, then iterate. This advice applies less strongly if you're working on an application area in which you have significant prior experience. It also applies a bit less strongly if there's a significant body of academic literature that you can draw on for pretty much the exact same problem you're building. So for example, there's a large academic literature on face recognition. And if you're trying to build a face recognizer, it might be okay to build a more complex system from the get go by building on this large body of academic literature. But if you're tackling a new problem for the first time, then I would encourage you to really not overthink or not make your first system too complicated, but just build something quick and dirty and then use that to help you prioritize how to improve your system. So I've seen a lot of machine learning projects and I've seen some teams overthink their solution and build something too complicated. I've also seen some teams underthink it and build something maybe too simple and. But on average I've seen a lot more teams overthink and build something too complicated. And I've seen teams, you know, build something too simple. So, um, I hope this helps. And if you are applying to your machine learning algorithms to a new application, and if your main goal is to build something that works, um, as opposed to if your main goal is to invent a new machine learning algorithm, which is a different goal, but your main goal is to get something that works really well, I'd encourage you to build something quick and dirty, use that to do bias variance analysis, use that to do analysis and use the results of those analysis to help you prioritize where to go next.
