Machine Learning Engineering for Production (MLOps)

Welcome. My name is Ryan Keenan and I'm the Director of product at DeepLearning AI. We really appreciate you taking some time out to join us for this event. We've got people all over the world joining us right now. So what we want to talk about today is that the field of artificial intelligence has seen incredible developments in recent years. The performance of machine learning models and the relative accessibility of powerful computers means that nowadays almost anyone with a little bit of coding skills and access to the Internet can piece together a really powerful implementation of at least some sort of proof of concept solution, ranging from things like computer vision to natural language processing, and much more. That said, the gap between any proof of concept solution that you might put together and a production ready machine learning system is really a substantial gap. And that's what we're here to talk about today. Machine Learning operations, or MLOps for short, is an emerging field within the AI space focused on closing that proof of concept to production gap. The field of MLOPS brings together the fields of machine learning, so data and modeling pipelines with the field of modern software engineering, so things from DevOps to deployment. So today we have with us a panel of mlops experts to discuss what are the most important aspects of production, machine learning and what MLOps looks like at companies today, from industry giants like Google to small startups. We organized this panel to celebrate the launch of the third course in our new specialization titled Machine Learning Engineering for production and MLOps. We've created this set of courses to help learners gain practical knowledge and skills in this exciting new field. You can learn more about the machine learning Engineering for production and MLOps specialization at our website, DeepLearning AI or by following the link in the description below. On the specialization landing page you can also find a link to our discourse community where you can connect with other learners and mentors and where we'll be hosting an AMA session soon with one of the course instructors. So with that, I'd like to welcome our panelists for today's discussion. We have with us Rajat Monga, co founder of a stealth startup and former lead at TensorFlow at Google. We also have Chip Hoyan, adjunct lecturer at Stanford University. We have Robert Crowe, TensorFlow, Developer, Engineer at Google, and one of the instructors of our specialization, Lawrence Moroney, who's an AI advocacy lead at Google, also one of the instructors of this and other specializations at DeepLearning AI, and our very own Andrew Ng, founder of DeepLearning AI. It's a great pleasure to welcome you all to this event today and so for the next 40 minutes or so I'll be asking some questions to our panelists and while I'll be directing my questions to a specific person among you, I'd like to encourage you all to jump in and offer your input on the conversation. We'd like to make it a free flowing discussion. So we'll start off by just a little bit more introduction from each of our panelists. So let's begin with you Rajat. Could you tell us a little bit more about yourself and your thoughts on MLOps to get us started?

Yeah, definitely. So as you mentioned Ryan, I spent a long time at Google most recently helping build TensorFlow and then run it for a long time. It's great to see you Lawrence and Robert, I guess we spent a long time time together since you know, over the last year or so I left Google to and I'm working on a startup that I'll reveal in a short while. But going back to the topic of mlops, something that you know, I spent a long time with with TensorFlow and otherwise, you know, the way I look at it is some things that you alluded to Ryan, where we've had this modeling where you're building a proof of concept. You want to see that. Oh, is it going to add value to whatever you're trying to do? Perhaps you see, okay, yay. It's giving you some predictions, they seem valuable. Now how do I use it? So going from that point all the way to using this model in an application, every bit that you need is part of mlops. I would say there are obviously lots of different things that we go into today, but just at a high level, that's where I want.

Thanks very much Rajat. Chip, let's hear a little bit more from you in form of introduction and our thoughts on ML Ops.

Hey, so my name is Chip, I'm teaching machine learning system design at Stanford and I'm also part of very small team that is building infrastructure for real time machine learning. So as in online predictions and continual learning. Previously I was helping so my background in tube ML Ops is actually starting from the first ML course was I took with Andrew Ng. So he was like the reasons why I got stuck in one of this. Not stuck, it's a good word. So I'm extremely, I'm a bit nervous right now because everyone here so I have been looking up for a long time so I'm just like very excited and a bit nervous as well. So I started in more research background and then I did More went to applied research at Nvidia where we built tools for company to experiment with a new models. But then I realized that oh I think the problems, of course there's a lot of problem with doing research but then solely a lot of smart people working on it. And I think an under addressed problem is with like how you bring these models into productions. So that's why I started joining a startup like Snorkel to work on it and then now I'm focusing on entirely on like how to like. So I believe that like training is a small part of the problem. The problem is retraining like once you have the model out there and how do you keep on like updating it too? Because like you know, have heard of things like data drift, concept drift, like the models performance decorates in real life, then how do you address a problem? So that's what I'm focusing on. So I think it's part of MLOps as well.

Absolutely. Thanks Chip. Robert, could we get a little introduction from you?

Hi, I'm Robert Crowe, I'm a TensorFlow developer, engineer at Google and work with Lawrence and used to work with Rajat MLOps. First of all, I'm not crazy about the term MLOps and there's a lot of disagreement. You talk to a lot of different people, everybody has a different definition of what it is. But essentially for me it's the idea of taking a model and really creating a product or service out of it and all the issues that come up when you try to do that, like drift. As Chip mentioned, the problem of getting labeled data not just for your first data set, but for continuing data sets as you retrain your model. Things like privacy and fairness and serving different customer sets, well, things like resource optimization, all that stuff comes up in a production ML setting that you don't have when you're doing research or academia or what have you. So for me that's the focus of mlops is making it possible to create and sustain a product or service responsibly.

Thanks Robert. And thanks for getting us started with the debate because I hope we can have more debate about just what this field is and what the most important pieces are. But Lawrence, disagreement. Lawrence, let me give you a moment to introduce yourself and disagree with Robert.

So yeah, I'm Lawrence, I lead the AI advocacy here at Google and I'm actually really excited about the fact that we're having these conversations now and we're having these conversations to try and figure out what exactly it is we mean by model ops or MLOps or is it a function of DevOps or what exactly is it? Because if you just think back just a couple of years, all of our focus was like how do we build a model, how do we hyper parameter tune, what type of algorithm should we choose, what type of model like architecture and neural architecture should we be working with? But now the conversations we're having is like, well, how do we use this effectively in production? How do we think about things like the serving infrastructure, scaling, process management and all that kind of thing? So that's just to me by the fact that we're having these conversations now shows that as an industry we're taking great strides forward in bringing ML and bringing AI to the general populace and to the general application developer. And whether it's a branch of DevOps or whether it's something in and of itself, I think is the kind of thing that we'll discover as we're going along on this journey. So it's just great that we're doing this now. Great.

Yeah, glad to be here. Well Andrew, I'll turn it over to you to wrap up our introductions.

Listening to everyone's comments. As a reminder, when I was once leading a speech recognition team and the engineers, the machine learning engineers and speech recognition team had a great result, very accurate speech on the test set, even better than human level performance. So they went to the business product owners and said, look at this great speech system we built. I don't know, give me a raise, celebrate my accomplishments. And the business product owner said, well we tested it, it doesn't work. This sucks. Look at all these users mistranscribing horrible transcriptions. And then my machine learning team said no, we did well on the test set, therefore logically it must work. And it was difficult to move the conversation forward. I think over the last decade, thanks to the rise of deep learning and other things, we collectively have become much better than before at doing well in the holdout test set, which is fantastic. Celebrate that. And I think that if you want all of you watching this, if you want your fantastic machine learning work to make into valuable production systems as well, I think ML Ops is this nascent field they're all trying to invent collectively that I think will help everyone through the entire life cycle of machine learning project. From scoping to collecting and managing the data, to training the model and improving the data, improving the model to then deployment, monitoring, managing, concept driven data driven model maintenance. I think mlops is exciting nascent discipline, solving that entire lifecycle of machine learning project. One thing I'M excited about this is when I teach online on deep learning and Coursera. A lot of things we teach are well tried, true concepts. That's widely agreed on, I think ML Ops and machine learning production, it is very much on the cutting edge, so I find that exciting.

Absolutely. Great. Well, that gets us started off in the right direction. Where we wanted to begin with this discussion of mlops in the first place is there is debate. There is. So mlops is short for machine learning operations or as Lawrence mentioned, sometimes you could see it as model operations, but that seems to be something differentiated. But anyhow, MLOps is really a nascent field and so it seems like it would be worth talking a bit more about just what this field is and perhaps having some debate and disagreement over what it is. But also to what extent do the roles, the current roles of data scientists or ML engineer involve MLOps? So we'll go around the room here and Robert, since you kicked us off in the debate, I'll give you the first say on what it all is.

What it all means. Well, so one thing I try to emphasize when I'm trying to explain, I like the term production ML. What I'm trying to explain that to people is the fact that the world changes. And your model, when it's trained, is a snapshot in time of what the world was when you collected your training data. And depending on what field you're in, that could be fine for maybe quite a while. But just like human beings, if you don't adapt to change, you don't do as well and things change around you and you're not prepared to deal with it. It's the same thing with a model. And in some domains, markets are a good example. If you're trying to use a model to make any sort of market prediction, markets change within hours. So that means your model needs to learn to adapt to those changes. And that usually means collecting new labeled data and retraining your model. So understanding that and the whole the domain, we lost everybody the domain knowledge around that is incredibly important. Are we still on?

We're still on. I think we had a little blip there, but thanks Robert. Raja, let's hear from you.

Yeah, so I think that makes a lot of sense what you said, Robert, around data is not static. What we're doing is not static. So the models which are a representation of the data do have to change as the data is changing, as the market or the world around us is changing. So that's sort of partly a data centric view. Let me give you or provide a different software, more software centric view. In some sense, if you think about models, they're also just functions traditionally with software we've hard coded those functions used to be heuristics or whatever has been replaced with predictive models today. Now instead of writing that software and hard coding it, saying this is what it is and obviously that's going to be a lot harder to change or learn or fix, we're saying, okay, let's just learn that function from the data and use that instead. Now if you take that software centric view a bit, then this is just a function in your overall program that you're building and suddenly you start to see, oh, if you were to think of not to mix it with DevOps, but there are a lot of things that we can learn from there. If you think of all the software engineering principles that you've thought of that we just don't have in ML yet, I think there's room to really learn a lot from there and take that here and apply that. Of course that's not the end of it. I don't think we can just apply those blindly exactly for the reasons what you said, Robert, which is this function is not static. It has to change over time. For certain kinds of things where, you know, you have static images, you just want to identify flowers. Yeah, we're not going to get too many new species of flowers in the next year or two. But on the other hand, in a business where you're relying on customer data or anything like that, things change all the time. And you want those models to be updated all the time.

Absolutely. Thanks Raja. Chip, I'd like to get your perspective on this.

So yeah, I think this is a very interesting discussion. So I think I agree a lot with what Robert and Raja just said. So I think what I heard from both is that mlops is like peculiar because machine learning is not just code. Like you can't just like finish your code and do a lot of testing in depth and then you deploy it and then the DevOps are going to take care of it as they get more system performance uptime and own the classic DevOps metrics. But like machine learning model is like part code and part data and data changes in real time. So maybe we don't have a new type of flower to predict. But like there are a lot of problems when I have new classes on the term. So for I was working with an E commerce company, right. And they wanted fewer problems to categorize their product and I have new Live products out on the time and it's like it's changing up like from like every once every week or every week it's like. But like can be like a few times a day. So so like it's very, very, very challenge a challenging problem. And you can't just like hand the models you DevOps people and just like hopes that like things turn out to be the best for, for the best. But they also you want to model not just like the system metrics, you also want to model how the model is doing. Because every time the data environment changes, the model performance changes. So like now the people will have to monitor on statistic like data. We mentioned that already like data like of drift and stuff. So like DevOps people may not be equipped to do it. And now we had to bring in like machine learning people should deal with like things in production. And maybe machining people are not like equipped to handle all the DevOps tools. So now we have like multiple teams from different domain experts to like try to work on the same workflow. So how do we like help them like talk to each other and communicate efficiently? Because everyone has different lingo and have different expertise. So it's a depth ML option. So about like bringing people from different disciplines, different disciplines should work together to solve like a common problem.

Thanks, Chip. Yeah, so you're, you're mentioning something that seems like it's core in people's minds when they're thinking about this. If they're, you know, maybe they were thinking about wanting to be a data scientist or wanting to be a machine learning engineer. And then how does that tie in with this whole, this whole world of MLOps? Lawrence, I'd like to get your perspective and also, you know, what, what does it mean for someone who's on a team and how they actually interact with the whole pipeline?

Yeah. So I think one of the things that I just like to highlight was something I think Rajat was saying and it's to me the, one of the more exciting things about ML and being able to use ML in production is that you design your systems fundamentally differently than you would on older based production systems which were entirely code based. And in particular that gives you the opportunity by being data driven, to give you a. Sorry, is my audio working okay? I just saw a message pop up.

It's a little rough. Yeah, but we can hear you.

Okay. What I was just saying is that by having ML based systems, what we can do is frequently update our models, frequently update our functionality in ways that would be much more difficult to do in traditional software basis. So with that ability to do frequent updates, then we can serve our customers better. And then that gives feature engineers, that gives ML engineers different workflows for how they can, like I say, serve their customers better. And we have one motto at Google is that if you focus on the user, all else will follow. And being able to do MLOps in NGA in. Sorry, being able to do ML in production and then having an MLOps system behind that so that you can focus on your user, that gives you new opportunities and new skill sets for your ML engineers and others to follow.

Absolutely.

Thanks.

Thanks very much, Lawrence. And Andrew, I'm going to rejoin.

Sorry.

Sure, no problem. Andrew, I'd like to get your perspective on all this.

Yeah, great points. I want to add one additional observation to the perspective, which is the importance of entering the data and a lack of tools for doing that systematically. So for example, Chip mentioned working on e com1e.comm problem I was chatting with friends about the other day was how do you decide whether the label products as hazardous or dangerous? So I have a two year old daughter, so she's inventing new ways to kill herself every day. I'm just going to not let her kill herself. And it turns out that when you get people to label products that you buy on an E commerce site, is this hazards for children? There actually isn't universal agreement. It's actually really difficult to judge what is hazardous and what isn't. And so I think that one thing I see in machine learning is previously we used to have people like us or like any of you watching this hack around the data set for six months or nine months and maybe we'll kind of figure out a way to label it or maybe throw up our hands and say, you know what, I don't know what has this. Let's just get three people or five people to label everything and take an average and hope for the best. And I think those processes, maybe they work if you have a giant living farm or something, but yet even that doesn't work that well. And I feel like if the nascent mlops teams can help with data preparation as well. In terms of deciding what data do you want to collect? Do I need more products in this category, label or that category? What are the standards of labeling the data? I think that there is a big gap today in tools to engineer the data so that when you feed this the code, you get the performance that you want. And I think that is important. In addition to the training and the model data Duration as well as the post deployment, your monitoring and maintenance.

Thanks, Andrew. Yeah, so now we have some of the data centric perspectives, some of the software centric perspective and some other ideas from everyone. I'd like to to take a moment then to think about what does this actually look like in practice? So you've all worked at a variety of different places and I can only imagine that this mlops pipeline or mlops infrastructure looks quite different at a place like Google, that it would look at some small startups. So maybe first I think it'd be really interesting for people to hear a little bit about just what does MLOps look like today at Google? Well, let's start with you, Lawrence.

Okay, hopefully you're hearing me okay now. So I think one of the things that how it would look like at Google is the easy answer is it depends. So in many ways, because mlops is such a nascent thing that there are so many different options for you to be able to design an MLOps based system that it really depends on the system that you're actually building. So one of the things that we want to kind of look at when Google is to say, okay, we have this little box which is our ML code, but one of the things that Google is how do we scale that, how do we make sure that we can focus on billions of users instead of thousands of users. And so a lot of the serving infrastructure that has to be in place for scaling is the similar type of skills that traditional ops folks would have. Then it comes to monitoring. Of course it's very, very important for us to make sure that our systems are up. We know we have whatever it is, five digits, six digits uptime. So to be able to build a decent monitoring infrastructure to make sure that our models are running inference at the requirements, required parameters, required speed, those types of things. We do want to have a good monitoring infrastructure in place, but because there's so many of these different dependencies, serving infrastructure, monitoring, scalability, process management, machine resource management, data verification. One of the important things is that there are so many people and there's so many moving parts to be able to keep all of these working together that we want to make sure that we have flexibility between these, we have standards based interfaces between these, we have open systems as much as possible between these. And we generally of these designer infrastructures that way. And out of all of that then came tfx. So a lot of the things that we've learned about building systems for MLOps, you know, we really focused into TFX And I know Robert's the real expert in that far more than I, so maybe I'll hand over to him here.

Well, thanks, Lawrence. So, I mean, I'm thinking about the small companies that I've worked in and how to contrast that against Google. Part of the situation at Google is historical because we were doing large scale ML really before tools were available in the outside world. So we invented a lot of things like TensorFlow and before that, disbelief in order to accomplish the goals that the business had. And there's still a lot of very powerful but very bespoke tools that are used in our Envelops infrastructure within Google. And gradually we're taking those and moving them into open source. Things like Kubernetes for example, really came out of the framework that was originally developed inside Google to manage containerized applications. Or as Lawrence said, well, both TensorFlow and TFX came out of that work that was done because of an internal need. I see this a lot at different companies where they've, because they needed something and they couldn't pull something off the shelf, they've invented it and it's caused a lot of problems for a lot of people because it's now something they have to maintain. So they're really anxious to try to adopt industry wide, you know, available things that, that are supported by communities, things like TensorFlow and, and TFX that helps them with their burden and it moves things forward as a community and not just, you know, their little team. So I don't know. Did I answer the question? I was kind of rambling there.

Yeah, yeah, I think that gives some interesting perspective. And so people are inventing tools to solve their own problems because oftentimes the tools don't all exist. Rajat, you're, you're at a small stealth startup, I'm assuming it's small, but you know, are you inventing your own tools? What does it look like?

There's so yes and no. Having seen the Google side of things and seen the craziness that ensues when you try to build everything from scratch, no, I don't want to build things from scratch where possible. That said, having been really involved with TensorFlow and thinking about where it's going and how it helps people and with TensorFlow extended, clearly want to leverage what's out there. And there's a lot of value in being able to use these standardized tools where we can. So as we grow, I think we'll use more of those where we are starting out. What I'm a firm believer in is really get that end to end thing working and then try to improve those. So what does that mean for us? The first thing was, okay, let's prove it out, let's build the model. That's what you need to do before you can build a pipeline around it. Of course, even to build a model, we had to do a bunch of data processing and stuff. It wasn't like you could get to the model without doing the data processing and getting things in order. But to start with, each of those was more bespoke. The data processing was a custom handwritten thing because we had a small data set that we could just process and that was fine. Then we had to start scaling it out. We had to run it across multiple machines. Now what do we just scale it out using, say, Kubernetes that came up here? Do we use Spark? Those are the kind of things that come up. And how do you then once you have the model and now you want to deploy it, how do you start to put those together? As we are picking each of these pieces, of course, for each piece individually, we want to use as much of the standards stuff as we can. In the long run, I think being able to use something like TensorFlow Extended is the right way for us to go connect it with the right kind of orchestration systems, perhaps like Airflow or whatever. So we can run all of these and make all of that work. But to start with, for a startup, I don't think it's necessarily the right thing to pick all the pieces together and start with the biggest thing you want to pick and choose. Add things as you go along, but try not to build those from scratch. There is a lot of stuff out there today that we can leverage and at our end we are definitely doing a lot of that.

Thanks, Robert. Go ahead, Andrew.

Yeah, I want to say something that may simultaneously annoy both Robert and Lauren, so we'll see. But Robert mentioned disbelief and then tfx and frankly I remember way back working with Rajat and Jeff Dean and others on disbelief and candidly, I certainly made a ton of terrible decisions. You can blame me for all the bad decisions made back then. Good Rajet and Jeff credit for all the good decisions, but I think that it was too CPU centric, where these giant crazy AZC linear algebra matrix multiplication things, trying to implement distributed CPUs. There's lots of bad ideas you blame me for. And then later Rajat led the development of TensorFlow and that kind of mixed version was much better. I think even though TFX is Still the shiny thing today. Part of me wonders if we're in the disbelief. Sorry, disbelief was a precursor to TensorFlow. But part of me wonders if collectively, even TFX, which is state of the art, is the disbelief of deep learning 10 years ago, where state of the art is the best thing out there. But there's something even much better to be invented. Late last night I was starting a Google Doc where I was writing down frankly a bunch of research ideas to share with one of my friends on new ideas to how to engineer the data. Because I was struggling. Literally yesterday I afternoon I was looking at some labor, some label image data. The labels are junky, but there are about 5,000 images. It was really hassleful to just browse and figure out what's going on and relabel the data. So I was annoyed. So last night I was thinking, okay, maybe you can develop a learning algorithm to make this much better. So I wrote up a Google Doc with three ideas to share. One of my friends ask him, he hasn't gotten back to me yet. So I feel like we're just. There's so much stuff to be invented again. Ten years ago, when we shifted collectively the community shifted the world to deep learning. I did not understand at that time how many tens of thousands of novel inventions and research papers and so on would be needed. And then there came disbelief in TensorFlow and other frameworks that laid a foundation. I think today as we think about MLOps and data centric AI, I think there are easily some tens of thousands of ideas to be invented and new frameworks. And I think, and actually, and I know, right. Chip's been busy out inventing these as well as some of our mutual friends like Chris Wray, Alex Ratner, you know, there's a lot of people just nibbling at this. I think it'll be a big movement.

Yeah, I think that's a great segue. Actually Chip, I wanted to hear more from you on. Before Stanford, you were at Snorkel, which is of course a tool for some of these MLOps data pipeline stuff that might be part of some of this collection of tools that people are using for what they're doing at their company. You've also written a post that captures what are all the tools out there? What are some of the trends? Really Love to get your perspective on some of these things. Are the tools really out there now or is much of it yet to be invented? What do you think things are?

So I think that before get there, I just want you to take a step Back. So I think the requirement for tools right now actually depends a lot on the company size, use case and also maturity. So I tend to think of like mlops adoptions based on two access. One is from like the size of company and it's very small. Like small startup like agile and then have like two large company and I think like the adoption is like look like something like this. So it's very big company like Google, Netflix, Alibaba. They like have incredible infrastructure engineers, they build incredible tools and they move very fast and they have state of the art ML Ops tool and they have like a smaller startup and who are like very, very agile and they like eager to adopt new tools. But they're like a lot of companies in the middle who like very interested but then like they are boggled out by either lack of like really good engineers because also really good engineers like to join like Google and stuff, you know like and then like they instead of buckled up by legacy systems. So let's say once you update. First of all a lot of companies like once you like switch to like do real time machine learning but say like a buckle down by when the system is set up to do like batch jobs as they can't really easily switch to like streaming. So like legacy system is one thing and then they have like adoption maturity. Like it's like the longer the company has been like adopting machine learning, like the, the more effective they are and they can and the more sophisticated. So, so in the beginning that you just want to go from like no machine learning to machine learning. So they just want to have some tools to help you get there. But then I want to have a couple of things in production. You, you want to like make the most of it. So you worry about regional monitoring and like squeeze out like the last job of performance. So like and this company require different tools. So. So yeah, I'm just. Oh yeah, I'm sorry. I like totally like off track on every time. But. But yeah, so, so, so yeah the true ones depend on it. And I think, I think for the. Because it's like so many wide requirements, it's really hard to have a standard pipeline for every company that works. But I do hope that in the next like five years or something like when everyone has been at the same maturity level then we can have more standardized pipeline.

Yeah, I wanted to follow up on that also because you're teaching this stuff at Stanford. So. So when you're telling students about how to think of all this and also in the article you point out that While modeling and achieving the best accuracy with your model was kind of the bread and butter of research in academia some years ago, it's evolved to more interest within research regarding MLOps or ML infrastructure. Could you say more about that and what do you tell your students?

So this is like actually when you're stitching, the course is very hard to come up with what you teach the student. It's not that like they are not things you teach. It's just like so, so currently a lot of ops is very tool focused, right? They have a pipeline, say like, oh yeah, you started like using this tool for modeling, this tool for monitoring, this tool for bringing to productions. Right. But the problem with tools that tools like evolve over time. Like you have like new packages coming out all the time, have a new version coming out all the time. You want to teach student tools like their knowledge, like what it takes to be outrated, like as soon as the new tool. So actually I learned that lesson because the first course I taught was TensorFlow and the first version was in TensorFlow 1.0. It's like, oh no, this like graph base, right? And then you switch like Rajat, I blame you, like you switching to like 2.0 and like ego is education. I was like, oh my God, I need to update like half of my materials. It's like, it's just so much work. So, so I want to try to like teach more on like, on like best practice, like on the more philosophical level. But then students like, oh, you can't, you can't really learn how to do ML ops by just like looking at philosophy like always, like best tense principles. So it's like, it's very hard to strike the right balance. And it's a course that still evolve, right. So I think it's also might be a challenge for students or like engineers trying to get into like ML Ops, ML productions. It's just like you don't want to like just learn about tools, but you also don't want to just learn at a high level either. So I'm very curious to see the course that mlops specializations by course by deep learning to see how you guys like try the right balance.

Yeah, I think that it was definitely something that took some evolution along the way. But also you sort of started us thinking about apart from courses or apart from theory, what is it that people should be doing if they're trying to prepare for this field? I think a lot of people watching today are thinking they'd like to prepare themselves to be a good candidate. For a role in the sort of MLOps world. So Rajat, could you extend that?

Yeah. So I think Anjip had some really great points there on how to think about it and stuff, but one of the things that she said was it's hard to teach and learn at a philosophical level, saying, okay, this is how things should be in general, that's great, but how do I apply it? How do I think about it in practice and stuff. And again, coming from a more software centric view, my perspective there has always been, okay, if we have ways of, we believe that this is the right way to do, let's say data labeling, or this is the right way to building models, or this is the right way to deploying models, can we build software to simplify that? That's where the tools comes, that's where TensorFlow came, that's where TensorFlow extended came and all of that. How do we codify that so it's easier for the next person where they just don't read a set of principles that here are three things to read and develop. You actually have software to do those. And yes, none of this is static. Things change over time. TensorFlow went from version one, which had certain things, to version two, which evolved and learned from that, and applied a whole bunch of new things as well. And I'm sure at some point we'll see TensorFlow 3 or something new as well. Totally. We should continue to iterate and live with that. But at the same time, for people to be able to leverage this and be able to do more ML, I think they need good, solid tools. If I, you know, going again, compare, comparing to software, where we were like where we've evolved over the last 40 years, clearly ML hasn't been around that long. ML as a field has been around that long, but not as being deployed in products and stuff. So there's a long way for us to go to really catch up to a lot of those and learn from them.

Absolutely. And so Robert, you're, you're the instructor for much of the machine learning, engineering and production and ML OP specialization that we're putting out the third course of today. Lawrence, you're also one of our instructors. You two have some perspectives as well on if people are taking these courses, what else should they be doing? Lawrence, what are your thoughts? Look like you were about to say something?

Yeah, sure. Hopefully you can see and hear me now. And I followed some good OPS principles and I had a backup ready. So I think one of the things that's really interesting and just Going back to stuff that Andrew and Chip and Rajat said is that it's something like TFX. It's at version 1.0 now, and we're dealing with 1.0 stuff. And if we think back to other technologies, Anybody remember Windows 1.0 or Java 1.0, and look how far they came. But we can look back and laugh at how naive they seem to be back then. But we needed to get through that to get to, to be where we are today. And I think, you know, in the mlops space and in the Model OP space, and we're at 1.0 with a lot of these products. And so to understand what it is that we need to learn, we also have to understand that many of the things that we're learning, many of the concepts that we're learning now, we may be throwing away in 18 months, we may be throwing away in two years as we're rapidly iterating towards, you know, version 2.0 and 3.0 and 4.0 of MLOps systems. So I think, you know, that's, to me, that's why I'm particularly excited that, you know, Deep Learning, AI, Google, we're all working together to be able to create courses like this. That Chip is doing the stuff that she's doing at Stanford because we need to get, like a critical mass of people with these skills in place so that we can learn from our mistakes, we can learn what's working, we can learn what's not working. We can see the opportunities out there that, you know, startups and other people in the industry are going to be able to take and run with, to be able to build an improved system for everybody. And so I think, you know, it's, it's, it's particularly exciting time to be working on this stuff. It's like I said, if those of us who are in the industry when Java 1.0 came out or when Windows 1.0 came out, look how much we've grown. And I would love like five years from now, ten years from now, to be looking back and saying, like, with MLOps and ModelOps, it's like, look how much we've grown.

Yeah. I think it's not like any of us regret it. Right. Learning things on deep learning or TensorFlow one when it was immature. I think people should learn TFX and then it'll get better and it will evolve with the tools.

Yeah. The more people using it, the more requirements are going to be driven and then the better we'll be able to build Whatever product, not just tfx with anything.

Yeah, I very much agree with that. One of the differences for CFX is that it's been used so heavily inside Google for so long that it is a lot more bulletproof than, and really well thought out than a lot of, you know, something coming from a different company that is just brand new. So there's, so there's that going on. But to get back to the original question, what, what should they be learning if they want to be in ML Ops? It's, it's a difficult skill set and it's a rare skill set. You, you, you really need to understand both the ML side and the software engineering side to be really effective and those people are hard to find. I can tell you. I've tried to hire and usually you have one or the other. You have people especially coming from like a mathematical or statistics back background who have, you know, very strong theoretical understanding of ML and you know, maybe know something like R or Python well enough to put models together. But in a, you know, in a production setting it's very difficult for them to create production level code and systems. So it's, it's tough. You kind of need both sides of things and it's, it's a lot.

And I feel like there are no jobs in, there are very few jobs in MLOps right now. You won't see that in a job description, but don't let that fool you. Lots of companies are trying to hire people they know how to build and deploy machine learning systems. So job interviews will ask, have you ever deployed a machine learning system? That's basically an ML OPS question, even though the word ML OPS doesn't appear in the job description. So I think this is an important skill, very useful skill for people to be learning today.

I know Chip just open sourced her book on like ML interviews and all that kind of thing. I'd love to know have you had any experience or have you seen anything while researching that book around the type of skills people are looking for an ML Ops or model ops.

So I think it really depends on like different company size and solely where they are in their adoptions or maturity. So I think what I'm trying to get my student to practice is just like don't focus too much on techniques or tools. I don't just go after this buzzwords like try to like, so try to be like problem focused. So like what is the problem you're trying to solve? And I just try to find whatever you can to like solve that problem. So like instead of looking like, hey, what's the latest terms or tools in ML Ops? Like, why don't I just try to do a project? I try to deploy a simple model on the phone. And in the process of doing that, they suddenly realize so many. And also there's so many tools that like, at least I most a lot of my students can't appreciate until they have run into the problems that requires those tools. So, yeah, so I think I would keep encouraging my students. People who read my books and interview is just like, they try to like do a lot of projects. I try to get involved. I know that like the project you do is a personal project, gonna be very small and might not as a scale required for a lot of companies, but that's where you have to start. And then you try to get involved more. I get the internships with companies when we allow you to try it out at a different scale.

I think what Chip alluded to a couple of times, I think is actually a very interesting point. I find that MLR's practices are quite different based on the scale of the data set and maybe company size. So I remember I once built a face detection and recognition system with over 300 million images that use a certain set of processes. Whereas at Land AI, which is working on an MLOPS platform for computer vision, we often work with hundreds of images. And so the tools and techniques are very different. The other gulf is structured versus unstructured data. With structured data, you can get humans. Excuse unstructured data that images and audio get. Humans examine it, label it. Whereas for tablet data or structured data, human judgment on it, it's hard to look at a list of transactions and figure out what the person was really intending when they did whatever. So I find that these gaps, big, big, big differences in practices. So, yeah, these days, just because of my work at Land AI, spent a lot of time trying to think how to innovate ML Ops for computer vision and how to help all of us collectively build and deploy computer vision systems 10 times faster. But I think there's actually a huge space and a very large family of techniques to be invented still.

All right, thanks everyone. I want to make sure that we spend a little time getting to questions that have come in from the audience. And so what we'll do now is shift to answering individual questions. And what I'll do is ask for one person to give a perspective and a brief answer, and then we'll move on to the next one just so that we could try to get to some number of questions first.

Off.

We want to start with a couple questions from learners who are in the current set of courses from DeepLearning AI, the machine learning, engineering for production and MLOps specialization. And so the first question we have is from Pablo Drummond. The question is about experiment tracking. And he says, I understood that the mlops pipeline needs to be data centric, not model centric. But when it comes to the experiment tracking, like models that might not have been deployed, do we really need an application or framework? Don't you think that we need to address this requirement as an mlops, a part of the mlops infrastructure? Does somebody want to take that one?

I can try a little bit with a bit of experience that I've had with these things. And primarily in my case it was with building models for use in mobile. So to deploy them to an Android or an iOS device. And we have an infrastructure at Google, a product, sorry, I hope this doesn't sound like a product pitch called Firebase. And one of the really nice things that we were able to do with Firebase was deploy multi models based on different data sets or based on different neural architectures and then have like an AB testing infrastructure that would say, hey, this cohort gets this model, this cohort gets this model, this cohort gets this one, and then tie that with analytics so that we could see the model performance, you know, based on, you know, different cohorts, how they did with the model, did they get better inferences, did they get faster inferences and those kind of things. So my only experience in like in that space is really doing it like that. And I found that it was really, really powerful way of really getting feedback from multiple users so that we could fine tune models so that we could fine tune the, like, the neural architecture used in the model, the data sets that we use in the model, and those kind of things. So you know, as well as data centric MLOPs, we could also have model centric MLOPs like that with, you know, a smart way of deploying models to different users. Sorry, Andrew, I think you were going to say something.

Yeah, thanks. Thank you, Pavel, for the question. So I think, you know, some of us have this experience where we run experiment find it does well on the test set. And then we talked to, you know, our colleague that had worked on and said, thanks for sending me that data set. Where'd you get this data set from? He goes, oh yes, it's on my laptop. And then he says, oh yes, but I built that data set because my other friend had emailed a data set and is on their laptop. And where's their laptop? Oh shoot, my laptop got stolen. Or accidentally delete the file and save this great result that's completely not replicable ever again. So I find that. And I remember when I used to do experiment tracking, I would use VIM to edit a text file on my desktop. So that was local to me. And then we upgraded to doing experiment tracking by filling in rows of a Google spreadsheet which could at least be shared. But then I'd forget some detail, I'd forget to fill the learning rate in some column spreadsheet and became non replicable again. Or we end up going back to redo the hyperparameters to figure out what exactly did we do to get this result. So I feel like in principle experiment tracking can be done in a manual way, but I do think that with tools for experiment tracking we should keep track of the model as well as the data, as was the data lineage and process. How on earth did all the data go through all this stuff to resolve in the model? Then that actually makes it more possible to figure out how we got the model and then also how to improve it.

Thanks, Andrew. And we have another question from someone. This is from Shahroz Aslam in the first course. This is something you mentioned earlier, Andrew, but there's, there's a discussion of kind of what is the agreement or compromise between the machine learning engineer and the business owner regarding the goals of an ML deployment. So for example, test set accuracy versus revenue. So the question is how, how should we educate? As I'm assuming that Chef Rose is coming from the perspective of the machine learning engineer, how should we educate industrial management regarding the machine learning cycle and how to tell them that the resources that they're very motivated to invest in might not give them the results they're looking for. Rajat, you've perhaps been a part of some of these conversations. I'd love to get your perspective on that.

Definitely. So today actually I'm on both sides of the conversation in the startup. I'm putting on lots of hats. Some of that done has included the ML engineering hat and oh, can we get the best model possible or even the data scientist had in that sense. Right. Two, at the end of the day we care about the product that the customer is going to see, what does it mean for them? And so that's like if I think about the business perspective, that's the perspective. Okay, Whatever we are doing doesn't matter if it's a model, doesn't matter if we're writing software, how does it help the business? How does it help our customers, our end users themselves? So again, going back to how to think about it from an ML engineer perspective, I think it's important to really think of that end to end. Can you get the whole thing working? Not just say, for the example that Andrew said, oh, the speech data set. I have this amazing results on the speech data set. I've seen way too many of those, perhaps even 100% accuracy because we are overfitting on that data set, of course. But you go to the real world and it's, it's not real anymore because in that product for what the user actually cares about, that that's not working. So you know, again, what's important is get to the most basic thing that you can do as an ML engineer first. Make sure it's having an impact at the end of the day, make sure it's changing somebody's life at the end of the day and then keep improving it. Then you'll see that works for both the folks and that's the thing that the ML engineer needs to learn. And from the business perspective, they need to learn that, you know what, Initially you might see only a small win, but it's worth investing because long term this is where you're going to get. It might be a sort of a local improvement or whatever, local minimize we might say. But if we keep investing, we keep improving things there.

I think in any business there's always a tension between the business decision maker and the technical decision maker. And with ML and with ML Ops in particular, I think the business decision maker has a pretty tough job because there's a lot of belief out there that AI will fix every problem and ML models can fix every problem and that kind of thing. And they have so much pressure on them to say, well, let's transform our business using AI. So then it's the job of the ML engineer, I think in this case also to manage their expectations about what is possible, what isn't possible, and to have that good conversation with each other so that you could understand what they, the external business requirements and customer requirements are and see how it is that you can drive that and build a roadmap to be able to drive something there, but also to push back on the expectations that AI or ML is some kind of magic pixie dust that will fix everything and create a new business. And it's in the tension in those conversations that we can actually really learn new things, that we can really drive any business forward.

Absolutely. Thanks LAWRENCE all right, well, we'll turn to some questions that have come in on the slido platform where we're collecting questions and people are able to upvote them. The most upvoted question is from Jerry Bai, and the question starts out by saying machine learning engineering is much more than just algorithms and models, something that we've been talking about throughout this discussion. What other sort of tech skills should one have to be considered A good machine learning engineer or MLOps practitioner? I'll add to that. So is this like back end and back end software engineering or something else? Robert I think you alluded to this a little bit when you were first discussing your thoughts on production, ML, MLOps, what it all means. Could you say more about that and answer Jerry's question there?

Well, so it's a mix of software engineering and infrastructure development and ML, but I think core to that really is an understanding of the whole ML life cycle and really, you know, understanding the different issues that are different when you put something into production and offer it as a product or service. So things like, you know, are you going to serve different parts of your data set if those are people, are you going to serve different groups of people you know equally well, or do you even know how well you serve different groups of people? And that requires a deeper level of analysis of your model performance? Or where am I going to get labels for the next data set that I need to train my model next week? How do I generate those labels and how do I generate enough and what label quality and how much drift do I see all those issues come forward? So it's a multidisciplinary thing that requires at its core, an understanding of the data and the model and the life cycle. So I think the most important group of skills is around ML. But in order to actually implement all that stuff, you need exposure and some experience with, you know, the rest of the of the deployment chain, really.

Thanks, Robert. And I think that one of the things that this is, that has been a question that's come up over and over. What are the, what's the total skill set that someone needs? And I'll just mention that one of our partner companies, workera, puts out an assessment that allows you to kind of see where you are at it with algorithmic coding or software development or ML or all the different pieces. So it's kind of a nice tool to see where you might.