A High-Level Approach for Solving MLOps Challenges

Sa. Sam. Foreign.

Hello everyone and thank you for joining today's webinar. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in a couple of minutes. We're just waiting so everyone has a chance to join. In the meanwhile, though, we'd love to hear from you. So let us know where you're joining from using the chat, the comments depending on what platform you're watching on, and yeah, tell us something that you'd like to get out of today's webinar. A few bits of housekeeping before we get going. The session is being recorded and the recording will be emailed to everyone that's registered for the event. If you haven't registered already, I'm going to send a link in the chat very shortly, so keep an eye out for that and then, yeah, head over and get registered and we will send you the recording as well as all the other resources we've got. We are also going to be sharing links in the chat throughout the session, so keep your eyes peeled for those while watching along. And we're also going to have a Q A towards the end of the session, so if you have any questions at any point throughout the session, then please let us know in the chat and the comments and I'll be saving them for the Q and A as well. Brilliant. I think that's everything from me for the moment. So yeah, I'll be back to repeat these messages for any new joiner shortly. But until then, enjoy the background music. Sa.

Foreign.

And thank you for joining today's webinar. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in a couple of minutes. We're just waiting so everyone has a chance to join. In the meanwhile, though, we'd love to hear from you. So let us know where you're joining from using the chat or the comments. And yeah, tell us something that you'd like to get out of today's session. A few bits of housekeeping for anyone that's just joined. The session is being recorded and the recording will be emailed to everyone that has registered for the event in the next few days. If you haven't registered already, then head over to datacamp.comwebinars or you can find the link that I've put in the chat as well. So yeah, keep your eyes peeled for anything in the chat. We are going to have a Q and A at the end of the session today, so if you have any questions at any point throughout the session, then put them in the comments and the chat and I'll be saving them for the Q and A.

Brilliant.

I'll be back to repeat these messages shortly for any new joiners, but until then, enjoy the background music.

Sa.

Hello everyone and thank you for joining today's webinar. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in about 20 seconds or so. We're just waiting so everyone has a chance to join. Let us know where you're joining from using the chat or the comments. And yeah, tell us something that you'd like to get out of today's webinar. For anyone that's just joined, the session is being recorded and the recording will be emailed to everyone that's registered for the event in the next few days. If you haven't registered already, then head over to datacamp.comwebinars or you can find the link that I've sent in the chat as well. To make your life a little bit easier. I'll also be sending links to other resources throughout the session, so keep your eyes peeled for those as well. We are going to have a Q and A towards the end of the session, so yeah, if you have any questions at any point throughout the session, don't feel like you need to wait until the end. Send them as soon as they spring to mind and I will save them for the Q and A.

Brilliant.

I think that's everything from me. So now I'll hand over to your host for today's session. Richie. Richie, please take it away.

Hi everyone. Welcome to the webinar. This is Richie. Now, one thing I like about data science is the tooling is fairly simple. So you choose Python or R, throw in a bit of SQL, pick your ID and you're pretty much done. But with MLOps, it's a different story. There are so many different tools, it can be hard to figure out what to use. So today we're going to help you solve this. But rather than giving you product recommendations, our guest is going to help you think about the problems that you and your team are facing and develop mental models for how to select the best tools for the task. Just really getting back to basics and like, why? Why do you want to do this and what are you trying to do? Now, that guest is Dean Pleben. He's the CEO and co founder at dagshub. This is a platform for managing data models, experiments and code. So if you've, if you're a regular at some of these webinars, we have had a few webinars from Dagshub before. We'll send links to those in the chat if you want to catch up on the recordings. Now before that, Dean was a researcher in quantum optics, so some pretty fancy physics going on there. And he's also the host of the the mlops podcast. So please take it away.

Dean thank you Richie, thank you for having me. And I'm really excited for this talk today. What we're going to be talking about is a high level approach for solving MLOps challenges. Or sometimes I like to call this talk solving MLOPs from first principles. One note sort of upfront is I am hailing from Israel and so you might have heard there is sort of a somewhat of a war going on. We might have rocket alerts and I'll have to leave for a few moments. Don't worry, I'll be back and hopefully that doesn't happen and we can go smoothly through this talk. And there's going to be quite a bit of time for questions, but please don't hesitate to send them through while I'm speaking. If they make sense, I might stop and answer or I'll get to them at the end. And with that, let's talk about solving MLOPs from first principles. And what better way to start than to talk about dogs? Man's best friend who doesn't love dogs. I hope no one in this audience and I think that it would be great to have a dog detection model that will help us point out in real time all the nearby opportunities to get to know dogs. Even better, this can be a constantly improving model where we gather new data from dog lovers in the field and as time goes by it might even achieve sentience and weight. Wait a second, why is it constantly pointing me to hot dogs? Who snuck hot dogs into our dataset? When did this happen? And how do I get my pristine dog data set back to prevent this catastrophe from happening again? I wish I had kept versions of my data, kept track of which model was trained on which data, and then I could track the first hot dog incursion. But how do I achieve this goal? Should I save a zip file of the data set every time a model is trained? Should I buy a $1 million a year SaaS? Or maybe I can build a not hot dog model to throw away bad data. There are so many options in this talk. What I want to do is I want to share a five step framework or mental model to guide your MLOps tool choices from first principles. I'm going to use a concrete example of choosing or solution for data versioning as seen in the tragic dog photo accident. And I think that the added value of this framework is that it's going to clearly define a process that you might have done in the past but you hand waved over and it's going to help you put limits on your time and your process so that you save that time and you save many tiers. And I'll talk more about that later and I'm going to go over each step one by one and explain them through. For each step I'm also going to share some of our personal learnings from performing this process with multiple MLOps tools and specific pitfalls you might stumble on. Now, many of these points apply to classical DevOps tools or for choosing tools and solving problems in general. And that's great. It means that even if you're not yet an MLOps engineer, this talk can be useful to you. And of course it can be useful even if you're a data scientist and you have an ML Engineer or MLOps Engineer rather than in your team and you want to either help them make decisions or make sure that you're sort of assisting in the process and not just throwing a bunch of product names on them and just like making everything more complicated. And to make it easy for you to remember, I created a mnemonic device and these are the initials of each step do guy and whether or not this is a coincidence that it seems like a dog, I don't know. You tell me. So Rich introduced me, but I'm dean and thanks for joining my talk. As he said, my background is a combo of physics, machine learning and computer science and I think what I want people to know about me that relates to this topic that we're talking about today is I'm really passionate about building tools that help machine learning teams and professionals work together. I am a strong believer in open source and the role it has to play in the world of machine learning. And I'm the co founder of dagshub. We're building a platform that takes popular amazing open source tools for machine learning and combines them into a single end to end platform experience, making them easier to use, more convenient to collaborate on, and then adding a lot of workflows that are needed. I'm also hosting the mlops podcast and the goal of the mlops podcast is speak with industry leaders and experts about their experience bringing machine learning models and projects into the Real world. And hopefully this answers to a certain extent why you should listen to me. But I think more specifically, we have gone through this process ourselves at Dagshub, and we go through this process with our customers and our users a bunch. So we've really done this a lot of times, and hopefully our experience can help again, save you a lot of pain and effort. So I mentioned first principles thinking a bunch of times. I know this term is now a bit more popular than it was when we started thinking about this and preparing this talk, but I want to explain what it is in case you're. You're not familiar. And the best way to explain what it is is by explaining what it isn't. So not first principles thinking is when you think in analogies. And so we want to avoid the mental model of I want what I think other people have. I think that's a great way to choose solutions that have really good marketing or copying what your friends are doing, but that doesn't really solve for the things that you specifically need. And so instead, what we want to do is we want to start with the fundamental truths that underlie the problem that we're trying to solve, the things that we know with most certainty are true, and then reason about the solution from there. So it's like bottoms up reasoning. And this might sound obvious to you now that I phrase it that way, but surprisingly, very few people that we've spoken to do this now. Later, after you've gone through this process, you can ask questions like, what do other people have? What are they using? Or search for analogies. And those are actually useful heuristics. But a meaningful result of going through this process first is, is that you're going to be more resistant to the different marketing messages and flashy features and actually find solutions that are better, better suited for the problems that you specifically have. Now, I'm not trying to claim that doing solving mlops from first principles is easy. I actually think it's pretty hard to do due to something that I like to call mlops fatigue. We have so many tools to choose from and everything feels really siloed with vague marketing messages and interfaces. And it's really hard to see how we can create a cohesive workflow. Then, of course, companies don't make it easy on us. They add marketing messages like, iterate faster, all of your machine learning needs solved, and get everything started up in just five minutes in one line of code. I think that that mess usually leads to a process that feels like four stages of grief. Maybe Some of you have experienced this. Let me know if this resonates. Step one is you start by doing everything manually. You might email pickled models, you might zip datasets, upload them to Google Drive and send links. When you get started, that's actually okay. It's a lean approach. And if you do this process manually for the first few times, you're going to have a much better first principles grasp of where there are actual friction points and what requires automation. So maybe you realize that pickling models and emailing them actually does work for you and your team members. Most likely not. But at some point you usually start losing either work or time or sleep over things that you could resolve with better tools. And that's when you graduate to step two, which is you say, okay, well we need a tool to solve this problem that we're having, so let's take a look outside and oh shit, we have so many tools to choose from. Everyone says that their tool is the best thing since sliced bread. So you get into analysis paralysis and this can be deeply tiring and depressing. And you keep on thinking, oh, the tool that I chose, maybe there is something better out there, that if I just spend a bit more time evaluating this other thing, it'll work better. And so you give up at some point and you adopt the mentality of let's solve everything at once. The best way to find a solution is to find something that solves all my problems, including my future problems. And we start by ruling out any tool that doesn't have anything and everything we imagine an mlops or a machine learning project needs. And you find yourself starting to say things like, my problem is mlops, I need to solve mlops. Now notice that mlops is not really the name of a problem. It's sort of a school of thought or workflow or mental model. So doing everything usually means being mediocre at everything. And edge cases will pop up that are only served by best of breed standalone tools. And that's going to be hard to integrate with this all in one tool. In many cases, and in the worst cases, this is going to actively lock you in into a product that later doesn't solve your problem perfectly. And then you're stuck. Now one underlying assumption here that I'll mention again in a bit is that machine learning work is fundamentally experimentation oriented and therefore more unpredictable compared to, for example, software engineering and DevOps. So this part of the grief cycle can be a real killer. And then the last part is usually you give up on that because you can't find a good end to end solution that's perfect for you and you decide, well, let's just build everything. I know the problems I have best and exactly what I need to solve those things. Now, this is the most resource intensive approach. You didn't build a GitHub clone, you're not building your own cloud service. So why does it make sense here? Many times companies that go down this path, you start, it sounds really rosy, you're going to get a custom tailored MLOps platform, but you end up with something that is suboptimal. Because making something that's perfect takes a lot of effort, both in time and money. Many organizations don't have those resources to do that. And so if you do by chance have a huge dedicated internal ops and development team that is just for you, maybe it's doable. But now let's talk about some of the assumptions that I'm making here. And the first one is that that's true if you're Facebook or Google or OpenAI, but you're probably not one of those companies. And if you try to solve everything in advance, you're going to find that you need 70 DevOps people for each data scientist that's running linear regression. It's very tempting to think about millions of requests per second on your model, but if it hasn't received one request yet, it probably makes sense to build a system that supports that first. And this approach is in fact the opposite of first principles thinking. If you assume that your team is analogous to Google and then building what they have, that's a great way to get something that's not really good for you. So the other assumption that I'm making here, right, is that mlops is still in its early days. You know, the relative maturity of this field is obviously progressing over time and I think we're in a much better place than we were when we started Dag Sub a bit over three and a half years ago. But it's still early and still things are changing a lot this year. Just to remind everyone, in the beginning of this year we had ChatGPT and now everyone is talking about LLMs and starting to call themselves AI developers and not machine learning engineers or data scientists. That's very recent. And that means that there's yet again another change in the tooling and everything. And that means that a lot of the tools and workflows that we're using today might not be relevant or might change drastically in the not too distant future. And that's also something that we need to keep in mind when we choose these solutions that we're looking for. So the last and maybe most important assumption and point to remember here is that. And again, if you take only one thing from this part of the talk, let it be this. It's all a matter of trade offs. We're trying to balance three, maybe sometimes more, but I think three main things. The first is getting our models to production at the relevant scale for us. The second is not to spend too much time and money on doing all of that. And the third is building something that's future proof so we don't need to redo our system every month. This is a trade off because many times you can't get all three perfectly solved. You're going to need to prioritize differently depending on your company and your specific needs. But it is important to note all these things and understand where you want to place sort of the weight in your specific case. And another thing to remember about this is it is better to make a bad decision than not to make a decision. Because I think not making a decision basically pushes the problem down the road. When you've invested more time and money and effort and into implementing something and then you might realize that it's not good for you. So now that we talked about the problem, let's talk about the solution. Okay, the first thing is we want to be problem focused and not list of feature focused. Sometimes I go on calls with users or customers. They tell me something like we need feature store. The first question I ask them is why? What in your workflow requires a feature store? That's not to say that they don't need a feature store or an experiment tracking solution or online IDE or whatever it is. The buzzwords change every now and then, but the idea is that you need to have a problem that you can clearly define and then you want to start from there and build a solution. On top of that, if you heard that feature stores are really good way to manage data and then you're looking for a feature store, then you might not know what parameters of a feature store you actually care about. Now, if we are problem focused, this is going to limit our search scope in two positive ways. When I say limit, I don't mean that in a negative sense. I mean we're going to spend less time looking in the entire landscape of tools and we can focus on what actually can work for us. So the two ways in which it limits our search scope is one, maybe there's no existing solution that actually solves the problems that I have. If I don't define my problem, I won't know that I'm just going to go with whatever tool exists and figure it out later. And the second is what I said earlier, you're going to be less distracted by buzzwords and marketing messages, so you can actually focus on the things that you care about and make sure that the solutions you're looking at are good for your problems. The second assumption is that the correct process is to start by building a simple pipeline to production. We want to think from first principles about which parts of what we're going to do from the start of our project, let's say data collection all the way to our models deployed and monitored. We want to find out what we're going to be repeating a lot, what isn't working for us right now, where we have unique bottlenecks. And this can be things like, we don't have a grid solution for data labeling. And that's taking a lot of time of our data scientists, which we want to dedicate to actually curating the data and building better models. So let's start working with a data labeling tool and the labeling service. It might be deployment. Maybe our data scientists are building models that are great, but the scale at which we need to deploy means that they can't use all of the data transformation functions that exist in Pandas, for example. And we need to focus on that and understand what makes sense. And then of course, it might be model training. Maybe we need more GPUs, maybe there's specifics to our model training process that we want to improve. But the point is that this is going to vary wildly and really depend on the business problem that you're solving. And so what I'm suggesting here and what I'm assuming about the problem moving forward is that the best way is an MVP mentality. You start by going through the process with the simplest model. You can, if you can make it an if else statement and not even train anything, that would be great. You just get to the end where you have a model deployed that's running even if it's not in real production, but actually getting data from production and you see that it works. And then you start iterating on the things that aren't working, like maybe we need to improve the model, maybe we need to improve the data collection, et cetera, et cetera. The third thing is that in many cases, machine learning tools are inspired by the equivalent developer tool for a specific use case. And that makes a lot of sense because software Development is older than machine learning, at least in the OPS perspective, and there's been a lot of blood, sweat and tears that have gone into building tools for developers now. Still, you need to remember that there are fundamental differences in the workflow. There's a more experimentation oriented aspect, and just to give one concrete example, the scale and costs of handling data put constraints on the workflow and the normal DevOps methodologies might not be practical. Imagine you have a CI CD pipeline that needs 8 GPUs and 48 hours to train every time you do a git push. This means that most of the tools and solutions need to solve for the ML twist, and we can't just say we'll use software development tools and it'll be okay. The last thing that I'm assuming is that I'm prioritizing open source software wherever we can, and the main reason for that is because the field is still fuzzy. In general, I'm very pro open source, but I think when things are changing quickly as they are, open source is much more important than if they aren't. For example, if you go to more classical systems, if you buy an Oracle SQL Server, you pretty much know that they're going to be plus minus the same thing with the same limitation in a few years. Shifting to a different SQL provider is going to be relatively easy, assuming you stuck with with vanilla SQL. But in MLOps the flexibility is extra important because the infrastructure is constantly changing and you want to be able to sort of dive one layer deeper and integrate your new data source into the tools that you're using. And that's going to be much harder if you're using proprietary systems. Now there are very few SQL like established conventions for specific needs in MLOps so far, but of course there are a few standards that have merged onnx, pmml, stuff like that. But mostly I recommend going with something that's open source. Okay, so I'll stop here for a sec. We're now going to go into the specific example of data versioning just to try to demonstrate what I mean with a framework, and I'll take this minute to see if there were any questions that came up while I was talking. Seems there was only one, which was about combining Agile techniques and mlops. I'll probably need clarification on that, so I'll answer that in the end, but if you can explain Wallace a bit more what you meant, that would be helpful. Going into a specific example, we talked about first principles, thinking, and now let's look at a concrete example of data Versioning and, and try to build out the method on top of it. You want to say, or you can say I want a tool to version my data, but I'm going to argue that that's actually bad form. That's not what you want to say because it's going to lead you down the feature path. You're going to look for solutions for your data versioning. You'll find those solutions because there are tools for that. But it's going to be harder for you to understand what you're looking for and you'll see a lot of tools that are not relevant for you and invest a lot of time in evaluating them. And so instead what I recommend you say to yourself, and we'll later talk about this, but write it down, is my data is regularly changing and I want to be able to revert back to older versions so that I can avoid, so that I can do disaster recovery. And clearly defining the goals that I, that I want to achieve so that the tools I choose are going to help me do that. And the first step here is basically defining the problem. Defining why something is important can help us prioritize this with other tasks that we have. Maybe you go through this step, you realize that this problem isn't as high priority as you thought and you should think about it. But later, or maybe you think your problem could be best solved by data versioning when in fact formatting the data in a different way. For example, adding timestamps could provide an easier but as effective solution. To make my point, here are three options that data versioning as a tool might solve. Each one leads to very different priorities and possibly different tool choices, but all could be considered data versioning from a non first principles approach. If you need a solution that helps you revert to an earlier version in case of bugs, if you need a solution to know which data is used where, if you need a way to add modify data without breaking downstream consumer projects, those could all be solved by data versioning. But you're going to actually end up with very different solutions. Two common pitfalls that we see in this stage is one, when you do this in a post hoc way, you start by saying I need a tool for data versioning and now you have a feature in mind and then you describe a few problems that you know that you will solve because you listen to this talk and you say, well, these are the problems that I have, that's exactly what I need to solve, right? And I don't recommend doing that because again, that's going to lead you down the feature path and that's not what you want. And the other sort of pitfall. I'm sorry, I'll take a sip of water. The other thing is when you think that all your problems are equally important. Maybe you saw the slide I shared earlier and you said, oh, I have all of these, that's okay. But you should take note that if everything is important, then nothing is. And you're just going to push yourself to analysis paralysis instead of focusing on the thing that is most important for you and that will unblock your ML workflow most. And so if you're not sure what is most important, it either means that you've already gone down the feature route and you're trying to do the first pit haul, which is post hoc rationalization, or you need to adopt more of a product manager mentality here and correctly identify your needs. So ask yourself, what am I most worried about that will happen within my project? Right? Is it which one of these problems that I described earlier? Right? Or what is the biggest recurring failure that we have in this realm within the company? Or something like that, right? So once you do that, you know, or you're going to have a better way of describing the problem that you have. And that's when you graduate to step two, which is, we defined our problem. And now you're going to be tempted to say, well, we need to get a flexible tool that supports data versioning so that we can revert in case of an emergency. But I'm going to say that that is also not what you want to do because that's still significantly more likely to get you into analysis paralysis. And so what I actually recommend instead is that you say, we need a tool for tabular Data that supports AD only changes, works with MySQL databases, and can support 100 gigabyte to 10 terabyte of data volumes, and we're only going to need to revert every few weeks. What I mean by this is that the next step of the process is outlining the problem parameters. In other words, we want to dive into the details of the challenge we're trying to solve and lay out the criteria and requirements from a solution. This translates in practice into something very simple, which is asking ourselves a few questions, as I mentioned before, like, what are your problems? Sorry, what are your problem specifics? So in the case of data versioning, are you working on computer vision and your data is videos or images? Is your data tabular? Do you expect the data to be immutable? That is, I Have a data set, it never changes, it's additive over time. So the actual data that I do have never changes, but I add more data or could it change retroactively? For example, I have an image and actual pixels in that image might change. How much data are we talking about? And if the answer is three, three orders of magnitude more than you actually have right now, that's not a good answer because you're not Amazon, right? So if you have 1,000 images and a gigabyte of data, don't look for a solution that works for billion images and petabytes of data. Right. The second type of question which I would argue is the most important to talk about is organizational constraints. So for example, is your company using a specific database that you have to use? It's not up to you, it's someone else decided and you're stuck with it. What other tools are already used in the organization and the solution must communicate with. Now these are most important because many times you can't really dictate them and they do dictate limitations on your decisions. So you want to make sure that you pay attention to these and not choose a tool that sounds really nice on paper, but then you can't use because the company won't like. The third type of questions is questions about yourself. And I don't mean this in a philosophical sense. Maybe you are a shell scripting wizard, maybe you're a sysadmin with little to no machine learning experience. And this isn't necessarily referring to a specific person. So you or your team members, it can be the average person who's going to use these tools, right? Or the company culture itself. This might add a lot of counterintuitive considerations to your choice of tooling. But the probably the most recurring place we see this is that some companies lean towards more analyst type data scientists and others more towards software engineering type data scientists. So the familiarity with more coding type and DevOps type tools versus more analytic and databases and things like that, that might really impact the type of tool you choose for the same problem that you're trying to solve. And the last thing. Oh, sorry. And another tool that I think is really useful for this is to define problem parameters through a user story. You imagine yourself or someone else within the organization and you tell a story about that person that has a workflow which includes the problem that you're trying to solve. For example, as a data scientist, I want versions of the data that I'm using to train models so that I can debug problems that occur in production. Are we going to version the database directly or just the outputs of queries used in training runs? Ideally, I think we want to write this in a document and share it with our team. I know there's a lot of recommendations to write things down and people take it seriously depending on their own personalities, but I think that this process, especially if you're doing this within a company and there are more people that are involved in the decision making process, needs to be written down. So you create a notion or a Google Doc that says here is our problem and here are unique problem parameters and, and then you share it with other people in the team, whether it's your team lead, your director, other data scientists that have strong opinions on these things. And you share this because you want to make sure that your assumptions aren't wrong. So maybe you think you have 1000 images, but actually the data scientist next to you is working with 50,000 and you just didn't know that, right? This does not mean that every tool decision needs to be a design by committee where everyone needs to unanimously decide on the same tool before you move forward. And one of the downsides of doing design by committee is that everyone has their list of features and maybe they didn't listen to this talk so they're not problem focused. And so you get a list of features like make sure that the tool can do this and this and this and this. Then you need to tell them like, no, wait, let's talk about the assumptions of the problems first and then we can write a list of things that we think are necessary for those problems and, and check back that those features are actually related to the problem that we defined as the one thing that we want to solve right now. So that's sort of step two and we're now mid process. I'll check for a second to see if there's any questions that make sense to answer right now. Ok, this is a question for later.

Oh, so maybe this one is sort of relevant. So Walt is asking, is there a way to combine Agile techniques and mlops? So I guess more generally you talk about asking questions about like what do you really need to solve? Like what's the cadence for that? How often do you need to ask these sorts of questions?

So I think that this is, there are two, let's say levels that we need to answer this in because there's agile in the sense of we're doing data science and machine learning, but we want to make sure that we're getting into a cadence where we can deliver new things not once every three months, but closer to what software engineering teams do, which is usually, at least in our company, for example, it's every two weeks. That's sort of the sprint period. So that can be something that you want to do. And mlops is actually a different layer which is, that is the enablement layer for agile. So basically the idea is if you decide that the timeframe that you want to deliver new capabilities on is, let's say two weeks, which my personal opinion is that for data science that doesn't always make sense. You do need to define which parts of the data science process you can do in week long sprints and which are more research oriented, in which case you might need more time and how you actually do checkpoints in that research process in a way that on one side psychologically doesn't make you feel bad about yourself because you're not getting new results every other week, but on the other hand does let the management level understand what's going on and make sure that the direction is the correct direction. And so mlops, the talk that I'm giving right now is actually the talk that you get to when you say, okay, we want to deliver a new experiment or a new. Or let's say I'll actually give a real example. So I was speaking to a data science team lead and he was saying every two weeks we meet and we discuss 10 experiments that we want to run and then we choose the top three that we think are most likely to move the needle on the things that our organization cares about. And, and then we try to run those three experiments in two weeks. We don't always succeed, but that's sort of where we're aiming for. That's their version of agile. And one thing that you might realize is you do this two week sprint, right? You chose the three experiments, it can be two, it could be four depending on your team. And then you realize, oh, wow, two weeks have gone by and we've done zero experiments or we've done three experiments but all of them have failed. Right? And that's maybe strange, maybe that's not what we expected. So maybe the reason for that, for example, is that our infrastructure is not stable enough and half of our training runs are crashing in the middle of training and that's why we couldn't finish any training run. So then you go to the MLOps layer and say, okay, we need to improve our training stability. This is a problem definition, right? So then we get to step one that I talked about earlier and Then you go through this process and you solve it and ideally you're now more able to execute on Agile. I guess I did answer your question on two different layers. I gave what I think is a good example of how to do agile in the practical sense and where I think envelopes fits into, let's call it Agile enablement within machine learning projects. Okay.

So it's about separating out the day to day work of running experiments and then sort of planning on how do you improve your operational performance longer term. So maybe though these sort of deeper questions about how do we do things better? That's like a once a quarter you're like, oh well, what can we do to improve stuff?

Yeah, yeah, so that. Exactly. So you can choose the cadence. It is very important to retrospect and that's sort of what's the instigating event for running through this process. You say we didn't meet our goals or we want to, or we did meet our goals, we want to go for more in the next quarter. How are we going to do that? We need to improve.

Improve.

Like our data is not good enough, our training is not good enough, our deployment process is not good enough. And then this is how you go through like platform, MLOps platform improvements as opposed to. Again, if I go back to my original point, which I'm going to make a lot in this talk, as opposed to saying like, well, I heard every mlops team needs an orchestration solution and a feature store. So that's what we're going to buy. Right. This is how I recommend personally approaching this.

All right, super. I think we can get back to the general flow of the slides. Now then, for anyone in the audience, if you've got more questions, please do drop them in the chat.

I'll answer more questions in the end. Don't worry, I won't miss any question that appears there. Now that we've gone through all that trouble, the next step is let's roll up our sleeves and build the tool we need from scratch. We just open up VIM or Emacs and we hack away. You got this right? And of course I'm joking, but the point I'm trying to make, and this is a serious point, is that as engineers, we do lean or sort of people with engineering mentality, we like building things and that's great to create something cool. It's even great, it's even better to see other people use the things that we build. But in most cases, reinventing the wheel is not the right approach. And so the actual Next step is to Google or I recently added this to GPT the problem. And this may sound trivial, but I think you'd be surprised by how many people spend five minutes doing research on existing solutions before deciding that they need to build something on their own, even if that's something that's going to take months to develop as an internal tool. And so let's go over a few tips on how to make this an effective process. I hope this is going to be useful to you. I think that a lot of these were counterintuitive to me when we started doing them. But the first thing is that you want to be self aware about the need to budget a reasonable amount of time, at least several hours, to just do research about the existing tools. You limit the time for this step and it's important for both directions. So on one hand you want to make sure that you're not spending too little time and just like, oh yeah, we're going to build something ourselves. And on the other hand, you want to make sure you're not getting into analysis paralysis. And the first dividend from the two earlier stages of this process is that we now have a defined problem and parameters and we actually want to search for those. So we don't want to search for tools. Even if we think that we know which tool solves our problem. We actually want to go and say our problem is that we want to revert tabular data at this scale for this reason, for whatever disaster recovery. How do you do this? Figuring out how other people solved this problem in the past is going to be very, very useful for you. Now, once we do that, you want to make sure that you're building an info page to which you dump the conclusions from this research so that other people in your team can review it and add tools and ideas that they have and have sort of a documented discussion to remember things yourself and sort of take notes as you learn. Because many times the way you go into this process is very different from where you end up. So you might have preconceived notions of which tools you're going to need to use, but when you Google it, you figure out that people are actually solving the problems you have very differently than what you expected. Now, it is very likely that one of the things that you're going to learn is that you were searching for the wrong key keywords. And that's part of why we go through this step. You might have thought that the keywords you're looking for are feature store, but you realize that what you actually are looking for is sort of like a data catalog, I don't know, whatever. And if you read blogs and forum posts and other content pieces to see how other people solve these problems, you're going to figure out that there are terms that you weren't aware of or possibly weren't aware of that people are using. And then you want to search for those terms again to see what tools they bring up. And last but not least, I guess this could also be GPT today. But you ask friends and colleagues and random people on Reddit about your problem. You use this as a tool to discover keywords to search for. And in the world of machine learning, many communities and tools don't have a stack overflow presence. So going through this process on Reddit might actually make more sense. And here are a few examples that people created that I searched for at the time. So you want to be as specific as possible on Reddit because they're real people that are going to answer your questions. You can see that all of these have a lot of comments and that's awesome. So instead of asking them, which tool do you use for something, ask them, I have this problem and I'm looking for a solution. How do you solve this problem at your company or in your team? And again, the goal here is to get as many responses as possible and not as many tools as possible. And so if someone responds to you with, oh, you should just use tool X without giving any detail, ask follow up questions like, why do you use tool X? Where do you think it's great? And where do you think it's not so good? Because that's going to improve your understanding of where and when you want to use a specific tool. And then finally you want to make sure that some of your searches could be sort of X tool comparison, data versioning, tool comparison. We have blogs like that on Dags app that's going to lead you to blogs that compare solutions. But you want to make sure that you understand the incentives behind the blog and not just accept it blindly. Are they explained? Is this a commercial post for some tool, etc. So to give a few concrete examples, this is like sort of going down in quality ranking, I think of search terms. So you should ask the first question on Reddit and then Google can be better for the lower rankings, sorry, the lower examples here, but the idea is really search for the keywords that you're looking for and see what you get and not for specific tools. We created a short list of tools. Let's say we have 10 or three or whatever. What do we do next? The natural step would be oh, I'm going to do the hello World tutorials of all of these tools and try to find out which one of them is better. But I think that we can do better. Ideally this list is going to be prioritized, but if not, we should choose a tool from the list and go through the following structured process. So we do pre technical evaluation trying to gauge the quality of the tool before we even bother installing it. We make sure the operating principles of the tool make sense, we try some hello World examples, and then we stress test it. Another dividend from our earlier process is that if we define our problem parameters it this evaluation is going to help us cut down on the tools that we're evaluating before we get to the hello World example. So we spend much less time trying things out without any specific aim. Let me give a few tips on each one of these parts. Now we're in the evaluation process. The first thing is people usually go to the project's GitHub stars and I think that that's fine, but it's not the best indication. I actually like to look at latest commit time, latest release time, number of forks, PRs and issues are more relevant. And for issues, what's the speed of response to issues? NPR's not just a sheer number of them. If there are issues from six months ago that no one ever looked at, that's probably not a perfect sign. You want to make sure that the operating principles of the tools are explained and make sense to you. Now to give an example of I guess this will show you what the creators of this tool care about. So if we're looking at one tool for data versioning, maybe the operating principles are it uses git under the hood to manage the pointers, the data files that are stored on remote object storage. It supports unstructured data, but does that fit my needs? Maybe I need my SQL database to be version, so that's not going to be a good solution for me. I think people when you get to the Get Started tutorial, people have a tendency to focus on how easy it is to get started. But I would argue that the ease of the hello World tutorial does not really matter that much. If it is a good tool and it takes an hour to run as opposed to two lines of code, that is totally fine. The question is how good is it after it's set up? And again, the importance of defining the problem and parameters here is critical. If the problem is defined as reverting to an older version in case of disaster, then making sure it does that without adding a huge burden to day to day use is important. But if you do this rarely, then you probably don't mind if it takes you a day to revert in the worst case, because you only do that once every, I don't know, six months. And if you do this on a regular basis because you need to experiment with older experiments, then you want to make sure that it actually works very easily to do the version revert, even if it maybe adds a bit of a bigger burden to day to day use. The last thing is stress testing. You want to kick the tires of the tool that you're using. If you're doing computer vision, and let's say my data is in the order of magnitude of thousands of images, we want to try the tool that we're using with 10 times as much. That's it. If we have 1,000 images, let's try it with 10,000, just to make sure that when we grow, we're not going to break it immediately and it's going to fail us. You don't try three orders of magnitude because you're not Tesla. Maybe in another case, you want to make sure that it supports streaming data, because that's something that you want to do. Now, what if you don't have your data set yet? You know it's coming. The, the people are labeling it right now, you're going to get it in a month. In that case, you can write a script that creates random images, you can scrape them from Google, or just create noise images and then add them to the tool that we're using to version your data to make sure that it works properly. It can be again like simulated tabular data, if that's what you're using. And this can also give you a good idea of the resource cost of using this tool, if that's something that you care about. So we're at the final stage. Okay, we're ready to start integrating the new tool. And this part is probably more intuitive, but I want to touch on a few important notes before we finish. Now, you might be tempted to start by integrating the tool into all your projects, or maybe get bogged down in defining success criteria for a POC or something like that. But what I recommend as a rule of thumb is to start simple. So you choose one project, one user, one unit of whatever the integration requires, one data set, one model, and you'll learn a lot from this first round, and that will save a lot of time. Tiers and money compared to if you do this with all of your projects at once. Usually this is the part where people say you should define clear criteria for success so that you can measure the cost benefit of the integration. And I agree in theory, but in practice, we live in the real world and sometimes you don't know how to define a clear criteria for success. You know it when you see it sort of thing. Don't let this get you into analysis paralysis. If you can define a success criteria, that's awesome. Otherwise, my recommendation is to add a time constraint. Say we're going to use this in one project for a month and then regroup and decide whether or not we want to incorporate this into other projects. The last thing is it is very easy to not review your work. You try to evaluate the process or sorry, try to evaluate the process and see what you can do better in the future. What I mean by process here, I mean the process of choosing the tool, not the process of using it. Try to estimate and note bottlenecks that you might have introduced through this new integration. So maybe you added data versioning, but. But we now know that we can't version our data if it has more than a million samples. So we need to remember that if our data scales up to that magnitude, we need to look for a tool before we go over that and then it becomes unusable. And you should set the review meeting when you start this process of integrating it, not after, because then you won't do it. So to summarize, we reviewed a process that has 5, 5 steps to solve MLOps problems for first principles. For each step, I hopefully shared some of our learnings from performing this process with multiple MLOps tools and specific pitfalls that you might stumble on. This framework is more widely applicable than just MLOps, which means that even if you're not an MLOps professional just yet, you can still find this valuable and hopefully use this tomorrow when you go to your work. And by adding structure to a process that you all have probably done to a certain extent in the past, but have done this in a hand wavy way. I'm hoping that this is going to be valuable both in enabling, for example junior team members, you have to perform a process with more consistent results and also to limit the time that you spend on choosing tools so that you can avoid analysis paralysis, save time, money and many tiers. And and let's face it, we don't want to spend our time on evaluating tools. We want to build cool things with data and machine learning models. And that's what we want to. Or that's what I want to help you do with this framework. So just reminding you, this is the way to remember it, if you ever forget. And thank you for joining this talk. I hope it was useful. You can follow me on Twitter or LinkedIn. And now I'm going to just make the screen a bit bigger so I can see and let's see what questions came up.

All right, thank you, Dean. That was absolutely brilliant. It's a really nice sort of way of thinking about, like, well, how do you actually solve your problems? Think about what's the problem in the first place, rather than just diving straight into procurement and buying some tools. Now, before we go to audience questions, I just want to highlight. So if you've enjoyed this, we have got another mlops session next Tuesday. So recent Pope post a link in the chat so you can sign up and register for it. So we're doing a code along on Tuesday on running machine learning experiments in Python. If you enjoyed this, then I'm sure you'll enjoy that as well. And there is one question from the audience so far from Patrick. So Patrick was asking about OpenAI's recent announcement. Now, there's a bit off topic for today, but I am running a webinar next Wednesday where we're going to go through the OpenAI announcements in detail and talk about what this actually means for you in practice. So, again, I think Rhys can post a link in the chat if you want to register. Please do show up on Wednesday if that's of interest. Yeah, all right, I'll also.

I'm happy to give a few thoughts on this. I feel like let me stop sharing my screen and then I'll be able to see everything a bit bigger. So I think no ML talk can now go by without needing to talk about GPT. I think OpenAI is doing a lot of really amazing work and proving that even when you're big as they are, you can still innovate really quickly. I think there's a lot of things right now that are in flux both because in general, I think GPT and LLMs have introduced sort of joker into the system and people are still figuring out how to best use them. And second, because OpenAI is to going is really building something that from a quality perspective, I think is unparalleled. This might be a controversial statement, but is unparalleled and people are still figuring out what you can and can't do with it. I do plan on writing a blog about sort of in A sense, right. First principles approach to how to use GPT. Because I think a lot of things that happened in the past year are a result of the fact that if you think about ChatGPT for a moment, GPT3 existed long before ChatGPT. And the actual innovation in ChatGPT, arguably, in my mind, is not the model, but the interface. And OpenAI figured out that interacting with this model in a chat format and fine tuning it for that purpose provides a significantly superior experience. The images that you saw in the last slide and in the first slide, I created them with Dall E. I did not do hacky arcane prompt engineering. I just told GPT what I wanted and it created them. Right? So I think that that's really incredible. But the result of that has been that many of the use cases that we've seen in the past year have just been, let's do chat 4x. I want to chat with my docs, I want to chat with my lawyer, I want to chat with my doctor, and that's fine. But I think that there's a lot of potential, if you think about GPT as a sort of general purpose tool, there's a lot of potential for things that you can do with it that are not just chat. So even things like asking it if you have a chatbot. Right. Even taking another GPT model that supervises the first model that is actually conducting the chat and asking it after every question, is the chatbot stuck in a loop? And then if it is, then raise a trigger that actually directs a person to talk to a real human being or maybe rating these conversations so that you know whether or not there are particular versions of your models that are doing better than others. So those are capabilities that I have not seen as much and I think are really cool use cases. And so there's a lot of exploration going on. And so to your question, who asked this? Patrick, to your question, I think that the answer is yes. And I think that there are a lot of use cases that are currently underdeveloped because people are not taking a step back and saying, what is this tool capable of? That's not just simple chat. Hopefully that's useful.

That's amazing. It's like meta chatbots. You have layers of bots upon bots upon bots, just all the way down. Fantastic. We'll have to get you back doing a webinar on that at some point. All right, so back to MLOps. So I have a couple of questions for you. The first one is it feels like we talk about sort of slightly abstract problems in this. I'm just wondering if you have a sort of mental model for what are the like, top types of problem that you have to solve in MLOps? Like, how do you think about, like the different categories of problems that are being solved? So I know you use data versioning a lot in this webinar. Are there others like that?

So I think that the way I'm thinking about categories here, I actually have a different talk that I gave. I don't think I ever gave it at a conference, but I gave it in some smaller, let's say, events. I was talking about our vision for the mlops landscape. The way I think about this is that you can think about components in machine learning and then you can think about tasks that relate to each of these components. And then from there you have sort of problems that you need to solve and MLOPs need to solve. So the main components, I would argue, are data code experiments and models. Those are basically the four sort of cornerstones of an ML project. If you're working on an ML project, you probably have all four of them. There are exceptions, of course, but mostly that exists in all ML projects that you see. And then for each one of them, you have different tasks that you want to make sure you do. So I can open up the slide and maybe share that, or we can send this afterwards. But for example, in the data part, you want to make sure that your data is labeled. If you need to label it, you want to make sure that it's curated. So now everyone is talking about the only undeniable fact that is true for classical ML and also true for LLMs and everything is that they improve drastically with curated data sets. So dataset curation is a critical thing that you need to do. And then of course, you want to be able to sometimes refer to older versions so that you can do a better job of managing the exploration process through data. You want to be able to subset your data sets, which is sort of a different version of curation, in the sense that maybe you have the core model that is trained on a larger set of data. But then the classic example that everyone use, likes to use is autonomous driving. Now you want a subset of data to make sure that your model does not fail when someone walking with a bike is crossing the street. So you want to create subsets for the code part, let's say I usually tie in the code and the experiment together, because usually you run the code and you get the experiments as a result. So you want to make sure. That you're tracking your experiments so you know which hypotheses you've checked and which you haven't, Things like that. And so for each of these, you have different problems that can come up. Again, I'll share the slide of how we break this down and hopefully that will help people think about this. This is a good point from you, Richie. I guess next time I'll add this to the main talk slide so people can see it there as well.

All right, super. Thank you. And another question for me. I'm curious how you think about what happens when people, different levels of technical ability, need to use tools. I'm thinking about something like if you've got something, if you've got an experiment, then maybe a data scientist wants all the technical details and someone from the business side wants to know which version of the landing page is best.

Yeah. So this is a great point that comes up in a lot of conversations with companies. You know, even within the data science team, there might be varying levels of technical ability. Again, the example that I gave, data science is a much more varied field compared to software engineering. So you have people that study computer science, but you also have people that study economics, biology, whatever. Everyone comes with a different tool set and they are going to be very good at different things, and that's fine. I think part of the richness comes from. From the diversity that we have in this field. But that means that you need to pay attention. And this relates to the sort of who am I? Part that I mentioned in defining the problem parameters. Things that we do see that are, I think, generally applicable is if you have a wide variety of technical ability within your team, you want to make sure that the tooling that you use enables you to solve the problem. But then sometimes you want to actually invest a bit of time with writing a thin layer that makes the core functionality accessible to the people that might be less technical. So one example that we did at Dagshub is we offer, I think, very good components for managing notebooks, versioning diffing notebooks, which is obviously something that a lot of data scientists want. But some data scientists are not as familiar with git, and that's fine. We recommend that everyone learns to use Git because I think it's like a core of anything that has code in it. But some data scientists want an easier component to manage those notebooks. What we actually offer is within our client, which is arguably a layer on top of the core functionality. We have a save notebook function that you can call from Python you give it the repo name and then it just appears there. And we know to overwrite and create new versions of your notebook automatically. That's something that took us not that long to write. And you can write internally if you're developing something else or using a different platform. There's just a small recommendation and sort of example of how to do this.

Okay, so you got like a shell underneath and then a nice user interface for people who don't like that sort of thing. All right, excellent. Super. Okay, we had no more questions from the audience. We are past time now so I just want to say thank you again, Dean. That was really informative. Thank you to Reese for moderating. Thank you to those of you who asked a question. Thank you to everyone who showed up today. I hope to see you again next Tuesday or possibly Wednesday for either of those sessions that we link to. Yeah, see you all again soon. Thank you.

Thank you, Richie, for.