#67 Operationalizing Machine Learning with MLOps

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

Hello, this is Adele Neme from datacamp and welcome to Data Framed, a podcast covering all things data and its impact on organizations across the world. You know, one thing we've definitely seen the rise of over the past year is the pressing importance of operationalizing machine learning models in production. COVID 19 exposed us to concept or data drift which with many models breaking in production due to the evolving nature of data being generated after forced lockdowns and completely different habits that emerged because of it. As such, we've seen the explosion of new methodologies and ways to describe operationalizing, monitoring and extracting value from machine learning and production. Chief Amongst them is MLOps. This is why I'm so excited today to have Alessa Visnich on today's episode. Alessa is the CEO and co founder of YLabs, an AI observability company on a mission to build the interface between AI and human operators. Prior to Y Labs, Alessa was the CTO in Residence at the Allen Institute for AI, where she evaluated the commercial potential for the latest advancements in AI research. Earlier in her career, Alessa spent nine years at Amazon leading machine learning, adoption and tooling efforts. She was a founding member of Amazon's first machine learning center in Berlin, Germany. Alessa is also the founder of R Squared AI, a global community of 1,000 AI practitioners who are committed to making AI technology robust and responsible for all. In this episode, Alessa and I talk about the machine learning challenges data teams face that spurred the need for MLOps, how MLOps intersects and differs from other terms such as DataOps Model Ops and AIOps, how and when organizations should get started with their MLOps journey, and what the most important components of MLOps are, and more. If you want to check out previous episodes of the podcast in show notes, make sure to go to datacamp.com community podcast Alessa, it's great to have you on the show.

Absolutely. I'm really excited to be on the show as well. Adele, thanks for having me.

I'm really excited to talk to you about MLOps, how organizations can get started with it, your experience launching yLabs and more. But before, can you give us a brief introduction about your background and how you got into the data space?

So my journey began at the University of Washington where I studied applied Math. I graduated with with a degree in 2008, which was really funny time, a lot of unknowns similar to now maybe. And I was really keen on going into grad school and getting a PhD, but decided to check out industry before I dive into that because of uncertain times. So I joined Amazon, which in 2008 was a pretty not well known company. In fact, my grandma commented that I went to work at a bookstore that doesn't even have a store. But the cool thing was that it was at a very interesting time. So the company was growing rapidly, the technology was growing rapidly. So I was very fortunate to join Amazon at the point where the whole DevOps culture was just exploding in the company. And I worked at a group called Platform Excellence where I ensured that the website is running fast, doesn't have anomalies, doesn't have issues. And halfway through kind of my career there, I was always looking for opportunities to get back to my applied math roots. And in 2013 when Amazon started kind of their first R and D machine learning R and D effort, so they started creating R and D teams internal in business units. I joined one of them and I moved to Berlin, Germany to kind of help seed the group, help start the group. The group was focused on deploying machine learning applications internally in the company. So there I was very fortunate to kind of end up in the right place at the right time. There were a lot of brilliant machine learning scientists that joined the group and we were tasked with building and deploying machine learning applications into supply chain personalization, into warehouses and so on. So I had the opportunity to essentially become the bridge between machine learning, engineering and the business. And in that role because of my kind of DevOps experience prior to that I also was the brave person who would often carry a pager supporting the machine learning applications that we deploy. So I had the pleasure and the adventure to kind of feel what it's like to run machine learning and production very early on for really big deployments. So the biggest deployment that I've ever operated was a demand forecasting pipeline that generated daily forecasts for a quarter of all Amazon global retail, which was mind boggling, like we were creating a forecast for hundreds of millions of products. On Amazon and supporting that in production was, was a big endeavor. So that's kind of how, how I got into, into the space.

I'm excited to discuss all the latest on MLOps with you. So while MLOps is a relatively nascent term, it has been a lot of the focus for many organizations today. And we're currently in what can be described as a hype phase in MLOps. I want us to discuss what MLOps is and isn't. But first, what I would like to do is set the stage and articulate the motivation for mlops and why it's important. So you're someone who's been in the machine learning space for a long time now. Can you walk us through the specific challenges data teams face when deploying machine learning models at scale that really spurred the need for mlops?

Yeah, that's a, that's a fantastic gateway into the subject. So first, kind of, to set the stage, I kind of want to take a step back and think about where the term came from. And unsurprisingly, it mirrors the DevOps culture that we have in traditional software. And really what DevOps is all about is continuous integration, continuous delivery, continuous deployment. And when I say these terms, what stands out is continuous. The term continuous. And here what it really means is that we want software to be a continuous experience to the customers, continuous and consistent experience, which seems very simple and kind of a basic definition of software. Right. But in the machine learning world, that is just becoming established. So in the machine learning world, when we look at machine learning models in academia, typically we take a data set and then we iterate on it to come up with better and better models. Where in the industry you have a model, it's deployed in production, and it's being hit with newer and newer data every day, and it needs to provide a continuous, consistent experience to the users. So really what MLAPS is about is tooling culture. Processes that help ensure that the machine learning models continuously deliver the experience that they were designed to deliver.

Okay, that's great. So over the past few years, there have been quite a few different terms that try to capture the operational side of data and machine learning work. Terms such as DataOps, AIOps, and now MLOps. Do you mind walking us through how these terms differ and intersect?

Yes, our industry is drowning in new terms. And to answer your question, let's first focus on the difference between AIOps and MLOps. That's the easiest one to define. So AIOps is a term that is being used to describe AI and machine learning techniques. That are applied in common IT issues. So say automating anomaly detection and clickstream data, or automating kind of monitoring and aggregation of alerts in the very traditional sense. So companies like datadog and Splunk are applying AI and machine learning to the data that the log data, the machine log data that they're collecting to identify and anomalies MLOps. Even though the difference is only AI and ML, and I do not know the history of how these terms came to be so similar, yet so different. MLOps is a set of techniques, and not necessarily techniques that are related to AI and machine learning, but techniques for operationalizing the machine learning and AI applications. So these techniques could be, you know, data versioning, reproducible data pipelines, logging and monitoring for data, and so on. So AIOps and MLOps are very different things. Now if we talk about DataOps versus ML Ops, the difference is a little bit less straightforward. So DataOps typically focuses on data pipelines, again, data versioning, data warehousing, and kind of Data Robustness, while MLOps focuses on machine learning pipelines and machine learning applications. Now the challenge is that machine learning pipelines are basically data pipelines. So where DataOps begins and ends and where MLOps begins and ends is still a little bit gray area, I would say.

Okay, awesome. And in summary, and please correct me if I'm wrong, mlops can be considered as a set of tools, practices, techniques that ensure reliable and scalable deployment of machine learning Systems, similar to DevOps in the software engineering space. Is that correct?

Yeah, I think that's a great definition. I would also add culture to that. So tools, practices, techniques and mindset or culture.

Yeah, I'm definitely excited to talk about culture with you. As we all know, that's what really differentiates an organization that makes the most of its data from the ones that don't. So I want to zero in on some of the best practices in MLOPs that are still emerging. What do you think are some of the principles, patterns, or key focus areas that make up a successful mlops practice?

Yeah, well, so I would say going back to kind of the term or pattern that I highlighted previously from the DevOps practice. So the concept of continuous, continuous delivery, continuous deployment, continuous integration, that pattern basically asks for an application to be consistent, to be reproducible, to be robust. So day in and day out, the experiences that the machine learning model is generating for the customers need to be continuous and consistent. And that's fairly challenging to do in the machine learning world for a few reasons. I think the biggest Reason is, you know, machine learning applications, unlike traditional software, they're not heuristical. Traditional software essentially is a set of rules that we produce. And you know, when customer comes in there, they satisfy one of these rules and they get some kind of experience. With machine learning, the rules are generated by the data that the machine learning model has been trained on. And so it's rather than having rules, the machine learning models are essentially building the rules by looking at the data. So that creates a few interesting challenges. First of all, the data really sways the experience. So the machine learning model behavior would be different even if no code changes have been made, even if no config changes, no deployments have been made. If the data that comes in as an input to the model is different, if the patterns are different, then the experience would be different. One of my favorite examples of that that I've heard recently is speech recognition software that is deployed out in the wild in the physical world, that recognizes certain words or phrases that are, that are spoken. When Covid started and everybody started wearing masks, the software started working much less accurately, much more poorly than before because the MAS kind of covers your mouth, which makes your voice come out not as clear, which makes the microphone pick your voice up not as clearly. And that essentially breaks the model. So that's a very concrete and very unique behavior to machine learning applications. So the data that comes in into the application sways the model. And I think that's something to keep in mind when you're defining those practices for successful machine learning operations. I would say in summary, kind of high level patterns that emerge from the desire for continuous behavior are reproducibility, which is a very kind of common and important concept and pattern to machine learning applications. So reproducibility means I ran an experiment on some data on my local machine or on, on the development environment. I have this really cool model. It's better than my previous model. Now I want to reproduce it in a different environment. How do I do this? Or I have this model running today exerting certain behavior. How do I make sure that it's exerting similar behavior tomorrow? That's reproducibility. That's one important pattern. Then robustness comes to mind. Robustness essentially means my model is in production, it's generating predictions. But you know, 30% of my users are unhappy with the predictions. Does it mean that my model is failing? What does failing mean for my model? Where, what are the failure modes and what does failure even mean in the probabilistic system? So answering those Questions is kind of the realm of the robustness pattern and then finally transparency. So you have a data pipeline, it's crunching gigabytes or terabytes of data. The predictions are coming out. How do you investigate undesirable model behavior? How do you explain the predictions? How do you debug the predictions that are being made? Transparency is a pattern that kind of captures some of these questions, tools and culture and mechanisms. So reproducibility, robustness, and transparency. I would highlight as three key patterns.

That's awesome. So we've seen in the past few years this shift from experimentation to operationalization. What do you think are the main roles that are responsible for operationalization? Is it the same data scientists who develop the model? Is it engineers, hybrid roles like machine learning engineers, or even specialist roles like MLOps engineers?

Yes, actually there is a very recent development of a brand new role that I think has been defined in the last three, four, maybe six months of an mlops engineer. So I like to call everybody involved in building and operating machine learning and AI applications. I like to call collectively all of those people AI builders. And I think among them are many different roles. So it starts with obviously, you know, the researchers that are building the models, but also the people with subject matter expertise that are defining the scope of the problem. So typically, you know, if we're talking about a personalization model, then it's often the marketing team. If we're talking about a supply chain application, AI application, it's probably like in stock managers. So it starts with the data scientists and the subject matter expert. And as the development of the model, the lifecycle of the model goes on. Typically there is, you know, a PM that's involved. There's an engineer or an engineering team that is involved. There's probably a QA team that's involved. And eventually I think the ownership of the model, once it's built and deployed and is in production, I think historically it's been owned by the team that built it. So a mixed team of data scientists and engineers. I think today we see an emergence of an actual MLOps engineer role. And there also are ML engineers that kind of come into the picture that typically sort of deal with data pipelines and actually the productionization step of the model. And today I see a huge emergence of the MLOps engineers. In fact, yesterday I was doing a little prep for, for the podcast and I went on indeed. And I searched for MLOps. I think it's just a, a fun thing that I do occasionally to see what's happening in the job market. And I found that there as a huge number of companies who are trying to hire somebody with MLOps experience. Companies like Peloton and McDonald's, traditional companies that are not even kind of well known for AI, but they do deploy AI and machine learning internally. So they're looking for somebody of that profile to essentially maintain machine learning models once there are in production. So I think that's, that is the new emerging role that, that we're starting to see.

So do you think in that sense that data scientists are geared towards becoming more and more specialized?

Ooh, I don't know if we can say with certainty yet. So I've seen both organizational structures be successful. So Eric Kolson from Stitch Fix is one of the thought leaders in the discipline of full stack data scientists or full stack ML practitioners, which basically means just like full stack engineer, the data scientist is able to build a model, deploy the model, operate the model and kind of own the entire cycle. I think that is still the case in many organizations. But at the same time we do see this parallel of very specialized roles. Data scientists who are machine learning scientists who actually develop novel techniques, data scientists who develop novel techniques with respect to data. Then we see machine learning engineers who typically productionalize or scale out the machine learning application. And now we see MLOps engineers. So there is a sign that kind of both of these, I guess, organizational structures have a place to be. And I would say we're yet to decide which one is better or more efficient or maybe both have space to be.

So in that sense, given that MLOps is still fairly emerging, what do you think is still needed till we have this massive wide adoption of MLOps as a practice practice in organizations today?

Yeah, I think we are still ways away. One of the biggest barriers to adoption that I see is culture and kind of understanding of the need for MLOPs, understanding of the need for tools and practices and processes and culture for operationalizing this technology. Typically what I still see talking to various data science teams is their thinking kind of ends at production. So they're running this marathon. And it starts with getting data, capturing the problem, building the model, iterating, experimenting, production lies. And it kind of ends when the model is in production. So that is, I would say kind of the most common thinking that we have been seeing in organization. And mlops is the movement that essentially says, look, you have to think about what happens to your model post production because that is pretty much the most important step. Once the model is in production, that's when it's delivering Value or not delivering value, that's when it's delivering positive customer experience or negative customer experience. So thinking about what happens post production is probably even more important than other previous steps. So I would say kind of realizing that this is important is one of the biggest barriers. Creating this culture, helping the businesses understand that there's need for that. Because a lot of organizations are still trying to adopt machine learning, trying to launch their first machine learning models, and they're spending a lot of resources on that, hiring specialized data scientists and machine learning scientists. And they're really excited to get to their first model. But then oftentimes they're surprised that there's still more resources and still more work needed. So I think changing that mindset is very key. And then I think as far as MLOps from the technology perspective, as a software category goes, I think we still are figuring out a few things. On the technology side, we're figuring out what does explainability mean for operating models, how do we take advantage of some of the explainability techniques that are coming from academia, from research to help the operations side. Then fairness is another fairness and bias is another big space that is still a massive area of research and I would say not inconclusiveness, but there's some metric then that we agree upon, some metrics that we don't agree upon in the fairness and bias space. So we're still figuring out what does that mean and how does fairness and bias kind of manifest itself in the MLOps tooling and MLOps process? And then of course, causality. So majority of machine learning systems are not causal. What does that mean for operations? Because when we think about software, software operations, we always think about cause and effect. With machine learning systems being not causal, like the root cause analysis becomes a little tricky and so on. So explainability, fairness and causality I think are still creating some challenges on the technology side of what do these tools look like and how do we ensure that these tools kind of service right when it comes to understanding the behavior of machine learning models.

That's really insightful. Thanks for that. So you mentioned here the tooling stack, you know, on the analysis side, it seems that the data science toolkit has standardized around open source programming languages and tools like Python's, Pandas, scikit Learn, and so on and so forth. When do you think we'll reach a standardization in the tooling stack in the MLOps space?

You know, I'm not sure if there is like an end period to a canonical stack. I think we are starting to move towards kind of Groups and organizations that are beginning to formulate their thoughts about what a canonical stack would look like. One of such organizations, which I am personally part of and I'm a big fan of, is an organization called AI Infrastructure alliance, short for ii. And that is the group of startups actually that are building an infrastructure alliance or building the canonical stack for machine learning applications. So not a canonical stack necessarily for just MLOps, but a canonical stack for building, developing, deploying and operating machine learning applications. So that's a fantastic move and I'm really excited to see what the stack is going to be like and, and not a spoiler alert because you can, you can go to the AI Infrastructure alliance website and kind of see how, how the alliance is approaching the problem. I would say it's not going to be a prescriptive recipe of, you know, you take these 10 tools and you add them into your process and off you go. It's very use case specific. It's specific to where the organization is at and their journey of adopting machine learning and AI. It's specific to what scale are you running things and so on. But I think we're seeing some positive signs that there is an emergence of a set of tools or set of, I would just say set of practices that are widely accepted as necessary practices to have in your organization in order to implement machine learning operations.

Well, that's exciting. So if you had a blueprint to provide organizations on how and when to get started, when do you think organizations should get started with their MLOps journey? And if you had a blueprint, what would that blueprint look like?

Ooh, as early as possible at the same time as they start adopting and building machine learning organizations and adopting machine learning technology. So I would start with defining a few themes and again going back to DevOps, because there's really no reason to reinvent, reinvent those wheels. I would say the important themes to think about, as I mentioned, reproducibility, transparency, robustness. I would also add the culture of quality and ownership to that. So those are kind of the five things to keep in mind when you are starting to develop machine learning applications. Those are the questions that the organizations should start asking themselves as quickly as possible. And then from there I would say a great process, lightweight process for an organization to figure out their own blueprint is to start with their DevOps team to figure out what are all of the activities, processes, mechanisms that this organization already implements for traditional software, and then extend their thinking to include data and to include machine learning. Non deterministic probabilistic applications. So that means that you should be doing everything that you're doing for traditional software for the DevOps side of the traditional software. And then you should have processes and tools that are specific to understanding the data, processes and tools that are specific to understanding the model. And that's not a blueprint per se, but if you go through this process, that would help you develop the very specialized blueprint to your organization and to your specific use case.

Now, circling back to the people element of this, how important do you view the role of data culture and data literacy when fostering an environment to scaling mlops, but also of solving the core issues with deploying machine learning models at scale?

I think that data culture and data literacy are absolutely key to creating the right environment for creating the environment that is ready for, I would say, responsible machine learning adoption. And the reason I say that is because it's easy when we say machine learning, it's easy to just think and focus on the algorithms and kind of take data for granted. And that's very dangerous because as we know and as we're seeing kind of machine learning failures that make the news, majority of unfortunate experiences that stem from machine learning come from poorly collected data for training poorly instrumented machine learning predictions that kind of allow unfortunate experiences to bubble all the way up to the customer. To be more specific, I would say data hides biases, data hides bugs, data hides machine learning failures. And if we are not thinking about the data, how this data was collected, who brought that data in, who is the data provider, who cleaned that data? Do we have an understanding of what segment of our customers does this data describe? Does this data describe all of our customers holistically or not? Does this data have personally identifiable information or not? And so on without having that culture and understanding of how valuable but also dangerous the data that you're handling could be if you handle it incorrectly. I think machine learning adoption wouldn't be successful in a company. And to that, to that note, I'm really excited to hear how Andrew Ng is approaching this whole concept of data culture. He's really outspoken on the subject. A lot of his talks focus on encouraging data scientists to think about their data first and foremost and models being kind of a secondary vehicle for working with the data. And then a good friend of mine, Joe Rice from Ternary Data, runs a podcast as well and is a great, well, outspoken individual who comes from data engineering side and kind of merging his experience with the machine learning culture brings on a lot of interesting insights where that capture how culture, or lack of culture around data can affect machine learning adoption. So I recommend listeners to go and look up. Well I'm sure Andrew Ng is really well known, so looking up some of his latest talks has a lot of interesting information and insight on data culture. And then Joe Rice from Ternary Data has a lot of interesting content on data culture and data literacy.

100% and we'll definitely include those in the show notes. Now I want to pivot to talk about YLabs and how it fits into the MLOps tooling stack. But before, how would you describe the current tooling space in mlops?

Well, I would say the tooling space is emerging rapidly. So I can see two categories of tools. One category is cloud providers who offer kind of machine learning building blocks, so everything from data streaming to data warehousing to scaling out the jupyter notebooks because that's still something that scientists use fairly often to build models and so on. So cloud providers are starting to also add tools that help you with MLOps. So tools for testing, versioning, monitoring your machine learning applications and data pipelines. So that's one category. And the second category, a bunch of startups that are emerging with purpose built tools for machine learning for operations of machine learning applications. So no matter where you are in your journey as an organization, yes, there are two big options to look at both the cloud providers and the startups. And there's definitely no shortage of options to try and to consider when you are kind of seeding the tooling of your organization for MLOps.

Can you walk us through YLABS and its mission and how it solves some of the pain points of deploying ML models at scale?

Sure. So ylabs is an AI observability company and our mission is to build an interface between AI applications and human operators. So what we help AI builders with is to ensure that their models are making an impact that they were built to make. And what I mean by that specifically is we have an AI observability platform that helps ensure that the models do not fail unexpectedly. That helps ensure that the models are fair to the maximum extent we can calculate that to help models realize ROI and to ensure continuous and automated delivery.

That's great. Do you mind walking us through some of the specific features that data teams can leverage today?

Absolutely. And actually what I would do is I would focus on the open source library that we've end that has been really popular in the community. The library is called Y Logs and what Y logs is Y logs is a standard for data logging which is very foundational to the entire mlops practice. So in traditional software we capture a lot of information about about how the software is running in the logs. When it comes to machine learning, you can capture a lot of information in the traditional logs. However, when it comes to data that flows through the pipelines and gets transformed along each step, there is traditionally no way of capturing what does that data look like and how does it change from step to step. So Y Logs helps you do precisely that. It's a purpose machine learning logging library that is open sourced by our team at YLabs and it's built to provide very lightweight, portable and configurable way of logging statistical properties of data and model predictions in both batch and streaming data workloads. So what Y Logs enables anybody to do, since it's an open source library, is to capture statistical properties of data and then build testing, monitoring and alerting on top of that to identify things like data drift, data bias, data outliers, and then identify degradations and model performance.

That is super exciting. Finally, Elissa, is there any final call to action you'd like to make before we wrap up?

Yes. So first, I'm really excited to be part of the MLOps conversation. I think this space is rapidly emerging. So you listener, if your organization is thinking about going and establishing an MLOps culture, would love to talk to you and every AI practitioner who is tuning in. As we at Y Labs are building the Y Logs standard for data logging, we'd love your feedback and your contributions. It takes a village and it takes everybody to define a standard for something so canonical as data logging. So I'll be sharing the link to GitHub. Would love feedback, contributions, issues, feature requests, GitHub stars, and so on. Join the movement that we started for building the standard for data logging and MLOps.

That's awesome. Alessa. Thank you so much for coming on today's podcast.

Thank you Adele. Really great to be here.

That's it for today's episode of Data Framed. Thanks for being with us. I really enjoyed the Alessa's insights on MLOps and how data science is evolving to meet the needs of organizations. If you enjoyed today's episode, make sure to leave a review on itunes. Our next episode will be with Maria Luciana Accente, head of responsible AI at PwC. I hope it'll be useful for you and we hope to catch you next time on Data Framed.