How To Evaluate MLOps Tools

Okay, so today I'm going to be talking about evaluating MLOps tools, but just a little bit about myself. So as Chip mentioned, you know, I currently work at a startup, but my general background is I've spent a lot of time building ML tooling. So I've done work on this at companies like DataRobot. I've been involved a lot with fastai. But also been building ML tooling at Airbnb and GitHub. I've been involved with open source as well. And I've worked on both what, quote, cool applied ML problems at Airbnb and GitHub. And then also like a lot of uncool applied ML as part of like when I used to be a consultant in things like retail, telecommunications, restaurants, things like that. So I've seen all kinds of different data science kind of flavors, and that really informs my views on tools. OK, so this talk is about how to evaluate MLOps tools. So I just want to share the motivation for this talk, like why am I even giving this talk. So actually, recently, there's a person who's an ML educator, author, very respected in the ML community. Came to me and said, hey, you know, like Hamel Husain Husain, people don't use PyTorch for production. Like, come on, no real applied ML occurs in PyTorch. They don't have like good tools. And then also, you know, kind of on and on and kind of, you know, a little bit of a provocative, in a provocative sense. And also like things like, hey, like certain problems are completely solved in and take TFX. And I thought this was really interesting and surprising, and it made me realize that this actually can happen. This actually is very common. If it happened to me, I'm an experienced person who has built ML tools, and I'm kind of— it's kind of known that I've done a lot of work with PyTorch or with fastai. That someone would come to me and kind of say that, it made me realize that this is happening to everyone all the time because this actually happens. Yeah, this happens to me quite frequently, and when I talk to other people, it happens to them also. And I think as you as a student taking this ML systems course, I think like one of the things that you have to think about is tooling. I mean, that's where the rubber meets the road. That's an important consideration when you go out into the world. And so you're gonna encounter people who are very passionate about certain tools, and, you know, what I would call zealots even. And I think it's kind of actually natural. It's not necessarily— I think it's human nature when there's a lot of entropy in a field. So this picture on the left is kind of this map of all MLOps tooling, and it's so gigantic you can't even see it without a microscope. And I think it's natural human nature to say— to try to make sense of that and say— and want to kind of cut the noise and say, well, I found one tool that is the best and all the other tools are wrong. That's kind of a very appealing thing to believe in. And it's something that, you know, it will kind of— you will encounter in real life. And it's actually a very difficult thing to resist, especially when you're first starting out. And it's kind of like, especially when there's appeal to authority, like, hey, this certain tool is used at Google or Facebook or Netflix or wherever it might be. And I think as someone starting out, it's good to have some framework or some example of like how to think about tools. So this is my criterion for evaluating tools. I'm not going to even read it. Just know that there is some criterion for evaluating tools. You can revisit the slides later. I don't want to read this, but I will touch on all of these during the presentation. Okay, so just coming back to TFX, TensorFlow Extended, and so like what I want to do is go through a concrete example with TFX, using TFX, just because I think that it's really hard to learn or gain an intuition of how to evaluate MLOps tools in the abstract. If I just gave you a list of principles and rules, it would be kind of hard to apply them without perhaps listening to someone else apply them, or at least have like some concrete sort of example of someone walking through that. So I'm going to be kind of focusing on TensorFlow Extended today. And again, it's not that I hate TensorFlow Extended or anything like that. It's just I think it's a really good motivating example for this general problem I described. So first, I'm going to talk really quickly about things I like about TFX. I'm not going to spend actually too much time about— on the things I like. I'm going to spend more time on the things I'm critical on because I think that's where it's more interesting. So just really quickly, so TensorFlow Extended has something called TensorFlow Data Validation, and it's very decoupled from TensorFlow. You can pass in a DataFrame and you get all— you get very nice EDA, for example, in a notebook, and it does a lot of boilerplate exploratory data analysis for you, and it's pretty nice. You don't even— like I say, you don't have to use TensorFlow, and just 2 lines of code It's very nice. And similarly, the same— similar library helps you validate statistics, validate schema. So you can quickly check the schema of your data, do some like minimal testing of your data at least, and it's very lightweight. Another thing I like about things under the TFX umbrella is TensorBoard. It's really nice. You can use it with any framework. People really like it generally, and it's nice and it's easy to use, it's well documented. And then also TF Serving, that's another thing under the TFX umbrella that I really like. It's very well documented. This helps you serve models in a nice, easy to understand way. It abstracts things very nicely, and it's also well documented and has really nice API— well-designed, thoughtful APIs. So things I don't like about TFX is everything else, and I'll talk more about that. So, so the first thing that I want to mention is tools need to get out of your way, especially in critical parts of the workflow. So one critical part of the workflow is iterating on data. So this is Andrew Ng, and as you know, Andrew Ng is a proponent of the data-centric modality of iterating on problems. I know that Chip actually touched on this earlier in the course. Not to say that data-centric modality is— or this data-centric view is the right view, but I do think that either way iterating on your data is very critical. And so kind of looking at TFX, what we see is They have something called TFX Transform, which is the library used to do feature engineering and transforming your data. The problem with this API is you have to use a limited, limited set of TensorFlow ops. So if you want to do simple mathematical operations like greater than, you can't really use plain Python code. You have to use you have to call a function, for example, that says tf.greater. And so this actually can slow down feature engineering quite significantly. This is because if you have plain Python code or using pandas, you have to refactor that. Or let's say if you're using spaCy to do some preprocessing or another library, you have to refactor all of that into TF ops. So it can be quite limiting. Another thing is if you want to iterate on data transformations, a lot of these things can't be performed interactively in TFX. So even the official tutorials, they write a Python file to disk and then execute that Python file via another thing that calls it. And iteration is really slow. So, you know, if you want to change some feature engineering in certain parts of the workflow, you have to— you can't, you can't do work interactively. And also the hello world paved paths kind of, you know, in the documentation when you're using TFX, they jump directly into distributed computing and things like Apache Beam. And Apache Beam is a very complex framework for distributed data processing, and it can actually be very difficult to reason about. And I'm a big proponent of what's in the documentation is, is the product. So while there might be another way to do things, if the documentation, you know, in their Hello World examples pushes you in one direction, then— and they don't really present other options very clearly, then it clearly is what the product provides. Another thing I want to mention is with regards to feature engineering and sort of understanding your data is it's important to visualize your data. I think we can all agree or probably understand that by now. But what you see here is Aniscom's Quartet. Basically all the data you see here has the same descriptive statistics. So same mean, variance, and correlation. And, you know, you need to visualize your data to really understand it. So TensorFlow TFX has a subproject called TensorFlow Model Analysis. And TensorFlow Model Analysis, the basic crux of it is you define these config files. So you define these JSON files that you see here, and that defines your visualizations. And again, it's very difficult to iterate. It's not interactive. If you want to make a change to a visualization, first of all, you have to learn this special DSL for visualizations, and it, it can very much slow you down. And so this is what the output looks like. It's its, you know, visualizations and it's of various kinds. But in the end of the day, it's not really anything terribly new with regards to visualization in Python. And this is a similar related library of TensorFlow Model Analysis. Again, you write these giant config files, JSON format, and you get, you know, you slice models and visualize them on various dimensions of your data. But again, it's very difficult to reason about, and it's a very steep learning curve. And so one thing I want to mention in this case, like Python has really great data visualization tools, things like Matplotlib, Bokeh, Seaborn, Altair. Altair happens to be my personal favorite. But kind of, you can't use this in, in TFX. You have to, you use, you can't use the tools that you're comfortable with. And that can be problematic. One thing I want to mention also is metrics on slices of your data is something that you can do with existing tools like pandas. And even if your data is at a larger scale, there are other things like pandas that offer consistent or similar APIs that are easier to reason about. And I'll discuss some of those later. Okay, so another thing I want to mention that you have to kind of really be wary of or kind of pay attention to is tool myopia can lead to blind spots with— when, you know, especially in applied machine learning. So when a tool offers a solution to a problem, can— and, you know, if you're, if you're really dedicated, if you're really in love with the tool, you might think that you have solved that problem. So one thing I want to mention is skew detection in TFX. So skew is when your training distribution differs from your serving distribution, or, you know, even like your training distributions differs from your holdout distribution or any partition in your data that you want to measure. So TFX comes bundled with some APIs for helping you detect skew. They offer these like categorical skew is measured with this L-infinity norm and numeric skew is with this Jensen-Shannon divergence. The details of that doesn't, doesn't matter. What I want to really highlight is it's not really clear in the documentation of how these are calculated and what summary statistics are included in the L-infinity norm, for example, but really like The main problem is it's not very actionable. So you might get a report, but you don't really know what to do next. You get a score of some kind, and something, something is above a certain threshold, but you don't really know— you're not really sure where it comes from or what to do next. And then another thing is this is done in a very univariate way. So it's one feature at a time in isolation. And so there's other things that can give you more visibility. So this is a screenshot of Great Expectations. Great Expectations offers you similar things, but it's a lot more human-readable and understandable. Another thing that I want to mention is skew detection. There are, there are more like practical approaches to skew detection that can be, you know, just as robust or even more robust. So this something called adversarial validation. That's when you train a model to discriminate between your data partition, so train— let's say training and serving, serving set, and if the model has any predictive power at all, you know that is indicative of, of drift or skew. And so the good thing about this is you can use all of your familiar machine learning interpretability machinery to then drill into like why that's happening. And so you don't need any new infrastructure or tools to do this. And then also like this is not limited to— it's not a univariate test. Basically, you know, you can catch complex interactions in your data that are indicative of skew. Okay, so the next thing I want to talk about is ergonomics. And so like one of my favorite machine learning tools is Keras. Keras. And I know when you get confused between TFX and Keras, I think Keras is actually very different in its design and philosophy, even though it's kind of somewhat related in the larger TensorFlow ecosystem. And I think it's like one of the most loved ML APIs. So one principle is like documentation is more important than features. And I think this is really like— this is really true. This really resonates with me. You know, something's not documented, it doesn't exist. And, you know, even François would go as far to say is like documentation is more important than features. And I agree. And TFX, you know, documentation is often missing. A lot of the APIs I see are not documented at all. And, you know, it's— and sometimes it's kind of funny, like you'll get documentation like this, and you'll even be asked if this is helpful or not. Another thing, another principle that Francois outlines is the APIs should reduce cognitive load in boilerplate. So, you know, you shouldn't have to look up things in a tutorial documentation to do very trivial things. So this is an example of how you remove an item from a set. So this is a— in TensorFlow Data Validation, you get a feature and then you need to use a not_environment, you know, you need to use this not_environment method and append it to that. So it's very unintuitive. This is nothing like how you would remove something from a set in Python, but you know, this is a DSL that, you know, you would— this is the cognitive load of kind of what the kinds of things you have to deal with. Similarly, configuring a metadata store is pretty— there's a lot of boilerplate involved. You know, this is just an example, but you have to do a lot of things that you wouldn't have to do in another framework, let's say like MLflow or Metaflow, that have sensible defaults. And even doing something like a train-validation split involves tons of boilerplate and kind of like a, like a bit of a new API that you have to reason about. Like, you have to think about hash buckets, and this is like a lot, you know, this is quite different than sort of what you are used to, and it forces you to like take on more cognitive load. Another thing is a progressive disclosure of complexity. So MLOps Tools should make it really easy to get started on like very easy things, but then scale up with you to do more complicated things and allow you to learn incrementally. And I mentioned this before, but like when you jump into TFX, it goes straight into Apache Beam, which is quite complicated. It is a different DSL that kind of mutates the Python language. For one thing, but also you have to reason about distributed computing right out of the box, which is kind of— it is a lot to take on. So kind of after all of these grievances, it would, you know, like, this person would say, wait, there's reasons for all this stuff. All the stuff you that you described, there's reasons why this is the way it is. So the first reason and the thing that's kind of thrown out there quite a lot is the scalability argument. So, you know, hey, like, we need scale. Like Apache Beam and things like that allows us to scale. And I think it's really important to keep in mind that scaling when you don't need to scale is really counterproductive. And this famous quote by Donald Knuth is true. I mean, premature optimization is the root of all evil. And so if you try to scale from the beginning, like using Apache Beam, you might never get off the ground. But then also I just want to mention, like, there's other solutions, like using large— single large nodes and using multiprocessing works for a lot of applied ML problems, and then things like Dask as well. So there's, there's other ways to achieve scale and data processing without cognitive overload. Another argument that's sort of put forth is if you don't try to tie your transforms to your model— so, so for example, you know, if you don't use those TF ops like the tf.gradient thing, you know, if you don't— the reason we do that is So you can tie your transforms to your model. The reason for doing that is to bake it into one portable unit is so you don't get train/serving skew, so they can have reproducibility. And so I think that is a weak argument because you have other tools that facilitate this like Weights and Biases, MLflow, Metaflow, and many other things that help you with reproducibility of your workflows. And then when I, when I pressed this person on all of these items, I mean, I did show this person many of these slides and I said, hey, like, how do you even get like TF Transform? I mean, how do you even do this? Like, how can you expect people to get started? So the answer was, hey, like, just don't use it until you're ready to use it and then just refactor everything to, to, uh, to TF Transforms. When you need it. This is kind of the definition of technical debt. You're just kicking the can down the road and you're gonna have to pay a pretty high cost to using a platform that forces you into this paradigm. And then finally, one of the arguments put forth was, hey, things like Apache Beam and other things are made really easy for you on Google Cloud. This is a general kind of principle that might be put forth by a lot of MLOps tooling providers out there. It's like, hey, if you use this cloud or use this thing, it's really easy. And I would be really careful of that. So in this case, hey, like Google Cloud only has 10% market share. So do you really want to gamble with your career by like investing all your time and skills and learning something that only applies to this only really effective on Google Cloud? I would think about that. And then kind of the ultimate argument, which actually traps a lot of people and it's very seductive, is, hey, is this appeal to authority? So for example, hey, like, Google is really smart. If they're doing it, then it must be the right answer. Like, you can, you know, of course they were successful with ML, just use their tools and you'll have the same success. And this is a, you know, it sounds, it might look silly on this slide, but it actually, you will encounter this quite frequently in real life. And so what I would say is like this is a religious argument. This is not a principled argument that makes any sense. And actually like this kind of, this is a kind of a very traditional marketing tool that you can be like somebody if you just use their tools. So this is an Air Jordan. Okay, so some final thoughts here. So, so Google actually produces really good research on technical debt and machine learning systems and kind of really has helped push the field forward in the— in like making us all aware of the importance of having tools. The one thing I want to say is like that doesn't mean that you should use their tools, and I don't think like their telling people that you need to use their tools either. I think there's a lot of, a lot of people misunderstand and they think they draw an equivalence that, hey, like these people produce the research, therefore you should use their tools. And so that's another kind of trap that people fall into in this. So that's great. So you might be wondering, okay, Hamel Husain just gave this talk, when would I recommend TFX? So actually, like, it's not binary, like I hate it or love it. I actually love some components and dislike others. But I do feel it's a bad fit for a vast majority of people just based on what I've observed and some of the things I mentioned. But despite that, it might be a good fit if you're, you know, really committed to TensorFlow, if you're running on Google Cloud already, or like if you need to deploy to edge— things like edge devices, if you want— if you really need that portability. Or if you are operating at a scale where somehow the optimization is really important to you. But even then, I'm not very convinced it's a great fit for many people based on just things that I've seen. Things like documentation, which is almost— lack of documentation makes it unusable. But if you take away one thing from this talk, it's this slide. One thing I want to emphasize is don't become a tool zealot. It's really easy to become a tool zealot. And, you know, as you form data science teams, as you, you know, go into the workforce and start working at all these companies, you're going to have to make decisions about what tools to use. And so one thing just to always keep in mind is like the tools that you choose, especially ones that are more monolithic and don't operate well with other tools. It will narrow the way you think about problems and also introduce bias towards working on certain things. Like if the places where the tool introduces lots of friction, let's say for instance in this case feature engineering, it will bias you towards working less on feature engineering and more towards working on other things. And then very importantly, if you are too myopic in your tools, and, you know, that will prevent you from hiring diverse talent. So for example, I don't think I could work with that person that was very convinced that TFX was the only right tool on the planet to use for machine learning. And, you know, that— I think that is really important too as you go out there and think about these things. But then also, like, you know, what tools you use can also lead to blind spots. And really, like, I think it's important to try tools often, especially ones that have different approaches to things. I think that can, you know, kind of the counterpoint to blind spots. And, you know, all the things I mentioned just now, I mean, these are the same reasons you should learn other programming languages. But even if you don't have time to learn ML Ops tools and different things, one thing I really want to emphasize is to keep an open mind. And then lastly, like, I hope that what you take away from this talk is kind of an example of someone evaluating a tool and providing some reasons why it didn't work for them, and then kind of seeing which one of these things I mentioned resonates with you. And having more confidence to either push back or sort of challenge certain, certain kind of zealots that may come at you, or even have better discussions about MLOps tooling. Because in practice, I think this subject is very important. All right, thank you and hope you enjoyed this talk.