Malleable ML Systems feat. Karan Goel | Stanford MLSys Seminar Episode 29

Hello everyone and welcome to episode 29 of the Stanford ML/Sys seminar series. Uh, so today I have with me Fyodor, Hiro, and today our guest is actually our very own Karan Goel. Uh, yeah, so as usual, you can subscribe to the channel, subscribe to our mailing list, click the bell icon to get notified when we go live every week. Um, so this week we'll be discussing malleable ML systems with our very own Karan. As usual, the plan is, uh, a 30-minute talk followed by a podcast-style conversation where you, the audience, can ask live questions. A little bit about Karan, our host. So he's a third-year CS PhD student at Stanford advised by Chris Re. His main goal is to accelerate the pace at which ML can be used robustly and safely in industry and across applications. So he leads the Robustness Gym project, which I believe we'll be hearing about a little bit more about today. He's also a recipient of the Siebel Foundation Scholarship. So with that, Karan, feel free to take it away.

Awesome, thank you so much for inviting me, and not that I didn't invite myself, but, but yeah, great to be here. And I will be talking today, as Dan said, about building malleable machine learning systems through measurement, monitoring, and maintenance. So the plan for this talk is going to be, I'll first highlight some places where I think ML is today, point out kind of interesting conceptual directions that I think are interesting at least in this area of malleability, and then I'll talk about a little bit of the ongoing work that I've been doing with my collaborators. So let's jump in. So here's— on this slide, I've attempted to make what is kind of my version of the ML software hierarchy. So it's trying to capture what I think are kind of the major areas where people are building tools and software and making progress and making machine learning more reliable. So at the bottom is something called program and run, basically, which is— How do we build models that as quickly and easily as possible that kind of get good average performance? And I think this has gotten pretty easy with Torch, with Keras, with basically a lot of these tools for handling and building models and building datasets. For example, Hugging Face has a great library for some of these things as well. And then next, once you know how to build a model, it becomes more important to understand how well is that model actually doing. So average performance can often hide a lot of problems that models have. And this is something that I think is a common experience in industry, but also is starting to become increasingly kind of a focus of machine learning as a field, where robustness and, and questions around making more robust methods have started to take kind of hold in the last couple of years. So you'll see more and more papers talk about this. And specifically, this talk will also kind of touch on these questions of like, how do you do fine-grained evaluation? How do you monitor distribution shift? And these are kind of, you know, the second-order kind of things that we want to care about now that we know how to build models easily. The third thing obviously is how do you improve and repair things. So once you've actually got a way to measure what's happening, you, you should be able to then, uh, go in and actually make fixes to your models. And we'll talk about this in the talk as well a little bit, but this is, I think, the area that's the least prominent. And, uh, very few folks actually know how to do this because there's a lot of unanswered questions in this area. And of course at the top we have, you know, various interfaces for actually going in and doing all these different operations. And this hierarchy is not necessarily capturing the order in which we do these things. I think interactive interfaces are always getting built. Pyro, for example, has been instrumental building some declarative interfaces for training and so on. So there's a lot of work that's always happening in this, but it's always very important to think about the human in the loop and how they're going to interact with all of these layers. And so just to recap, like we're doing this pretty well, I would say today, but there's a lot of stuff to do in this bucket of 3 things. And this talk will hopefully try to talk a little bit about that, but there's a lot to do. So let's talk about malleable ML systems and what I mean by that. So I'll try to give a working definition and then, you know, it's not necessarily precise, but I think it gets the job done. And so here, what I'm defining as malleable is the ability to modify the system over time to perform bug fixes, hit performance targets, and meet some sort of ethical standards. So some sort of improvement of the system over time, making sure that we're hitting whatever targets we have for the system in terms of user performance or application performance, and then also making sure that we're hitting accountability and governance. And then what would the system components here look like? Well, What you would want is that the stakeholders of the system, which might be engineers, developers, researchers, ethicists, whatever they are, they can interactively, the keyword being interactive here, measure fine-grained system capabilities and performance so they can actually understand what the system is doing. And this is a critical, I think, bottleneck to actually building valuable systems because you need to know what the system can do in order to actually improve it. Well, second, one system component would be monitoring it. So as I talked about in my first slide, maybe you want to track distribution shift and its effects over time because these are machine learning models. So the inputs are basically data. The code is the model itself. And so as time passes, the main things that are changing are actually the data and the sort of environment in which this model is deployed. And so you want to understand the effect of that. On your system. And then third, of course, you want to perform these atomic updates to data models. And the emphasis here being on atomic, because I think right now we have maybe less of an understanding of how do you actually change a model in small incremental ways to actually make it better over time. And this is, I think, a hallmark of software systems where things don't get built in a day. You don't ever build a model, or you don't ever build a large piece of software in one day and then say it's going to work. You obviously incrementally keep adding bug fixes. People find issues. And so this is something that, you know, is interesting to borrow from software engineering and bring into, I think, machine learning a little bit more. Okay. So that sets up this kind of high-level idea. And one of the things I want to point out is like, there are organizations and folks who actually have started to build these types of systems, especially in industry, but there are very few of them. And they kind of have this knowledge hidden inside these few organizations. So very large organizations like Google, and Microsoft or Facebook, they have large systems that are obviously deployed and they've worked very well, but, uh, but they've kind of built tools to do a subset of these things, and there's a lack of repeatability in this process right now. We don't really fully understand all the issues, and so of course we haven't built tools that generally work well for this. Um, so in this talk, I'm going to talk about these three things, which are understanding fine-grained system capabilities and performance with measurement, uh, tracking distribution shift and its effects over time with monitoring and then performing some sort of atomic updates to data and also maintenance. And, um, you know, all three, uh, question directions are obviously pretty nascent and there's a lot of work to be done. So, you know, fair warning, uh, this is all kind of recent work that I'll be presenting and there's, there's more to do here. Um, so let's talk about measurement first. Um, so this is probably the biggest bottleneck because I think it's pretty obvious that you can't fix what you don't measure, right? If you don't really know there's a problem. You can't actually do anything with monitoring or maintenance at all. So, so we really need to pay close attention to how to get finer-grained metrics and model performance across important slices of data that we care about. Uh, so for a system, we want to understand, you know, how is the system doing on certain slices? Um, and one thing I want to point out as a simple example is that it's very, uh, simple to imagine how powerful this idea is because Imagine you have a model and you're only measuring, let's say, average accuracy. And then all you do is measure two subpopulations. Let's say you measure performance on men versus women. Let's say it's some healthcare system. And you notice that there's a large performance gap between those two groups. Almost immediately, you'll have ideas on what you can do to fix that. Well, if your average accuracy is 90%, you're not really going to know what to do next. But if your two subgroups of men versus women is like 95% versus 65%, you might be very concerned. And you would go in and say, OK, maybe we need more data for this subgroup. Maybe we need to do something different. So this is why measurement is important, because it immediately kind of reveals problems and also lets you have a sense of what possible solutions might look like. And here I'm going to talk about very quickly, you know, point out 3 works. I won't touch on the second much, but I'll talk mainly about RobustStream, which, as Dan mentioned, is a project that I lead on building finer-grained evaluation toolkits. Something I'm pretty excited about, which is Mosaic, which is this kind of other data substrate that we're trying to build for kind of doing interactive machine learning that we hope can power some of these measurement interfaces that we're building. So let's talk about RobustSystem. So here, the idea of RobustSystem is that there are a lot of evaluation strategies out there. So people do evaluation in all kinds of ways, and, you know, there are people who care about, let's say, critical data slices. So if you're an industry, you might care about performance on tail queries queries that, that, you know, have high value, or maybe you care about certain stakeholders and you want to make sure that your models are doing well against those. You might also have, you know, bias fairness concerns. So a lot of commercial systems, uh, including recruiting tools and other things, have been pointed out to be, you know, unfairly kind of stacked against certain subgroups of folks. You might want to understand, you know, sensitivity. For example, you might be concerned if your dialogue or, or chatbot is overly sensitive to spelling mistakes, for example, right? So you want to understand these types of, uh, these types of, um, performance, uh, of your, of your model. And then, you know, invariances and other things that maybe researchers care about, which is what does this model actually understand, or what are the capabilities of this model. So there's a lot people do in actually trying to measure this, and what we wanted to do is bring a little bit of uniformity to this by having a toolkit that you can to actually support all of these different ways that you can do evaluation and then pop out visualizations that capture or summarize this with something they call robustness support. So basically what we have is a Python toolkit that helps you build these evaluations and then generate these visualizations. And at a high level, we want to be able to support, you know, these different strategies that folks have. And the toolkit has, has support for a lot of them, and some of them we're still building out, but, but hopefully what we want to do is capture this, this wide spectrum. Um, so let me, you know, show you what an example, uh, in code, and I'll walk you through this. You don't have to read the code yet. I'll just kind of show you the steps that robustness stream users take in order to build evaluations. Um, and you know, like, there's a lot underneath the covers, and, and I won't talk about a lot of the detail, but you'll get a sense of what's going on here. So first thing first is obviously loading some sort of data and model in. And, you know, we have support for mostly text so far. And so you can load in a bunch of text datasets and models from Hugging Face. And here, you know, like we have, we support text classification, token classification, text generation. We're also working on vision and time series that we'll talk about in a little bit. Then there's the kind of this important piece, I think, which in, in kind of interactive analysis workflows, which is that you often want to do some preprocessing or processing step. To compute some information that's useful for analysis. So for example, for text, it's very common to want to tag text, to want to tokenize text and compute, you know, part of speech tags, for example. And all of this is something that robust systems helps you do with these ops that we provide that you can just run on your data and then immediately get out, you know, these additional pieces of information that you can then use for analysis. So for example, maybe what you want to do in processing is merge metadata and tag, compute predictions for your models, compute embeddings, and basically any kind of side information that goes with your model. And then next, what we provide is a bunch of these abstractions that actually help you create and, and store these evaluations that you run. So here we have this DevBench object that basically can be used to store all the evaluations that you actually create in the course of your analysis. And the evaluations themselves can be, you know, subpopulations, so filtering your data to get certain subgroups Transformation, so transforming examples, for example, trying to simulate, let's say, keyboard character errors for dialogue typesetting or things like that. Adversarial attacks, where we kind of outsource some of this to other libraries that do this really well, for example, TextAttack. Text generation, again, we are building this capability out, but we'll be able to support you writing down examples and then actually evaluating against them and hopefully also integrate some support for other great toolkits. And of course, any datasets you have lying around can all be added to this DevBench so that you have like one object that contains all these different ways that you would like to evaluate the model. And on the left, I show a simple example of how we can write a simple sort of, you know, subpopulation for looking for like horror movie reviews if you have a word list running around. So it's a couple of lines of code, you can do it pretty quickly. And then of course you can add all these evaluations into this DevBench object. You can evaluate your model by writing down, you know, custom metrics. We provide a few metrics ourselves, but you can write pretty much any metric you like, and it will be easy to then go in and visualize it. And of course, one of the goals was to support this sort of exploratory evaluation. So you can keep adding metrics in and then inspecting and visualizing your intermediate results in some sense. And then next, we let you create these nice-looking reports, which we call robustness reports, which are basically trying to capture all the evaluations you did and put them in one place so that you can look at them. Nicely. And so you can kind of, you know, visually summarize kind of these multiple types of evaluations and also have some degree of standardization. This is something that we hope we can, we can push more in the future where we have standardized evaluations for certain tasks that folks can just run out of the box. And, you know, one thing that obviously as a researcher I care a lot about is reproducibility. So you can save and then reproduce things so anybody can, you know, take your DevBench and then run Create report and they will get the same result as you did. And so this is great for continuing your own evaluations or sharing your results. And all these abstractions, of course, we've iterated them a little bit, but we would love feedback if you use them and we are happy to improve all of these things. I think it's a process to build this in the right way. Here's an example of a robustness report for a particular task that I think natural language processing researchers care a lot about, which is NLI. Maybe folks in industry are less familiar, but it's a text classification task. And I just want to highlight a few things here, which is, you know, on the right we have these 4 different ways that we're evaluating these models. So we have 4 sections in this report. One is subpopulations, so those filters, attacks, which are adversarial transformations, which are kind of perturbations, and these evaluation sets that, you know, are commonly used in the literature. And then what we do here— sorry, yeah, so on the left what you see is all of these different slices that are bucketed into these these categories. On the left, we've kind of— what we tried to do is take slices that people in the literature evaluations, that people in the literature were said were interesting or important, and then try to implement them in a restless dimension, show that you can actually do this for, you know, a span of these different things and put them in the same report using our tool. At the top, you can obviously report a bunch of metrics. We report a few for text classification, but, you know, you can report any that you like. And then here I just want to highlight that these types of reports are very helpful for even state-of-the-art models, where you can see that they're often sensitive to certain perturbations. For example, this model, this BERT model, is sensitive to keyboard errors, and, and so might actually be something that you want to look at when you're, you know, if it was in a dialogue setting, for example. And so there's, there's been a lot of recent work, for example, talking about how tokenization might be important to this type of stuff. So, so, you know, connecting all these these, um, conceptual ideas with the evaluation is very important. What we are hoping is that this gives you a nice visual summary that helps you do that. So obviously we want to make this even easier to do and unified for a lot of different modalities, as I mentioned. So this is ongoing work where we're trying to actually make it possible for you to do this for images, for videos, for time series data, and more. And we want to just have, you know, kind of a standard data substrate for which we're building this mosaic, which is kind of like a pandas DataFrame, but easier for you to work with high-dimensional data and the kind of standard things in machine learning like pre-trained models, embeddings, and, and so on. And so we want to make it super simple for you to do kind of data loaders and lazy loading and so on from, from disk. And, and then on top of that, then be able to run things like Robustness Gym, which will then be kind of dataset or modality agnostic. And you can then run evaluation for images, you can write for text, you can run for multimodal applications. And so this is kind of ongoing work that we're very excited about. There's a demo link there that you can check out. It's a Colab notebook that kind of describes the Mosaic workflow, but I think it's, it's kind of exciting and early, so we'd love feedback, love users. And if you want to participate, of course, like, reach out. We would love that. Um, so next I'm just going to jump into, uh, the next phase of my talk about monitoring. So, so far we looked at measurement. I've talked about RobustSim, which is a toolkit for actually measuring fine-grained performance across different ways of doing evaluation, and then I, you know, mentioned briefly this idea of trying to unify this type of toolkit for different modalities. So now, uh, the next part of the talk is about monitoring. So now that we know how to measure, one thing that's very important is, can we understand, does performance degrade over time, or how is performance changing? And so what we need here is really to be able to calculate metrics on unlabeled data. So as data is coming in, it's going to be unlabeled. Your model is going to actually make predictions on that. And what you want to understand is, is my model doing a good job on this unlabeled data or not? And so for this, I'm going to talk about Mandolin, which is very recent work that's going to be upcoming in ICML with a bunch of really great co-authors. So the idea of this work really is to— and I just want to point out before I say anything, this is also something that you can use in RG. We have code for this. But yeah, the idea of this is that typically when you train a model, you learn on a train set and then evaluate on a validation set. And those are both sampled from a source distribution, P sub S. And this is your labeled data. This is your, your standard machine learning setting. And often the assumption is that your target or your test time data will come from the same distribution. But in practice, when you deploy your model, the distribution shifts. This is very common. And then your target data distribution is actually P sub T, which is a different distribution, and it's also unlabeled. So how do you understand what your model is actually doing on this on this data that you're getting. And so for this, we built Mandaline, which is a user-guided framework to evaluate models using slices. So in the previous section, I talked about robustness and how you can filter data to look at performance on small subpopulations or small slices of data. And so Mandaline is trying to take this information and actually use it to monitor your model as well. So a slice here, just to give you more grounding, is just a user-defined grouping of data, so a filter. And so here, for example, we have negation, which could say, let's look for these phrases in a piece of text. So if a piece of text contains not, then it's marked as a negation example. And then you can say, OK, it's a 1 instead of a 0. You could look for things like male pronouns. Or let's say you could look for strong sentiment words. So these examples are obviously for sentiment classification type setting. And so in the third bucket, you have maybe strong sentiment words like love or adore. And then male pronouns. And the idea is— I'm just going to show you kind of the process pictorially. But the idea is on the left, what we have is some sort of source dataset, your label validation set, the dataset that you basically use to actually validate your model. And this is the standard dataset you have. It's labeled. And so we show a few examples here and how we kind of can mark out which examples fall in which slices. So you can see, like, I love eating ice cream would fall under strong sentiment because it has the word love, but maybe not under the other two slices. So you get a -1. 1 for those, and a 1 for those. And maybe the model gets that right, so you have a correct check mark there. And so you can do this, you know, this process for this entire dataset and basically get a matrix representation of your data that is n by d, where n is the number of examples and d is all the slices you have. So this, you know, this data, maybe your model is doing like 90% on this, on this validation set, so you're pretty happy. And now let's say you get some unlabeled test set, so unlabeled target data. Maybe it's streaming in, maybe it's coming in a test time in one go, but, but you have some data. And then what you can still do, even though the data is unlabeled, you can still mark these slices. You can still tell me, you know, this example belongs to this slice, this example belongs to the other slice, and so on. And then what we do in Mandolin is we take this information that you give us, which is, uh, the slice matrix for your source and target data, and then we use some variant of importance weighting, which is a common technique to handle these types of settings, in order to actually give you an estimate of your performance on the target dataset, which is actually unlabeled. So you get some sort of metric that tells you, oh, your target accuracy is 84%. And the intuition is that the slices are some sort of like reweighting of your source data. So they tell you, let's say you had, you know, very few, um, or, or a large number of like love, adore, kind of strong sentiment slice in your, in your source dataset and much fewer of that in your target dataset. And the performance on that in the source dataset was high. Then if you have a lot less of that in your target dataset, you expect the performance to drop because the model is doing well in that slice. So that's some of the intuition on how this works. It's kind of reweighting based on these slice— slices how things are working. Of course, we have some, you know, interesting theory in the paper that shows that we can do a lot better than existing techniques for importance weighting. And we also in fact show that you can be up to 5 times more accurate than, you know, standard importance weighting baselines that people use all the time. And, and the other thing that I'm really excited about in this is the fact that you can actually leverage this side information of slices that you're already computing when you're doing evaluation. So this is kind of a neat way to tie everything up where you're saying, I'm going to evaluate my model by actually filtering my data down, but those, those subpopulations that I compute could also be used for this other thing, which is understanding and monitoring your model's performance. So this is something that, that's pretty exciting, and there's a lot of interesting directions here to understand what slices are actually relevant or useful to do all of these different things. And so there's, there's a lot of work to be done there, but I think the early results are pretty, pretty interesting. Um, so last I'm going to talk about maintenance. So, you know, we talked about measuring performance. We talked about monitoring how the model changes over time. Can we actually measure and make fixes to undesirable behavior? And this is definitely the part that's, uh, you know, the, the hardest and the, the least studied, I would say. Um, and here we have a couple of papers this year on understanding kind of how do you change a model to make it better in some way. So the first paper is what I'll briefly talk about, which is called Good Will Hunting. We presented it this week at NACCO, actually. And the other paper is talking about how you can learn data augmentations in order to actually close the gaps between different subgroups of your data. And so I would talk about that work, but you can check it out if you're interested. So for the first piece of work, let me give you a quick brief on named entity linking. Which is this task where you want to map strings or text to things in a knowledge base like Wikipedia. So let's look at a quick example. So let's say you have a sentence called, like, when did England last win the Football World Cup? In this case, what you would want your model to do is link England to the English national football team and Football World Cup to FIFA World Cup, because these are the grounding of these entities that you found in the text. Um, and then you might have a downstream system, let's say for question answering, and what you would do then is pass these entities to that question answering system, and then it will produce some answer. And the key thing to know here is that pretty important that you get the right entities because the question answering system would then use that information to actually give you the correct answer. So these things have to work in tandem really well. Uh, so in this work on Goodwill Hunting, what we did was we basically analyzed a bunch of off-the-shelf commercial and academic named entity linking systems. and we use Robustness Gym for this. So I won't talk about this, this particular part because, you know, I'm talking about maintenance right now, but, but you can check this out in our paper. And we look at, you know, commercial systems like Microsoft, Google, a variety of academic systems, and, and we kind of understand how they behave off the shelf. And then second, what we do is actually repurpose the bootleg system, which was actually developed in our lab, to guide its behavior for sports QA applications. So let me talk about that very quickly. We want to fix poor performance, so that's the goal here. We're doing maintenance, so there's some problem and we want to go in and maybe improve or fix it. So here, the particular problem that we're looking at is, let's say you have a sentence like, English player Harry Kane scored a goal in their match against Germany. So there are a lot of contextual cues here that tell you that Germany here refers to a national football team. It does not refer to the country. This is a common kind of confusion in, in named entity linking because there's a certain degree of disambiguation you have to do, and often you have to rely on context in order to do that. So what we found is the off-the-shelf system that we are considering here, Bootleg, was, was actually making a lot of errors in these types of cases where it was actually predicting the country type instead of the sports team type, which might be, you know, something that's undesirable for, say, a sports QA application where we might want to make the model actually be more likely to predict the the sports team instead of the country. So this is where we came in and we said, okay, we want to make the model prefer the national sports team instead of the country. What can we do here? We tried a very simple approach as a proof of concept, which is we just relabeled the training set with a simple heuristic. And the simple heuristic was, you know, let's go in and say if there's a phrase like match against and there's some entity that follows that, we will relabel that entity if it's a country type, to be the national sports team type. That's all. So we made this one little change that basically just gives the model a little bit more contextual cues about what to expect when it sees these types of phrases. And then what we found is that we could actually just— same model, retrain the model, and we found we could actually, uh, a) reduce a lot of errors that this model was making. So 25% absolute accuracy improvement in actually making these sports-related errors. And second, that, you know, like in the particular subset or subpopulation of interest, you can see that it was much more likely to actually predict the sports team instead of the country. So here, what I'm saying is if the gold label was this team and if it was earlier predicting country, it now is much less likely to predict country because we've kind of guided its behavior a little bit. And so I think this is an interesting kind of more atomic level fix. We're actually changing the low-level small behaviors of the model to make it more suitable for the application the application that we have in mind. And so I'm very excited about directions that kind of take this and kind of scale this idea up a little bit more to actually make kind of malleability a little bit of a system property that's kind of first order. So I'm coming close to the end of my talk, so I just want to conclude by saying, in this talk I've so far talked about measurement, I've talked about robustness shim, I've talked about monitoring with Mandolin, and I've talked about maintenance with some of our work in Goodwill Hunting and model patching. And I just want to kind of maybe as a last attempt just talk a little bit about a wishlist for, you know, methods and tools for malleability and what's going on in this area. So I think there's definitely a lot to do, and especially I think the gap is going from measurement to fix. And I think my common experience talking to folks is that they will often say, you know, you can measure it, but what else can you do, right? And so that's something that people care a lot about. And so those workflows I think are something that need need a little bit more study because I think there's a lot that's unclear there. And so, uh, you know, in, in even in the Goodwill Hunting paper, we characterize some of these different paradigms that people might have to actually change their models. You might have approaches that are more data engineering, like actually going and, and making modifications to your data by doing augmentation, maybe adding more data, doing some sort of active sampling approach, maybe doing weak labeling, maybe even changing the preprocessing pipelines. This is something that that a lot of folks have found have to be very sensitivity-inducing in your models, that your preprocessing is particularly important. And then you might even have these model engineering approaches. You want to go in and change the training paradigm. Maybe you want to do multitask learning, meta-learning, whatever it is, standard classification, the training algorithm used, the features they use, the architecture used. So a lot of this stuff, I think, needs a little bit more of a realignment from this lens of malleability. If you start thinking of these as kind of more atomic things, you might think, okay, maybe we need to design architectures that are actually updatable in some sense, because a lot of the architectures we have now don't have an immediate way to actually update them very quickly by adding, you know, more parameters or any of that sort. A lot of that work is more or less based on heuristics. So, so having more kind of knowledge about this would definitely help. And of course, I think there's certainly like a lot to do in kind of understanding how to make models again atomically develop, or systems atomically develop over time, much like software. You might want to, you know, do some sort of semantic versioning where people file bug fixes or file reports, and you can actually do atomic fixes in order to go from, you know, model 1 to model 1.1 instead of straight from model 1 to model 2, because that's, you know, kind of a little bit more expensive, and it seems like a little, uh, you know, less elegant in some sense. And I think in the long run we'd probably not want that. So the, the conclusion here is that understanding, you know, a lot of these trade-offs between methods is really a research question and is critical. So you want to understand, you know, should I fine-tune more? Should I retrain my model? Should I add more parameters? All of these— this design space is something that's kind of very understudied and really requires more structure and more understanding. Um, and so I hope like one of your takeaways from this talk is obviously there's a lot of exciting work here, but, but there, there's a lot more to do and there's kind of fun, fun problems in this, and that would be obviously happy to chat about. Um, So this is, you know, my conclusion slide, um, just to kind of recap some of the work that I've talked about. Uh, so malleability kind of equals, you know, measurement and monitoring and maintenance and kind of understanding each of these buckets very well. And, um, some of the work that we're doing is trying to kind of make progress on these questions. And, uh, super excited to chat more with anybody who's interested in these areas. Uh, of course, um, thank you so much for having me, and, uh, that's the end of my talk.

Cool.

Thank you so much, Karan. Um, yeah, so we're, we're super excited to, to get started with kind of the, the podcast portion, which, um, you know, where we'll be able to, you know, ask you questions, have a bit of a conversation. Uh, just a reminder to the audience, you can, uh, you know, ask questions in the chat and we'll summarize them, uh, pass them along to Karan. Hopefully we can have a nice, uh, interactive session. So I, I think I'll just, uh, go ahead and get started here. So, um, The first question that I had kind of throughout this talk is, I think we've seen throughout this talk a couple of different approaches— or sorry, throughout the seminar, maybe a couple different approaches for model evaluation over the, you know, past 30 weeks or so, how people think about it. I was wondering, how did you come to the Robustness Gym abstractions and kind of those specific interfaces, um, when you were building Robustness Gym?

Yeah, that's a great question. I think, um, yeah, so I, I remember like Marco's talk, uh, very early on in the seminar, and he talked checklist. And, you know, he described some kind of interesting abstractions there for doing more sort of software engineering-inspired unit tests. So I think in robustness, Jim, the reason why we even built the toolkit was we started to see that robustness was something that was very important to a lot of researchers. And I was also working on stuff in robustness, and it became clear that a lot of the problems that we were seeing people work on were things that they kind of anecdotally somebody else had discovered. So somebody had come in and said, there's a problem with this dataset where certain subpopulations are underperforming with the common methods, and let's try to fix that. And so I think a natural question that I had was, well, if we already know that the problem exists, it's not that interesting. We need to actually figure out how to discover these problems in the first place. So that was kind of the motivation for actually building RobustSystem, which is like, we need measurement tools that actually go in and help you figure out where the problems are. And then you can hook them up with these more robust training or, or these other methods that actually help you improve the model. So that was kind of the motivation. And of course the abstractions themselves were kind of like, uh, a way for me to do all of these different ways to evaluate very easily. So what I felt was like, I don't want to reinvent anything that anybody else has done before. I don't want to reimplement, you know, adversarial attacks because TextAttack did a great job at that. I don't want to reimplement or kind of try to rederive things that people have already noticed. So we want to wrap as much as we can. And so what's the most natural way to do that? And that's kind of what led to the abstraction. Of course, we iterated using these kind of papers that you saw were kind of, I think, proof of concept iteration cycles that we did, where we tried to apply robustness to named entity linking to understand, okay, what needs to change now? Applied it to actually summarization tasks for text generation, where again, we noticed gaps and And that inspired another direction, which is something I didn't talk about in the work, which is SumViz, which is an ACL demo where we realized that it's not enough to look at these metrics. Even you need to look at the examples themselves. So there's a lot of, you know, layers of complexity here. And so we were trying to just capture as much as we can and give, you know, personally me, a lot of the tools in one place. That was the idea.

That's great, Karan. I can actually follow up on that because I think, you know, the idea behind RG is really great. And also, I'm curious about Mosaic now. I have to— one thing that— one question that pops up in my head is that one thing that you touched upon is the fact that you want to go from, let's say, a text-first kind of tooling to be more data type or data agnostic with respect to modality agnostic.

That's what they use.

So I'm curious how you're thinking about it, because from the text perspective, it's relatively simple and straightforward for someone to write something that is like of the sort that you showed, which was like a function in that case that was figuring out if one data point was— if a text contained words that were associated with horror, right? But when you go to time series or when you go to images, I don't have a strong intuition of how you would go and do it. So maybe you can talk about it.

Yeah, yeah. That's a really good question. I think that's really one of the— been the big bottlenecks. I think one is obviously the fact that the modality is different and you need to handle the data in a slightly different way. But the other major difference, as you pointed out, is that in images, you can't just write, search for this thing in the image. But I think now the thing that I'm hopeful about is that pre-trained models and embeddings are kind of closing that gap a little bit. So what you see is, for example, things like CLIP actually allow you to search for images. So you can connect kind of language to, to vision and then use these kind of embeddings to actually do that. And then the, the language interfaces will still exist. So that's something that I think is interesting and exciting about this, which is like connecting up these kind of pre-trained embeddings and, and using that information to actually parse or understand, uh, the input. So in the image case, you would compute some sort of, you know, image embedding and then Hopefully that would allow you to cluster or group your images and then, and then do things on top of them. So these operations are not necessarily, you know, searching based on phrase lookups or things like that, but they're really maybe machine-driven things where the machine says, okay, I'm going to group all these data points because they all look similar. And then you can click and point and do certain operations on them in order to, you know, peek in or say, I don't, uh, you know, maybe find me more like this or, or so on, right? So there's kind of an interactive loop to build there, which is still a little bit, you know, difficult to do. And the other side, of course, you can use site information. So, you know, metadata, you can join that and you can compute, let's say, bounding boxes on top of images to get more information about what's in the image. You can do optic segmentation and so on. So there's a lot of site information that I think is interesting to push in. And so this is one of the principles I think in RG, which is like you want to do this workflow where you load your data and then you immediately want to compute some site information so that you can actually use it in the analysis. Processes. So we're still thinking through some of those questions, which is why images are not in our GM. But yeah, that was a great question.

That's super interesting. And I think that actually makes a lot of sense also because you're already doing it for text too, right? You're using the models in spaCy, for instance, for tagging and other things. And those models, the fact that those models are not 100% accurate is not a huge deal because even if your slice misses a couple data points or gets a little bit like false positives in the set, the overall story will still be, you know.

Right.

Yeah.

And I think even there, like there's definitely more to do in characterizing like reliability in some of these evaluations. So understanding, you know, confidence intervals or how much can you trust the numbers that you see and are they really indicative of underlying problems in your model? So yeah, I think those things are still kind of being worked out and not quite sure, but I think, you know, the fuzzy heuristic here is that, yeah, like you said, On the whole, you'll get the right picture about what's going on, and then you can do more there.

That's great.

Thank you for your answer.

Hey, Karan. So kind of related to Piero's question here, when I was watching the segment on Mandolin, I was wondering, how do you think about how people should be interacting with slices in Mandolin and writing new ones? Like, the workflow that immediately pops in my mind is that you notice some problem and then you write a slicing function for it. Is that how you imagine people should be kind of interacting with Mandolin, or is there, you know, pre-built slicing functions that can use— what's kind of the iterative workflow there?

Yeah, yeah, that's a great question. I think that is probably the biggest unanswered question in the Mandolin work, where I think the question of what slices are the most relevant or the most important is kind of hard to answer a priori. So as you said, there's definitely an interactive loop there where you go in and you, you check, you know, which slices are actually relevant to this model's prediction. And in the paper, we kind of, you know, describe slices that we thought were— or were described sizes that were either derived from metadata. So we, you know, looked at metadata that was attached to, let's say, an image dataset. And, and so you can— if you're in industry, for example, you have a bunch of metadata or tags associated with your with your data, and those would be the slices that you use. Or we looked at, you know, doing programmatic slicing, so writing down slicing functions. And that's where things get a little trickier because again, like, which— there's a huge space of things you could write down, and which ones should you do? But one thing that we found which was pretty interesting, I think, was like, uh, there's a bunch of cheap slicing functions that are derived from the model's outputs that are actually fairly performant and fairly generic. That do pretty well inside Mandolin. So they're basically these slicing functions that bucket your examples based on how uncertain your model is on those examples. So they are roughly capturing kind of the proportion of examples that have uncertainty on them and how they shift across time. And so if you use those, you really get, you know, good estimates of target performance. At least that's what we found. And, and yeah, I think like the, the question of like, how do you How to push this into an iterative workflow is something that we're exploring right now, so that you can help kind of guide the user towards the right set of slices and so on. So that's still kind of in progress. But what we found is that at least reducing the dimensionality of the data, right, like getting from your inputs to these slice functions is not losing anything. In fact, it's helping with some of the theory. It actually helps you get good estimates with importance weighting. Remove some problems in participating with support shift and so on. So, so that's, that's, uh, that's something that I think is interesting to explore more.

Yeah. Karan, when you, when you're on that slide on slices, you know, the first thing that bumped in my head was that this looks a lot like a very partial but user-defined embedding of the input, right? And I know embeddings have been used a lot, especially in like active learning literature to help kind of at least group subsets of unlabeled data. And I thought that, you know, maybe that would have been an interesting place to start, was rather than having like these user-defined partial embeddings, just use the natural embedding of the input and then use that basically to decide. And this is exactly kind of what you talked about when answering Piero's question about like different modalities, right? Does it leverage those embeddings directly? And I thought, you know, what, what made you think that like starting off with user-defined things was the way to go? Like, how did you come across it?

Yeah, it's a great question. I think, um, what we wanted— of course, like, there was, there was this, uh, this thought that it's easier for us to get, you know, um, algorithms that, uh, you have theoretical guarantees when we work with some of these user-defined sizing functions because you know, the output binary things, then it's kind of easier to model them in graphical models and reason about the kind of correlations between those, those slicing functions and the noise inside them. So that was obviously one motivation. And I think it, it also kind of fit more cleanly with the abstractions that I had in RG at the time, which was that, you know, RG is allowing you to write down evaluations. And so it's kind of this dual purpose thing. You write the evaluations down and then you push them into Mandolin in order to do to this monitoring kind of thing. But yeah, you're right that I think the next natural step is to try to actually use these pretrained embeddings. And I think the example of this sort of uncertainty and the model outputs that we were using for slicing functions is actually one example of this, where we actually tried to compute the last layer's embeddings, which is basically the logits of the model, and use that. And so it did pretty well. And so this was something that we were also kind The first thing that crossed our mind after we wrote the paper was like, okay, now I think there is this kind of direction where you can use embeddings, discover slices, you know, actually understand, as you said, cluster the unlabeled data, understand the regions where there are actually problems that your model is encountering, and then use that information inside something like Mandolin. But yeah, I think having this framework set up in this kind of more generic way with Mandolin is, is pretty useful so that we can inject now other ways that we can set up the slices and slicing functions and so on. So I think that'll be an interesting kind of next step for that project.

Yeah, that makes a lot of sense. You know, it's almost like if, if you were to do the embedding approach and you notice that like you aren't really seeing many errors across the slices, but in your predefined kind of, you know, slices, uh, you were seeing like large discrepancies, what that really tells you is that your embedding really isn't rich enough to to tell the difference between these kind of things, you know?

It's like, uh, yeah, I guess like, you know, be mutually helpful there. Yeah, I think often that doesn't happen because, um, if your model is degrading on some slice, then, um, it's actually because the model is representing that information differently from some other slice. So you can actually separate those slices in the model's representation. Um, so it actually will be sort of aligned in some sense. Uh, this is something that, um, you know, there's some work, a great work from our lab on hidden stratification. Where they were trying to understand, you know, how does a model that's trained on some set of classes, you know, how does it perform on these hidden subclasses of each class. So for example, if you train on dogs versus cats, maybe the model is not very good on certain dog breeds. And the fact that it's not good means it has learned to differentiate those breeds and is doing something different on them than it is on the other dog breeds that it's doing well against. So that's pretty interesting. I think there's definitely, yeah, that kind of interesting finding, and, and that way of thinking is definitely going to help in, in these problems too, I think.

Karan, actually, still on, on Mandolin, I actually have a couple questions. Um, one is, so you correctly say it's 5% more accurate than other methods, right? But I'm curious, um, do you think that the accuracy that you can get, or the correlation with the actual accuracy is at the level that it can be used right now for, for like practical purposes? And the other question, which is actually correlated, is can you give us also an intuition of why the distribution of the slices is better at being correlated with the actual performance than the distribution of the data itself? I'm, I'm interested about the intuition there.

Okay, yeah, so, uh, to your first question, I think, um, I think you, you asked is it something that I think would be really reasonable to use, and does it actually capture, um, I think I would say directionality in terms of performance change. And I think actually, uh, the answer to that question is positive. I would say I'm not going to claim that it's, you know, perfect and it— and you should use it immediately in every industrial application ever. But, uh, but I think one thing that we actually did try to do was understand for baseline methods, do they move in the right direction, right? Does, does the performance go up when it's supposed to go up, or does it go down when it's supposed to go down? And in almost all cases, what we found is a lot of the baselines would wander off in the other direction, which is really bad because it's making you think that the thing is actually better than it is. But in almost all cases, what we found is our method would go in the right direction. It's just the degree of change that would be, you know, causing some errors. And so that was something that I found really interesting, which is— and we have a, actually a couple of tables in our appendix that talk about this for many different models and how we observe this across, you know, we did this for text tasks and we observed this across, you know, models that are more complex like RoBERTa and models that are less complex, I think DistilBERT or something like that. And so the kind of the spectrum held consistently and this trend held consistently. So I was pretty hopeful seeing that, which is that it is going in the right direction. We just need to figure out how to make the estimates better over time. And to your second point, I think basically the main way in which this helps is especially if you know for an application that certain slices will be important or will shift, then it's very useful. So for example, if you're, uh, you know, building and deploying a system and you anticipate shifts along certain attributes because you kind of understand the environment in which the system will be deployed. So for example, let's say, uh, your users have a certain, you know, makeup and it changes, uh, seasonally or something, or it changes over time, uh, you know, or you anticipate that that change will happen, then then you can actually use this because you can track very precisely those particular slices. And as soon as they shift, you will have— or Mandolin will, will kind of downweight or upweight depending on which direction they shift in. And so, you know, having the ability to kind of reduce the data dimensionality and focus on the attributes that are actually probably going to be important for this model's performance in the future is what allows it to work so well, um, in, in the settings we consider. And of course, like, uh, the main drawback is that you have to be able to understand a little bit more deeply how your domain is shifting, which I think can be somewhat unrealistic in a lot of applications. So, so that's where, you know, some of these automated methods are probably going to help.

Right. So this sounds, this sounds great because, you know, it's, it's a way to introduce domain knowledge evaluation criteria that you're using.

Exactly. Exactly. It gives you a little bit more control over the evaluations that you're doing, a little bit more control over your monitoring pipeline. And of course the automation comes in after that. You automate away a lot of that effort.

Amazing, amazing.

Yeah, thank you very much.

Hey Karan, one thing that kind of strikes me about, you know, slicing functions like you talked about in Mandolin, is that they seem very similar in some senses to the kind of like the guided corrections that you were talking about in the third M, maintenance, with Bootleg. I wonder, have you had any thoughts about connecting those, you connecting the slicing functions to some specific corrections that you might make or anything like that?

Yeah, yeah. So that's a great question. I think for sure. So for example, in the model patching work that I did not discuss, this is kind of— I would say some of the idea there is to actually use the fact that you know which subgroups are underperforming. So the setting there is you're doing classification and somebody tells you that a certain class has, let's say, a few subgroups and some of them are worse performing than others. And so you want to fix this problem. And, and so there we actually leverage the fact that we know these subgroups and we train or learn these data augmentations in order to fix the problem. So there we're very particular in targeting performance on certain subgroups of data and actually leveraging that in the, in the fixes that we do. So for example, what we do is we say, okay, uh, let's say you again like have some, um, you know, classifier where it has a subgroup imbalance across gender. So maybe it's performing poorly on men versus women, or performing poorly on women versus performing well on men, you can actually train some sort of model that actually adds more data to the women subgroup using data augmentation. And that improves performance because it tends to balance your data better. Because often a lot of these problems are caused by data balance issues in practical scenarios. And so you can use this sort of thing to add more data. And then when you train with that, you definitely get lifts in performance. And that's an example of a place where we were trying to target certain errors. And it was obviously also one of the motivations for actually doing robustness because there somebody told you that this is where the errors are. And so you need something that actually helps you discover that.

Well, that sounds really, really exciting. I love that model patching stuff. It seems really— as we kind of start getting closer to the end of the hour, you know, as you know, one of the things that we kind of like to ask our guests is kind of what's next? Like, what's next in the Robustness Gym pipeline? What's next? Kind of what are you most excited about working on next? If you could talk, you know, a bit about the future directions that you're currently—

Yeah, for sure.

Yeah.

Always excited to talk about the future directions. I think, I think, yeah, I'm pretty excited about, you know, for example, this summer spending time making a lot of improvements to Robustness Gym, doing a lot more in terms of helping users get more off the shelf out of it. So I think right now we've been kind of focused on understanding what abstractions actually make sense and doing some proof of concept studies that we saw in some of those papers. But next, what we want to really do is kind of make the whole process seamless, add more support for the kind of evaluations I think that you guys talked about, which is, you know, being able to automatically discover problems in your model and be able to actually use that information as part of these evaluation pipelines. And, and of course with Mosaic, I think the other angle that I think is very interesting is bridging the gap between text and vision and like all these other modalities. Because I think evaluation, at the end of the day, the, the larger issues in evaluation are not very different across modalities, but the kind of communities still are pretty different because they have, you know, their own kind of ways of writing code, their own libraries that they use, and and so on. So I think that's really the, the thing that's bottlenecking that, not necessarily anything technical or conceptual. So it's just a case of putting it all together and, and, and pushing it out. So that's what we're working on right now. But yeah, I'm pretty excited, uh, if for anybody watching, to, uh, to get more users now. And we're starting to push more on, on actually giving you kind of more off-the-shelf capabilities that you can just plug in, plug and play with. Um, so if you're, you know, interested in this in these general questions or want to use robustness gym, definitely feel free to reach out and we can, we can chat.

Um, but yeah, well, that seems to me like a good place to, to call it as any. Um, so as always, you know, well, we'll thank our speaker, which in this case is now Karan. Uh, thank you to the audience as well. Um, so thanks everyone for watching online. As always, if you go to our website mlsys.stanford.edu, you can to check out our future schedule and see all about our seminar. There you can also sign up for our mailing list, which, where you'll get, you know, nice non-spammy emails from us every week. Lastly, you know, subscribe to the channel, hit the bell icon to get notified when we go live. Next week we've got Koji Inoshiba and Jeroen Singer. So we're very excited to have them chat next week. And with that, we'll say goodbye to the audience.

Bye.