Getting in the Flow with Snorkel AI

These noisier sources of supervision can be much more scalable. I mean much faster to execute, easier to version control and iterate on than individual labels are. And if you can layer a number of these on top of each other and basically then let their votes be aggregated by an algorithm, one that we developed at Stanford, you now have the ability to get maybe not 100 perfect labels, but 100,000 pretty good labels. And it takes about the same amount of time. And as we've seen time and time again in recent years, the size of the data set seems to keep winning the day when it comes to getting high performance with these models.

Bandwidth for changelog is provided by Fastly. Learn more at fastly.com Our feature flags are powered by LaunchDarkly. Check them out at launchdarkly.com and we're hosted on Lino Cloud servers. Get $100 in hosting credit at lino.com/changelog this episode is brought to you by Digital Ocean Droplets Manage Kubernetes Manage Databases, Spaces Object Storage, Volume Block Storage Advanced networking like Virtual Private Clouds and Cloud firewalls Developer tooling like the Robust API and CLI to make sure you can interact with your infrastructure the way you want to. Digital Ocean is designed for developers and built for businesses. Join over 150,000 businesses that develop, manage and scale their applications with DigitalOcean. Head to Do Co Changelog to get started with a $100 credit again.Co Changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community and follow us on Twitter. We're Racticalai FM foreign.

Welcome to another episode of the Practical AI Podcast. My name is Chris Benson. I'm a Principal Emerging Technology Strategist with Lockheed Martin and unfortunately Daniel was not able to join us today, but I have a guest that I'm excited to talk to today. I have Braden Hancock, who is the co founder and Head of Technology at Snorkel AI. Welcome to the show Braden. How's it going today?

Thanks, Glad to be here. Doing well.

Well, I was wondering if you would start off telling us a bit about your own background and let us understand how you got to where you're at. And then I'm looking forward to asking you more about Snorkel AI.

Yeah, absolutely. So yeah, as you mentioned, I'm currently co founder and Head of Technology at Snorkel AI. The company's been around for about a year and a half now, maybe coming on two, and it's been a blast. Before that, I was a Stanford PhD student along with all the rest of my co founders. That's actually the origin story of our company, and that's what brought me to the Bay Area in the first place, was actually not in computer science. Originally, I came from mechanical engineering and just found myself consistently being drawn to machine learning and then finally saw the writing on the wall and made the jump myself going into grad school.

So I'm just curious, because we hear that kind of story a lot where people are coming in from an industry that you may or may not expect that to. When you were still doing mechanical engineering, what was the draw into machine learning for you at that point? I'm just curious what it was that started that process of sliding over.

Yeah, no, I completely agree with you. I think we see a lot of people make that shift, and it's perhaps not too surprising just given how rapidly the field is growing, that the people have to come from somewhere. But for me, I think part of it's just how much faster science is in computer science, the fact that you can iterate so much more quickly. An experiment can be run in seconds or minutes, certainly set up and run in a day, often for certain experiments. Compared to when you've got a mechanical rig and there are parts and one bad transistor, and the whole thing's kind of suspect. There's just so many more failure modes in such longer timeframes that I kept coming back to. I want to be able to answer questions quickly, and I could do that so much faster in cs.

And do you think that's going to be a situation that we see over and over again with people in various industries pulling in? We've seen a certain amount of that. I tease Daniel a lot. We end up talking to people that come from physics a lot, and one way or another, they found their way over. So do you think that's going to be very typical with mechanical engineers constantly finding the need to use machine learning to get their jobs done? And whether or not they jump over to the dark side or not, I'm.

Sure we'll continue to see plenty of people jump in from over there for different reasons probably, right? I think for a lot of people, you end up finding that the best way to do your job is to use machine learning, and then you realize, hey, this is actually a really cool tool. I think I'd like it to be more than Just a tool for me. And then you really lean in and start diving in a more permanent way, rather than just in sort of an applied sense.

Gotcha. So I'm just curious, what was your first experience as you started getting into machine learning before you made the full jump? What was that thing that was drawing you in? What kind of models were you doing? What was your tooling that made you think it might be time to make a shift?

You know, this maybe just shows how thick I am, that it actually goes all the way back to high school, that I was. That I had my first dabbling in machine learning and loved it and just didn't realize then that I should have just embraced it wholeheartedly from the get go. But there was a lucky break for me. There's an internship program for high school students near Wright Patterson Air Force Base in Ohio, where I grew up. And so I was on a project using matlab, of course, the lingua franca of mechanical engineers, not the python of machine learning engineers. But yeah. So the task that I was assigned to was using genetic algorithms to design better airfoils. So some non gradient based optimization. And I thought it was so cool that even after I lost my matlab license during the school year, I had to go back to high school. This was after my junior year. I used Excel and I had a separate tab for each generation of the genetic algorithm and tried to recreate it there because I was still, of course, a lousy programmer, but just thought the ideas were so neat.

Very cool. So is it? Genetic algorithms were what actually pulled you in. And going from that and thinking all the way to now. As a co founder at Snorkel AI, what was the crossover right there that got you to Snorkel AI?

Yeah, I'd say from the very beginning, one of the ideas that drew me in was there should be this different interface for getting things done, for transferring information from an expert into program that can now do work for you. And I think historically that's very imperative. Code very much like describe exactly what you want done step by step. And that was less interesting to me. Felt a little bit more like your job's just to translate from one language to another. But the cool thing about machine learning or AI in general, I think, is that you get more of a sense in the right setup of if you can tell me what's good, then I can find it. There's this better synergy between the human and the computer where now I can show you what I want, even if I don't know how. To get there. And you can get there where you here is the computer, of course. So I think that's the broader idea that it was really appealing to me all along the way that had me coming to machine learning. And then throughout my PhD, I kind of dove into that problem much more deeply of what really is the best interface for getting domain knowledge from an expert into a model. And those are themes that I explored for multiple years. That along with my co founders, ended up being what led us to snorkel and then snorkel AI.

Now, just to dive in there a little bit, was there a particular itch that you were scratching in that context that actually led to snorkel AI? Was there something you can relate where it's like, well guys, we got to solve this particular issue. This is something that we need to dive into. That might have been the specific genesis.

Yeah. So I'd say one thing that my PhD advisor was fantastic about was Chris Ray at Stanford. And he. I think it's very good at making sure that the problems you're solving actually will matter to people, actually solve real problems. And part of the way that you do that is by. On most papers we would try and have real world collaborators work with another company or research organization or government entity or something where we could make sure that this actually solves your problem. So people are more likely to care, this is likely going to stick and have a potential to make real impact. Very early on in my degree, as we were looking at what is the effective bottleneck for new machine learning applications, what is it that stops people from solving their problems quickly, as quickly as they'd like to, the realization came that that bottleneck is almost always the training data. You know, we saw kind of the writing on the wall. Deep learning was blossoming. Right about then we saw these super powerful models. Feature engineering is becoming a lot less necessary. A lot of that can be learned now. But with the one caveat of you can do all this if you have just mountains of perfectly labeled clean training data ready to go for your specific task. And that in reality never exists, of course. And so that I'd say was the real impetus for this line of work was this is what stops people. In academia, it's download the data set and then do something cool with it. But in industry, it's get the data. I mean, steps like 1 through 9 is where am I going to get my data and do I have enough of it and is it clean enough? And these annotators are doing the exact wrong thing. I can clarify the instructions is this good, now it's iterating and 80% of the work is making that training set after that, like pulling off some state of the art model in the open source and running that. That's the easy part.

Yeah. It's funny you would think of AI as, I think people outside our industry look at this and think we're doing this dark magic of AI and producing the model. But every time we talk to somebody, it's always trying to get set up to do that. It's getting to that, getting to the starting line of doing the actual modeling itself that people are struggling with. So tell us a bit about Snorkel AI. You know, how did that blossom out of this experience that as a co founder you were having and as well as what, you know, what the others were driven to do as well. And can you tell us a little bit about your co founders and just kind of how the whole thing got started?

Yeah. So we feel very lucky at Snorkel AI to have the founding team that we do. It's a little bit larger than you typically have. There are five of us. It's Chris Ray, who I mentioned was my PhD advisor, myself and then three other previous students. We were all sort of in the same cohort, Alex Ratner, Paroma Varma and Henry Ehrenberg. And all of us began at about the same time our grad school experience in picking up different projects. And all of us were just drawn to these ideas and ended up collaborating in almost every combination you can think of between the four of us on different papers through those years. And it was going to be in the beginning like, this is an interesting idea, let's run a quick experiment, pull up a jupyter notebook, test some of these ideas. And then it really seemed to work. And so then it became a workshop paper and then a full paper and eventually a best of paper and an open source project and then an open source ecosystem and other derivative projects and lots of collaborations. And we helped a few different organizations make sort of industry scale versions of this internally to really prove out the concept of paper with Google, for example, that we were able to publish. And by the time that we were at the end of our degrees, it was clear that there was just such a dramatic pull for this. I mean the ideas were very well validated at that point over probably 35 different peer reviewed publications, but maybe more importantly a whole bunch of different organizations that independently had seen success with these approaches, almost always from working with us to kind of help lead them through the process. And so we just learned so much through that time about what you would really need to take this proof of concept and make it something that could be repeatable and with a relatively low barrier to entry that doesn't require a room full of Stanford PhDs to make successful. And that's part of what motivated the company is the chance to now make this a fully supported enterprise, ready and able to be shared with a whole bunch of different industries and company sizes and in different work areas.

Before you dive into the specifics of the product and service offerings, could you talk about a little bit about what you did learn? Because with that opportunity to be doing the academic work and to progress through that over time and kind of have that insight before you ever actually start the, or the start the new company. Can you talk about what that learning process was like and what were some of the things that had a big impact specifically conceptually? And then from there I'd like to kind of go on into how that was realized in the company itself.

Yeah, absolutely. And I completely agree. I think it was a huge, huge advantage for us to have that, I mean, really much larger period of time than you would ever get as a startup to do the learning phase. Right. We were able to succeed and fail and try different variations and really push the boundaries and like intentionally try to find where does this fail? Because as an academic, that's the hat you wear is like, let's really suss out, let's do every ablation we can think of. Let's figure out does this work for text, does this work for video, does it work for, you know, very dependent in correlated data, does it work for, you know, the whole variety of the space that you can imagine we were able to test. And so it meant that by the time that we were building now the, you know, the air quotes final version, like the enterprise version, we were able to bring all these different learnings to bear as part of that design. So if I was trying to structure categorically the lessons that we learned, I think one of the big ones was interfaces. I think as a grad student supported open source project, you don't have a lot of time to polish up a front end for people, right. So it's in the form of a Python package. And if there are unit tests, you're lucky because that's of course we cared about that, but it's not necessarily in your incentive. Unit tests don't lead to papers. It just means you have some more stable development as you work. So it was fine and it worked well, and we of course did support it as much as we could, but I think one thing we did realize is we were writing a lot of the same code over and over again. There were certain things, templates for labeling functions. I think we'll talk more about those later. But you know, third party integrations or like just sort of patterns of sequences of steps that people would try that get lost if in like the forest of scripts and notebooks. Whereas if you can set up a properly structured interface in GUI as well as other access points, you can really dramatically improve the likelihood of success. So that's one category, the interfaces, I'd say.

Okay, what else did you learn along the way? Was interfaces the primary driver there or were there any other key lessons there?

Yeah, no, I mean interfaces was a big one, I'd say if I was grouping it into other areas, I'd say there was also infrastructure. There was like intuitions that we gained and baking those in and then sort of like user profiles or like interaction points. So I can say a word about each of those. On the infrastructure side, I think that one's fairly self explanatory. If you're going to have your as a company, if you're going to depend on a piece of software, you need it to have certain things like I mean basic security and logging, encryption and compatibility with the data formats that you care about and dependency management and parallelization. All these things that of course, of course you want in software you're going to depend on. But that again just aren't necessarily a part of research code that's meant to be more of a proof of concept.

Sure. Kind of making it real comes down to really kind of classical software development, things that you need in place to deploy remote software. And I think that comes back to a point that we, that we run into a lot on the show. And that is the fact that you can't really separate the AI from the software the AI is running in. And it sounds like y' all had a realization about that even before you got the organization launched.

Yeah, no, absolutely. I mean, I'd say another big piece of this is again, as an academic, you test often these ablations. You'll test a very specific problem and can the model learn what I need it to. But in the wild you often have actually just a problem you need to solve and you don't necessarily care how that's solved. You just want a high quality system. And so you typically, you don't just have this one model that's ready to go with the data that you care about that has an output that is exactly what you care about. There's always, it's a pipeline, right? You've got pre processing steps, you've got business logic, you're chaining together multiple models or multiple operators. Some are heuristic and some are machine learning based. And so this actually gets at one of the big differences, I'd say in terms of fundamental value out of the Snorkel Open Source versus Snorkel Flow, the business product now is that the latter is much more focused on building AI applications like an application that solves your problem from end to end, rather than just a point solution for a part of a pipeline that is making a training set or training a single model.

Gotcha. So Braden, just a moment ago you were talking about Snorkel Open Source and Snorkel Flow. And could you now define what each of those are and describe what the differences in the two are?

Yeah, absolutely. So if you go to snorkel.org, that's the website for the open source project that again began almost four years ago at Stanford and served as sort of our testing ground and proof of concept area for a lot of these ideas around can we basically change the interface to machine learning to be around programmatically creating and managing and iterating on training sets? And so that's what that is. It's pip installable, you can pull it down now. It's got 4000 something stars and it's used in a bunch of different projects. Snorkel Flow is offering now the primary product of Snorkel AI and that is based on and powered by that Snorkel Open source technology. But then it just sort of expands to much more. It is now a platform, not a library. It comes with some of those infrastructure improvements that I mentioned before. It also bakes in a whole lot of the intuitions that we gained from the years of using the open source. There are certain ways that you can guide the process in a systematic way to creating these programmatic training sets or improving them systematically, really completing the loop. So that at every stage of the way you have some sort of hint at what should I focus on next to improve the quality of my model or of my application. So that application Snorkel Flow is, as I mentioned, sorry, that platform is meant to be this much broader solution for supporting end to end pipelines, not just the data labeling part. Baking in a bunch of these best practices tips and tricks that we learned over the years of essentially writing the textbook on this new interface to Machine learning and then includes also some of those interfaces, like an integrated notebook environment for when you do want to do very low level custom one off stuff, but also some much higher level interfaces like those templates I mentioned for labeling functions. There are a number of ways where it can be a truly no code or very low code environment for subject matter experts who don't necessarily know how to whip out the Python and solve a problem, but do have a lot of knowledge that's relevant to solving a problem.

Gotcha. Actually, to dive a little bit deeper into both sides of that, let's start with the open source and build on that. What would be a typical use case where somebody would go to snorkel.org and do the PIP install, read the docs and what are you offering with that, and through those libraries, what's available, and then in a minute I'll obviously ask you the other side about, about taking it to that next level. But if you could kind of give us a sense of what the open source side experience is like, what the benefit of the libraries are, that'd be fantastic.

Yeah. So if you go actually to snorkel.org, there's a section that is tutorials and we walk through a number of different, fairly simple, right, but meant to be sort of instructive tutorials for different ways you could use the library. So often one of the most intuitive places to start with that is on text based problems. There also are a couple of demonstrations there for how to apply it to images. And then we've got research papers as well we can point people to for working with time series or video or things like that. But one very simple example, one that we actually rely on in our primary tutorial, just because it's very interpretable and almost everyone has the domain expertise necessary for it, is training a document classifier. In this case, we could say the document will be emails and you want to classify these as spam or not spam. One way you could do this in a sort of traditional machine learning setting is get a whole bunch of emails that are sort of raw and unlabeled, look at them one by one and label them as this one's spam, this one's not spam, that one's spamming. And eventually you'll have the thousands or tens of thousands or hundreds of thousands of emails that you need to train some very powerful deep learning model to do a great job. But when you do this process, if you've ever tried to label a data set, you do find that very Quickly there start to be certain things that you rely on to be efficient or that are basically the science to you for why you should label things a certain way. An easy example here might be lots of spam emails try and sell you prescription drugs, Right. So you may see the word Vicodin in an email, and that's pretty clear to you. This is not a valid business email. This is spam, and you can mark it as such. And you might eventually label over 100 emails that have the word Vicodin and all of them are spam for approximately that same reason. Among other things, there's other content in the email, but that's what tipped you off. And so if you could instead just one time say, and if you see the word Vicodin in the email, good chance that this is a little more likely to be spam rather than, we'll call it ham or not spam.

Right.

You could write that, apply that to hundreds of thousands of unlabeled data points and get in one fell swoop hundreds of labeled examples. And those labels may not be perfect. There may actually be a couple examples in there, some small portion where it actually was valid. Someone was asking, did you see where my fyke eviden was put? I'm not sure, I won't guess. But basically these noisier sources of supervision can be then much more scalable, much faster to execute, easier to version control and iterate on than individual labels are. And if you can layer a number of these on top of each other and basically then let their votes be aggregated by an algorithm, one that we developed at Stanford, you now have the ability to get maybe not 100 perfect labels, but 100,000 pretty good labels. And it takes about the same amount of time. And as we've seen time and time again in recent years, you know, the size of the data set seems to keep winning the day when it comes to getting high performance with these models.

Yeah. So essentially that open source library is helping you scale out your labeling so that you get to the meaningful thing, meaning that you're actually starting to create models faster. So a way to overcome that.

That's right, yeah. You can think of it as, you know. Yeah. Essentially as a way of building and managing training sets, you know, very, very quickly, often at a much higher sort of rate of production as well as just much larger magnitude.

So at what point, if you've been doing this for a while and you found that utility in the libraries and such, what is a typical scenario that you're finding with Customers where they do need to level up. Maybe they've used the open source software for a while, maybe they had already been doing it even prior to you creating the company, but now it's time you mentioned platform specifically. What is it that they are now facing that it's a clear step up and they need the enterprise approach at this point?

Yeah. So I'd say there are a number of different reasons for this and it's a little bit different which elements of the grab bag are most important for different customers. But I can list a few of those. One of the big ones is just the guidance. I think with the proof of concept library, the open source, over the years of using it, we knew what to look for. How accurate is accurate enough for a labeling function? How many do I need? How should I come up with ideas for what a valid labeling function could be? How could I integrate external resources that I may have, like a legacy model that I want to improve on or maybe an ontology that belongs to the business that has information in it and how should I integrate that? So there's a lot of what would otherwise be folk knowledge if you're using the open source that you just only get through experience that we've been able to really bake in and support in a native first class guided way in the platform and that's a big difference maker for a lot of people.

Gotcha. As we're talking here, I'm looking through your website and I went into the platform and I noticed that you're kind of segregating out the different processes with label and build, integrate and manage, train and deploy, analyze and monitor. Why that particular segregation? What is it that the platform brings to each of those capabilities? How are you guys envisioning this process and if you have any insight, what is separating that from other options that you may see in the marketplace?

Yeah. So I'd say that label and build is probably the piece of that pipeline that overlaps most with the open source in the sense that that's the area where you're going to write labeling functions and then likely aggregate these right into effectively training labels, confidence weighted labels for these unlabeled examples that you can now train on that manage and version piece. Up next. That speaks to when you have not just a one off project, when your goal is not just to fill a table in a paper, but really to build something that you have confidence in that you can come back to, that you can point to in the case of an audit or whatnot, there's extra value in Managing all these different artifacts, you've got often many applications that you care about, many teams working on it, many different, again, just artifacts that you create, whether that's models or training sets or sets of labeling functions. So there is an element here that's as well just the data management side of things and tracking and inversioning and supporting all of those types of workflows on the modeling side that is entirely unique to the platform. With respect to the open source that we have a bunch of industry standard modeling libraries integrated with the platform. So if you do want to train a Scikit Learn model. Sure. Or some of the hugging face transformers right there. Flare is another one. XGBoost so a lot of these libraries we've kind of unified behind a simple interface so that it can be a sort of push button experience to try out a number of different things in hyperparameter tune and whatnot. But with the goal really being of you'll find most of the time you'll get the biggest lift by actually improving the training set rather than the model. I guess that actually moves us onto the fourth part, which is analysis. We have a whole separate page with a bunch of different components that effectively take a look at how your model is currently performing and where it's making mistakes and why it might be making those mistakes, and then makes concrete recommendations for what to do next. And so in some cases it's yes, actually your training set looks pretty good. The learned labels that we're coming up with actually line up pretty well with ground truth. And so if you're making mistakes here, it's probably because it's your model now. So you need to try a more powerful model or hyperparameter tune a little bit differently. I think that's where a lot of machine learning practitioners naturally go immediately to the model on hyperparameter tuning, when in reality almost always the far larger air bucket is. There are whole swaths of your evaluation set that have no training set examples that look at all like them. They are basically just blind spots that your model has. And now in the platform you can go ahead and click on that air bucket, go look at those 20 or 100 or however many examples where none of your labeling functions are applying. So this is not reflected at all in your training set. And write some new supervision that will add effectively examples of that type to your training set so that the next model you train will know something about those types of examples and can improve upon them.

Sounds good. I'm also looking at some of the different solutions that you have that are listed from document classification, named entity recognition, information extraction. I'm kind of curious since as you're looking at this and you guys clearly found a gap in the marketplace from the perspective you were coming from, what makes your approach to each of these problems, because these are fairly classical problems, sentiment analysis and anomaly detection. What are some of the ways that you think you're adding value that you weren't finding out there? What is that special sauce to some degree that you guys were really looking to introduce into the marketplace with this platform?

Yeah, I think what really moves the needle is the fact that with this approach and with this platform, machine learning becomes just more practical, more systematic, more iterative. And so all of these different problem types you mentioned, different ones, I think on the website right now we mostly focus on the text based ones, but again we've seen these used successfully and we'll continue to build out the areas for applying this to other modalities as well. But this paradigm is really agnostic to the data modality and most problem types. Right at its heart it is a machine learning problem where you have a training set and you have a model. And when your model is making mistakes, it's often due to what is or isn't reflected clearly enough in your training set. And so for any of these problems, there are different types of labeling functions that you write for a classification problem versus an extraction problem or whatnot. But fundamentally, once you scrape off that top layer, it looks very similar. And so this platform really is meant to solve a wide variety of problem types and work in a whole bunch of different industries and verticals and whatnot. Because again, it's sort of under the hood. They're all relying on the same basic fundamental principles about how machine learning works. And it was with that in mind that we built the platform. Foreign.

Is the best way for you to directly support practical AI? Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and helps sustain our production of practical AI into the future. Simply follow the changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog it's better.

So that was a good introduction. I am curious though, earlier in the conversation you talked about some of the third party integrations and along with that I'm kind of thinking from a workflow standpoint. So, you know, could you describe a little bit about how you might integrate in with other tools that are, you know, widely used within this industry? What kind of integration do you have and how that kind of really helps the practitioner get through the process of modeling that they're trying to do?

Yep. One of the things that we learned from the Open Source project was the importance of having intuitive, natural, modular interfaces to different parts of this pipeline. The labels, certainly, the labeling functions as well, the models, all that. And so we kept that design principle very much in mind as we designed the platform. We've made sure that every step of the pipeline can be done either in the GUI or via an SDK that we provide. And so that means that you can write labeling functions via these nice GUI builders that we've got, or you can define completely arbitrary black box labeling functions via code in the notebook. Push those up and then they're treated the same way in the platform. Same thing with the training sets. You can create a training set and then go to the models page and identify the model that you want, set a few hyperparameters and train it there with a button. Or you can use the SDK to export your training set, train your own model, and then just re register the predictions, push them back up, just very lightweight. Assign certain uid, certain labels, and then use the analysis page to still guide you. It means that we're able to interact with a whole lot of different customer types and workflows that have different requirements. For some people, no, we really just need to use our proprietary model. We know that nothing works as well as this does. That's totally fine. At that point, you can pull things down from the platform and then push up the results when you're done. For other people, it's actually, we've got a lot of training labels already available from crowdworkers or it's just as a natural part of our product. We're always getting feedback that we can use, but we'd really like to be able to be systematic about how we patch up failure modes that we have. And so we want to use the platform, the analysis tooling especially, but maybe also the models. And so for them, they're able to start in that way. So really it's that any piece of this can be. That's the test we use for ourselves is can I complete an application in Snorkel Flow without ever opening up that tab of my browser? And the answer is yes, which makes it just sort of ultimately flexible, I guess, platform for integrating with other workflows. You may have Gotcha.

So you guys are, even though you're several years, given the work ahead of time, getting into the company you mentioned that you're about a year and a half into the company's existence, so which is pretty early in the lifetime of an organization recognizing that it takes time to get things out the door and stuff. What other gaps are you seeing in the industry that is, you know, more of that itching that you want to scratch, whether it be short term or longer term. What are you envisioning snorkel flow evolving into and what kinds of problems that you're not addressing today necessarily are you thinking about addressing for the future? What, when you guys are getting together and hanging out and talking about what ifs, what are some of those what ifs that you're willing to share?

Yeah, so a few different things come to mind. One of them is that as I mentioned, I guess a couple times, there are different modalities to consider. And the way that you write labeling functions over images is fundamentally different than the way that you write labeling functions for text. And so just given where the market pool was initially, we've started focusing on text. But we absolutely plan to bring in some of that other research we've done as time goes on over the coming months and years. I'd say, in addition, another area that's really interesting to us and where we would have this sort of unique leg up based on the approach that we're taking is the monitoring side of things. When you acknowledge that most applications are going to go deploy, it's not great. I've got my model now deploy it and set it and forget it. Test distributions change, the world shifts. People talk about different topics, different words get different meanings. Covid was not a part of the discussion a year ago and now is a huge part of the societal fabric of what gets talked about on social media. So the fact that you do very frequently need to iterate on your models, improve them, as well as just sort of, you'd like to know, preferably more than just a single number of the accuracy of my model, is that going up or down? It's really interesting to see what types of examples am I starting to get more right or more wrong? What subsets of my data are diverging basically from what they were when I was trained. And so what's really interesting is after you've written these labeling functions, they're essentially a whole bunch of different hooks into your data set. They each observe different slices of your data that have different common properties. And these could effectively become monitoring tools for you because you can now observe how those labeling functions increase or decrease in coverage over time when applied on the new Data that's streaming through your application and inform you when you can basically set up automated alerts showing you now is the time to go and update things or here's some suspicious activity going on based not just on did the number go up or down, but we're seeing movement in different parts of the space where your model is operating. Take a look. That maybe appeals more to the technical nerdy side of things, but I think it's a really interesting problem, one where you've got that information. Right. You have already identified for you these very interesting angles on your problem. And so why not use those to help guide the sort of post deployment life of a model?

I guess at this point, as you were answering that, I want to ask within the limits obviously of what you can share customers that you have, what are some of the really interesting things that you've seen customers doing with this that maybe particularly things that were outside of what you might have expected, expected the kinds of things, you know, we all have problems. Everybody in this industry has areas of focus that we're addressing. What are some of the things that made you surprised? And you know, people went, oh, okay, hadn't expected that to, to see that or just for plain cool, just something that someone's doing that just like, wow, I love having our platform involved in that.

Yeah. You know, two things that I've found personally very cool. One of them is the privacy preservation aspect of this approach. That was not necessarily a top priority or top of mind when we were developing these techniques at Stanford. It was often on problems where it's just, I'm trying to get a good result, I want high quality. How can I get high quality? But it's been really cool to see different companies that have the very desirable goal of we'd like to have our data being seen by fewer humans, we'd like to have fewer people reading your emails, fewer people seeing your medical or financial records. And how can we do that while not sacrificing the quality of our machine learning models? And so it's been really interesting to see them in working with them, coming up with these setups where now they can take a very small approved subset of the data to give them ideas for how to write labeling functions or to label like a test set to give them a sense of overall health quality. But then the vast majority of their data never gets seen by a human. Now they can take these programs they've developed to go label those automatically, use them to train a model and then get back just sort of the final weights of the model. And it's really neat to see, and I'd love to see that sort of thread continue because, I mean, not just for the privacy preservation standpoint, but also we keep seeing articles about the PTSD almost that you get as an annotator over these awful domains. I mean, you hear about some for social media. Yeah, there's some horrendousness, actually. Even during the Stanford days, we worked with DARPA on a project for human trafficking, where in their case it was more out of, I think, necessity of keeping up with a very rapidly moving environment where it's this adversarial setting. And so your training sets always losing its value because things are always changing. And so they needed to be able to create training sets very quickly, and they did with Snorkel, which was cool. But also conveniently, now there are that many fewer people who need to spend a day sitting in front of these awful human trafficking ads. So I think the privacy standpoint is very cool. I think another interesting application we've seen was there, we had one customer who, and I'll try to appropriately obfuscate here, but they had an application that was affected, we'll say, by Covid when you suddenly have the stock market plummeting and there are certain risks associated with that for different businesses. And we were in the middle of a POV engagement with them, sort of they're test running the product to see how it worked for them, and they came to us and said, okay, here's actually, this was not part of our scoped work, but this suddenly matters a lot to us. And our typical process would take about a month. Do you think you could help us? Could we try and use Snorkel to get some result faster? And so since it was very early on and we hadn't necessarily had a lot of time to train them yet using the platform, we said, sure, we've got some ideas, give us a sec. Threw three of us in a war room for the day, ordered some burritos and hacked away. And by the end of the day, we were able to extract the terms that they needed with over 99% accuracy on their application. And that was achievable with a model that was trained on, you know, tens of thousands of examples which we didn't need to label. We were able to quickly come up with what are the generalizable, you know, rules and principles here that we could use to create a training set to train a model that now, you know, can handle edge cases and things much better than these rules. And then get the high quality that they needed. So that sort of, you know, live action, the nerd saved the day, right? Kind of moment.

No, it's a good story.

Super cool to see.

You've raised several interesting points there, you know, one of which is the fact that in real life, as this technology is more pervasive, that these dynamic, ever changing data sets are a reality we have to contend with. I mean, are you seeing the industry getting more flexible at large in addition to, you know, obviously you guys are, you know, in terms of thinking about the fact that that's something that has to be accommodated, but I would expect that that is something that has to be addressed more and more. Do you have any, any insight into kind of or any thoughts into where we're going in terms of us moving along this curve from these, you know, static label data sets that we were talking about historically at the beginning of the conversation to this dynamic? You know, especially since COVID has struck the ever changing world on a day to day basis. You know, what, what's that trajectory look like and how are you guys preparing for that?

Yeah, so I think we're definitely seeing a increased awareness of some of these issues. I think a lot of companies are still trying to figure out how to address it in the right way. We see companies realizing that, you know, schema lock in is becoming this problem for us because real problems, you know, change, our objectives change, we learn more about the problem. What we thought was a positive or negative classification problem is actually positive, negative or neutral. And then all of our old labels are garbage now because we don't know where the neutrals are and the positives and negatives. So people are being burnt by some of these problems. And so I think that's part of the reason why we've had such early success with sort of inbound interest, more than we could even handle at first because people are aware now of some of the costs that come with machine learning. The promise of machine learning is very much being broadcast and how it's the future and solves a lot of problems. But there do end up being these very practical, I won't say necessarily limitations, but gotchas or costs really that you need to be aware of. So I see this reflected a little bit in the way that companies are starting to prioritize more, the ability to see the like, where did my model learn this? Like that auditability kind of.

Yes, we're kind of touching on kind of AI ethical issues. You've talked about auditability and privacy and such. Yeah, totally, totally. See that as you're kind of maturing your way through the process here.

Exactly. So yeah. So I mean they realize that that's important in a way that they maybe didn't before. I think they're also realizing just from an economic standpoint that there's this, that the training data is not a one time cost, this is a capital expenditure, this is an asset that loses value. There is a half life to these things. And so you start seeing these ongoing, regular cyclical budgets to get the training data even just for a single application. We need more data to train more models for more applications, but to keep this application fresh and alive.

It's a great insight right there.

Yeah, well, it's super interesting. And it changes the way that you account for the cost of different applications you might use. Right. Because there's a certain way that you maintain imperative, we'll call IT software 1.0. And there's a different way that you maintain this machine Learning based software 2.0 Way of Solving a problem. It's something people are learning. And I think that's part of all an interesting part of the conversations that we're having with different customers as they realize how this can maybe change the way that they approach their machine learning stack in general.

Okay, so I guess as we wind up here, I'm finally getting to ask what is always my favorite question anytime we're getting to talk to someone such as yourself and that is, you know, blank sheet of paper. What are you excited about right now? In the space of machine learning and AI, you know, what is the thing that has captured your imagination, whether it's work related or whether it's not work related and just something cool out there. What's got you going? That's the thing I'm really interested in tracking either on my own or through the company or whatever.

What's cool? That's a very good question.

There's a lot.

I know so many things. I'd say there are a number of areas that are super important, super hard, but super important. And I'm glad to see that they're getting the attention that they deserve. At least that we're trending in the right direction. And that stems around the privacy, the fairness, the bias, a lot of that. I think it's just super hard. If anyone says that they've got a solution to that problem, I'd be very dubious. But I think we are marching toward progress there. And that's something that I'm certainly going to watch with great interest and hope that we can be a part of the solution there. That's one piece. When I think maybe a little closer to my personal research agenda and history, a lot of that's centered around how you get signal from a person into a machine. A lot of my research through the years has been seeing how high up the stack can we go. Like there's this figure in my dissertation that compares basically the computer programming stack to like the machine learning stack. So, you know, computer programming computers run on these ones and zeros, right? They run on individual bytes and bits, but nobody writes ones and zeros code. You write in higher level interfaces like a C or even like a SQL or something that compile down sometimes multiple times into this low level code that you're then going to actually run on. And I'd say similarly, machine learning runs on individual labeled examples. That's how we train it, that's how we express information to it. But it feels fairly naive actually to one by one, write these ones and zeros, write these trues and falses on our individual examples. And so I think that there's a lot of really interesting things that can be done around higher level interfaces of expressing expertise that then in various automated or just sort of assisted ways can eventually result in the training sets that have the properties you need to actually communicate with your model. You know, use the compiler. Right. Essentially the optimization algorithm that's in place to transfer that information. So that's a fairly high level description, but I think there are interesting things yet to be done there.

Well, thank you for sharing that. I appreciate it. Braden, thank you so much for coming on to practical AI. It was a great conversation and looking forward to our next one. Looking forward to having you back sometime soon.

Absolutely. Thanks for having me.

Thank you.

This is our final episode of 2020. We're taking a couple weeks off to relax, re energize and gear up for the new year. If you want more practical AI goodness during the break, I personally recommend the Waymo episode with Drago Angelov. The one where Peter Wang drops a bunch of history and knowledge on the guys, and the Human Compatible AI episode with Stuart Russell. I'll link those three up in the show notes for you. Practical AI is hosted by Chris Benson and Daniel Whiteneck. It's produced by me, Jared Santo. And our music is brought to you by the Beat Freak Breakmaster Cylinder. We have awesome sponsors who get it. Thanks to Fastly Linode and launchdarkly for their support. That's our show. We'll talk to you again next year.

Sam.