Killer developer tools for machine learning

I feel like with deep learning your runs often take days, right? So you want to get a sense for if they're working right away and you want to really have a good record of everything that happened from like system metrics to like exactly what was the last commit that went into your code to. And you know, everyone has these like notebooks, you know, like in their Emacs or VS code or like maybe a Google Doc if you're really advanced, you know, where they're just kind of typing like, okay, here's all the things I did, you know, and here's what happens.

Or in file names, extra long file names.

Yeah, yeah, that's a really good point as well, is the fact that as you move through the different phases of your workflow and you know, you're kind of, you know, doing research in a notebook and then you need to deploy it as a software component stuff, there is still friction in the process in terms of moving through each of those very distinct phases of trying to get your workflow from the very, very beginning to the very, very end and deployed. And I think people still everyone kind of customizes that quite a lot. That's a. There's another opportunity out there for someone. Bandwidth for change log is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is.

Brought to you by Digital Ocean Droplets Manage Kubernetes Manage databases, spaces object storage.

Volume block storage Advanced networking like virtual private clouds and cloud firewalls Developer tooling with the robust and CLI to make sure you can interact with your infrastructure the way you want to. Digitalocean is designed for developers and built for businesses. Join over 150000 businesses that develop, manage and scale their applications with DigitalOcean. Head to dot co changelog to get started with a 100 credit again dot co changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changeog.com community and follow us on Twitter. We're Racticalai FM foreign.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson who is a principal emerging Strategy technology strategist at Lockheed Martin. Every time, Chris, I've got to get those words in the right order. It's an impressive title.

It's a terrible title. It's a mouthful. So, yeah, it's funny. I'll go ahead and make fun of him. I'm hoping he doesn't listen to this. My boss will be introducing me in a meeting and he can't even say it. And it just makes, you know, I'm like, no one's like, who's this guy that joined us?

Yeah. So, yeah, I mean, you could do an acronym that's like pet. Pet Strategist. That sounds also very animal friendly. You know, pet.

That's very applicable.

Yeah, Principle Emerging technologies.

Yeah, exactly. I love animals for anyone that. Yeah. So anyone who's listened for a while heard me carry on about it.

So how are things going for you, Chris? You excited about a crazy week in the US at least?

Yeah. So as we record this, it's the day before election day, so, you know, I guess everyone is holding their breath no matter what side of the aisle you're on, and we'll see what happens through the rest of the week and maybe beyond, depending on how things go. I'm remembering the election of 2000, and it went on months to get a resolution, so we'll see. It'll be interesting.

We'll see. Yep. Well, we decided that at least, you know, on my end, I wanted a break from anything political and a little bit of a chance to be practical, as we are here at Practical AI. So this week we've asked Lucas Biewald, who is founder and CEO at Weights and Biases, to join us and talk about a lot of the things that they're doing in terms of developer tools for ML and. And maybe some other stuff. So. Welcome, Lukas. It's great to have you with us.

Thanks so much. Great to be here.

Yeah. Could we maybe start by just having you give us a little bit of information about your background and how you got interested in AI and computing and ended up where you're at now?

Yeah, totally. I mean, I always was interested in AI. I mean, ever since I was a little kid writing software to play video games against me and. And then, you know, I wanted to go into academia and study AI and machine learning, and I got as far as a master's degree and did some research, and then I left to. To do more practical AI. So. Love the title of the. Of this. This podcast. This is. I mean, this is like, back in, like, 2003, 2004. And it was super fun because it was like, you know, the really early days of ML systems and, and deploying big ML stuff in production for Yahoo back when that was a super relevant company and then at some startups. And then I got really interested in data labeling, which you'll probably understand just because it felt like actually in research and in industry, it felt like the quality of the data labeling completely determined the quality of the ML systems deployed.

Oftentimes still does.

Yeah, totally. And so I started a company originally called Crowdflower that was. Changed its name to figure 8. We were really maybe the first data labeling company for ML practitioners. And I ran that for about 10 years. It sold to a company called Appen, which is a big Australian public company that also does data labeling. And then in that process I saw kind of deep learning take off and I saw suddenly all these applications, these very awesome practical applications across tons of industries. When I was first coming out of school, it sort of felt like all the jobs available was ranking ads or ranking search terms or going to go into Wall street and be in quant. And I mean those aren't horrible jobs, but I feel like the really practical stuff is more exciting to me. And so it felt like there's just so many real world applications happening. And kind of what I saw was the tooling just seemed nascent and it felt like a lot of the developer tools that we all take for granted don't quite translate into machine learning applications because it's inherently statistical. And I go on and on why. But basically they don't seem to translate. And you know it because you see ML practitioners not using a lot of software best practices. And so me and my same co founder from Figure 8, and actually a third co founder, we decided to start a company that was just focused on making great tools for ML practitioners.

Yeah, and we should say too. So you mentioned the name of this podcast, but you've also hosted a podcast with a pretty great name, Gradient descent, spelled D I S S E N T. So. So great work with the, with the naming there.

Excellent.

Yeah. So definitely check out, check out that podcast as well. We'll put the link in the show. Notes. Has it been interesting in terms of, you know, after kind of being a practitioner, being a founder for a while, has it been interesting having conversations on your podcast with other practitioners? Have there been sort of viewpoints that have surprised you or insights that you've gained that maybe you didn't expect to? As, you know, as you started, as you started having some conversations with people you know, in various parts of the.

Industry, you know, when you run a company as, especially as I think as a founder CEO, most of what you do is you talk to customers. It's a huge part of what you do. And so the podcast is really kind of just recording the conversations that I was having and giving me an excuse to kind of ask a little bit higher level questions to people. You know, we kind of get into the details really quick, you know, with the user. But we thought it would be fun to sort of step back a little bit and just have kind of open ended conversations, which you're kind of doing anyway when we're starting weights and biases because at first we weren't really sure what the first best tool would be to make. And so we had a lot of these and they were so much fun. We kind of thought maybe we'll record these and other people will find them useful because we were citing them a lot. But I mean, I would say it's been absolutely fascinating and it's been really fun to step way back and ask questions because you hear surprising stuff like I'll say Anantha Kancherla from Lyft was a longtime customer of weights and biases and he's a great guy, but he was basically like, ML is just the same as anything else. It's just a discipline of engineering. Which I thought was a bold statement from a guy running the level 5 team there. Or I remember Jeremy Howard was like, basically, python can't possibly be the future of machine learning, which is another kind of controversial topic. So it's funny, you ask people and usually they say the same stuff, but then every so often you get these gems and you really think, huh, interesting. And dig in.

Yeah, yeah. Statements that would provoke a lot of discussion at any conference.

Yeah, yeah, exactly.

It's like people just assume certain things like, oh, you're going to use Python. That's kind of what people do. But people really thinking about the foundations of those things and why we're using those things is really interesting. We've had those experiences here as well, for sure.

Cool.

So I'm actually want to take you back for a moment to your introductory bio a little bit and hit a particular point that you kind of went right over there and when you were talking about founding Figure 8, which used to be, I can't say it now, Crowdflower, I apologize. Which is really how I think of it because I've worked for several companies which have been your customers.

Oh, fantastic.

So there you go.

There you go.

I'M just kind of curious at that point in time. Relatively early, certainly based on where we are with machine learning today. And figure eight is a well known name in this industry and you have a lot of big customers and stuff. But going back to when you found it, at that moment in time, what was it that made you say, this is the thing I'm going to go do? What was the problem that you saw? How did you want to solve it? You know what, I'm kind of. If you can take yourself back to that year and tell us a little bit about what that moment was like.

Yeah, totally. Well, I think it started in my master's program in grad school and I was doing research on a thing called word sense disambiguation where you try to figure out if plant means the power plant or the living plant. Right. Kind of classic problem in nlp. And I spent all this time with a data set that's called WordNet is the ontology. But there's a data set of documents labeled with the meaning into WordNet. And it turns out word sense disambiguation is kind of tricky. It's a tricky thing to know really what the senses of a word should be. And I think my personal opinion is WordNet makes way too many senses. So they'll have 15 meanings of plant. Like, you know, there's sort of two main ones, like I said, you know, like plant, the living thing and the power plant. But they'll also have like, you know, plant a block into someone like in football, it's like its own meaning, which is like questionable and just all these kind of like.

Yeah, like specific or like very niche metaphorical.

I mean, it's almost like it's. It sounds so obvious, the senses of a word when you talk about those two. Yeah. But when you think about like planting a block or planting your foot into the ground, it actually gets a lot trickier of what meaning is different from what. And you kind of only learn that once you spent a long time with that ontology. Anyway, what happened was I was finding all these patterns in the dataset and I was really excited to publish a paper. And then I realized that actually what I was doing is not finding interesting patterns. I was basically figuring out who the annotators were. So I was doing this sort of cross document classification. I was realizing if you could look at the classifications in one part of the document, it can help you classifications the other part of the document. And I thought what I was doing was topic detection. But then actually what I realized I was doing is I was basically doing annotator detection, right? And it was so heartbreaking because I had spent months doing this research and it seemed really promising. And then suddenly I realized that I was just finding this artifact of the labeling. And the labeler shouldn't really matter for documents, but of course it does. And then so that was really like top of mind. And then when I was working at Yahoo, I was actually switching their search engine from kind of a rule based system to an ML system. And I was doing it in all these different languages. And it was funny because, like, each country that I'd go into, sometimes they'd be super happy with me and sometimes they would be like really pissed off. And it had nothing to do with me because I was doing the same algorithm in each country, but it was totally based on how seriously they took the labeling process. Right? So like some countries, I mean, I guess now I could probably name names like you said, like, China was totally not taking the label process seriously. They're just trying to do it as fast as possible. They're always like mislabeling. And so it's really hard to get a system out where like Japan was, was taking it very seriously because they had like, they had a kind of a. For historical reasons, they just had a bunch of laborers that really cared and they were labeling it much more carefully. And so it was much easier to launch something there. And I was sitting there like, hey, I'm the ML guy. I'm doing the same thing in each country. But you seem to have very different takes on it. And so it just made it so obvious that the labelers are kind of what matters. And so I sort of felt like if you want to make ML real, and that's something I've always really cared about, like, it's so exciting when you get something really deployed for a real application if you want to do that. Most of what you should spend your time on probably is the labeling process. I think at the time what was happening a lot was ML teams were kind of relying on some other function in the company that was already kind of doing a lot of data collection to do the labeling. And so the ML teams would be kind of divorced from the labeling process. And so I think what, you know, crowdflare at the time then became Figure eight. What it really did was it gave the ML practitioners control over the labeling process. Right? So it wasn't like an outsourcing firm that is really like finance using. And then you're sort of like, you know, kind of like sneaking your project like into their budget. But it's like now you actually are running a labeling process and you're really looking at the data and you're writing the instructions. And I really deeply believe that that's essential in almost every case for making a machine learning thing work in production.

Yeah, it almost seems like when you were doing your research work, you really found these issues with the foundational, the underlying data itself and the annotation process. And then like assuming you have some data to work with, it seems like the next sort of level up are these developer tools that were either non existent or really rough or not being used or something like that. And that seems like at the level of weights and biases, I'm curious if that's how your thought process went because it seems like a natural progression to me. But I'm not sure if after you had thought about the data, the data and annotation problems for a while you just, you know, your sort of next pain point or your next obvious layer in that stack was that layer up of developer tools.

Well, actually, I'll be a little more specific since we're on a long podcast. I normally kind of skip over these details, but honestly what happened really was I got kind of worried when I was running figure eight that I was getting out of date. Like, I remember at my first job there was a big fight between rule based systems and machine learning systems. And I remember kind of looking at the mid-30s guys or 40 something guys who had kind of invested a lot in rule based systems and they were kind of like, oh, this machine learning stuff is stupid. It's hard to scale, requires a lot of training data, hard to deploy. And I remember just looking at them thinking, you guys are idiots. You've just committed yourself to this one thing and you can't see that this new thing's better. Right? And I kind of found myself, I hadn't done a lot of, you know, training models in a long time. You know, I was running a company and I was kind of saying the same stuff about deep learning. Like, I was like, you know, like, is this really better? Like it seems really hard to deploy. And I was kind of looking in the mirror like, man, I am like out of date. Like, I don't know, I don't really know like what TensorFlow does. Like I pretend I do, but I haven't like messed with it. I was feeling really bad about that. And then I got myself in a, like an internship at OpenAI where I worked for like, you know, a really really smart 24 year old for a little while and I just was like, just make me do all your dirty work. I just want to kind of get back into doing something real. And it was kind of a shocking experience, I think, because I think the tooling had been bad for ML, but it's like strikingly, shockingly bad for deep learning. I think it just sort of exacerbates the problems even more. And so, you know, quickly, like, my entrepreneur brain is just like, I have to, I have to build tools to fix this up. And so I didn't last very long as an intern because I just really wanted to fix these underlying problems for everyone else. But I'd love to kind of go back. I mean, training models and doing research is so fun. I guess I just, I don't stick with it very well.

Before we go on from that point though, I just wanted to draw. I mean, it's very insightful of you to recognize that that thing that you had noticed before about, you know, the older, you know, contingent at the work and then recognizing that you were in danger of doing that yourself. And I know I'm definitely older than Daniel and have certainly found myself in the same position when I was younger, saying, oh, look at those older people that are doing it. And then when I got older, I did that. And I recognize, and I think that's a great lesson for people out there, is don't ever do that. Don't ever, no matter what age you're at, don't ever lock down on your technology. Don't ever be in love with it, because it's going to change. It's going to change and there's going to be a new best thing out there.

Yeah, I think that there's, there's definitely a balance. I mean, maybe I'm saying this because, like, I'm too, too stubborn, but I think there's definitely a balance because there's like the side of things where, like, if you're not passionate enough about something and, and stick with something long enough, it takes a good amount of time to like, really like, make a difference in a certain area. But then also, like, you have to be willing to sort of jump ship onto other things occasionally. So, yeah, I definitely think that there's a cycle with that and it's hard to do. It is hard to do.

It's hard to find that sweet spot of committing enough to becoming very good at what you're trying to get done at the moment and knowing when to shift. Yeah, it's not an automatic thing So I just want to draw that out. It was a great point you made there.

Yeah. I'm curious if you could go into like, we do like to get into the weeds sometimes here. But like in that state, when you were doing that internship with OpenAI and when you're talking about the tooling quote unquote, like what was your workflow like during that time in terms of training models and that sort of thing? Like, what were the tools at your disposal for you to use?

Totally. Yeah. I mean, it was funny actually because I also was like, you know, kind of working for a 24 year old or 25 year old. I felt really embarrassed that I was like using Emacs, which like, now that I play, that's kind of cool. But like I, you know, I was like, I'm going to use VS code. Like I'm just going to use the latest stuff. I'm not going to, you know, I can't be using these crap.

We're not offending any of the Emacs users out there. Right? Right.

I gave up my Emacs files pretty crap.

Now it's super, you know, it's hardcore.

You know, I know now people think it's hardcore, but come on, like, VS code is good. I don't know.

Yeah, VS code is good.

Yeah.

A lot of people have moved off of VIM and Emacs finally, you know, after resisting for years to VS code. I mean, it's made a big difference in the industry.

I strongly recommend it.

That's what I use. I moved away from the other stuff into that. So.

But it's actually, it's another one where, you know, you build up these like, you know, your, your Emacs file and you don't want to give it up.

In my hands.

You've been wasted so much.

Yeah, yeah, exactly.

Yeah.

But, you know, so OpenAI was unique in that they had a ton of compute resources, but at the time it was actually basic stuff. Like just trying to train across multiple GPUs was an enormous pain. I mean, even setting things up was painful. I think still is way more painful than it needs to be. Just like you buy a GPU at the store and then you want to train a model on it. I mean, expect to wait a day while you go through the latest iteration and then you have to find something that came out in the last three or four months or else the instructions are going to be wrong. And this is totally Nvidia's fault.

Look at it squarely there. But yeah, I mean the startup Workflow definitely has room for improvement. Absolutely. I found that as well overall.

And you'd love for those artifacts that like or implementations from like a year and a half ago. It seems like this is exactly what I want to do but like I haven't got to start over because it doesn't, you know, it's some compatibility issue or something.

Yeah, like I thought, you know, I thought I would never see like a linker error in my life, you know, after like 2006, you know, but like, you know, now you see them when you're just trying to, you know, get your situation set up. And you know, I think like collabs and stuff have really helped with this and made the onboarding smoother because even this is, you know, three, three years ago, maybe two, three years ago. So look at me, I'm out of date again. But this wasn't even the problem that I was trying to solve weights and biases initially. And actually the thing that, the thing that was really striking to me was just that you. It's really hard to go back and look at a run that you did in the past. And I mean this is something that's always been true like any kind of long running thing. But I feel like with deep learning your runs often take like days. Right. So you want to get a sense for if they're working right away and you want to really have a good record of everything that happened from system metrics to exactly what was the last commit that went into your code to. And everyone has these notebooks like in their Emacs or VS code or maybe a Google Doc if you're really advanced where they're just kind of typing like, okay, here's all the things I did and here's what happens.

Or in file names, extra long file names.

Yeah, yeah, that's a really good point as well, is the fact that as you move through the different phases of your workflow and you're kind of doing research in a notebook and then you need to deploy it as a software component stuff, there is still friction in the process in terms of moving through each of those very distinct phases of trying to get your workflow from the very, very beginning to the very, very end and deployed. And I think people still, everyone kind of customizes that quite a lot. There's another opportunity out there for someone.

I think there are a lot of opportunities right now in the tooling space for ML. I think it's like companies are really seriously doing it and I think every step is basically broken right from you Visualizing your data, to tracking your data, to tracking your experiments, to deciding gating deployments with continuous integration to production monitoring. I actually think all of these things are huge pain points. And I think there's going to be a lot of really interesting companies that spring up solving them in different ways, I think.

So I'm curious. You talked a little bit about like, you know, wanting to go into this internship to figure out, or once you started, you know, figuring out that there was this problem, like you had to start somewhere with like the tooling, like for a specific thing. Where. Where did you end up landing in terms of like the particular rough edges that you wanted to smooth off in the tooling? Where did you focus when you first started thinking about weights and biases?

It's funny. So the thing that we first started focusing on is now the thing that weights and biases is known for. I expected to do a couple different iterations and hit walls, but I think this is an example of the second time you do a company when you've been in the space for 10 years, you have better instincts about what the needs are. So the thing that, that I started with, which is really what weights and biases is known for, is essentially tracking, tracking your training. And it was basically like, you know, tensorboard was doing this, but it really only worked with tensorflow at the time. And it was hard to look at multiple runs, which is really what you want. And so, you know, they, they sort of have a way of doing, putting hyperparameters into the name of the run, which I think is kind of awful. I really don't recommend that. And then it's clearly not designed to show you runs in the context of other runs. The fact that your model has this accuracy after eight epics, it's kind of not that interesting. What's more interesting is how did the accuracy change from the previous thing you did and then also being able to dive in and look at exactly what was that previous thing you did that was the core of the beginning of weights and biases. And still, I think the thing that people appreciate most about us is saving all the metrics of a run, you know, all the performance metrics, the system metrics, and the sort of like accuracy loss and those kinds of performance metrics and being able to quickly compare them with lots of other runs.

Could you describe that a little bit about how you know that, how that contextual comparison is realized and how that's different from previous tools people might have used?

Totally. And I think tensorboard is the best comparison but you know, I mean, let's pick a concrete example. Like say you're trying to train a self driving car. So say you're doing perception of self driving car. So you're looking at these images and you're doing semantic segmentation where you want to label each pixel with what's in that pixel. Is it a pedestrian or is it a road or what's going on? So the first cut of what you can do is as the model trains, you can look at the loss function and whatever that loss function is, you can see if it's going up or down and hope it's going down, because it's what you're optimizing. And you might want to compare that to other runs you did, maybe with different hyperparameters or different state of the code. But say you have one with a higher learning rate and a lower learning rate. You want to see which is better right? Now what turns out the second you start to do this is you realize that better or worse is more complicated than you think, right? How do you define that exactly? So at first you're looking at the loss and you're like, oh great, this run has better loss. But then it actually might be that when you look at accuracy, which is a more actionable thing, it completely flips. So the lower learning rate has actually higher accuracy at the end. But then you might say, well, what is accuracy really? Like, maybe it's just predicting that everywhere is a road and just ignoring the pedestrians because there's not many pixels that correspond to a pedestrian. So now you say, you know what, I don't want to have Anything with under 99.9% pixel accuracy on pedestrians, right? And so now maybe you didn't check that the first couple times you, you change your model. So you want to go back and look at what those were. And so like, you know the requirements as you, as you do this in the real world, they keep changing as the businesses change and you realize, you know, that things are different. And so fundamentally what you know, the, what the kind of core weights and biases tool does is it tracks all this stuff for you and it shows it to you in graphs. But then why you might really want this is like, you know, six months from now, you've deployed this model into a car and the car crashes. Right? Now you really want to know, okay, what was that model trained on? What was happening in the training that made you think you could deploy it? Where could things have gone better? And if you don't do this in a systematic Way, like with weights and biases, you can imagine it's going to be really hard if you're just tracking notes in your Emacs file. It's probably not going to include all the detail that you wish you had. What we try to do is capture lots of stuff passively, right? And we think that when you capture everything passively and you don't put a burden on the person training, you get a lot more useful information, because usually you want this information after the fact.

And when you say passively, you're meaning you don't want a user to decide weights and biases. Save this parameter and save this parameter, but not these other things you want to enable or disable, rather than having a user click a bunch of boxes for all the things that they want to save. Is that what you mean?

Yeah. I'll give you an example. Just fundamentally, the way this works is you import our library, and then our library basically collects lots of system metrics and other metrics as it runs. Kind of similar to tensorboard, but we collect a lot more stuff. I'll give you an example of the passiveness of it. One thing you should probably do before a big training run is commit your code into git. And so we could just connect each training run you do to the commit shot. We do do that if you want us to. But one thing we found, because we're talking to people all the time, is most of the time people don't actually commit their code before it runs. So we capture not just the commit shot, but also a diff against the latest commit shot. So that way, every time you do a training run, we keep track for you the exact state of the code, the way it was when it trained. Right? And that way, a month later, you want to go back and look at something and be absolutely sure what the state of the code was. You can do that, but you never need to think about it. Once you turn that on one time now, you'll be sure that every single time your model trains, there's a snapshot of exactly the state of the code. And you might say, well, that's a waste of space. But come on, if you look at the size of training data sets, they're going to be petabytes. You could do this all day long, capture every metric you might care about, all the state of the code, everything going on, like you know exactly what kind of hardware it was on. You can capture all this without really making a dent into a modern machine's storage.

Would it be fair, as you say, that, you know, I know that certainly Daniel and I come from a developer background in addition to a data background and you know, something that we've talked about lots over time is kind of the maturity of DevOps relative to the maturity of Data Ops. And you're clearly, you know, as you just, as you were just explaining that, you know, you're clearly kind of addressing some of that, is that maybe the core participants in those two arenas are coming from fundamentally different places, do you think? Or those of us who have been developers prior to doing machine learning and we're kind of carrying some of that in and we try to put that in place. I mean, is this just different constituents that this arises from? And maybe a data scientist wasn't worried about Data Ops because it wasn't a mature enough field prior to this, but now it's, it's important. What's your take on that?

Yeah, I mean, look, I think like, I'll say a couple things on that. So first of all, I think data science trains you to be bad at DevOps, right? It's actually a different thing than engineering. And most of the code that you write as a, as a data scientist or an ML practitioner, most of it's throwaway. And I can really relate to this, right, Because I came from this world and I was kind of surprised when I got, I remember I got to Yahoo and I checked in some code that actually crashed search engine and they're like, man, you are moron.

That's a claim to fame though.

That is. That's a good one.

You can work that into conversation now.

Yeah, I mean, they should have put some more controls around me, honestly, but I just really like didn't. You don't get trained in like testing. And not only that, but what you do get trained is that like most of your stuff is throwaway. So you have this very, very 8020 rule of I just want to write a lot of fast code, kind of figure out what's going on and then like toss it over the fence, someone to kind of harden it. Right. I think that that breaks now that AI is getting used in so many mission critical applications, even applications where someone could get seriously hurt or really bad things could happen. And so I think the other thing that happens is that DevOps doesn't really understand machine learning and data science that well and they're not really prepared for the statistical nature of it. Right. So for example, continuous integration tools, they crash if any test fails. And that totally makes sense from a normal engineering perspective, but it's just not realistic to have a suite of continuous integration, like real world continuous integration tests for a machine learning thing. The machine learning stuff that we put into production, we know that it has a failure rate that's non zero. And so instead we have to deal with that versus trying to drive the failure rate down to exactly zero. And so I think there's a real culture gap and there's very, very few people trained in both DevOps and machine learning and those people make tons of money as I've seen. So it's cool when someone has both skills, but it's natural that people don't have both skills. And I think the skills in a way teach you very different styles of development.

As a two second follow up to that, as we get to where we're deploying out there and you're addressing it with the workflow that you guys have been creating, how do you address the use cases, the variability in the use cases in terms of if it is something that is more just kind of typical consumer then totally see your point. But there are use cases where you have these models that are going into, that are starting to go into life and death, you know, issues, you know, where people's safety is becoming dependent upon it and they are trained. 99% is not going to be acceptable. Is there a point there where you evaluate or cross over or acknowledge that in some way or how, whether now or in the future, how would you address that kind of variability in you're starting to get models in so many different real life situations that you have to account for that issue.

Well, I mean look, weights and biases mission is to support the ML practitioners to see what's going on. Right? So you know, we're not trying to tell you if 95% accuracy is good enough. What we're trying to be sure is that you know what you have and you know, if it's 95% or 99% and then you can also dig in and say okay, this is 99% on roads but only 8% on pedestrians. And that's, you know, got it, I should flag it. I also think like, I think it would be nice. It's hard to acknowledge but I think the industry does have to acknowledge that 100% accuracy in most applications is not attainable even in life or death ones. Right. So I think it's really hard to come out and say like look like you know, this self driving car is going to get it wrong, you know, 0.001% of the time or whatever. But you know, people Talk about miles between intervention and. It's not infinity. Miles between intervention and humans also, by the way, are not 100% accurate. So that's a little bit outside of the scope of what we try to do. But I think fundamentally, as we do more and more complicated things, it's better to acknowledge that errors are inevitable and have plans to deal with them than try to claim that there are no errors.

Gotcha.

I'm curious about a sort of different side of variability. Not so much on the performance of the actual model side, but as a package that someone imports into all sorts of different workflows. There's such variety now in terms of, like, the tooling that people are using for their ML workflows. And I see like on your website, you're talking about, you know, kind of integrating weights and biases into all sorts of workflows. TensorFlow, PyTorch, Keras, SciKit, learn, you know, hugging face, transformers, all sorts of things. You know, could you speak a little bit to that challenge in terms of, you know, I'm sure as a company and as a, you know, sort of product roadmap, it is extremely difficult to sort of understand and navigate, like, how tightly you're integrated with various things versus other things and how to reasonably support the variety of tooling that's out there in AI right now. How do you, in general, do you think about that and approach that?

Well, it's a conversation with their customers. So when you sign up for weights and biases, we really only ask you two or three questions. And one question is, how often do you train models? And if you answer that you train models frequently, then we take very seriously answer the next question, which is, what frameworks and other tools do you use? And so when we see something like JAX being used by lots of people that train their models frequently, we say, okay, we should prioritize a good JAX integration. And you're right, it's tricky. I mean, I think it's nice that you appreciate the work that we do. I mean, a lot of people look at what we do and it's just like we generate these graphs really quickly and making these graphs, I hope that it's kind of a simple experience for the user, but it's a tough challenge for engineers to make sure that those all kind of reliably generate in a way that's useful to our users, regardless of what framework. I mean, forget about framework, like Python version, also like PyTorch and TensorFlow, both incredibly cavalier about making breaking changes to their libraries. So it's definitely a challenge.

Yeah.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of practical AI into the future. Simply follow the Changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com changelog is better.

So, Lucas, I would be curious to hear a little bit. So we talked a lot about experiment tracking, which I know is like you say where you kind of started with this journey and maybe what weights and biases is most known for. But as I'm exploring the website and the feature set that you're supporting, there are some other things in particular I see, like sweeps, which I think is related to hyperparameter tuning, and then artifacts, which is related to data set versioning. I'm really curious about the latter one, actually. I'm really passionate about data versioning, actually. And so I was curious, you know, when this came about, it seems to me like maybe a couple of years ago, if I was talking about like data versioning, like people just like had no idea what I was talking about. And now there is this sort of acknowledgment that there's something here and maybe that's fed into the support of that and weights and biases. So, yeah, could you tell us a little bit about your support for that and also your thought process around data versioning and how you think about it?

Sure. I mean, I would say all the things that we add are 100% customer driven. So if you're a practitioner and you want us to make tools for you, come into our Slack channel and talk about what you're doing and tell us what you need. And we would love to make things to support that. So that's how all this stuff came about. I would say, you know, hyper parameter, we call it sweeps, hyper parameter search or hyperparameter sweeps. We built because everyone kept telling us, yeah, we know this is best practice. We know there's a lot of tools available. But when you ask them, okay, which of these tools are you using? How are you doing it? They all just be like, I'm kind of doing a random search with my own home grown stuff. And when I hear that as an entrepreneur making tools, it's like, okay, these tools aren't working for you, why aren't they working for you? And I think it's because the reality of hyperparameter search is messy than the theory of it Right. Like the math behind doing Bayesian optimization or things like that. It's not so complicated. What is complicated is the real world where you start to do it in a distributed way and six of your runs break and then you decide to change the code midway through your search. But you don't want to throw away real data. And so we've really tried to focus on that and kind of making it simple to do and simple to see what's going on. And in fact, you can use other optimization libraries that would even tell you maybe better than ours. Right. So you can use raytune, which I think is a fantastic library to do advanced stuff that you can't do with us. So that's how we think about that. And I would actually say data versioning is almost the same thought process where people know that they should be versioning their data. Right. They know that it's dangerous to not version their data. You know that you can't have real reproducibility unless you version your data. But I think that all the existing solutions were kind of causing problems. So I think, you know, Git has like the large file store system, which seems like it should be made for this kind of thing. But when we talked to practitioners, most of them weren't actually using that. And so we kind of dug into why and it's like, well, a lot of data sets these days are in object stores. So supporting an object store is incredibly important for versioning data, I think. And then another detail that shouldn't be overlooked is most ML practitioners are not super fluent with git, right? They find Git to be kind of a scary interface. DevOps people find it trivial. In fact, my co founder Sean comes from less of an ML background and I'd say more of a kind of DevOps background. And he just constantly makes fun of me for how bad I am at Git and how in his view, I just kind of type in random commands into Git until I like break everything and then like ask him to fix the stuff. But I think that was a little bit eye opening for him when he was designing the versioning because he's like, I have to really dumb this down to make something that like Lucas can understand and use. Yeah, I think that's a real thing, you know, I mean, you're representing your.

Customers in that way too though. I mean, there's something to be said for that.

Yeah, no, I mean, I remember this a couple years ago, our conference, New York, which they're going to be having another one here soon. Which Chris and I are going to lead a panel or moderate a panel on. I remember in that conference they had like a git workshop sort of deal and it was like one of the most popular. People were just so into it because there is this sort of like, yeah, I can create a repo and maybe make an initial commit. But then all this stuff happens and I have no idea and get scared, it gets scary.

And when stuff gets weird, it's just like, ugh, no, really.

I mean, if you haven't been doing professional software development for a while, Git is scary. It really is. I mean, I've seen a lot of people getting into the field and they struggle with it and stuff. So I totally. I'm very empathetic to people who are coming from a data science background and they're trying to figure out how to do this, either git or something similar to that. It's. It's a new concept. So there's a cultural barrier that has to be surmounted there.

Right? Right.

Yeah, definitely. So when you are kind of seeing, and I'm sure I was looking through your website there, there's lots of good stories and that sort of thing on there. But in terms of like success stories with weights and biases and like the workflow that people used, you know, could you give us like tease us with sort of example? I know I'm putting you on the spot, but in terms of, you know, maybe a team that was doing something, you know, rolling, trying to roll their own before, and then was able to switch over to some tooling, maybe a combination of what you offer and other things to accomplish something. Do you have one of those stories you could share?

Yeah, totally. I'll tell you one of my favorite ones because it's so practical and just so great. Although it's been told the application has been told a couple times. You may have heard it before, but I feel really proud of it. So this is a team at John Deere called Blue river, and they basically make a system that looks down at lettuce and tractors and if it sees weeds, it sprays the weeds. I believe it's with fertilizer. So it burns the weeds and then it becomes fertilizer for the lettuce. Right. So instead of blanketing the fields with pesticides, they aim a sprayer at the videos.

It's amazing.

Yeah.

And they did a whole case study with us, so I know I can go into detail and you could maybe post the case study where they actually show their weights and biases, runs and how they think about it. But you know, when, when we went in there, they were, you know, having the problem of they were training tons and tons of models and trying to keep track of which models they were making and what they were learning and then also kind of what was getting put into production. Right. Because it's kind of high. I mean, it's kind of high stakes. It's very high stakes. If you're a farmer, you can't have your crops get killed by your AI eating machine.

And a single single percentage over hundreds of thousands of acres is not trivial.

Yeah, yeah. So I mean, they're a super smart team, super technical, actually. I remember when we went in there, we had this funny feature where we would try to pull up a web browser to do authentication and they actually had links installed on their system. Do you remember this? It's like a Unix terminal based web browser that it then popped up and I thought, oh man, there's like a bug. Yeah. Anyway, it was kind of a rough initial meeting with them, but they got it working and so then the whole team can look at what they're doing. And I should say maybe the most important thing for them is that each unit of work, kind of each thing that they try, they put it into a report which they attach to a pull request. And so they can go and they can look at kind of. And it's not like necessarily one run, right. It might be like, okay, I tried this new thing and I tried 10 different sets of hyperparameters. Like I tried a new data set. Like maybe there's a new camera that picked up like UV light. And then I tried these 10 hyperparameters with it. And here's what I concluded. Like, the camera was good, we should explore it further. And so they'll actually kind of write that all into a report and then attach it. And they actually put an example report that they gave to us to use for a case study. So it's really cool to look at. You can see they're trying different types of gradient descent on their data set. So that was like the initial case was them using it to track their experiments like that. But then they started to use it to look at their data sets too, because they have so many different data sets and it's constantly being collected in these fields. So they use our versioning system and they attach it to our experiment tracking system to really know kind of end to end reproducibility of the stuff that they're deploying into tractors.

Pretty cool. I guess as we start Winding up. If you could wax poetic a little bit for us for a moment about kind of what you are aspiring to in terms of future of tooling. What are some of the things in your head that you have this vision for you would like to move into? You're not there yet, but you realize what a big impact it would make. I'd love to see kind of where the future of tools tooling is going because I know that will directly affect both Daniel and my lives. And so any insight there?

Totally. I mean, what I love to do is make simple tools that really help the day to day lives of people trying to do practical AI. So I would say the thing that we haven't done that I'm kind of dying to do, but it's a huge undertaking is stuff around production monitoring. You know, we do have some stuff today or we have a lot of customers use us implicitly for continuous integration, but we don't support that case explicitly yet. And I think that that's a big deal to. Because you know, AI safety is so important and I really believe that all the ML practitioners are good people that care more than society gives them credit for. AI safety actually just think it's really hard. And so I think, I think these ethical issues are real, but they're hard. And I think actually tooling can help here by showing people what's going on. And, and yeah, it's like developers, when you, when you deploy a bad, a bad piece of code, right? It's like, you know, you should improve the systems. I think a lot of the tooling is missing for ML practitioners and so it's dangerous, right? So continuous integration is a big one. You know, production modeling, things, detecting things like model drift or data drift and things like that, you know, I think makes me nervous. Should make anyone nervous, I think. You know, Andrej Karpathy had this really evocative talk he gave at one of Figure 8's conferences around sort of like a new IDE for ML that I keep kind of kicking around in my head where it's like, what would that really mean? Right? To really be able to tweak all the things that go into the model. Not just the hyperparameters in the code, but maybe even the data that flows into your model. So those are kind of ideas I have. But these ideas always take forever to actually get into the world. So I don't know what we'll get to.

That's what we're looking for. Not today, but down the road. What are people aspiring for today? That they might work on in the future without commitment there.

Just.

It's always interesting to hear that.

Yeah, well, definitely link to the website of Weights and Biases and some articles and other things and of course your awesome podcast which people can find and listen to. I definitely encourage people to do that. It sounds like Weights and Biases has a Slack community that we can link to where you can connect with them. If you're having trouble finding that, you can always reach out to us as well on our Slack community and find that@changelog.com community or on LinkedIn or Twitter or anywhere. But really appreciate the insight today, Lucas. It's been a real joy to talk to you and I'm looking forward to all the great things that will be coming from Weights and Biases. So thank you so much.

Thanks so much. That was fun. If you enjoy practical AI, we would enjoy a 5 star review on Apple Podcasts, a blog post in response to something said on the show and or a recommendation to a friend or colleague. Those word of mouth recommendations really do make a difference. Practical AI is hosted by Chris Benson and Daniel Whitenack. It is produced by Jared Santo with music by the mysterious Breakmaster Cylinder. Thanks again to our partners who support this show's existence. Shout out to Fastly Linode and Robar. That's all we have for you today. We'll talk to you again next week. Sorry, I think you're frozen. Can you, can you hear me still?

Yeah, we can hear.

Just totally still. Yeah. Amazing.

That's the thing about the Zoom meeting.

I'm like that. I'm just listening.

We're just on the edge of our seats, you know.

Oh good. Okay, you had me wrapped.

Riveting story. I'll try to wrap it up.

I'm gonna like tick tock a little bit just to make sure that you see, I was, was like really into your story as I was sitting still and listening. The audience can't. Yeah, we're watching each other on Zoom. We can see each other's facial expressions. So it's not just audio for us as we're having it. And, and yeah, I think with Daniel and I were really, really still.