Escaping the "dark ages" of AI infrastructure

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@robar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean. Guess what? Digital Ocean recently added MySQL and Redis to their list of managed databases. Their full managed databases lineup now includes the three most popular databases out there for developers, Postgres, MySQL and Red. Eliminate the complexity involved in managing, scaling and securing your database infrastructure and instead get back to focusing on building value for your users. Learn more and get started for free with a $50 credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence pract, practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practicalai FM. And now onto the show.

Welcome to another episode of the Practical AI Podcast. My name is Chris Benson. I'm a principal AI strategist at Lockheed Martin and with me, as always, is my co host Daniel Whitenack, who is a data science with SIL International. How's it going today, Daniel?

It's going really good. It's Thanksgiving week here in the States for those that are listening from the States, so a little bit shorter week. I'm working through Wednesday, so it's a good week and feel like I've been reasonably productive. So what about you?

Same for me. I'm working through Wednesday, but I'm looking forward to having a long weekend ahead. Do you have any special plans for Thanksgiving?

Well, just Thanksgiving dinner, but then I'm going to help out my wife who has a candle business and Cyber Monday weekend is pretty insane for them. It's a company called Antique Candle Company and they're going to ship out a kind of insane number of orders. So I'll probably be packing boxes with candles, which will be a nice, you know, break from staring at a screen and something completely different.

So no AI in that one?

Not as of yet, although I mean, so I help them with some like marketing and Facebook ad stuff. And obviously in advertising it is interesting to kind of come from from the AI perspective because you see certain things like in Facebook ads where it's talking about, you know, optimization and learning. As you kind of kick off the ad there's like a learning phase where it's kind of figuring out how to optimize the placement, the placements and the. And the cost and all that. And so it's interesting to think about it from that perspective. For sure.

Fantastic. Well, we need to include a link to your wife's business in the show notes.

I certainly will. Shameless plug.

There you go. In case there are any AI people who want to jump into candles. So I guess for me, I'm just taking a breather. Next week, I'm at Carnegie Mellon University for an AI conference to do a panel.

Yeah, that sounds great.

Yeah. And then finishing up the week with two things in a row. I'll be in Philadelphia doing an AI and ethics talk as a keynote at an ethics conference. And then I'm finishing Friday night in Austin where the final Alpha pilot, which we've had an episode on the world championship race, will be there. And at the end of that race that evening, we're going to hand out a $1 million check to the winner.

Exciting stuff.

Pretty big deal. If anyone's interested in hearing more about that, we have an Alpha pilot episode from not long ago, and you're welcome to tune into that. But turning to today, we have a fantastic guest. We have Evan Sparks, who is the co founder and CEO at Determined AI. Evan, welcome to the show.

Thanks so much for having me, guys. It's a pleasure to be speaking with you today.

Pleasure to have you on the show. If you could just kind of start us off giving us a little bit of background about yourself, kind of how you got to where you're at at this point before we dive into Determined AI.

Yeah, absolutely. So as it pertains to my sort of career around machine learning and AI, I really kind of got my start in that space kind of fresh out of college in quantitative finance. So this is the mid 2000s. I was working for an asset manager based in Boston, where we were doing applied machine learning to the stock market to pick stocks and trade client portfolios.

When I was in, I did my PhD in physics, and this was like before the sort of data science hype. But the rumor I always heard for, like, people that got out of academia was there was like, oh, you can go do all this cool math stuff in finance, but I never quite figured out how to do that.

Yeah, I mean, there definitely it was a very common career path. And it's funny, you know, probably a few years later, everybody then went into ad tech or something like that. And, you know, now it's probably autonomous Vehicles or something. There's always an interesting corner in a hot area to be, to be doing this stuff, which is one of the things that I find super fascinating. So the few years of that quant finance thing found that other people really, you know, liked looking at PE ratios all day. And that wasn't for me. I was much more interested in sort of the technology problems we were solving. Ended up going to work for a startup in the, in the NLP space called Recorded Future. So we were, you know, building, taking the web and throwing it through this massive NLP engine and building kind of structured data products based on it and trying to figure out how we sell that kind of structured data to places like trading firms, but also like the federal government and so on. And ultimately that company sort of found a good niche in threat intelligence, basically trying to build predictive indicators of where cyber attacks are going to happen and so on, but again with the same kind of data driven sort of machine learning technology. And so in many ways the roles were pretty similar, one being kind of in financial services, but the other being in this kind of totally different startup kind of environment, but always building models and kind of driving kind of forward data products. And in both cases though, I found I was spending much more time building and maintaining my own infrastructure than I was kind of worrying about the modeling problems. And it was really the case in those days. This is kind of like figure around 2010, 2012, as Hadoop was kind of becoming popular and so on, where as soon as I was tasked with analyzing a data set that didn't fit in memory on my laptop, my world just collapsed and you were forced to figure out how to write MapReduce jobs and so on. And I took that as kind of a good signal to go back and invent the world that I wanted to live, live in in grad school. So I had the good fortune to join the AMP lab at UC Berkeley right around the time that Apache Spark was born. And my co founder at Determined AI, Amit Talwalkar, and I got to work kind of right away building out kind of the machine learning ecosystem around Spark. So we were among the designers and initial contributors to mllib, which is the standard library for machine learning in that ecosystem. And the rest of my PhD was really kind of focused on how do we build sort of how do we give people tools to build end to end machine learning applications and optim them in a large scale and sort of distributed fashion.

So this is a slightly less formal question, but it must have been a perfect fit in Terms of working on Spark and being named Sparks, I'm assuming Spark was not named for Evan Sparks.

No, absolutely not. It was funny. I sat next to mates Zaharia, who was the creator of Spark in the lab. Spark was around 0.3 when I landed in the lab and the first few days we sat next to each other, there were these kind of long weird stares going on back and forth until finally we kind of broke the ice and made a joke about it. But yeah, it was kind of fortunate coincidence from my perspective. I guess there was a long running joke that my real name was Evan Apache Sparks, but not so much. So yeah, it was good timing and honestly the AMP Lab was a great place to be for what I wanted to study, which is really thinking about where does this intersection of huge volumes of data and machine learning really get real and how do we build out kind of supporting systems to enable this. And so also while at Berkeley I met my other co founder at Determined AI Neal Conway, who's more kind of from the pure distributed systems kind of part of the world. So he'd been a postgres committer. He was working at kind of core Apache mesos for a while around kind of distributed resource management. Meanwhile, Amit on the other hand, is more kind of dyed in the wool theoretical ML student and he's now a professor at CMU in the machine learning department. In some ways you think of me as the person who takes the what those guys do individually, figures out how to mash them together and then hopefully can figure out how to build interesting applications on top of kind of that intersection of system and machine learning. So while at Berkeley, and I promise this is getting into what we do here at Determined, while at Berkeley, one of the big things that we saw, the big megatrends that was happening within academia first was this shift to deep learning as a primary way that people wanted to be doing machine learning, particularly in industrial settings. And so it started with computer vision and speech. And obviously more recently we've seen amazing advances in things like NLP and text. This meant people retooling, learning how to use tools like TensorFlow, buying GPUs en masse, figuring out how to take this, what had been like a tiny corner of academic machine learning and really kind of make it into an industrially viable technology and stubbing their toe on a lot of serious problems along the way. Right. So you go from, from logistic regression that trains on my Spark cluster in a couple of minutes to big week long training runs for large scale image classifiers on a massive cluster of GPUs, for example, you start to have a lot of design decision baked into your modeling choices that you didn't have before. Things like just how many layers should this architecture have, how does the model capacity relate to my training data set, and so on in ways that are not really intuitive and end up being really highly empirical. So we saw that. And we also saw that the frameworks, the TensorFlow and the PyTorch and so on of the world are really good at sort of their individual task they're tasked with, which is helping you describe what your model is and get it training on, say a single or maybe several GPUs on a machine, but really bad at helping model developers through the rest of the workflow associated with getting one of these applications into production. Stuff that you guys have covered on your show before, around data labeling and so on. Do any of that at Determined AI. But there are other pieces of the workflow around hyperparameter optimization, architecture search, getting your models to train really fast across a wide variety of different hardware platforms, dynamically managing resources in the cloud, say, so that you can pay for the GPUs only while you're really using them. All of that stuff is sort of handled right now on a manual basis, honestly, with bash scripts and duct tape in many cases. And people don't really have a good way to support their more general workflows as they're in this model development process. And so at Determined AI, that's really the gap that we serve to fill. How do we enable you to do the rest of the pieces of your workflow while still using the tools that you know and love, your TensorFlow, your Pytorch, your Keras, et cetera, but make you much more productive as kind of an individual engineer, but more importantly as a team of engineers. How do you share your results in a reproducible fashion? And how do you make sure that I can get the same model out of my infrastructure as you do? And so at Determined AI, that's really what our mission is.

Yeah. So I'm curious to dive into a few pieces of that, but you mentioned in one of the blog posts on Determined AI about people still kind of living in the dark age of AI infrastructure, where certain larger companies have built sophisticated AI native infrastructure for their own use, but everybody else is kind of struggling. I'm curious if that sort of dark age that you're seeing is due to the fact that, like you say, that there's all these other pieces of the AI Workflow that might be data pre processing, model deployment, model optimization, all of these other things. Data labeling, like you mentioned, is it that the tools for those other pieces of the workflow are not. There aren't good tools for those, or is it that they don't play well together in a sort of all in one workflow? Or just that people haven't had enough time to develop standardized methodologies around these things? What do you see as kind of the main contributor there?

Yeah, I think it's a little bit of both. So I think that you hit the nail on the head. In many cases there are individual tools and point solutions to some of the problems that you mentioned. There are toolkits for model compression, there are services and open source libraries for just hyperparameter optimization and so on. Even sometimes whole companies built around these things. But in our view, what ends up being a result of that is that you get these tools that are isolated and aren't designed to work well with one another. More importantly, you then miss broader opportunities that might exist around optimizing the entire workflow. If you can step back and look at that rather than individually, like how do I make this particular piece of the puzzle go? Absolutely as fast as possible? Sure, you eliminate that bottleneck, but you might still be completely bottlenecked on ETL or data collection or training time, for example. And so you have to be careful as an organization about where you're investing your time and your resources in terms of making those things better. We think that a more holistic design, that is one where the pieces are kind of designed and know about each other, opens the door for certain types of optimizations. So to give you an example, we have our resource manager that is built into our product at Determined AI is totally AI aware. It's aware of the fact that what you're doing with running your jobs on our system, all of the jobs are somehow related to training or running inference on deep learning models. And you can start to make a bunch of interesting assumptions about the workload that you couldn't if this was just general purpose compute. For example, the idea that these things are iterative and that they have intermediate state like model weights and state of the optimizer that can be used to sort of checkpoint and understand where the computation was and then reschedule it, say to run on another device. Now we have that kind of design in the resource management section. But then when we're designing our hyperparameter tuning algorithms, for example, and implementing them, we can take full advantage of knowing what that internal scheduling layer looks like and use properties of that scheduling layer that we couldn't if we were just running this as like a black box job on top of something like Spark or Kubernetes or whatever. And that power of sort of these components being designed with one another in mind allows us to do this job much more efficiently in a much more fault tolerant and resource aware kind of way than we would be able to otherwise. If you're spending 90% of your time kind of starting up the cluster and getting it done, that's a lot of wasted cycles for your GPUs that your data scientists really want to be putting to work, you know, finding good models and solving your problems.

So I'm curious, you mentioned kind of a more holistic view of AI infrastructure and I know that something that can happen because there are so many pieces to this that you can end up in with scenarios and companies where you have a data engineering team or something that's in charge of all of this, like pre processing and getting data sets ready. And then you have like the modeling group and then you have like deployment and app people, app integration group people. Do you see that trend disappearing as, as things are kind of tighter, more tightly and better integrated together, or do you think it's reasonable that a data scientist could take something all the way through that whole cycle? I guess could and should. Should they be doing that?

I think it depends on the company and kind of the scale of the application that's under development. For example, if you're building a self driving car, that's probably not a job for a single data scientist, I would really hope. Right? You know, that's, it's sort of call it a generational moonshot, if you will, there. It makes perfect sense that you're going to have massive team of people just worrying about data labeling and data ingest and etl. Another set of people just working on the perception pieces of the job. Another set of people just working on maybe a different component around path planning and so on There in those scenarios you really want to think about, okay, what are the various teams and what are the Personas of the users of a broader machine learning platform? What do they care about and how do we facilitate coordination and communication between those teams? In other cases, companies have done a really good job of cleaning up their data, putting it into massive data warehouses and making it, even making the feature catalog, say self serve and the kind of thing where a data scientist who says, hey, I'm looking for a fraud model for mobile purchases in Southeast Asia, We've decided we're losing enough money on that particular area that a specialized model on this particular part of the world makes sense in those cases. I do think that proper infrastructure can enable a data scientist to go from start to finish all the way. And ideally you want to get that person to the point where they don't have to work directly with a data engineer to get the features flowing through the system and so on. And in my view, almost more importantly or place that we see people get tripped up is around deployment and monitoring of those models. We see people often taking models that are built in PyTorch or TensorFlow or whatever and completely rewriting these things in C or Scala or whatever, because that's what fits into the production serving environment. That side of things. We see these deployment engineers. That's a job I would love to see go away. In the common case. If the infrastructure gets better, you want data scientists to be able to get to the point where they're confident that it works well enough on test sets and maybe even start to a b test it and then, you know, hit a button and deploy it more broadly to the fleet. Reducing that friction in that time is definitely kind of a central thing I think we need as an industry in order to make these technologies more viable and successful.

Foreign.

If you like this show, I bet you'd enjoy listening to go time. Not working with go. Don't hit that fast forward button quite yet. GoTime covers a wide range of topics including cloud infrastructure, distributed systems, microservices, kubernetes and Docker. Not only that, but they have a ton of fun doing it. Listen to this clip from a recent episode on on Security. I'm very excited about this.

Before we start, I'd like to just try an experiment.

This is a security podcast.

I just want to try something.

Bear with me.

Hey Siri, Play Never Gonna Give youe.

Up by Rick Astley okay.

Google Play Never Gonna Give youe Up.

By Rick Astley Alexa Play Never Gonna Give youe Up by Rick Astley so I just want to see if that does hack anyone's home devices. And please let me know in the Slack channel or on Twitter if it does.

Do you just hack yourself?

I just hacked myself. It's not hacking if you hack yourself, is it?

Go time is a riot. Check it out@changelog.com go time or subscribe to our master feed@changelog.com master and let your podcast client download all the shows that we produce.

Then you can pick and choose the.

Ones you're Interested in the most and skip the rest. What have you got to lose? All right, back to the show.

So, Evan, I'd like to ask what are some of the unique challenges that are related to team interactions that you're seeing in terms of sharing data sharing, GPUs and other aspects of jointly utilizing AI infrastructure? Could you speak to some of those challenges for us?

Yeah, I think from our perspective. So the data piece is one that every organization faces, particularly organizations who are dealing with sensitive data. And that is something that we've seen users kind of figure out on their own. They have a version role based access control system on their primary stores of data, at least the interesting data, oftentimes the data that contains PII and that sort of thing. They really tightly regulate who gets access to those resources and those data resources and when, as they should. From our perspective, it's really about integrating with those various kinds of authentication mechanisms and supporting security on those data stores. So we do that out of the box. The second and third pieces I think that are harder for organizations that most people don't really have an answer for are first sort of resource sharing. So the rude awakening that many people get into with GPUs in general is they're really expensive and you're talking about spending upwards of 150k on say a DGX1, which is one of Nvidia's latest servers filled with V1 hundreds. And one of those might be good for two data scientists. But in order to enable your team to really be productive, you need several of those kinds of servers. We see people doing really immature things with these systems. We see people managing them with either like static allocation, meaning Joe gets GPUs 1 through 4 on this box and Kyle gets 5 through 8 kind of forevermore, or they've got like some kind of Google Calendar system set up. And this is some really sophisticated organizations that we run into where that's the way they're managing this expensive resource.

Do you think that's just because of like the mixed background of people working on this sort of technology, that a lot of people are coming from science or maybe non computer science or non software engineering background, or do you think it's more than that?

Yeah, totally. I think that is. So that's a big piece of it. And honestly, people who are really good at thinking about convex holes and the right shape of your loss function and so on probably shouldn't be wasting their time honestly thinking about the right way to do resource management. That problem has been solved in a bunch of different domains and that should be a layer of abstraction and that's one that we provide folks. There are other solutions to this problem as well. Some of the cluster resource managers that I mentioned earlier, like Kubernetes or we see people using queuing systems like Slurm from the HPC world, those things all have their drawbacks. But in general this is a problem that modelers don't want to be thinking about. And more generally I think we need better abstractions for these folks.

So that's certainly a challenge. I mean I've been at two large organizations, one that I'm still at Lockheed Martin, where we have of many DGX systems within the enterprise and we are from a kind of an AI oriented, high performance computing context trying to make these resources as broadly available as possible. Kind of conceptually. How do you think about that? Obviously you will see organizations that start off doing this. You know, you get a GPU and you get a GPU and all that, but that's not, that doesn't scale against the workloads that, you know, certain teams, they only need one GPU at a time and it may not take very long and others might need dozens for a much longer period of time and everything in between. Conceptually speaking, how do you approach differentiating between users and the various differentiated workloads that they're having to contend with?

We love to see people that try and plan for this sort of thing, right. They try and get a sense of, okay, I know I have this data volume coming in next year. I know roughly speaking it's going to take me this long on this many GPUs to train my models. Let's set aside budget and bring those resources on prem or secure them with long term leases on one of the cloud providers for the most part. Now that does a good job at helping you plan for your base load. But then as always there's going to be things that come up like towards the end of the quarter or a new model family comes out, or a new project takes really high priority that you've just got to ship, in which case we see real benefits to bursting onto cloud resources. And so within the context of our system, that's a core feature that we offer. We call it elastic AI infrastructure. And the basic idea is that if the system is configured and there's budget within the organization and so on, you can do that dynamic sort of provisioning of those cloud resources spilling work over onto them. We handle sort of the data transfer and other aspects of that planning for you and then as the workload goes down, those resources are released and the organization can save money. So we think it's a combination of having good planning, but also maintaining some flexibility in your systems and in your processes are required to really help AI scale within the enterprise.

I know one of the things that I've talked to people about, as they've talked about this particular problem, is the fact that the data transfer as you're trying to scale new GPU nodes in the cloud or something, if you have to transfer 200 gigabytes of data very frequently, that could be a downside. Are there ways around the sort of data management piece while still keeping things elastic?

Yeah. So when we see people kind of in sort of hybrid cloud and on premise environments, we like to take a look at what their infrastructure is for replicating that data. And we'd like to see it be sort of continuous, where the copy of the data that lives on the cloud and the copy of the data that lives on premise are maintained in a way that they're not exactly identical necessarily, but very, very close, or there is a path for them to become identical very quickly. So that sort of incremental process ends up being important. The other side of things I'd say is with all this discussion about just how big the data sets have gotten and how much data you need to fuel deep learning and so on, we are mostly looking at customers where the upper bound on the size of the training set they're dealing with is like order of terabytes. And that is a lot easier to manage and transfer and move around. It's still hard. You don't want to do it 100 times a day or whatever. But it's easier than say, moving petabytes, which is the scales that a lot of people are in the Hadoop space and so on will talk about. And so that gives you a little bit more flexibility and makes the data transfer being the big bottleneck in our experience is often the exception, not the rule.

Yeah, so it's good to hear that terabytes is small data now. It's only big data when we get to petabytes.

I guess I'm just kind of curious. I'm pretty fascinated with kind of as you've taken us through the approach. I'm curious as you're looking out kind of at the competitive landscape, as you see different organizations tooling up everything from the giant companies like Google and Microsoft and Amazon and such to smaller startups in the space like know you, how do you think about yourself in A competitive advantage mode. Like, what do you really think differentiates yourself from those out there? How do you think about that in your head?

I think there are a couple of key things. One, we've got some pretty unique expertise on the team kind of in this space. These are problems we've been thinking about really deeply, both in an academic but also professional setting for collectively the team dozens of years. Right. And we've got a track record of delivering some really popular and influential technology in this space. The other thing I'd say is I think the cloud vendors are there to build their platforms, to help monetize their hardware. You know, the GPUs that they've invested in, they want to get people using and so on. And so all of it is, you know, Google pushing Google's cloud or Amazon pushing their cloud and so on. Where we come, where we differentiate ourselves is by being really neutral to the vendor. We will give people access to the best, cheapest, correct technology for their particular workloads. And you're already seeing signs of vendors getting kind of custom hardware for these particular tasks. So Google has TPUs now. Microsoft just announced a partnership with Graphcore. There are sort of, and you can bet that Amazon is working on AI specific hardware. There are going to be a bigger menu of hardware choices to be available to help you solve these problems down the road. And we think that developers, in the same way, they don't want to be worrying about the resource management and the calendar system. They definitely don't want to be worrying about reprogramming their applications and figuring out which chip is best for this version of my language model and so on. And we think that layer of abstraction from a systems level can offer that kind of flexibility. So you submit your job to us, we figure out what the best hardware to run it on is, we go acquire that for you, your job gets built and run and then those resources are released. That basic idea, I think is something that we can do and we'll be able to do better than the larger cloud vendors because we won't have these exclusive ties to one or the other.

So that kind of leads me a little bit into a next question, which is around automation and I guess more specifically around AutoML methods. So I see AutoML mentioned quite a few times on the site. And also, I mean, there's been kind of a general trend of, of sort of AutoML platforms being released like Google Cloud, AutoML or H2O driverless AI. And it seems to be a lot of focus in this area. I was Wondering, you know, it probably makes sense to people, like one problem that AI people are going to have is managing their like GPU infrastructure. But maybe people think that the like hyper parameter tuning and the modeling side of things is kind of their baby and they don't want to mess with things like that. What do you see as some of the major advantages of automating some of that piece of things and utilizing some of these AutoML methods to kind of automatically figure out architectures, automatically figure out the right hyperparameters or automatically do other things? What role do you see that playing in the sort of future of AI infrastructure?

Yeah, so I think the way we think about it right now is that you've got these experts who are highly trained in their particular fields. You know, maybe they're really great at understanding the physics of solar flares or, you know, understanding how robotics works or whatever it is, and yet they're spending a lot of their time doing highly tedious tasks. So manually, you know, hitting, looking, telling the end of the log files, figuring out what the loss looked like, deciding, is this an area I want to keep investing in or should I try radical different model architecture, that sort of thing, and then writing the same 50 nested for loops to tune over my parameters over and over again. And when there are better algorithms out there for this stuff, either they don't know about them or they don't have time or interest in implementing them. They don't quite realize it's easy in the narrow to miss the fact that much of this work could be totally automated away, or at least partially automatically. And so our view is really, we want to give these practitioners power tools, right? Instead of saying like, we're going to build a robot that builds a house for you, let's take a carpenter and equip him with power hammer and the circular saw and so on. That's kind of the phase where we think that we're in when it comes to AI development. And so if you can equip experts with tools, again, new layers of abstraction that they can reason about, say, move from, you know, fiddling with the knobs individually to reasoning about search spaces and budgets around how many, say, GPU hours you want to put into solving a particular problem and then letting the system pick the right algorithm for say, hyperparameter optimization or the right way to approach that problem, we see really terrific gains. We've had customers tell us that they were able to replicate what had been a two month process of manually tuning hyperparameters and selecting model architectures in a single overnight run of our system. And that's leveraging kind of best of breed algorithms from active learning, developed primarily by my co founder, Amital Walker, around hyperparameter optimization and architecture search. And that to me, sure, if you could do that 50 times a year, I'd be printing money right now. But even if you can save somebody a couple of months, a few times a year, that ends up being really powerful in the way that they get their work done and how quickly they ship their applications. And again they start start thinking about the data problems and the modeling problems that they have and not so much how do I write out this infrastructure and that sort of thing.

So I know one of the things that determined AI is working on has been a lot about making AI work reproducible and being able to track experiments. And within the larger body of literature in AI, we're always hearing about explainability and transparency and such as that in AI. So I guess what I'm asking is why do you think that this is important to have this reproducibility built into AI infrastructure going forward? What kind of benefits do you see it offering and what do you think might be missing in terms of the things that we are tracking or parts of the conversation that haven't really been addressed yet?

Yeah, I mean, I think that if you told a software engineer that their code wasn't going to be tracked and that even if their code was tracked, they were going to check it out from GitHub and try and build the system and there was only like a 2% chance that they were going to get the same artifact out at the end of the day as their peer who downloaded the same repo that afternoon. They would look at you like you were completely crazy. Right. But that is very much the state of reality and the world when it comes to machine learning practice. It's because we have all this stuff under the hood that we need to track and get just right in order to get our algorithms to converge to the same level. Doesn't help that the optimization problems we're solving these days are non convex. And so there's a bunch of stochasticity embedded in them and so on. But the idea that I need to collect and understand every random seed that lives anywhere in my system, I need to understand what are the right hyperparameters for this particular run, what are the settings of the optimizer and so on in order to. And how is my model even initialized in the first place. Those are all necessary ingredients. I also have to Keep track of what my data is. Now, once you have built, built a solution or a system for ensuring reproducibility across runs of different machine learning models, and this gets to your point of, why is this important? Now you have the kernel of something that can be used to enable, like, very direct and repeated collaboration among data scientists. You can say, hey, download my version of the model and you can reproduce it. Exactly. Okay, great. Reproducible. Done. That's cool. Reproducible. Built. But now I can also use that to say, hey, why don't you extend my model? Try training it on a different data set, try running it on 64 GPUs and make sure that it converges in the same way. And I can begin to sort of riff with my colleagues on the next great idea. And I think that's sort of the dream. It's one thing for a single developer to be able to continue to innovate, but once somebody has a good idea, and now you can broadcast that idea to the entire rest of the, the organization, and everybody incorporates that into their solutions, now you've got a flywheel going that can really help an organization accelerate. And, you know, again, we see these, these kinds of best practices and properties emerging at places that are really sophisticated in their AI infrastructure. The bigger, you know, companies, the Googles of the world, and so on. But that hasn't yet hit the mainstream yet because our tools don't have support for that. And so that's one of the main things that we try to drive at determined AI.

All right, Evan, I, I'd like to kind of switch gears a little bit here. So we've been talking a lot about practical things around infrastructure, which I think is great because this is practical AI after all, and those things are super important. But I was also curious to hear some of your thoughts on another subject. I saw that you wrote a recent blog post about AI leadership and positive impacts on things like the economy on. On human labor. Labor and other things. I was wondering if you could share a little bit about the motivation behind that article and why you thought some optimism needed to be brought into that conversation.

Yeah, I mean, it's funny. The company is headquartered in San Francisco, and as I get outside of the San Francisco kind of AI bubble, or whatever you want to call it, at dinner parties with friends outside of this world, a common theme that comes up is, isn't AI all about automating jobs away? Isn't it all about taking away kind of my livelihood? And, you know, it's scary as we move into for people who are even in skilled jobs, they're looking at, hey, is your algorithm that is really good at tech summarization going to replace the need for the training programs in my law firm of an army of freshly minted attorneys doing discovery work and that sort of thing? Right. And the answer is right. It's like, like it's maybe. But when I think about technology, I like to look back on kind of what has technology done for the economy over time and how has this story played out previously? So in the blog post I use an example of how Japan recognized that their population demographics were going to shift in the 80s and started plowing a lot of money into robotics. And of course now they're a world leader in robotics, but it was in service of kind of planning for a world where the majority of the population was going to be over 65. Right. And building out infrastructure to support that. So I think that a similar kind of view needs to be taken of AI. Here we look at the Industrial Revolution. We've been automating things for like a century and a half at this point, and probably longer than that, depending on how you want to think about it. It always does lead to sort of short term job displacement, but in the long run quality of life and standard of living across the globe has risen dramatically. And so I think we kind need to take that view on technology as a whole in that we have to be careful about what it does in the short term to people and making sure that we've got social policies in place to help folks out. But it's good to be optimistic. These technologies can be really, they can enable things that felt like science fiction 10 years ago to be real, like the self driving cars we see on the streets and so on. But they can also really help in a bunch of ways that are otherwise unexpected around helping environmental health. We've got a customer in kind of the waste management space that specifically uses AI to help do recycling much more effectively. We've also got, we're working with folks in pharmaceutical drug discovery that are using AI to cure new diseases. So there are ways that these technologies can be used broadly for the social good. And that was really the motivation behind this piece that I put together.

Yeah, it's really great to hear that actually. I know, just because it's a brief tangent. Daniel and I are both very focused on using AI for good and we talk a lot about it during various episodes. And Daniel is focusing on bringing, making language availability with an AI, you know, more broadly available because there's so many languages out there in the world that are, that are not getting attention from technology and I focus on animal welfare issues and such. And so I love your optimization in this space. So I guess turning to the next thing is obviously with the potential for AI to continue to increase productivity at large, despite some of the bumps in the road obviously for society that you already addressed. And given the fact that there is a tremendous concern right now about privacy issues, how do you look at that dynamic tension between productivity and privacy? Are the two, are they always at odds with each other? Are they mutually exclusive in the context of AI, or do you see a more optimistic path where you can be productive and yield privacy at the same time?

It's a really interesting question in a broad area and with my kind of recovering academic hat on, I think it's a really interesting question from fundamental research where we can set up this, what you're calling. Perhaps we can formally study whether there is fundamentally a privacy and productivity trait. And first we try and answer that question and then if there is indeed this trade off, maybe there are ways that we can come up with that will give us precise control over that trade off as we make it. So an example I like to talk about is federated learning where users could potentially remain completely in control over their data and it stays on their edge devices. And yet the collective wisdom of all of the users through AI and things like homomorphic encryption and so on, could be used to, in a differentially kind of private way, help update models that globally make use of lots of users data without leaking individually private sensitive pieces of information. I don't think this stuff has been completely figured out, which is why I think it's still a really interesting research area. But I'm hopeful that as consumers demand that their data be kept private and so on, which I think we're seeing a lot of, and look no further than GDPR and the European Union as evidence of this, that we will start to have to get clever with how we navigate that trade off space. And I'm really excited. I watch the research coming out in the field pretty closely because I think there's some really exciting stuff happening.

Yeah, I know that in the most recent versions of TensorFlow and a bunch of other projects there were very certain things around privacy. And of course you have things like federated learning like you're talking about. I was wondering kind of, as we get near to the end of our conversation here in terms of practicalities for AI practitioners, whether that be someone that's, you know, working on Some of their first AI projects, maybe as part of a startup or something or a larger company. What are some of the best things that, that we could implement to help our workflow? What's the biggest bang for the buck that we can do? Maybe that's looking into things like automl or maybe that's implementing experiment tracking. Where do you think people should start changing their workflows first to make the biggest impact?

So I think if you're kind of in the early days of your projects and just kind of getting your feet well with the technology, my advice would not be to go try an auto ML solution off the shelf. It might work for you, but you're going to be in a position very quickly where you don't understand what's going on. One layer of the stack beneath you and as data problems come up or the next model needs a new tweak to it, or something like that, you might be at a loss and might be at a place where you're stuck. Instead, what I tell people is invest heavily in your data production, tracking, versioning to make sure that you're in a spot where you can go back and replay the past as it was exactly at that point in the past and build your models in that particular way and begin to invest in tracking and understanding your workflows from a code and data and kind of models perspective. So that is some level of experiment tracking. The other thing I'd say is start simple. So start with the simplest model that could possibly work and solve your problem problem. And that will do two things. It will both. One, maybe your problem is really simple and you don't need a fancy 50 layer convolutional neural network with an LSTM bolted on the side to solve it. Which is a good thing to learn. But at the very least it gives you a baseline for okay, this is the baseline sort of no signal to noise ratio kind of place I need to be. I need to make sure my models are at least as good as this. And it gets you in the head habit of targeting a metric that you can use to evaluate whether or not your model is good enough. And I think that's a really important lesson for people getting started.

Yeah. And I think those are amazing tips in terms of like the experiment tracking one. I think you're right on the money. That's a huge benefit that people can have for people that are maybe not coming from a software engineering background in your experience, maybe they're not quite to where they're ready to invest in, in the Full determined AI solution. But what would be some like practical ways for them to track certain experiments, you know, initially just a matter of sort of metadata and naming things correctly or getting into good version control habits with GitHub or where do you see people struggling the most or what are some simple ways that maybe they can benefit themselves.

Yeah, so I would say that for sure get used to using software version control tools for your code and versions of the models that you, that you got for data. Things like S3 for example. On Amazon they offer version data store. You can turn it on on your bucket and start using the version numbers as you're pulling data off of it. And then for the last piece, honestly, or a big piece is around metrics and so that in the early days can be recorded through either some pretty ad hoc processes, so structured log files where you write down what you think are the key parameters of a particular experiment or run. So think of it as maybe a JSON blob that records the keys and values that you care about and store that somewhere where you're sure you can get access to it and so on. There are also projects, open source projects out there like MLflow tracking, which can help facilitate this and give you dashboards around this as well. And so that might be another place that I'd recommend people check out if they're interested in another open source option in this area.

Awesome. Yeah, that's great. I should also mention Joel Grus was on the podcast and we'll link his episode in the show Notes as well. He talked a good deal about responsible AI development practices, bringing some of that expertise from software engineering into the AI research and AI development workflow. So we'll definitely link that and I.

Guess to close out for listeners who might not necessarily have all the skills in infrastructure and back end engineering and they're wanting to kind of level up and maybe they're even a little bit intimidated by kind of diving into this new area. Do you have any other, any other ideas to close out with on how they can level up those infrastructure skills?

There are a number of great online resources. It's funny, I've never really liked. I thought about that side of things needing to be leveled up. In fact, that's kind of why we provide the software platform that we do, to try and keep people from worrying about that.

That's fair enough.

But yeah, I think that the various cloud providers do a good job of providing education around things like Kubernetes and so on. That can be helpful as you're thinking about what's the modern way of building out this infrastructure? But I don't have specific, specific resource recommendations in mind right now.

No worries. Well, Evan, thank you so much for coming onto the show and telling us all about determined AI and infrastructure, and it's a fantastic conversation.

Sure thing. Great. Great speaking with you guys. Thanks so much for having me.

All right, thank you for tuning into this episode of Practically AI. If you enjoyed this show, do us a favor. Go on itunes, give us a rating. Go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you got to do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more@fastly.com and we catch our airs before our users do here at changelog because of rollbar. Check them out at robot.com changelog and we're hosted on Linode Cloud Servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@changelog.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.