From notebooks to Netflix scale with Metaflow

We can potentially apply ML and AI to all aspects of the business. So I think like we are entering a world where like no area of life and business is totally kind of a safe from ML in a sense that like we can apply ML to all kinds of things. And now I don't mean like a crazy general AI, but I mean like a tiny little optimization problems here and there and they are like really everywhere in all lines of business.

Big thanks to our partners Linode, Fastly and launchdarkly. We love Linode. They keep it fast and simple. Check them out@linode.com changelog Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by LaunchDarkly. Get a demo at launchdarkly.com this episode is brought to you by our friends at rudderstack and we're calling all data engineers to check out rudderstack cloud and start building smart customer data pipelines. Rudderstack is warehouse first. No more silos. Rudderstack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API first. Rudderstack can be easily integrated into your existing development processes and because they're open source you can see all their code so you don't have to worry about vendor lock in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. Rudderstack is free up to 500,000 events and pricing scales transparently. From there, learn more and get started@rudderstack.com again. Rudderstack.com that's R U D D E R S T A C k dot com.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changeog.com community and follow us on Twitter. We're racticalaifm.

Well, welcome to another episode of Practical AI. This is Daniel Lightnack. I am a data Scientist with SIL International and I'm usually joined by my co host Chris Benson, who is a tech strategist at Lockheed Martin, but he's got the week off. But I do have a wonderful guest for today's discussion which I think is very timely and also very practical. Vila Toulouse is with us, who is CEO at Outerbounds and was previously leading the Data Science Infrastructure group at Netflix. He's also the author of a really interesting and new book called Effective Data Science how to make Data scientists more Productive. Well, that sounds super practical. Ville, welcome to the show. Great to have you here.

Yeah, thanks for having me.

Well, I know even before the show we were talking about your background and how you're sort of background has shaped how you think about data science infrastructure, but also how you think about like AI and what really is new, what isn't new, what trends have happened and developed over time. So could you just give us a little bit of an idea about your background and how you started in this field?

Yeah, well, I wonder like how far back I should go. But embarrassingly enough, I studied at my first startup that worked on artificial neural networks already back in 2000. So I mean that's kind of a long time ago. And believe or not, I mean this of course predates all deep learning and so forth, but people did artificial neural networks at the time. And this startup in particular, it was focusing on a very peculiar kind of neural networks called self organizing maps that were quite popular like in late 1990s, early 2000s. And the commercial idea we had at the time is that we could help enterprises with kind of a search, like enterprise search, the intranet search, stuff like that. So it happened that yes, I mean companies needed help with that, but I mean they didn't need neural networks. But interestingly enough, ever since then I've been really focusing on this question that how can we actually improve the tooling and the infrastructure for people who built these models since already back in the day I saw that a big challenge that we had is that we had all these amazing researchers, data scientists whose task was to kind of build these models that were supposedly useful in practice. And it took many iterations, it took a lot of thinking and trying out, experimenting. And of course back then the tooling was really quite horrible and now it's much better. But I think it's really interesting to see the whole like trajectory like where we came from and like where we'll be going in the future. And yeah, I mean it's kind of scary to see that like also there are many things that like haven't actually changed that much and like kind of we have much work ahead of us.

Yeah, maybe could you share some of those examples like what is the same now? Maybe a challenge that you're still facing now that was the same as back in those days and maybe something or a couple examples of things that have really dramatically changed.

Yeah, well, I mean, starting with some things that are the same amazingly enough already back in the day we were using Python and of course that was highly controversial at the time. So I mean Java was pretty new and of course actually everything that needed to be high performance was written in C and C. So that's actually a big change between then and now that of course we had to build all the libraries by ourselves. And it was even seen as a big kind of differentiator and a competitive moat that we had a more performant, more scalable algorithms to optimize these models than anyone else. Nowadays of course everybody can just use off The Shelf, PyTorch, TensorFlow, even like XGBoost, Psychic Learn just released 1.0. So the Lbry ecosystem, I mean it's way ahead like what it used to be. Now another big difference is that we actually spent considerable amount of time just setting up the hardware. Amazingly enough we had a rack of servers kind of at the company and we had to set up the networking and the storage and the compute and the operating systems and all that stuff. And now with the cloud again, that's another thing that has massively changed that you can get this infrastructure that used to be available only for the largest companies, even at the startup and you can get clusters of machines with GPUs and it's just mind blowing. And now what's really exciting today is that on the one hand you have this high level modeling libraries, on the other hand you have all the hardware available. And now I think that it's kind of mind boggling that still things aren't that easy. It kind of feels that everything is possible, but I mean nothing is really easy enough. And I think that that is still kind of something that we need to work on. And of course one interesting challenge is also that there are way working on these problems. It used to be only a handful of people, I think we had three people at the company at the time who could actually build these models. And now, I mean it's like probably 100 if not thousand times more people. So definitely feels like a new field in that sense.

Yeah. And how did your perception of data science and the infrastructure in particular needed for data science, how did that shift as then you led the team at Netflix and I mean now we're talking of course about a, a much bigger scale. Right. But also, yeah, these models are I assume even at that point a really critical piece of what makes Netflix Netflix and the value that's added by these models. So how did your perception of infrastructure change as you worked on those sorts of problems.

I think one interesting thing that maybe not everybody realizes is that the few things that many people know about Netflix, especially the recommendation and personalization systems, they are kind of the tip of the iceberg. And there are so many other things where Netflix wants to apply potentially ML and related technologies and also wants to experiment with new ideas. I think one thing that's really interesting is just the kind of the diversity of different use cases. If you think about computer vision, natural language processing, even things that are not technically machine learning, like operations research, how we can optimize schedules, stuff like that. So it's just the fact that there are so many different types of problems and they come in all shapes and sizes. It's not always about scale also. I mean, it's not always that everything has to be like a super kind of a high SLA business critical. I mean they're like experiments, like a crazy experiments. But the interesting challenge is really that, like, how do you manage like kind of like the diversity of all these things. So that's really interesting. Now when it comes to the more technical side of things, Netflix, like for the longest time has been 100% on AWS, like it's a cloud first company. And Netflix has been trailblazing like many architectural patterns, like when it comes to storing data in the cloud, way before like other companies were doing it, also microservices engineering. So it was really fascinating to be at this company that had on the engineering side, like kind of some of the world's leading cloud infrastructure. I think, honestly, I mean, what makes Netflix interesting is that it all runs on aws in contrast to, let's say Google and Facebook, who have their own infrastructure, which is really kind of an island and nobody else can do it. But I mean, Netflix is in that sense like a closer to everybody else in the world that it is the same AWS at the end of the day that Netflix uses that everybody else can use as well. And it's just that they have all these practices and ways of thinking about things and ways about building services that makes them really effective. And now when you layer something like a data science and machine learning on top of it, I mean it's really interesting. And of course, I mean that is kind of all those learnings are reflected in MetaFlow, which is the open source library that we started building there.

Yeah, I do want to get into Metaflow, but before I do, maybe what is your perception of how maybe back in those days when there was sort of like a building hype around data science, a lot of people initially getting into IT companies experimenting. What is your perception of like from then to now? Now, how has the average data scientist ability to work with these different pieces of infrastructure that maybe come across their path, like the various services in AWS, whether it be EC2 or object storage or things even all the way up to Kubernetes and EKs and that sort of stuff. How has the average data scientist, from your perspective, what they're required to know or maybe what they come in with knowledge of, how has that shifted over time?

Well, my kind of initial reaction is that I still think that we are in the early days. Yes, I mean, of course. I mean the fact is that like maybe five, 10 years back, if you wanted to do anything in this field, you basically had to know C and you had to know the depth of knowledge about, let's say distribute systems was much deeper. And even, let's say the kind of recommendation systems at Netflix, they run on Spark and like many people use kala. And that's a bit of a different Persona, a bit of a different profile than what we see now amongst data scientists who are building a new set of very diverse models like using these Python based libraries, maybe directly using the cloud and so forth. At the same time, I do think that all these things and especially like being able to leverage the cloud is still harder than it really should be. Just thinking about the kind of the amazing amount of computational power that you have there and still, I mean, there are most companies I talk with, there's the feeling that, well, data scientists kind of need to know about Docker files and maybe they need to know about the CI CD systems. And I think that there are different points of view. I mean if that's actually a feature or a bug, I think that there are so many questions related to the modeling itself that at the end of the day, all of us, all human beings, you kind of need to manage your cognitive bandwidth. And as interesting as it might be for everybody to know about CI CD systems, I do think that it kind of takes a bit away from the bandwidth that you should have available for thinking about the modeling problem itself. So I do think that, and I do hope that we manage to kind of raise that level of abstraction even more.

Yeah, I guess in this case abstraction could be a good thing. And even though it is interesting to dive into these different systems and containerization and all that, it does take a lot of time. Like you said. I remember there's like a tweet from Eric Bernardson who said, like having a data Scientist sort of learn about some of these things like Kubernetes and Docker and Terraform and these things is kind of like having web app developers learn about the Linux kernel. It's just like so far apart. It's very tough to expect that. Yeah. At the same time, like I have benefited from those times where I've been able to maybe push something further on my own at least into like a prototyping stage within a company.

Yeah.

And get it in front of people to see that value without sort of like reliance on, you know, passing something off to a data engineering or a software engineering organization to even create a prototype. So maybe there is some like tooling that's improving around that. Like I know there's things like Streamlit and other things where you can create something that's very compelling very quickly in terms of like a prototype. But I don't know. One of the things Chris and I discussed on the podcast a little while back was maybe why many data projects fail in certain cases is because people aren't able to push a project far enough into a prototype stage for people to see the value of something and actually get buy in from the organization. From your perspective, I know also being the CEO of a company that is trying to help people with their ML infrastructure, where do you often see when you first maybe engage with clients or when you maybe just making an observation about the industry, where do you see the problems in people not being able to get value out of machine learning and AI? Where do things get blocked most often from your perspective?

I think it's definitely a combination of maybe three different factors. One, I mean maybe starting from the kind of the easiest one is technical, that there are technical hurdles still. I mean it's just like putting the infrastructure together, it's just building the models. Although technically all the ingredients are there. Many companies are still struggling kind of putting the pieces of the puzzle together. But I do think that that is in a way, I mean like the easiest one of the three. So then the second one is definitely the kind of the skillset of the people involved. And it's not only that they wouldn't be skilled. I mean, I don't think that that's oftentimes the problem, but also kind of what are the things they should be focusing on. And especially when we start talking about actually producing business value using machine learning, really understanding the problem domain, understanding the business needs, I mean that takes a lot of bandwidth. And still much of the time these practitioners, they spend on either engineering problems or maybe modeling problems. That may be fun, but I mean, ultimately might not really kind of affect the company's bottom line so much. But then the last problem, I think, and it's the big one, is really the organizational. It's kind of like a leadership question. That kind of. What you mentioned before is that, okay, so why do many of these projects fail? Well, I mean, they don't get close enough to production. I think that that is absolutely a key. And I think that that was one thing that Netflix did really, really well, that they have, even at the highest level of the organization, they have this experimentation culture and they have this idea and understanding that like, well now first we can potentially apply ML and AI to all aspects of the business. So I think like we are entering a world where like no area of life and business is totally kind of a safe like from ML in a sense that like we can apply ML to all kinds of things. And now I don't mean like a crazy general AI, but I mean like a tiny little optimization problems here and there and they are like really everywhere in all lines of business. But now the problem is that you may have thousands of ideas that, okay, we could do this and that, but how do you know which one of those work? And nobody really knows in advance and you can't really ask anyone. And the only way to know is that you really need to kind of start experimenting. And not only experimenting in a sense that you hack something, a prototype in a notebook, but oftentimes really the only way to know is that you basically push these things to production. And now the production meaning not so that you have a huge team working on something for in six months, but actually kind of getting something to a B test. I mean, if you have. And honestly, I know that this is really not that easy, but I mean, the idea that you can test these ideas, you can test different prototypes and pipelines in production alongside whatever system you have in place today, and then you compare the results, that it's hugely, hugely powerful, and then have the understanding that you can interpret the results and decide what to do with that, and have the understanding that actually it is by design that most of these things, things fail. I mean, that's kind of the whole point of experimentation. If you knew that everything is going to succeed, I mean, you wouldn't have to experiment. But the idea is that you can afford making so many of these tiny experiments that then you can quickly decide that, okay, this doesn't seem worth it and maybe then you redirect resources to something else. But I mean, this is also a question of product management, oftentimes having product managers who really understand how to work with these ML systems. I mean all these organizational muscles are missing at many, many companies.

SignalWire is real time video tech to help you create interactive video experiences previously not possible. It gives you access to broadcast quality, ultra low latency video that's proven and trusted by Amazon, Ring Doorbell Zoom and others. See why the future of video communication is being built on Signal Wire? They have easy to deploy APIs, SDKs for the most popular programming languages and expert Support from the OGS of software defined telecom tech. Try it today at signalwire.com and use code AI for $25 in developer credit. Just visit signalwire.com that's signalwire.com and use code A.I. to receive that 25 bucks. Once again, that's signalwire.com code A.I.

So file you started to mention as we were talking about trends that you're seeing in infrastructure and ideas around where things get stuck in production. You mentioned Metaflow a couple times, which I know is a big piece of the puzzle in terms of how you solve these problems, but also a big part of your career in terms of what you've developed. So could you give us a little bit of the backstory of metaflow? Sort of the origin story? I guess.

Yeah, I guess the nice thing is that it's actually quite pragmatic, quite bottoms up in a manner that As I mentioned, Netfl joined Netflix back in 2017. This was before SageMaker, this was before MLflow, this was before Kubeflow. This idea of having any kind of, let's say especially open source machine learning infrastructure was quite new. I mean of course there were products around like you had Datarobot, you had Domino Data Labs, obviously like you had Databricks and Spark and so forth. But I mean like the idea that what does the full stack for ML look like? I mean that was quite new. So now when I joined Netflix back in the day, I saw that. Okay, so, well, I mean obviously like the company had like all these like basic foundational pieces of infrastructure in place. So they had a large Data Warehouse, an S3 based data lake. They had a team managing a large scale compute infrastructure, basically something like Kubernetes. They had also teams who had been thinking about workflow orchestration for a long time. So you had all these pieces. So again, I mean like technically everything is impossible. So it didn't seem like the challenge was that okay, we need to invent some new pieces of tech that we could do something that nobody else has done before. That didn't seem to be the problem. But then the problem was really that they had the organization of data scientists there who constantly complained that like, well, getting anything done was too hard because exactly for the reasons we discussed that, like, okay, how do I run compute? I mean, you go to the compute team, they say that, oh, I mean, you just have a Docker container and you put the container here, image here and tag here. And then maybe you better go through the CI CD system and then it already at that point in time, you had lost the data scientist. Does this make any sense? And the workflow systems, of course needed lots and lots of YAML to kind of define what you want to do and even kind of thinking like what are the kind of the patterns? Because remember, these people are not software engineers by training. So kind of how you actually architects software like this was hard. Really. The origin story and the idea for MetaFlow was that, okay, assuming that you have this foundational infrastructure available, how can you kind of stitch them together in a manner that would present an API to data scientists that would kind of help them to build these applications that they have been asked to build for the company? And now the other interesting side of the coin was that Netflix has this culture of freedom and responsibility, which meant that we didn't want to take away all the freedom from people saying that, well, I mean, here's a training API and you can only call this one API to train your model. It was well known that different people preferred different tools for the job. I mean, some people prefer TensorFlow, some people prefer XGBoost, I mean, depends on the application, of course. So the idea was that, okay, we should allow them to kind of at the high level, kind of exercise that freedom and exercise that domain knowledge and expertise in choosing those modeling tools. So we kind of started having this idea that, okay, we need to be quite opinionated about the lower layers of the stack. How do you do compute, how do you access data, how do you do orchestration and then leave a lot of space at the top of the stack? That okay, so what kind of modeling libraries you use and how do you do your feature engineering and maybe even what are the KPIs that matter for you when it comes to monitoring models in production? And then we started crafting that stack and again, I mean Netflix, there's no top down in anything. I mean, there's no cto, no VP of Engineering saying that everybody must use this thing. But we started solving these very practical problems and then kind of in A very organic manner. I mean metaflow started spreading inside the organization because people thought that, oh, I mean this is quite a no nonsense tool that helped them to solve exactly the types of problems that they had been facing on a day to day basis.

Yeah, so there's a whole variety of things that people have created around workflows, but then there's also a whole set of platforms and projects out there related to ML Ops and other things. There's like all sorts of things that maybe data scientists care about from like making sure that they can run their workflow not on their laptop, which is more maybe infrastructure compute related all the way to like hey, how do I version and control experiments? How do I access data? So how far does metaflow reach in terms of these different things that data scientists this might want to do? Like what pieces of the puzzle does it try to solve? And how can data scientists think about it in terms of those various buckets of things they're trying to do?

Yeah, that's a good question. I think like because we were faced with this great diversity of different applications, we couldn't think, let's say, I mean at my previous company, before Netflix, I mean we were doing real time bidding and like for targeted advertising. And then like in a context like that, you know exactly the application and you know that okay, maybe we build a feature store, maybe these are exactly the workflows everybody follows. And that's one type of a challenge. It actually might be a great engineering challenge, but I mean different type of challenge. In our case with Metaflow, the challenge was that we didn't know exactly the type of machine learning applications people wanted to build. So we started thinking very much from the bottom up that what are commonalities across all applications? And really it starts with that. Okay, so I mean really the question of data and now the discipline ways of accessing data. Okay, so how do we do it quickly enough? Let's say you have some kind of a data warehouse, you have a database. How do you get the data out quickly so you don't have to wait for 40 minutes for your SQL to execute. So that's one thing that we were thinking about like working with arrow. Metaflow comes with the custom S3 library so you can get your data super fast from S3 small things like that, then on the compute again, all these models, I mean not even the ones that require huge amounts of data, require a lot of compute. So you may want to do hyper parameter search. So maybe you have a model ensemble, like maybe you want to build A separate model for every country or maybe for every customer. So you need to be able to fan out this compute to the cloud. So definitely wanting to solve that part as well. Then we saw that oftentimes it's really, really sensible idea to structure these applications as a workflow. So I mean, there's a lot of confusion about these DAX and workflows these days that okay, what does it even mean? There's so many different workflow systems but, but purely as a way to kind of express these ideas. The idea that you structure things as a dag, I mean it makes a lot of sense. So we kind of took that as a core way of implementing things. But then we definitely wanted to separate the idea that once you deploy these workflows to production, running workflows, that scaling production is actually an engineering challenge of its own. And we didn't want to claim that. Well, I mean MetaFlow is the most production grade scalable workflow scheduler in the world. So we integrate it with other systems out there and also to ease that path due to production. So that was really another thing that what we discussed earlier about how important it is to kind of test these ideas as close to production as possible. We knew that, okay, we need to provide a path all the way to the end and that's why we integrated with the existing system so we wouldn't get the resistance from engineering teams saying that, oh, you have this piece of Python code, but I mean, no way we are going to run this in production. So kind of really like thinking about the production best practices starting like with very mundane questions like dependency management, like what if you need a very specific version of TensorFlow? Again, we don't want you to kind of write Docker files by hand. It's surprisingly hard to do it in a reproducible manner. But I mean, how do you let you use the exact version of the library you need? And then you mentioned versioning as well. I mean first there's the idea that, well, you should maybe version your code, maybe using git, but I mean, how do you version your models, how do you version your experiment, how do you verse on your data even? And we felt that these are such kind of foundational concerns that we should also provide an out of the box solution for them. I think that that's definitely kind of helps because then the data scientists don't have to think about it too much. That's kind of what we have been doing this far. So if you think about compute data orchestration, versioning and kind of all Kinds of questions related to pushing things to production. Then there are things at the top of the stack that we haven't been so opinionated about. So I mean many of our users today, they use other model monitoring tools. They are amazing model monitoring tools. You mentioned streamlit, weights and biases, many others out there, of course, specific tools for model explainability, if that's important to you. And then of course a feature engineering that's a complex topic of its own. There are some customers who use MetaFlow with some feature stores that works and of course the modeling libraries is something that you should absolutely use the best of the breed tools off the shelf.

Yeah, so it sounds like part of the philosophy here is people are going to be opinionated in their own teams about like oh, we use weights and biases to do this, bits of the monitoring and experiment management. But that's not going to solve these sort of scale and infrastructure problems. And they like workflow running problems that you mentioned as well. So being able to pull in what you need I think is, is really a cool idea and having that sort of modular nature of it is really great. So I do want to get into like the actual workflow with MetaFlow, but in terms of how it works under the hood, like let's say that you're setting up a flow and you've got a series of steps that processing steps, eventually something has to run on a server, right? And maybe certain, like you're saying Maybe I'm running TensorFlow over here and it needs a GPU or maybe I'm doing this pre processing of images and I just need to crank through a bunch of stuff in a sort of batch or maybe even parallel way on CPUs. How do things in code that is using MetaFlow, how do those things eventually end up running on the servers? Is there some sort of containerization or something going on under the hood or, or how have you built that abstraction layer?

Good question. Well, I mean now Metaflow since the day one, I mean it has been built with this cloud first mindset. I think that kind of when it comes to things like compute and storage, I think we live in a bit of a post scarcity world that it's actually interesting when you think about it that many of the systems that we even use today, like databases, even things like Spark and Hadoop, they were built with this idea that you have a constraint resources. And really the engineering challenge is that okay, how do you allow people to kind of run compute given that you have only like 200 servers or something like that and you have to do the resource management very carefully. Then the mindset that we adopted with MetaFlow, which I think it's really, really useful, especially in the people's productivity point of view, is that you work with the cloud. The cloud provides you at least this kind of abstraction of having basically infinite scalability. So you can use some cloud based platform again, I mean we rely on existing systems like Kubernetes, like AWS batch, like what have you to kind of farm out the containers to the cloud. You can specify the resources you need. If your function needs GPUs, you can say that I need GPUs in this case and in other cases maybe you need a lot of memory. And I think that the interesting challenge with machine learning, I think that this also sets machine learning apart from many previous data intensive applications, is that the needs are so different. Some types of models are really IO sensitive, like maybe you need to read like tons of images, tons of videos, but the model itself may be simple. In other cases it might be something super compute intensive, but not I O sensitive. In some cases you absolutely need GPU or maybe even like some custom crazy hardware. In other cases it is cost prohibitive to do that. And it's very hard to have any kind of like uniform, one size fits all. So I think like having that like scalability that goes in all the different dimensions, like vertical, horizontal, you name it. So that's super useful. And cloud makes that possible. So we rely on these systems and we take care of like then packaging the code, sending it to the cloud, executing on the container, handling retries, all that like kind of a basic, basic plumbing there and then. Yeah, I mean the same thing like with the orchestration, I mean like the DAG execution at scale, if you have 100,000 dags running in parallel, I mean it's a thing of its own. And like there are some systems that do it well. We integrate with AWS dev functions. Now we are integrating with Argo, maybe one day with Airflow. And I mean what have you. And the idea is that you should be able to test these things locally. So MetaFlow always comes with the local mode. So you can kind of test and iterate the same way you do in a notebook. But then kind of when you want scale, like when you want kind of something that's like production ready, you can use your existing production infrastructures.

So you were just getting into talking about, hey, maybe you're doing some experimentation locally in a notebook. And then eventually you kind of of go beyond that and Scale up and all of that. I guess first off, to set the stage, MetaFlow is an open source project and people can go ahead and try it out and we'll include links in our show notes to where people can find it and try it out. But let's say I am a data scientist. I understand what we've been talking about so far, that hey, I'm going to experiment locally, but then eventually I need to run all of this sort of workflow and series of processing steps on infrastructure that is in the cloud. Could you maybe just walk through, like, what does it look like for a practitioner to use MetaFlow? Let's say they've written some Python code, they're used to working with notebooks, maybe they're sometimes used to writing Python scripts that they run on. Maybe they log into a server and they run it. What does it look like for them to install and integrate MetaFlow into their workflow? What are the prerequisites and how does the integration happen?

Let me start with the kind of a data scientist point of view before getting to deployment and stuff so I can give you a timely example that just yesterday I was actually creating an example for the book using Keras and this was a new data set. I was actually using the NYC Taxi data set for the example. It's fun. Data set, publicly available, everybody probably in this situation, I actually started exploring the data in a notebook. Of course, notebooks are still great for visualizing, exploring data and so forth. And even when I started drafting the kind of the model architecture in Keras, I mean, it's really quite nice to be able to iterate that in a notebook quickly. I was very conscious about, I wanted to kind of introspect that. Okay, so I mean, what are the things that work well in notebooks? What are things that work well on the metaflow side? And I kind of faced the same problem that many other people face when using notebooks, is that after maybe three hours of prototyping, my notebook had this kind of a mixture of cells that I had been executing out of order.

And who knows what the state is.

Yeah, exactly. It was super convenient. It was like I kind of felt that I'm in this garage hacking something together and everything is kind of on the table and it's kind of a messy setup at that moment. I mean, it made me super productive. But I mean, it was, was absolutely 100% obvious that there was nothing in that notebook that I would dare to run in production. I mean, even the idea that I would somehow run that because it was the kind of my experimentation process that was reflected in that notebook. I would like to kind of think about production a bit differently. So then the idea, what happened at Netflix as well, and what we recommend people doing, is that they by all means use notebooks for experimentation, for exploration, for building prototypes. But then at one point, when you kind of have a rough idea what that workflow could look like, and really the threshold for that shouldn't be too high, you can almost start copy, pasting the snippets, let's say in this case, I mean, just the kind of 15 lines of code that define the kind of the cross architecture, like kind of in a step, in a file. And now if you use something like a Visual Studio code, actually it's really easy to have both the kind of the IDE as well as the notebook side by side. So I can use notebooks for exploration and then I can still have that really proper IDE for writing Python code. And again, the idea with MetaFlow, what we have had since the very beginning, is that it doesn't require that you know anything more than what you would need to know in a notebook. So there are no new concepts, no new paradigms, you don't have to change the code, the same libraries and all that stuff. So I was able to then kind of take the best parts of my experimentation and kind of put them to this workflow. And now thanks to MetaFlow, I'm able to kind of start running it at scale. So, I mean, of course in the notebook, it was rather small data set that I was testing with, and now I could take the kind of the same concepts, the same code, then like start testing that with larger scale. I mean, like, I didn't have to wait for my tiny workstation to kind of crunch all the data, but I could like farm it to the cloud. So overall, I mean, that is quite nice pattern. Like, you kind of get the best of the both worlds. I mean, you can use notebooks where they really shine and then on the other end, like in the end, an artifact that like you really dare to run in production. Now, I guess the other side of your question was that, okay, so the deployment. So, yeah, I mean, like, indeed. I mean, the easiest way to get started is that you run PP, install MetaFlow on your laptop. It works out of the box. There's nothing else that needs to be done. We have also had this belief that the kind of the needs of an organization grow over time. You don't necessarily have to kind of have the most battle hard and most scalable setup on the day one. But you can start with something simple. And probably one of the simplest things is that you can sign up up to AWS batch and there are three configuration things that you have to set or if you don't want to do it by hand. I mean there's the terraform cloudformation template. You go to the ui, you click a button and it sets up the kind of stack for you. And then you can start running compute at scale and that's really great. So if you have more than one person working on these things, you probably want to have this centralized metadata tracking so people can share their results. That's quite convenient. It comes as a part of the cloudformation of not too hard. And then there's the orchestration system again, I mean part of the stack depending how you want to do it. I mean you have freedom there to set up in a few different ways. And then of course the larger the organization, now the largest organizations there, they might care about setting up the data governance rules like lifecycle policies, thinking about how do we harden the kind of the service deployment so it's highly available, stuff like that. But I think realistically these infrastructure stacks need to start small and they need to be able to grow with the organization. I think many systems have the problem that either they are super easy, but then they don't scale. I mean, as your company grows. So I mean you kind of outgrow them at some point or then they are way too enterprisey and you have to scratch your head about the kubernetes deployments and whatnot before you can get even the simplest thing done.

Yeah, and I'm looking through your documentation, which is great, but it seems like there's a sort of concept in metaflow where like you said, in a lot of these workflow management systems, you're writing like YAML, maybe you're writing JSON, you're writing config files and docker files to sort of manage these various steps, which is definitely doable if you want to get into that. But in terms of an approach that you're taking is this sort of decorator pattern in Python where like you're defining maybe a class that's your dataflow and some steps within that class that are decorated with a metaflow decorator of step and then you're connecting those different steps within your actual Python code to create your workflow which is then maybe farmed out to some infrastructure. Did I get some of that right?

Yep. And like the interesting thing is That, I mean, there are like so many kind of systems that look like that. And I always say that like the devil is in the details. And many systems, let's say, I mean tons of systems that let you specify workflows in YAML and oftentimes they say that, oh, I mean, you can run any docker container and you can run any code inside the docker container that makes a step which kind of like as a user, well, I mean, it kind of pushes the hardest problem to you that. Okay, wait a minute. So I mean, how do I define what code runs in this container and where do I push my container and what are the dependencies I need inside the container and how do I move data between these containers? So in a way, I mean, just like having a workflow, I mean that's kind of the easy part. That's like why wanting to have this like a self contained thing in metaflow, that you have everything in one place. So you define the code, you define the dependencies, you define the resources, you define the workflow. Because then you kind of get the unit that you can actually run the full thing in production and it does something useful. And by the way, I mean, like, one thing that I definitely want to mention, which is really important is that this is never a waterfall. I mean, it's never so that you prototype and then you deploy and then you declare mission accomplished. But I mean, if the project is successful at all, what inevitably happens is that either something fails in production, in which case you have to go back and start debugging, or the business stakeholder, whoever comes to you saying that, okay, now we want better results, can we improve accuracy, can we add this new data set, whatnot? So you kind of have to start iterating again. And that's really the challenge with many systems that even if you are able to do that one deployment, how do you come back from production? Everybody always talks about going to production, but how do you come back from production and then keep iterating and maybe start having multiple versions running in parallel? So it's all these small things that really matter a lot when you think about running ML for real.

Yeah, definitely, I totally agree with that. So we talked about how you might set up your workflow in MetaFlow. You can pip, install MetaFlow locally, connect it to AWS cloud resources. In terms of that step from, let's say because you talked about being in the notebook and then kind of moving into this MetaFlow workflow, but then eventually, let's say that I create a pipeline that I really like, and I know that I need to run it triggered when this happens, or I need to run it every Friday to do this or that. What does that look like? Because I assume you could run your Python code locally with these metaflow decorators that gets sort of farmed out to resources in the cloud and tracked in various ways. But what does it look like to go from that to some sort of automation or something that's running kind of hands off and you're not running, you know, Python, whatever, locally?

Yeah, I can paint you a picture that I saw at Netflix that worked really well. Now I know that like not many companies are yet at this stage. I do hope that like kind of the world will advance over the next three years or so. But I do think what's really, really useful is that you have some kind of a centralized workflow scheduler. I know that many organizations are struggling with this question that should they have many different infrastructure stacks and kind of different departments have their own, and ML has its own, and data engineers have their own. But the fact is that ML is not an island. And especially if you want to use these things in production, I mean, producing real business value, you have to integrate with whatever is the outside reality out there. So there's a lot of value in having kind of a decentralized system. And what the centralized system needs to do is only to take care of this seemingly simple task, which is that you have workflows. And this system needs to keep the workflows running, farming out and the compute due to kind of whatever is your compute backend. I mean, kubernetes or AWS patch or whatever. And now the beauty of this setup is that oftentimes these data science workflows run in tight conjunction with data engineering workflows. So you have an etl and you can imagine that you have maybe daily ETL that takes some raw data, takes some whatever streaming event data, and kind of massages that to new tables. And whenever that table updates, then maybe you want to update your models. And then in the best case, there is a triggering mechanism that automatically, whenever the data updates, then triggers the ML update as well. And there's maybe some piece of information carried around saying that, okay, these are the new partitions available, or this is the latest hour, or however you want to do this. And then if you have this centralized scheduler, if you have this triggering mechanism in place, you can start constructing this almost like a web of workflows that comprises both of the data engineering side of the house ETL and Even now, if you want to use something like a DBT or create expectations, you can kind of tie that really nicely upstream. So you have a really nice etl, always data quality. And then you have something like let's say the ML workflows managed by metaflow. And then oftentimes even what happened at Netflix is that there's kind of the ETL after that. It might be that let's say you do produce some batch predictions and now you kind of have to load those batch predictions to another place or let's say in some cases some decision support systems even like wanting to have those predictions in tableau or airtable or something. So you have another piece that then takes those results and pushes them to something else. And now of course as the complexity grows, then you want to layer observability tools on top of these alerting tools that okay, what if something is late? How can you trace what's going on? And of course there's a lot of additional infrastructure that you need there. And now if I'm looking at this workflow orchestration landscape in the kind of the world overall, I think we are not quite yet there in many companies, many companies have maybe multiple airflows that are not connected connected. Many companies still use this like cron based scheduling that like it's always like runs at 3am no matter what it like runs at the same time, which is kind of silly. Also the observability parts are missing. But I think that that vision is really great. I think it really helps a lot in like kind of tying the ML really close to the rest of the organization. So it's not like island like in some walled garden somewhere.

Yeah, well I appreciate that very much. I know that we've covered a lot of different topics today. I do want to mention again your book which is Effective Data Science Infrastructure how to Make Data Scientists More Productive. We'll include a link in our show notes to that because this is something our listeners I think will really get into because it is so practical. I'm just looking at the flow of your book right now, which goes all the way from sort of notebooks to workflows to metaflow to production and scaling up, which I think is a super, super practical book. So thank you for your work on that our listeners. You'll definitely want to check it out because we do have a 40% off discount code for the book from Manning. You can use Pod PracticalAI 19 the code Pod PracticalAI 19 for 40% off of the book, which is pretty cool. Well Maybe to end, we usually like to ask our guests some future looking question and I think you've already started to go there in terms of where you would love to see infrastructure go, but maybe in terms of data scientists and the infrastructure that they're working with, what you're hoping to see maybe in a couple years in terms of data scientists workflow, how do you see that abstraction layer advancing and changing over that time period?

Yeah, I think that there is work to be done at all layers of the stack again. I mean, as mentioned a few times during this episode, I mean I'm excited about the fact that we have so much compute available, so I think we can make that even easier. That's exciting. Definitely a lot of work to be done on the orchestration side, tying together etl, ML. I mean there's a lot of work to be done there overall, I think at the higher levels. I think that of course the fact is that many companies, and if not most companies out there, are still struggling with the questions that how do they use ML to power their business, not only produce demos. And I think that that goes back to that. Like even organizational mindset change with the experimentation culture. And how do you divide work between engineers and data scientists? Data engineers. So I'm super curious to see how that evolves. And I'm already now when I'm talking to companies, I'm always fascinated by all kinds of ideas and all kinds of business opportunities that people are coming up with. And some ideas of course don't end up working so well. But I mean, there are some amazingly promising ideas out there. And I'm sure that NFTs will only grow tenfold hundredfold over the next three years. So I think it's pretty much inevitable. And the kind of a parallel that I always draw is kind of due to E commerce and the web. I mean back in 2000, even setting up an E commerce store, it took a lot of engineering work. Today you just sign up to Shopify or kind of, you go to Squarespace and you don't have to write a line of code and you can get something that works amazingly well. And I think it's inevitable that we will enter with machine learning infrastructure as well. But maybe it will take another five to 10 years.

Yeah, well, I definitely look forward to that time. That'll be a good time. But thank you so much for joining us, Vila. It's been really wonderful to chat about your projects and your thoughts on data science infrastructure. I look forward to seeing how metaflow grows and what you do in the coming years. Thank you so much.

Thanks Daniel.

Thank you for listening to Practical AI. We have a bundle of awesome podcasts for you@changelog.com including our brand new show Ship it with Gerhard Lazou, a podcast about getting your best ideas into the world and seeing what happens. It's about the code, the ops, the infra, and the people that make it happen. Yes, we focus on the people because everything else is an implementation detail. Subscribe now@changelog.com shipit or simply search for ShipIt in your favorite podcast app. You'll find it. Of course, the Galaxy Brain move is to subscribe to our Master feed. It's all Changelog podcasts including Practical AI and shipit in one place. Search Changelog Master Feed or head to changelog.com master and subscribe today. Practical AI is hosted by Daniel Whitenack and Chris Benson with music by Brakemaster Cylinder. We're brought to you by fastly on stream Darkly and Linode. That's all for now. We'll talk to you again next week.

King love.