MLOps is NOT Real

You know, the tools for model deployments today are largely not super accessible to data scientists. They're more accessible to folks that are machine learning infrastructure, let's say machine learning engineer. I feel like with the right tools you should be able to get a data scientist to export their model into a well defined container that contains a model that then you can hand off to an existing DevOps team and IT infrastructure that should not be specialized to machine learning. That step from a model to the thing that the existing DevOps teams and IT infrastructure could use that can be automated.

Hello friends. Jared here to tell you about changelog, our membership program for those of you who want to directly support our work. Your plus plus membership gets you closer to the metal with extended episodes, makes the ads disappear, and takes our audio to the next level with higher bitrate MP3s. You can join today at changelog.com. Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join us at PracticalAI FM community and follow the show on Twitter. We're PracticalAI FM. Thank you to our partners at Fastly for shipping our pods super fast all around the world. Check them out@fastly.com.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International. I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

I'm doing very well, Daniel. It's a beautiful spring day here in Atlanta and we are going to have a good time for the next hour or so.

Yeah, definitely. You know, it's interesting, the topic of MLOPS has increasingly come up on the show and we've had different takes on the topic and I'm really excited today to have a sort of different perspective on that and to welcome back to the show Luis seze, who is CEO of OctoML. Welcome back, Luis.

Thank you, Daniel. Thank you Chris. It's great, great to be back here. Had a lot of fun, you know, almost a year ago now.

It was almost a year. It's crazy.

Lots happened since then and right now it's also a beautiful spring day in Seattle as well.

So yeah, it was about a year ago we talked through some things about Apache TVM and OctoML. Do you want to give just like a quick update on maybe the Apache TVM world and then maybe circle back over to Octoml and what's been happening with Octoml in that meantime.

Yeah, absolutely. So on Apache tvm, lots of progress there. The community kept growing nicely and steadily with a lot of fantastic people doing work on machine learning systems, compilers and so on. So tvm, there's a lot of progress on automation and better performance we call performance automation to make it easier to get to high performance machine learning code in different hardware. We held our TVM conference in December last year, the largest ever. We had just about 1600 registrants and 700 people actually attending live, and then more folks that consumed the content after that. And it was really nice to see contributors to tvm, but also folks from the general machine learning acceleration community and hardware vendors participate and cloud providers participate and so on. Also related to Apache tvm, we announced the TVM Unity effort, which is essentially an effort on bringing together all the key threads in performance automation, extensibility and so on, and integration with the rest of the ecosystem front and center on tvm. So our view there is really not what we call, not too opinionated on how you actually get a model to run well on a harder target. It's about how do you actually enable people to do what they want productively, including using other pieces of the ecosystem. So yeah, TVM is moving along really, really well. So it's great to see. And now on the octoml side. So since May last year, we more than doubled. The team is about 130 people now. We've made significant changes to our platform, to the SaaS platform that uses TVM as one of its key components to automate the process of deploying machine learning models. And we recently released also a private accelerated model hub, which is a set of models that are pre accelerated to a bunch of different hardware. So folks can see the power of the platform in the process of getting models from the hands of data scientists into deployable artifacts. Yeah, and we also formed a lot of former partnerships with key hardware vendors like amd, arm, Qualcomm, and cloud providers like Microsoft Azure.

Yeah, that's awesome. You sort of mentioned this idea of the hub. We've definitely seen model and data hub sort of like just grow huge over the past year. Probably one of the things over the past year that we've really seen explode is like, you know, hugging face with, you know, 30,000 plus models now and other sort of hub environments. So yeah, it's interesting to hear how that that kind of idea is impacting a lot of whether it's people trying to optimize models for certain hardware or people just trying to try out things and that's. And that sort of thing has that impacted the. The types of clients and customers that are coming into Octoml because in general they can access models much quicker and then they realize, oh, these are slow.

Right? Yeah, no, great point. They absolutely has affected. So not just customers that comes our way, but also I would say the entire ecosystem. Because the way I see the maturation of these model hubs is that it's much easier for folks to find models to start from or even find models that already do what they need to do and just get them to deployment. So they really make it a lot easier for folks to get to a working model that does what they need to do. Which means that a lot of the action now shifts to how do you get these models to production, to add value as part of an application. Right. It's great to see our friends at Hugging Face making incredible progress in democratizing creating new machine learning models and creating communities around it. And our point of view on the model hubs is to complement that. So we're not talking about not a place where people come and find new models like Hugging Face and refine those models to create other models. It's more about here's some popular models that people can come and see pre accelerated to a bunch of different hardware targets and see how they compare across edge devices and cloud instances and so on.

Yeah, and maybe as a reference, I like to bring this up occasionally in the podcast because a lot of people are focused on that sort of training new models side and maybe like coming up with cool demos and such. But you know, the bulk of what happens in industry in terms of how you run models is inference. Right. So have you seen people like come in, they're really excited about like the demo that their data scientists, you know, created, but then just really blocked on that sort of like what is the, what is the typical kind of process and viewpoint that you see come into Octoml? Where do they already know what they want to do and they're just really blocked on scaling that up. Or is it something else?

Yeah, the part of the flow that we cover is you have a model that you want to deploy and then you have to navigate all the paths from a model to what kind of hardware you're going to deploy it on. And for that hardware, what kind of libraries and compilers and tools you should use to actually arrive at a deployable artifacts. And even just extracting the model from what the data scientists produce from model to working piece of software is something that takes manual work and then we automate that. But now this might be a good moment for us to step back and talk a little bit about MLOps. So thinking about the entire flow from data to a deployed model, that's actually adding value to a business or adding value to a user, there's several steps. You curate the data to create training data sets, and then you think about model architectures and you train models or you do some architecture search to find the right architecture for the model that you want. And then after that, once you arrive at a model that has the right statistical properties for what you want to do, you need to turn that into a working piece of software that you can deploy. And that step is very labor intensive. You have to extract code from. It could be a jumbled mess of Python code to go and extract a model that you can put in a box with a clean interface. Right. So you have to extract that, that.

Terrible button in your notebook that's like, export this notebook to a Python script.

Yeah. And then from there to a working piece of software you go and deploy is a lot of work. And then you have to go and optimize, make sure it has the right performance properties, such that it has the right latency in case it's interactive or has the right throughput in case it works in batch all the way to, if you're going to deploy it in the cloud, make sure that you find the right cost effect way of doing so with the right. And also have the right reliability and the expected behavior in deployment. In our opinion, by the way, is that model creation and model training does have its special place in the flow. I can understand why in ML Ops people think about those steps, but I think everything else comes after that, like how do you process a model into how to put a model in a container, and how do you monitor that model in deployment, how do you build CI CD integrations and so on. All of that should just be DevOps people are building and calling that MLOps as well, I feel like creates a lot of confusion because the way I think about machine learning models today, if you really zoom out a million feet, is they are an integral component of any intelligent application today, which is pretty much any application that we're excited about today can be called an intelligent application. So it has natural user interface, can recognize your voice, recognize gestures, it has rich media and it has machine learning components as integral part of them. But machine learning models are not treated as any other piece of software. They're treated as this special thing that's just hard to deploy, hard to integrate and so on. And we need to get past that, I think, to improve the cadence of innovation of these intelligent applications so people don't have to treat machine learning models in any special way. And this colors a lot in how I and we see at OctoML. The value that we can add is really enable folks to treat machine learning models as if you were any other piece of software. Right.

So you have no idea how relieved I am to hear you say that because that's like a huge hot button issue for me. I mean, that's like Daniel's heard me rant about this repeatedly over time is, you know, the model is just part of the software at the end of the day, you know, that's. It's a model that's not, is not usable. So, you know, as you're looking at deploying it out as maybe MLOps kind of, you know, outgrows its diaper and gets into the big boy pants of DevOps and actually like becomes part of the real world around it and usable. How is that changing? You know, we talked to a year ago and kind of had some of the same conversation, but I know this is a very fast moving ecosystem and the evolution at it. So can you talk a little bit about, you know, like, you know, I shuddered to say it as MLOps is kind of growing up and as hopefully it gets further and further recognized and integrated into DevOps. How do you see that evolving over time into maturity? What does that look like to you?

Great question. So I'd say that one thing that's changing fast, that I think people are starting to understand or have some common view of what ML Ops is. Because if you ask 10 people in this space what MLLOPS is, you're probably going to get 12 answers because some people are going to have multiple answers to that too. I think there's clarity being brought there on what is it that should have a different name? If you allow me to be cynical for just a second. Sometimes people like giving names to things because it makes it easier for you to grab attention from the investors, investment community from investors. It's easier to grab attention from folks that are more on the, hey, the cool technology side and let's give a name to something that looks different. But I feel like sometimes if you give a new name, something that already exists, that you can just do better can cause a lot of confusion. Right? And I think that there's some maturity just that's starting to happen on what people call MLOPs. And I'm glad to see that most of it is going more and more towards how you deal with data and how you create models. Because everything else I think should just be called DevOps, Chris, and shouldn't even be called ML Ops.

We're building solutions, aren't we? We're building big solutions to solve real world problems and these are all parts of that solution.

That's right, exactly. And then to answer your I completely agree with that 100% and to answer your question more directly, what I see changing even in that space is a couple of things. Like one, and I would say a year ago there was a lot more emphasis on end to end fully integrated platform like platforms like you get SageMaker or you get Azure ML or some other big tools. But now I think there's a lot to best in class for each one of the steps in this flow and have clean integration points so have tools to deal with data that has a clear integration point with how you move on to the training step, how do you move on to say network architecture search to how do you package the model and how do you actually monitor the model in deployments? Where we sit in that flow is again and we have an API that allows you to we take a model as input and we produce your deployable artifacts. There's a lot of evolution. I can't talk about all the details because otherwise my marketing folks are going to be mad at me. But you're going to hear more about what we're up to soon. But basically in a word, we're all about automation. So automating the manual steps of getting working software model from say your Jupyter notebook, your Python scripts and put into a container that you can deploy. So that's highly specialized to the hardware target. But once we do that, if you put this in the right format, you should be able to use your regular DevOps flows. If you have the right API, you can use GitHub Actions for example to do CI CD on your model. As you change your model, you run through this flow and then if you put in the right container format, you can use existing microservices to serve your model. And then if you also put the right hooks for monitoring, you can collect data from these deployed models and put it in a datadog way of visualizing your data. You can put views on top of that to look at model behavior. So that means that I think a lot of the work being done in model monitoring today is really, really important, but I think it's less about the MLOps part of it. Of like, how do you collect the data? But it's more about how to abstract it away and find higher level behaviors for models that you should go and debug because those things are different than how you debug software today. Right, sorry. There was a very long answer to your question, Chris. I don't know if it was an answer. It was more like a quick tangent there.

We are going to ship.

3, 2, 1.

I'm Gerhard Le Su, host of Shipit, a show with weekly episodes about getting your best ideas into the world and seeing what happens. We talk about code Ops infrastructure and the people that make it happen. Like charity majors from Honeycomb.

We act like great engineers, make great teams. And it's exactly the opposite, in fact, it is great teams that make great engineers.

And Dave Farley, one of the founders of Continuous Delivery.

Start off assuming that we're wrong rather than assuming that we're right. Test our ideas, try and falsify our ideas. Those are better ways of doing work.

And it doesn't really matter what work.

It is that you're doing.

That stuff just works better.

We even experiment on our own open source podcasting platform so that you can see how we implement specifically tools and services within changelog.com what works and what fails.

It's like there's a brand new hammer and we grab hold of it and everyone gathers around, we put our hand out and we, we strike it right on our thumb. And then everybody knows that hammer really hurts when you strike on your thumb. I'm glad those guys did it. I've learned something instead.

Yeah, I think that's a very interesting perspective, but I don't see that way. Okay, it's an amazing analogy, but I'm.

The sure that applies here.

Listen to an episode that seems interesting or helpful and if you like it, subscribe today. We'd love to have you with us.

So one thing I was thinking about, Luis, as, as you were talking about, this sort of transition from ML Ops to DevOps is just, I guess, the team dynamics that are at play here in the sort of human dynamics. And I'm just thinking like in my own experience with the teams that I've been on and working on AI models and different applications, there is this real sense that you can do a lot, like you say, GitHub Actions. I love GitHub Actions. I do so much with that. But that sort of onboarding into that for someone coming from PhD scientist route, and they have no idea this thing exists. Right. And as soon as they find out, they can you know, they're smart, they can grab onto it and use it and figure out ways to use it or things like datadog, like you were talking about other things out there. How much of this sort of confusion in the terminology and the workflows here is caused by this sort of mismatch of what teams are aware of and their sort of spheres of knowledge versus actual functionality differences?

Yeah, no, this is a great question. So I would say that a key aspect in the human and team dynamics today is that you have folks that create models, typically data scientists or some people call them ML engineers or data engineers. And then once they arrive at a model, they hand off to a team, if the company is big enough that goes and turns that into the deployable thing. And that team is still special compared to the DevOps team. So these are folks that are more sophisticated engineers that understand machine learning, understand the tooling involved and then put the work and then put that in a format that you can actually go and deploy. And you write that the tools for model deployments today are largely not super accessible to data scientists per se. They're more accessible to folks that are machine learning infrastructure. I'd say machine learning engineer. And I think this can also change. By the way, I feel like with the right tools you should be able to get a data scientist to explain exports their model into a well defined container that contains a model that then you can hand off to an existing DevOps team and IT infrastructure that should not be specialized to machine learning. So I think that step from a model to the thing that the existing DevOps teams and IT infrastructure could use, that can be automated. And I think there's still a lot of work to make that automated because it involves human engineering today. But fundamentally it can be automated. And once you do that, I feel like you make both teams productive. Data science scientists can focus on making the models and then the DevOps teams can focus and continue making their applications run well and integrate models. If you were any other piece of software and bring best practices to machine learning deployment. Right.

It's funny, when you were talking about that a little bit, I was thinking you're solving two problems. You're solving the problem that you're describing, but you're also solving the problem that these data and development capabilities in organizations, it sounds like high school, you have the job and you have the nerds and you have different groups socially that are doing stuff and you're going to automate the whole thing and it's going to bring everybody together, which is a good thing, it's a good thing in this world to do that. So I'm curious, as you were bringing everyone together, how does that change the dynamics for organizations working for these different individuals that have different functions right now, and they're off in their little silo and they're trying to interact. As you get those automations, it sounds like it'll make it more efficient. How does that look to you? What kind of workflow are you striving for as you achieve this?

Good question. And again, this is all happening and maturing fast. Our current view is that with the right automation, again, you could have the folks creating models do the best job they possibly can in creating the models that have the right properties. And then on the DevOps side, they can focus on continuing models and deploying software the way they do. And I think that the dynamics are going to change. That organization wouldn't have to go and look for people that, that are actually specialists in both, because that's the reality today. So this teaming between folks that create models and folks that run the regular DevOps, these are people that again, as I said before, understand machine learning. They understand the tooling around all the guts of say, TensorFlow and PyTorch. And what are the right libraries to use depending on the hardware target, what are the right compilers to use? For example, should you use TensorRT for Nvidia, should you use TVM in case you want to have broader options of hardware? And then you have to do performance evaluation to make sure that you're getting the right performance from your model running on the chosen hardware. Often they have to make decisions about the procurement. So if you're going to deploy it in the cloud at reasonable scale, chances are that your model is going to have a line item on the budget because it's going to cost a lot of money to run at scale. So you have to go and understand what are the cost implications. And these are the kind of things that none of these teams are used to. It's not their strength. So DevOps don't understand enough machine learning to do that. And then data scientists don't understand enough of the systems aspects of to do that themselves. So I think that the dynamics that's going to change is if we automate this right, companies wouldn't have to go and look for these kind of people that are specialized, that understand this intersection. And I hope this is going to make it even more accessible for users to put models into production, because they don't have to go look for people like that. They don't have to have the systems expertise coupled with machine learning to make use of machine learning.

Yeah, this point about automation is a really key one and I'm trying to find like other. You may have some examples outside of the machine learning world where there's reasonable parallels. But I was just thinking about like cyber, something like cybersecurity or something like that. Like at a certain point, I think if you look at kind of the history of how things developed, you really like had to have very specialized people having very specialized knowledge to understand like what sorts of vulnerabilities are in my software, cybersecurity wise. And to some degree those sorts of people are still very valuable and they have their place. Then you come along and there's now systems. I was just looking at one called Snyk where it's like you can just run this automation suite on your software and figure out all the various vulnerabilities and the open source packages and the dependencies that you're importing and where you're doing something wrong in terms of exposing this or that. And that allows any kind of person that can stick that in a kind of DevOps workflow to be able to enable better security for their application. Maybe not prevent everything, but you certainly can do much better. And I wonder if, I don't know if that's a good parallel for what you're talking about here, where some of these things are still at that stage where you really have to know about all of these granularities, but as automation kicks in, a lot of those things are going to be taken care of.

Yeah, no, that's a fantastic analogy. Really fascinating. Daniel. So, yeah, I was thinking, when you described it, I was thinking about Snyk. Exactly right. So because a lot of the tools like Snyk can plug into your DevOps workflow and say whenever there's new code committed, you're going to kick off your static analysis to find for vulnerabilities. You're going to make sure that all the open source code used there has actually been vetted and has checked all of the security properties. Right. So. And that automation, I think, brought a significant progress in producing secure code. The parallel there is valid. The only thing that I think is so different is that you have very, very different kind of people, like people writing regular code today that is subject to the flow like Snyk would use to go analyze the code are typical software engineers. And then you may not need a security engineer. But, but even if you did a security engineer, it still kind of Thinks like a software developer, except that they know what all the best practices for security are. The kind of automation that we are talking about in machine learning, I think it's deeper and different because of the following. So first of all, the difference between somebody who can create models and somebody who can write system software and then can write software to go and deploy, it is so much wider than what you typically have in between folks that worry and don't worry about security today. And then second, the kind of automation that's needed, it's still pretty deep, right? So as you're saying, if you're going to export your model from a jupyter notebook, to turn that into a workable piece of software that you can go and deploy requires a lot of manual software engineering that hasn't been automated yet. So there's still a lot of work to be done. I think that kind of automation is not quite as well defined, I would say, and as clean as the automation required going and analyze code for security vulnerabilities.

So as you're talking about this, I'm kind of trying to visualize the description and I'd like to ask you, can you give me kind of a concrete example of how this is evolving now that either would have been much harder, like a year ago when we spoke last, or at least, or maybe even not possible in the sense of given the constraints that most organizations are dealing with, it could be anything. But what's a typical use case that you feel that you're enabling at this point going into it?

Good question. So let me give you one specific example. Suppose that you have a computer vision module in your application today because you're going to go and verify whether images don't have anything inappropriate in them when you upload to a blog interface, let's say, okay, so the way you do that today, you have to do it happened in the past, like let's say a year ago, is to go and maybe you find a model, you're going to probably put quite a bit of work on this model to make sure that is actually classifying appropriate content correctly and so on. This is all done by the data scientists, data engineers, I would say, and machine learning creators. Let's use that label now. And then once you arrive at the model, you have two options. You just go and say, you know what, let's just use the regular say PyTorch or TensorFlow serving mode and just hope that that's fast enough. If it's not fast enough, then you're probably going to hire A consultant to go and help you. Okay, so you're probably going to hire or go hire folks to do it. And now. So I think with tools that like tvm, some things that we're building, and some other folks that are in this space as well, I'd say essentially enables one to take that model and go through not just the default path and package with existing libraries that are not optimized for the Hatter, but help you choose. All right, if you're going to deploy on an intel processor, what is the right libraries to use? If you're going to deploy on Nvidia, should you be using a Tensor IT compiler to generate a more performant version and wrappers around that to go and run that for you and more easily produce a higher performance output? Even in the last year that's already getting a lot easier. But even then after that, you still have to get that output and put an interface around it that has just the right API to integrate in your application. That's what we firmly believe that we can automate even that really. Just go from you upload your raw PyTorch or TensorFlow model to the service and then you get a package ready to be deployed with the right interface that you define. Right?

Yeah. This might be like a sort of off the wall question, but I know also, Luis, that you do some teaching and lecturing and other things. And I'm wondering in a lot of the workshop, even the ones that I've taught at various places, a lot of the focus is on that sort of model creation pathway. And I find increasingly the people that I interact with in industry, once they're sort of in a position, are really not even aware of some of those components that you just talked about, like model optimization, like the different ways of serializing models, the different ways of serving models, like batch inference or other ways of applying models. And yeah, I'm just wondering if you have any thoughts on is that true across the way that we're kind of bringing up the next generation of machine learning practitioners and are there ways that we can maybe shift the balance a little bit?

That's another great question. So you're right that a lot of the way that folks are learning about machine learning today and getting started with it, they're not thinking about model optimization deployment because luckily we actually create pretty good tools to arrive at models and test them and make sure that they have the right accuracy, it has the right properties. But chances are that most of those models created wouldn't actually see deployment. They wouldn't make it to deployment. So it's good for people to learn, but chances are they wouldn't be deployable. And honestly, I wanted to stay that way even for the sophisticated users. So let me talk about the other users. These are folks that have already done this for a while and now they are improving models in a significant way because they're doing things that haven't been done the way they wanted before. Right? Today, people already have to start worrying about performance too early. They have to like, okay, if I make this change in my model, am I going to be able to deploy it? If I can deploy it, should I deploy in these CPUs or GPUs and so on? People should not worry about this. I want human power, I want human creativity ingenuity to go into making a better model and not have to worry about systems aspects too early because that's going to constrain the way they see the model. It's as if as you're designing a new feature for a car, you're all thinking about all the different ways it's going to be used too early and then you're not innovative enough because you get constrained by practicalities before you're able to unlock your creativity. If you don't mind, just go. On a quick side comment. I like where Hugging Face is going, where it makes it very easy folks to go. Start from an existing model, make modifications from a foundation model they talk about, and then specialize specific use cases. And then you don't have to worry about some of the systems details yet when you actually put into deployment, chances are you're going to have to worry about it because performance is going to come and bite you if you don't. But at the conceptualization stage and model creation stage, I don't think people should worry about details of how these models are going to be implemented. I want it to stay that way, but I want to stay that way and still allow these models to actually be deployed. Right.

So I'd like to follow up. It's actually kind of combining what you were just addressing with, with Daniel's previous question a little bit, because the two of you together really got me thinking about a different way in that sense. And that is, you know, even now with all the model creation and all the detail on that, and yet that's not an industry. You know, we're, we're doing, we're using foundation models and there's a lot of transfer learning. And typically students aren't doing this off the bat. They're. They're Kind of going through it the hard way to teach the stuff, but. But then they get into industry and they're struggling because they only have a narrow focus of the picture. So as you're doing this work, and as Daniel brought that point up, I'm wondering, does it make more sense then to, by using these kind of tool, these kind of tools to get more of an end to end learning process going and you're, by providing the right ecosystem and tooling as you're doing, you're essentially saving them from wasting what creativity they have on the wrong thing because you're kind of helping them through that, but at the same time you're crucially helping them understand the full end to end workflow on how to get it out there. Is that the way of teaching AI going forward in your view?

Yeah. So I'm so glad you brought this up because when I said that they shouldn't worry about all the practicalities of deployment, that meant that they shouldn't be constrained, but should still have an idea of what would it take to take their models into production. And the more automated we make that, that the more in the loop of model creation. You can put it, for example, succeeding where we want to be on taking models, fresh models and deployable artifacts, evaluated, benchmarked and so on. If you make that fast enough and productive enough, you can actually put this in the active loop of as people are developing models, they can just go and try them out. But not the way it is today. Where I have to think about is it a CPU or gpu, what kind of gpu? And then go and benchmark. If it's just completely automatic and seamless. You can always get for all the versions of your models as it evolves. You can see how well this model would do in production and in various scenarios and just doing model creation. So think of it as an outer loop of what they call network architecture. Search even outer loop from that is for all the candidates models for what I'm thinking about doing. Just give me an idea of how well this way of thinking about the original model would do in production. If you don't mind me, add one more thing that's not related to what you talked about, Chris, but it made me think of another thing that we've been observing that I think is interesting. Even though we're talking about a model, say model A or model B and how to take that model to production, the reality is, as we're talking about models are integral part of applications of bigger applications and it's not Just a single model, by the way. It's all an ensemble of models. You have computer vision, you have language and you have decision trees and you all combine this ensemble of models that they might talk to each other directly because there's a data flow from model A. Output of model A could go to input of model B. But also even if they don't, chances are they're actually in a package running some machine in some container in the cloud that interact with each other because of performance. These are all system aspects that we have to worry about because in the end you're going to have to package all of these into modules that it can actually deploy. Right.

That's a great point you're making.

The ensemble of models is something that's important. Right.

As you're talking about these things and going from this sort of ensemble and systematic thinking and these things interacting with each other. I'm wondering about some other jargon that has come up on the show in the past and is, I think related to some of this automation and workflow sorts of things. And that's this sort of low code, no code type of type of stuff. There's increasingly this like messaging around, like we're making AI and machine learning systems low code, no code, et cetera. And you know, I have maybe my own sort of opinions on where we're at on that spectrum and where, where things could go or might go. I was wondering, you know, maybe your perspective on that and like, how far can we push the automation aspects and the things that are. Where are the machine learning engineers and the DevOps engineers really going to sink their teeth into the lower level like code? And where are those opportunities really for the low code and automation pieces that are maybe a little bit hyped at the moment.

Yeah. So good points and juicy topics. So I would say that for model creation, I see a path to low code, no code, and it's kind of going that way. At least low code, I can see a path because there's ways of you defining the data set, how partition it. There are tools now to better label the data and create that. And this all has very little to no code. And then when you think about classes of models, you can imagine just being a high level choice and not a programmatic thing, that you're writing a piece of code and I see a path there. But now from that to deployment, I have a harder time. I can see some code. I don't call it low code because you still have to figure out, okay, what is the API again going Back to thing that we keep repeating every five minutes. Models being part of applications, they still have to define an API, it's going to call that model and then to be part of the rest of the application, that part I still feel like again, maybe the rest of the code is already we succeeded and all of that is already low code and no code. I don't know if that's going to be the case, but you have to have a very well defined API for that to work. And that involves some significant amount of code, fundamentally, in my view. And then there's some other aspects that when you're actually taking a model to deployment that you have to think about that I think are important. Like for example, is it latency sensitive or is it throughput? Like for example, are you going to care about how long each prediction, each inference takes or are you going to care about oh, if I do overall, this piece, a bucket of data, that you're going to have the right throughput. And these are things that you have a lot deeper systems thinking there, that even if it's low code, it still requires a lot of deep understanding of what you're doing. By that time you're going to require specialized people. That doesn't really matter whether it's low code or no code. Right.

I got a follow up and that would be if we go back to addressing the low code, no code approach to model creation or more specifically optimization. There are a lot of known things and you can train a model to do the optimization along those lines and we're seeing that.

Great to see Autonas Automl works finally happening, right? Yeah.

Yeah. I'm wondering if one of the challenges as we're addressing deployment and recognizing that people focused for a while on model creation way more than getting the model out there in a usable fashion and that we're getting mature about that now. Your organization and there are others out there too that are thinking very deeply about this. Do you think that there's an opportunity for maybe code, no code approaches once that we arrive at more kind of standardization? Right now it's still very early, it feels like in terms of different approaches. So there's a real custom feel to how you're going to deploy, especially when you combine it with all sorts of edge targets, with edge being a catch all phrase that could be almost anything. And so there's a ton of specificity to your target at this point. Do you think over time as those get categorized and kind of best practices emerge, that there might be more opportunities? Opportunities or do you Think that's still going to be a challenge given some of the API concerns and such that you mentioned in your previous answer.

Yeah, I would say that again. Once how the model is used in an application as well defined in APIs is settled and we're talking about more the evolution of the model, I can see that being very low to no code, because by that time you've already seen a path to deployment, that you find the box where it fits in and then I can see the model evolving and having very little code for the model updates. And to your point specifically, if you're going to deploy it in the edge and you have to see how it works on a wide variety of devices, I also see a path to automation for this model that you created. It's going to work on 85% of the phones, for example. I think going where automation is going, having the ability to benchmark it across all sorts of scenarios where the model is going to be deployed and validate that it's going to work across the set of devices where you care about. I can imagine a feedback loop with the model create that says okay, for these decisions that I've made, it's going to work on these classes of devices. It's a great point. You made me visualize that in a clean way. I can see that happening, but again, only after you've defined where the model fits in the larger application.

I'm relieved to hear that because I work in an industry where absolutely everything that we target would be on the edge. Oh, on the edge. So if you hadn't given me some hope right there, I was going to start crying on that. So thank you, I appreciate it.

I always like to step back and think about models being part of a bigger application. That big application was written by someone I had to put a lot of teams that had to think clearly about where is the model going to fit. So once that is defined, I can see a lot of the rest being automated and being very low to no code. I think that one way of actually summarizing this, if we nailed how to turn models, trained models, and by that point you can do this in low to no code into these agile performance, reliable pieces of software that you can integrate throughout the application. Once we nail that automation, everything's going to get easier in my opinion, managing applications and also creating better models, because then you have separation of concerns the way that I think needs to be done here. Given that machine learning folks are going to think creators are going to think very differently than the hardcore software engineers. That are on the other side of the application building. So we'll be sure it stays that way.

Right?

Yeah. So maybe as, as you look at hopefully that future, what is your sense of like over the next, you know, the next year when we have you back on the show a year from now, what are those things that you would really hope that like are maybe enabled that aren't at the moment and are, but are sort of achievable within the that kind of timeframe.

Yeah. So I would say again, being able to get a model that was freshly created by a data scientist, a machine learning, without thinking about systems aspect of deployment, having their models benchmark, knowing how it runs, you're going to know how much it costs and going to click deploy, it's going to produce a box that the DevOps folks can go and deploy, integrate with the application. I think we have line of sight to that. And I hope next year in the show here we're going to be talking about all the ways that people are using that. But now if you ask me to think about what about five years out, I think what's interesting to think about here is that how do you even stop thinking about what's edge, what's in the edge, what's in the cloud, what runs where, and just think about, hey, I want to solve this problem that involves machine learning. And yes, so users are on the edge, computers are in a building somewhere, spread all over the world. Right. And I think AI, an application creator, I don't want to use the word developer. An application creator here should say, should be able to specify here's what they want to do and then the system should automatically figure out, okay, so we need this kind of models and this one should run in the edge, this one should run in the cloud, this one should, might run in a cell phone base station and automatically split what should run where that should be done automatically because again, that's an optimization problem that once you define the constraints, you should be able to plug in place the right piece of your model in the right physical place automatically. And if we nail the automation that we just talked about from a model to the thing that runs in the hardware really well, that part is already also done. So now we can go and work on a higher level problem like how do you break down your model into the pieces of what's in the edge, what's in the infrastructure, what's in the cloud, just the right way. I also think that that should be possible. Right? So because hey, we have machine learning, designing chips Today that's pretty hard. Right. So you have machine learning design, you know much better ways of doing power management in large scale data centers. These are all low level things that we used to do by hand or with heuristics. Right. And we abstracted that away as this goes up and up the stack. I'm very excited about the future of creating exciting applications without having to worry about all the design constraints that we have to worry about today.

Awesome. Yeah, well, I'm very excited about that future and as always, it's a pleasure to talk to you, Luis. And I'm really excited by the things that you and your team at Octoml are doing. Make sure and listeners, make sure and check out the show notes for some links to some really great stuff both for TVM and for OctoML. And yeah, thank you again for taking time. Luis, it was a pleasure.

Thank you. Daniel, thank you, Chris, always a pleasure. And again looking forward to coming back and talking about all the other new stuff the next year. So thank you. Thank you. You guys are fun. So bye bye.

All right, that is Practical AI for this week. If this is your first time listening, subscribe now at PracticalAI FM or just search for Practical AI in your favorite podcast app. We're in there. And if you're a longtime listener, please do share the show with your friends. It is the best way you can help Practical AI succeed. Thanks again to Fastly for shipping our shows super fast all around the world to break Master Cylinder for the beats. And to you for listening. We appreciate you. That's all for this week. We'll talk to you again next time.