Apache TVM and OctoML

There's literally billions of ways in which you can compile the same model on the same hardware target. The question is, how do you pick the fastest one? You cannot try them all because they'll take too long. So you have to use intuition. That's what software engineers do. We replace that intuition with machine learning based optimizations that learn about how the hardware behaves. In the press of optimization uses those models of how the hardware behaves to tune and search the right way of optimizing and compiling your model to the hardware target.

Big thanks to our partners Linode, Fastly and launchdarkly. We love Lino. They keep it fast and simple. Check them out@lino.com changelog our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by LaunchDarkly. Get a demo at launchdarkly.com this episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal tech books and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos and all their conference talks. Plus you can learn by doing with live online training courses and virtual conferences, certification practice exams and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics, machine learning, AI, programming languages, DevOps, data science, cloud containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your or yourself up to speed on their tech skills, Then check out O'Reilly's online learning platform. Learn more and keep your team skills sharp@o'reilly.com changelog again o'reilly.com changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community and follow us on Twitter. We're @practicalaifm.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a principal Emerging Technology strategist at Lockheed Martin. How you doing Chris?

I'm doing very well. A little bit chilly here in Atlanta. I don't know what's going on with that. Mid May we're supposed to be sweltering heat.

Yeah, it's a little bit chilly here as well. I don't know what's chilly for Atlanta, though. Depends on your reference frame.

I don't know, I'm guessing it's like 60 out there or something like that, you know.

Okay, that sounds pretty delightful to me.

Actually, but I mean, if it's not 85 by now, it's just not normal, so.

Yeah. Well, what are you keeping busy with these days?

Just lots of technology stuff at work, having a good time with it. And I'm super excited to pause in my day job so that we can have a good conversation about aiml.

Yeah, yeah. And specifically, I know in my job recently we've been working on a few different deployments of the same model. One to like an edge device which is disconnected from the Internet, one cloud deployment and then another one for an on prem system. And that carries with it, of course, all sorts of joy in terms terms of fitting into resource constraints but also optimizing a model for different deployments. And I think that's definitely one thing that I don't know if anyone ever pitched being a data scientist to me, but I don't know if that was part of the pitch, like optimizing for certain hardware and all that. It's cool, but it's definitely sometimes laborious and hard. I don't know if you've run into this in your past.

Yeah, you have to be a data scientist and a software engineer to do that. Right.

So you know what struck me as you were saying that, Daniel? The thing is, I'm kind of doing a similar, similar thing, but being in the defense industry, we have a completely unsoftware, like name for it. We call that joint All Domain Operations. Nobody outside this world would ever. They're like, that's software, really? That's AI. Really?

Wait, what acronym is that?

Yeah, that's what we call it. We call it jado. Yeah, we call it Jado and stuff, but you'd never know. I don't think anyone outside my industry would ever associate JADO or join All Domain Operations with software and kubernetes and AI deployment.

I would have no idea what that meant if you gave those words to me. But I think we'll both be enlightened today quite a bit on this subject. I'm really excited because we have Luis Seze with us who is co founder and CEO of Octoml and professor at University of Washington. Welcome, Luis.

Thank you. Thanks for having me in the show. It sounds like you guys are fun to talk to. Can't wait for the conversation here. And it's A nice sunny day in Seattle.

By the way, we don't take ourselves too, too seriously, trust me.

Oh good. I love that. That makes even. Even better. Just to finish the weather theme of the conversation. It's a nice sunny day in Seattle in the 70s, you know. So I'm wearing short T shirts here.

What's up with that? He's in Seattle and he's having the great weather and you know, Seattle's famous for rain and we're here in these other parts of the country and it's just kind of.

Isn't it though? I mean Louis, correct me if I'm wrong. I mean summertimes in the Pacific Northwest are quite delightful, aren't they? It's just like the winter, like rain and fog and.

That does sound nice.

Yeah. Yeah, that's great.

Yeah.

Winters are long, but the summers are awesome. So there's some balance in the universe and nature here, especially for a native Brazilian like me, that matters, you know. So happy. A nice summer. Yeah. Well, thanks again for having me on the show. It sounded like from your adventures with machine learning model deployments, we're going to have a lot to talk about here, so.

Yeah, for sure. So before we get into all that, maybe you could just give us a bit of a background on yourself and how you got interested in this topic and ended up working in the areas that you're working now from your moment.

Of birth right up until this recording.

Exactly. Great. Perfect. So as I was mentioning before, I grew up in Brazil and the funny thing is I like to joke that I'm in the 20th year of a three month internship in the U.S. i was in school in Brazil. I was recruited by IBM Research, this is in the early 2000s to work on the Blue Jean project. There was a three month internship that became a year long co op that do anti next and I went to grad school in Champaign Urbana, Illinois, close to where you are and then after that came to University of Washington as a professor and next thing I know 20 years have gone by. So anyways, at IPM I worked worked on harder software co design. We were part of this team building the blue gene machine, the first blue gene machine. And this was like hardware and software designed for very high performance molecular dynamic simulation and a lot of it like if you look at the kind of workloads that we ran there was high performance linear algebra and then you have HPC systems with all sorts of internal communications reductions, pairwise communication and so on. So after IBM I went to grad school in grad School, I did work on hardware and compiler support for speculative parallelization, essentially making it easier for folks to write parallel code by not having to prove that the code is actually parallel. And then I came to uw. At uw, I started in my faculty career, started working on harder software code design for emerging applications. So I had done a bunch of work on approximate computing. Essentially the idea is to take advantage of the fact that some applications don't have to have perfect execution to have a meaningful and useful answer. So right. Simulations are like that, there's multiple valid outputs. And of course machine learning is a huge workload in that category. That's how it started, actually. Started working on hardware and software optimized for machine learning, for energy efficiency, better performance. And then about six years or so ago, I started collaborating with folks in the core machine learning group here at uw. Carlos Gastrin, who's a friend of mine, also Brazilian, Brazilian, transplant to the U.S. although he's Brazilian, Argentinian, transplanted to the U.S. and has a long history in machine learning systems. And we were chatting six years ago or so, there was already a growing set of machine learning models that people are interested in and a growing set of hardware targets. It's kind of crazy to think about that six years ago GPUs were just starting to get popular for machine learning. It feels like that has always been the case. But six years ago that was just kind of picking up and people were starting to think about.

A lot's happened since then.

I know, yeah. And then now there's cpus, gpus are picking up. We start talking about fpga's they had for a while, but at that time was getting heated again. And then people started talking about building accelerators. So we were wondering like, oh, kind of interesting that there is this growing set of models already starting to fragment ecosystem with TensorFlow, PyTorch didn't quite exist yet, but there was already some fragmentation going on. And now growing set of hardware targets. So now imagine this cross product of models, frameworks and hardware targets. It's already becoming a pretty complex cross product. So we start thinking about, oh, wouldn't it be nice if we had a common intermediate representation that allows you to do high level mod optimization do and generate specialized code for that model on specific hardware targets. Tianqi Chen, who's an incredible machine learning systems researcher, was Carlos's grad student at that time. And Carlos and I started co advising Tianqi and that was the genesis of let's look at machine learning, model optimization and compilation as A way to actually optimize and deploy machine learning models. And I had other grad students there, Terry Moreau, who, who was working on specialized architectures for machine learning, got interested as well. So he quickly became part of the mix and started working on FPGA backends for machine learning models as part of this mix. So that's how I got into that. In the end, everything boils down to high performance linear algebra, safe for machine learning. And I had a history with that at IBM and everything kind of came together. Approximate computing, high performance linear algebra and machine learning in the genesis of the TVM project.

Awesome.

And maybe we could just sort of take a step back. You talked about like machine learning compilers, we talked about intermediate representation maybe for that person out there that has taken the Coursera course. And they went through the Coursera course on deep learning something and hey, it never talked about machine learning compilers. They learned maybe how to do a training thing and they created an image classifier and then they learned how to do an NLP thing and do autocomplete or something and then they end the course and they know how to use TensorFlow and do some data pre processing stuff and maybe even a little bit about GPUs, but they never heard this. Machine learning compiling. Could you just sort of.

Yeah, absolutely.

Yeah. Where does that fit in?

Yeah, I'm going to translate all of that. You're basically saying, wow, I'm really missing the audience here. Let me translate all of that here. All right. When you write a typical machine learning model, by the way, I'm really glad that machine learning tools are becoming more and more accessible. More and more people can build meaningful and useful machine learning models, either starting from an existing one or tweaking and creating new ones. I'm so glad that the tools are getting much better for that. But now when you create a machine learning model though, in the end you need this model to actually run well on whatever hardware device you want them to run. Today this is typically done by having a set of libraries that implement parts of your model. Like different layers will have different, often hand coded implementations. And then your high level framework like TensorFlow or PyTorch just gets your model and calls these hand tuned libraries that either Nvidia provided or AMD or Intel, the hardware vendors typically provide those libraries and then the frameworks stitch everything together to produce the thing that runs on your hardware. Which is all well and good, it's nice and easy as being able to build pretty complex systems that way. Except that now as we build richer and richer and more complex machine learning models that really need to make the most out of the hardware they are deployed to, which is great. So it's kind of crazy that you can build gigantic language models or very complex computer vision models that do a ton of computation that's really pushing at the limit of how fast you could run just by writing on Moore's law. So you need to be able to squeeze much more performance out of that. So that's where a machine learning compiler comes into play. When you write a piece of code in language like C, for example, you run through a compiler and you run it on your hard attack, you don't even think about, yeah, you run through a compiler, you run it, but for machine learning we don't do that. So you just interpret the model. What a machine learning compiler does is essentially treats this process of going from your model to what runs on your hardware as a compiler problem. What do you mean by that? I mean by translating your model into a representation, as we call the intermediate representation, that enables optimizations of your model. For example, you could fuse a layer with the next one, say a fully connected layer, which would be matrix multiplication followed by a convolution that you chose in your model. Potentially it can fuse that them, treat them as a unit and generate nice new fresh code that specializes to your model to run on your hardware target. So, and then as a machine learning compiler, what you get as an end user, the benefit that you get is your model becomes a highly optimized executable for your target hardware. And the difference of performance can be huge, like 2, 3, sometimes 30, 50x better performance than your stock high level framework, execution time and speed. Right. So does that make sense essentially? You treat that basically what you take for granted in writing regular software. You know it does that for you in machine learning, right? To produce a fresh, highly optimized binary for your model.

Yeah, and that makes a lot of sense. And it sounds like a lot of what we're talking about here is performance in the sort of maybe latency and resource consumption type realm when we're thinking about compiling, Is that mostly what we're concerned with? Or do some of these compilations like for example, does the actual performance of the model's predictions, does that ever come into play when you're doing this sort of optimization as well?

Great question. So performance here is ambiguous. For a computer systems person, performance typically means how fast it runs. But for a data scientist, performance also means how good is statistical properties of your model. So by and Large machine learning compilers do not change the accuracy of your model. There are optimizations. For example, you can do automatic quantization, you can quantize parameters in your model that can change the behavior of your model. In that case, you do trade off some accuracy for better execution time or for better system performance, right? But for the most part, in fact, what we focus on in Apache tvm, our transformation optimizations do not change the accuracy of your model at all, even though we do support quantization so on. By and large, the way it's used is it just compiles our model faithfully to run without any changes in accuracy, right? And to tie back to your comment earlier of deploying multiple hardware, say, you know, deploying on the edge, getting the right performance and using the right amount of resources that fits in your heart is something that's an extremely laborious task. If you have a model that's going to run on a Raspberry PI like device, say it's computer vision and the first version of your model runs at half a frame per second and uses too much memory and doesn't leave any other compute resources. For the other things you need to run on your device, you can't really deploy it. Our experience has been with several users and customers. Is that the process of getting a model that is ready from a data science point of view, from the accuracy point of view, to be able to be deployed, it could take weeks to months of hardcore software engineering work. And that's what we want to automate with. Apache TVM and other machine learning compilers do similar things. But Apache TVM is especially good at doing that because it uses essentially machine learning for machine learning. In the process of translating your model to an executable, like a deployable artifact, it's a search problem. So there's literally billions of ways in which you can compile the same model on the same hardware target. The question is, how do you pick the fastest one? You cannot try them all because that will take too long. So you have to use intuition. That's what software engineers do. We replace that intuition with machine learning based optimizations that learn about how the hardware behaves in the presence of optimization. Uses those models of how the hardware behaves to tune and search the right way of optimizing and compiling your model to the hardware target.

This episode is brought to you by Snowplow Analytics. Snowplow is the behavioral data management platform for data teams. Maximize the value of your behavioral data using Snowplow Insights, a managed data platform that's Built on leading open source tech leveraged by tens of thousands of users. Capture and process high quality behavioral data from all your platforms and your products and deliver that data to your cloud destination of choice when marketing needs to make data informed decisions, when product needs next level understanding, and when analytics needs rich and accurate data. Snowplow is a solution for data teams who want to manage the collection, processing and warehousing of data across all their platforms and products. Get started and experience no Plow data for yourself@snowplow analytics.com again snowplow analytics.com.

So Luis, I'm curious. There's one of the hardest problems in running this podcast, I think, is just the wide variety of jargon that is used throughout all sorts of different areas of ML. And I know that on this podcast we've talked before about model serialization. So we've got something like an ONNX project, for example, where, you know, maybe I could quote, save a model to onnx and maybe like load it into Pytorch or a pytorch model into TensorFlow. Or it's this sort of overlapping serialization format. So as people are thinking about kind of saving and serializing their models, where does maybe this compilation fit in in terms of the developer workflow? Like, let's say that I have my model created or I've trained it. Let's say I'm in TensorFlow and I've decided on a way to serialize it. So I've created a. I've output a file that corresponds to my serialized model. What happens next? What's the workflow look like after that in terms of Apache tvm?

Yeah.

And the compilation process.

Great question. So there's many ways in which Apache CVM ingests your model. One is exactly what we talked about. So there's a front end to Apache TVM that ingests a model that has been serialized into onnx, right? So you just import some model and then you specify the hardware target, and then you wait for a little bit and then you get your artifact ready, your executable for your model, and packaged in various ways. But there are also ways in which you can call TVM directly from the code that specifies your model. So you can do that from TensorFlow, from PyTorch, from MXnet or Keras and so on. You do essentially import TVM and you load your model graph into the TVM representation, then you choose your harder target and then you compile. So there's two ways in which you can interface with tvm, either via serialized model or by just embedding a few lines of code to call tvm. So in the flow from data to your deployed model, you create the data, curate the data, create a training set, specify model architecture, sometimes a little bit of architecture search. You arrive at your model architecture, you train it, and then you're able to actually test and validate your model. Then you're happy with it. So at that point your model has the right statistical properties. You know, it does what it's supposed to do and what you want to do, you want to make it run fast, right? So now you have a trained model at that point, that's what you hand off to say Apache TVM, either via OnNX serialized models or via calling directly the second. Then you specify your harder targets and then you click compile, right? So you call compile. That's when TVM does its high level and low level optimization magic and also uses this machine learning for machine learning engine to tune the code for your specific model on your hardware target.

So I'm curious, I got maybe several questions that I'm going to try to combine into one a little bit and you can segment them any way you want. If you go through that optimization process, a what does that output look like? How is that different from the model before you took it through the optimization process in terms of how you approach inference on it? And what are the limits in terms of that target architecture that you're trying to hit? You mentioned the Raspberry PI and there's this day and age, there are tons and tons of kind of low capability or low power targets that you might want to run a model that otherwise would have been impossible. Can you describe kind of what that looks like after that optimization and what the limits are on it?

Yeah. Great. Okay, so let me tell first what it looks like. So the output is really a executable. It's just an executable code for your model that includes your model executable for your model plus a runtime. Like the runtime would be support for your model that's tuned for that hardware target. And with tvm, you get a custom packaged binary for your model. The way you call it, you load that binary and then there's an API to call. It could be a shared library, for example. One way of packaging this whole thing up is this a SO library or a DLL in Windows where your model is just, just fresh executable code with an API that you call to do inference on. Does that answer your question? This is what it looks like as the Output. And then there's many ways of making it standard. Like you can put in a python wheel and have python bindings for it. There's many ways to make it easy to call it.

Yeah, yeah, it does. Like as a follow up, and I'm going to say something extreme. As we're building bigger and bigger models and we're taking, you know, things like GPT3 and things that are, that are very large and we have this once upon a time unrealistic expectation of putting them into places that you would be like.

I think Chris wants to run GPT3, his smartwatch.

You are closer than you realize on that one.

So absolutely no. And we had people asking about this. Yeah, there's a limit to that. Right. So you can't get around physics. Right. So even if you compress the maximum, if it doesn't fit in the memory that you have, it just doesn't fit.

Understood.

Or if it uses, you know, so much compute that it's going to take too long to do the inference, you know, then it's not going to be a useful output. Right. And of course, in the process of searching what hardware makes sense, you can use a tool like the Octomizer. Let me just put a quick plug in here. So Octomizer is a software as a service platform that Octoml built that uses Apache TVM as its engine and it's like a very easy to use tvm. As you can tell, business conversation is likely to be a more sophisticated stack for more general data scientists. And general data scientists already have enough to worry about making models that do the right thing is what they should focus on. With the Optimizer, we are raising level abstraction to match that so you can upload your model as a nice graphical user interface. Where you upload your model, it tells you what the layers are, you tell what the input layer is, you can click on the harder targets that you want or you can have it choose for you by running across them all and see which ones it runs best at. So you can get the highest throughput per dollar or you can hit the. You know, we do support Raspberry PIs now. So you said, does it run a Raspberry PI? If it doesn't just going to say, hey, it couldn't run this model there. So then you know that even doing all this magic, it does not run at a higher target anyway. So back to what you're asking. So now on the limits, how do we know we can get around physics? Right? So there's only so much Compute resources that you can afford in an edge setting. But there are plenty of new techniques that actually can twist and turn your model to make it fit. One, for example, specifically for language models that you brought up, something that we've done quite a bit of work on is support for sparsity. Right. So these language models are giant, but there are a lot of zeros in it. So as we all know, lots of zeros, easy to compress. Right. So if you multiply something by zero, you don't even need to multiply it. Right. And if it's a bunch of zeros, why are you going to write a bunch of zeros in memory? Just say like hey, only write the non zero values. Sure, you can do that. And that actually goes a long ways in using less memory and using less compute. I wouldn't go as far as saying that we can quite run GPT3 like models on cine device, but we're getting close to running pretty large models because of good sparse computing support and also other forms of compression and quantization that makes big models fit in edge devices.

So just to follow up on the inference side of things, so maybe this is part of what you're building with Octoml, because like you were saying, Apache tvm, maybe it's a lower level thing for real experts. Octoml maybe is some more accessible. Yeah, has a bunch of convenience built in. Yeah. So I'm curious on that inference side, maybe you could contrast the two. Like if I compile a model with Apache tvm, you mentioned sort of of Python wrappings around that output model and maybe there's other language wrappings. Is that as simple as sort of importing a Python library and then importing your compiled model and running an inference or what other sort of workflow changes might you have to do to run a patchy TVM compiled model in terms of the Pythonic ways that people sort of are used to doing things.

No, it's exactly what you said. It's two lines of code. So the results in Python, will you import it and you call inference on it and that's it.

Awesome.

Yeah, and that's the experience that we offer with the Optimizer. And then the Optimizer also has an API call by the way. So if you want to embed the model optimization compilation to your workflow, you can use the Optimizer. I don't mean to give a hard sell here, I'm just saying it's super easy. Like TVM itself, you have to set up your task framework, you have to spin up some instances to run this auto tuning and this optimization tuning component. You have to collect the data for the machine learning, model based optimizations and so on. And it's work, it pays off, but work I have to put up front. Right. So with the optimizer you can replace that with two lines of code in the API. You upload the model via the API and then you specify your harder targets and then you start the optimization process and then when it's ready, you can download the resulting wheel that you can import into your workflow. Right. So you can do all of that either via the web interface or via this API.

Awesome. Yeah. So I know I'm super interested in OctoML and I know that, that you're kind of in beta now. It doesn't have such a long history, so maybe you could tell us how that came about. If it was sort of natural feeling some of these pains of the lower level things, expertise in Apache TVM and the desire to have that be more automated and have some convenience around it. Could you give us a little bit more of the motivation there and your thought process with Octoml?

Yeah, absolutely. Yeah. So maybe I can do a quick once upon a time on Octoml as well. So we had worked hard on Apache tvm, started as a research project and got quite a bit of traction being deployed in Amazon, Facebook, Microsoft and so on. And when running one of our conferences we're like, oh wow, there's quite a bit of interest here. But the folks that were attending were really machine learning researchers, engineers or folks that have data science and machine learning experience as well as software engineering experience. We said, oh, it's cool, we want more people to use it and of course want to continue catering and growing the open source community. So that's why we formed Octoml to, to invest in Apache TVM and grow the community, grow the ecosystem, work closely with hardware vendors to enable more hardware targets on tvm. TVM today has a pretty broad set of hardware targets. Like it has server CPUs from Intel and AMD and ARM CPUs and mobile and server. Right. So Nvidia GPUs, ARM GPUs, Intel GPUs, AMD GPUs, AMD APUs. We have FPGA support, have a broad set, analog and, and a lot of these were done with us in partnership with hardware vendors and some hardware vendors that go and put the work into the open source package. And the reason that I keep emphasizing open source here is that I don't think it would be sustainable to have a project of the diversity that TVM needs or any sort of machine learning compiler needs to thrive. Because you want diversity of models that it supports, frameworks and harder targets. The only way you make the diversity manageable, in my opinion, is by having an open and welcoming community that even competitors can collaborate. Kind of like Linux. Linux is a great story of success there and that's what we want with tvm. And what we wanted to do with Octoml is enable more hardware targets and work with hybrid vendors, but then also make the power of TVM and machine learning model optimization, compilation accessible to the broadest set of users possible. And the way we approach that is by building a high level service that makes it very, very easy and fully so natural that you'd never choose not to use. And that's really the goal of the Octomizer. And we made it into a SaaS offering because machine learning moves fast, as you know, right? So models change fast, there's new harder targets almost every other week. Right. So the best way to do that is actually package that into a service because then we can take care of all of the, you know, complicated lower level things that you don't want your data scientists spending time on.

So you've got me pretty excited about it as we've gone through this and how with different people having their practical workflows that they're using and you know, whether it's TensorFlow or PyTorch and they have a deployment string attached to it and I know that you've mentioned it targets several frameworks we've talked about, you know, the kind of the two biggies and stuff. Can you talk a little bit about how first on the Apache TVM side and then talk about where we optimally can use Octoml but try to give me a sense of practical sense of how do I get that into my Org stream, what do I need, what are limitations, what should I avoid, what should I do? I'm just trying to get a sense of how do I get started. If I'm listening that you've sold us on what to do here and Rubber meets the road kind of moment.

Yeah, absolutely. So if you are so inclined and you want to do some software engineering adventures, you should definitely go to TVM AI. There's plenty of tutorials there on how you can get started. You know, there's a TensorFlow example, there's a Pytorch example, there's a non example and you can just go there. That's how you use the open source offering there, right? So now there's many ways in which you can use use open source Apache tvm so you can use just as a compiler, know what we call auto tuning? Autotuning is the machine learning based magic that I told you about that searches for better implementations. For that you have to do a little bit more work, right? So you have to set up a benchmark infrastructure. It could be just your machine, but then it takes a little bit longer because it needs to run a bunch of experiments. And so there's tutorials for that. If you have more experience with the lower level parts of your framework and you're ready to use, you know, a stack like Apache tvm, I'll start with that route, try and run through the examples there. Now if you have some work to do and you want to get your model ready quickly and you want to enjoy some automation, you can use the Optimizer because It's a full SaaS offering. Then what you do today, the way you support is you serialize your model to onnx. You upload the model and then once you upload it, there's all sorts of hardware targets that you can click. There's a Checkbox for Nvidia GPU, Intel CPUs and Raspberry PI. You can choose the ones that you want and then you click Optimize and then you get a notification when your workflow is done. You get all the performance comparison across all the hardware targets and across even different ways of compiling your model.

So between the Optimizer and having the baseline open source project, you can kind of choose the level of abstraction that you want to get into for what your workflow is. So you have choice that way, right?

You do have choice, yes. You have choice of how you want to take advantage of that. You can go through the Apache CVM routes. You do it and we'll support that. There's a thriving community that will help you. Now, if you really want to get started from day zero and not have to worry about that, then you go to the Optimizer, either using the web interface or the API and then you can use our support and you also you get access to our ready to go machine learning models for the hardware targets that you care about.

ForeignLog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of practical AI into the future. Simply follow the changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog.

It's better.

So Luis, this is super interesting. What I'm wondering. One of the things that we mentioned a few times is all onnx and I think that sounds like some of what you're sort of centralizing around with Octoml is onnx maybe as a recommendation, since you've mentioned it a couple of times, could you maybe just give those who aren't familiar brief definition of what we're talking about with onnx? And also maybe from your perspective as someone maybe not working on ONNX day to day, but working on something that depends on that, how you see that project progressing and the momentum with that.

Great question. I want to emphasize that even though use onnx several times, we do support directly if you go through TensorFlow or PyTorch and so on. And there's no. I do tend to like what ONNX aims to do because ONNX is just a way of representing your model essentially at the highest level is a description language. So you can have your model, you built it in memory, you just specify it in whatever framework you like to use and you want to store it, right? So you need to describe it somehow. So onnx is agreed upon way by multiple players in the space that this is a good way in which you can describe a machine learning model that includes the computational structure, your layers, what each operator does as well as all of your parameters gets serialized in one single package. So ONNX is evolving and it has its ups and downs and I think right now people are getting more excited, so excited about it, and I think it's an uptick. Now I'm sure there'll be other model description languages and exchange formats, right? So that would pop up and we are ready to support those as well. I do think that it's good to have at least one format of storing models that is generally adopted, like widely adopted. Because then if you keep your models that way, chances are that the software components that you need in your workflow will support your model. Right?

This field is evolving so rapidly right now and you have not only each framework's kind of way of doing things, but I'm looking through the TVM AI website that you referenced and going through and there's like they get started with TVM and there's so many different options and I can't help but wonder, you're covering so many architectures and with all this changes happening and this is happening at light speed, all the time. We're constantly getting bombarded with new things. And I know that I as a practitioner struggle to keep up at times with all the things. How do you do that? How do you keep the project going, keep it current, keep all these new things coming out? I'm assuming you don't sleep.

Yeah, it's a great, great question. I do sleep and thank you. And one thing that makes me sleep.

Yeah.

Is that I have to my natural neural network working. The way we do that is by having good sleep and a glass of wine on Fridays, you know, so glass of wine.

A very important part of that.

Very important. Yeah, absolutely. So the answer is, how do we keep up? Is really having a strong community and nurturing it and being player in it, encouraging more folks to participate. And looking back, I mean, I'm very grateful to the community and I think we were lucky to have been involved in helping catalyze the community because somehow, luckily, TVM was able to capture the interest of folks that build the frameworks, folks that build models. Because new classes of models say when you do recurrent neural networks, we had to go support that in tvm, once you have dynamic models with dynamic shapes, all these things that you don't need to know what it is, but essentially different aspects of your model that make them more powerful and more general needs to be supported in tvm. All of that were actually contributed by community members and we help make that happen. We put a lot of work ourselves too. But then the hardware vendors, so the hardware vendors are the ones that actually really feel the pain, to be honest, of this growing complexity of the ecosystem that you put very well, Chris, you know, so hardware vendors today, they have to write libraries, low level libraries that are tuned for their hardware targets for each one of the major operators in these models. Every time models change, they have to go and tune it again. So they're always like having to update. That's not sustainable. Right. So that's why Civium automating a lot of that made it very attractive for them to contribute to tvm. And they want it to be open source because they also want to enjoy this amplification effect that the community has. So since we incubated Apache TVM into the Apache Software foundation, there was even more interest and industry became more comfortable in contributing because now there is professional, independent governance of the project. Because before was a few grad students and a couple of folks, some contributors sitting in a room or sitting in a virtual room or folks at the University of Washington sitting too. Anyway, so there was a long answer to your question, but basically it is by having a open source community and having the writing technical incentives for folks who contribute to it. That's how we deal with the growing diversity.

So I'm curious more about that open source side. So could you give maybe there's people out there listening that are working on what they feel like might be the next really cool AI practitioner tooling or data science developer tool or something. They want to get this project out there, there, they want to have it be an open source project and get other people involved. Do you have any tips for those sorts of people out there that are working on tooling, working on new things in terms of helping them understand how they might foster a community around those things and maybe get a little bit of momentum going. What are some of maybe the key points with that?

That's a great question. I'll say first, recruit early users and truly listen to them and make them feel like they're part of your adventure here. And then you're helping them succeed and their success is your success, right? So I'd say like we got lucky and we're fortunate that we had early users that were very involved in giving us feedback. And by showing that you care about their feedback and implement it quickly and then that catalyzes the process, right? So you kind of have to treat them as customers paid with love, right? So they're giving you feedback and you respond to that by making their lives easier, right? So that's the first thing, really treat your community as well as possible and respond fast. And then second, whenever picking the general theme, try hard and find what are all the other open source tools or maybe not open source tools that exist there that are very adjacent to what you do and have a lot of clarity on what is your differentiation there. So what kind of new problem are you solving? How do you communicate that? And if it's related to research project, it might be a little bit easier because you write a paper about it and people read the paper, it's like, oh, you're solving a cool problem, they come and take a look at your work, right? So those are two things, clear differentiation and then recruiting users as early as possible so you can iterate fast. And if you solve their problem, chances are they're going to tell their friends and colleagues and they start using it too. And that's how you catalyze it.

So I'm kind of curious, kind of in the same strain of that last question, you're affecting so many other communities out there that may be commercially based with a hardware vendor. There's a lot of communities involved in the targets that you're, that you're compiling to. And I can't help but wonder how do you see those types of communities getting involved? Because you are essentially pretty significant influencer in how those targets get used. Because if the compiling is working, if it's really awesome, that benefits them in a big way. How do they choose to engage you? Do you need more engagement from those target communities to do better? And what kind of value can they add?

I just.

As you're hitting Raspberry PI, I would imagine the Raspberry PI community would have to be keenly interested in working with you on this.

Yeah, it's a great question, Chris. So ARM is already fully bought into the TVM ecosystem. They've built their cpu, GPU and NPU compilers on top of tvm. They're very active contributor to the open source community and they work closely with us at Octoml as well. So some of the big players, of course, like for example, let's say Nvidia, right, Nvidia has a very mature, probably the most mature system software stack for machine learning on their hardware and arguably that's a good chunk of their success. And even Nvidia is interested in working with vm. You see some commits from them. We support Nvidia pretty well. And the point I wanted to make here is the following. For hardware that's really popular, the community is going to do anyways, even if the hardware vendors themselves are not involved. So we have really good support for Nvidia because we did a bunch of work. The community did a bunch of work because Nvidia Hardware manager in terms of we have very good performance on Nvidia hardware because of that. And the point here is not to just say compete with CUDNN and tensorrt. Of course they know they're hardware, they do well in many cases, we do really well. Some other cases that was not in their radar. And we want to offer users a choice of which one to use and sometimes even choose it automatically for them. So in tvm there's something called best of both worlds. When you process a model through tvm, you can choose to either use CUDNN or TVM native code and it picks the best of each part of your model model and you compose this for the end user. What you care is your model runs fast, right? So the point here I want to make is that for this big hardware, the company supports it. For emerging hardware vendors, honestly, I don't know if they have an alternative. The alternative is to build everything in house. So think of, let's say you are a new AI chip company and you have a great idea for a hardware mechanism that's going to make a certain class of models run really well and that's your business. Are you going to cater to this set of users? So it's upgrading your hardware, then you go and look at the ecosystem, you see, say, oh, now I had to support PyTorch, TensorFlow, Keras and MaxNet. I had to support this type of model. You start looking at everything they need to support to connect to the rest of the ecosystem. It's a daunting task. So that's a huge incentive for them to come and use tvm because if they support a very clean code generation interface that we made, very easy for new hardware vendors to come and be part of the ecosystem, I don't think they'll find a more compelling alternative, to be honest. Because how are you going to spin up a team of 15 compiler engineers to go and build your own internal compiler? It's just hard to imagine. Right, so basically what I'm trying to say in a long way is that the way we made it easy to add new hardware target and the fact there's a community around it and the fact there's a lot of momentum already sells itself to the new hardware vendors. And we see that by having hardware vendors come wanting to work with us to do some enablement work for them directly. We do some of that for some hardware target targets. But a lot of times you just see Hyder vendors going directly to the new, smaller emerging ones going directly to the Apache tpm. So does that answer your question, Chris?

No, it does. That was a really good answer, actually. Satisfied it. Thank you.

Yeah, definitely, I agree. I have sort of a strange question as we get more towards the end here, what else out there, sort of across the AI industry do you have your eyes on in terms of things that really excite you in terms of where the industry is going or maybe particular groups that are innovating or particular technology that you have your eye on? What else do you have your eye on as you're looking kind of towards the future of the AI industry?

Yeah, well, there's just so many things. Let me start with the shorter term that goes longer term. So in the shorter term, I think it's really exciting what's going on in doing more and more harder aware network architecture search, there's some companies doing it, but essentially as you evolve how your model looks like with Network Architecture Search, you do that in a hardware way. I think it's super important because it's complementary to everything that we talked about here. Right. So what we talked about here is having a model that is ready to be deployed, use, go compile it, but now having the model in the first place to actually be better suitable for your hardware grade. So that's one of them. The other one is more automations in the data management side. For example, I love what those Norco AI folks are doing, which essentially I don't know if you've heard of them, but they essentially have tools that enables you to construct synthetic data sets in a programmatic way that essentially automates a lot of the work that's required for you to start building a data set to train models. And then on the hardware side, what I think is happening right now that's really exciting is just to see more and more reconfigurable architectures coming into the mix. Right. So I'm sure you've heard of CPUs, GPUs, right? And CPU like accelerators. You've probably heard of FPGAs as well, which essentially hardware fabric that you can program very much in the same way that you design hardware. But it's not quite as fast as a truly application specific chip. But it's pretty general and then you can do a lot of meaningful things with it. I find it exciting that those FPGAs are getting more and more tuned for machine learning. So Xilinx has offerings that way. Altera is enriching their FPGAs with more blocks that are more suitable for new models. I find this exciting in the short term. So now in the medium to long term, I just love what's going on in between machine learning and life science. So just seeing machine learning enabling very large scale genomics studies that conscious crazy amounts of data to go and make sense of data that is incredibly complex because nature is a complex beast, right? And then also using machine learning to design systems. So people are using machine learning to design molecular systems, People are using machine learning to design aircrafts, People are using a reverse engineering like was it again reverse design where you give the properties and you synthesize something that has has those properties using machine learning. These are all things that I find really, really exciting to think about because machine learning itself is also systems. So using machine learning for machine learning improvements is pretty interesting. We do that to some extent, but I feel like we're just scratching the surface, right? So you can use machine learning to design machine learning chips, you can use machine learning to optimize machine learning training systems. And when you close the loop there, that's when you should embrace and let it evolve. Right.

So I remember a while back we had some guests from intel and they were talking about just what you just said, using machine learning for chip design. And yeah, it's. The sky's the limit that you can use it for so many different things at this point.

Yeah, absolutely. And machine learning itself as going back full circle now as a workload, it's so tolerant to we call noisy execution. Right. So by that I really mean if you have flaky hardware and then there's just no way around it as you go to 2 nanometer technology, which I don't know if you heard, IBM just announced they're getting 2 nanometer process, make a lot of progress with it. That's kind of crazy to think about. Right. And it's likely to be very, very noisy and have very flaky transistors. And the way we make use of that is not just by doing the typical systems design of layering error correcting nodes. We should do that too. But with machine learning you can use that directly because it's so tolerant to noisy execution that there's many interesting possibilities there. So for better energy efficiency and machine learning wouldn't get as much bad rep by using a whole lot of energy. Right. Which I'm sure you've heard that before. Right?

Well.

That'S the whole topic of a whole other conversation, you know, so.

Yeah. And I hope we can have that conversation sometime. Well, Luis, I appreciate you taking time to help Chris and I fully optimize our discussion on the podcast to maximize interest. Interest. It's been, it's been great. I've really enjoyed it. And we'll have links to TVM and OctoML and all the things in our show notes, so definitely encourage our listeners to check those things out. I know I'll be playing around a bit after the episode, so thanks so much, Luis. Really appreciate you taking time.

Thank you. You guys are really, really fun. I can't wait to hear your other episodes for other topics as well. So you and Chris are really fun. Thank you.

Thank you for listening to Practical AI. We appreciate your time and your attention. If you enjoyed this episode, help us out by spreading the word. Think of a friend, think of a colleague, somebody who would benefit from listening to it and send them a link. We'd really appreciate it. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Breakmaster Cylinder. Thanks again to our sponsors, Fast, Lastly, Linode and LaunchDarkly. That's our show. We hope you enjoyed it and we'll talk to you again next week.

Sam.