Stellar inference speed via AutoNAS

What we are doing and specializing is kind of the model design approach where you need to select the right model that is optimized both for the data to reach the accuracy and also for the hardware to reach the latency. And by optimizing the model specifically for the inference hardware that you are looking to use, you can get significant boost in the performance compared to having kind of an open source model or off the shelf model that you just took from an academic paper or from GitHub or something like that.

Big thanks to our partners Linode, Fastly and launchdarkly we love Lino. They keep it fast and simple. Check them out@linode.com changelog Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by LaunchDarkly. Get a demo at launchdarkly.com this episode is brought to you by our friends at rudderstack and we're calling all data engineers to check out rudderstack cloud and start building smart customer data pipelines. Rudderstack is Warehouse first. No more silos. Rudderstack builds your customer data lake on your data warehouse, not theirs. In enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API first. Rudderstack can be easily integrated into your existing development processes and because they're open source you can see all their code so you don't have to worry about vendor lock in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. Rudderstack is free up to 500000 events and pricing scales transparently. From there, learn more and get started@rudderstack.com again. Rudderst that's R U D D E R S T A C k dot com.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community and follow us Twitter we're racticalaifm.

Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co host Chris Brinson, who is a tech strategist at Lockheed Martin. How you doing Chris?

I am doing very well. It's dog days of summer as we record this and just try not to melt.

Yeah yeah, it's hot. Like you're sitting right on Top of a bunch of GPUs or something.

That's exactly right.

Yeah.

Well, speaking of hardware, today's episode is kind of connected to that. Really excited to have Jonathan Gaifman with us, who is CEO and co founder of the deep learning company dese. Welcome, Jonathan.

Hi.

Thank you for hosting me. How are you?

Yeah, doing wonderful. So I know a lot of what your company does is related to productionizing models, optimizing models. There's a bunch of things that you do related to optimizing inference, and I'd love to dive into all of that as time goes on. But maybe let's just start out by talking about why should we care so much about inference? So a lot of effort and I think screen time, if we want to put it that way, in terms of Twitter and other places, is put on the training side of things and scaling up training. But why do deep learning and AI practitioners need to be concerned about inference?

So actually the first thing to be concerned with is the training. But after you finish building a model, then you need to deploy it somewhere. It could be in the cloud or in the edge. And when you are going to deploy it and do serving or inference at scale, you must be concerned or think about how this model is going to perform in terms of latency or throughput. What is the operational cost of this model? Or are you actually able to deploy it on the edge device that you want to use in order to serve the inference of that model? And it's kind of the second phase of the development cycle after you reach the accuracy. But you must think about it at the beginning when you choose the model in order to know that you are going to finish the project with a model that meets the SLA of going to production in your environment with your characteristics of performance that you are looking for in production. If you think about it, you'll see that a model is trained once or once in a while. But the inference workload is huge because it kind of linearly scales with the amount of data in production. So if you think about an organization that for example, develops a self driving car, you'll think about the number of data scientists that doing the training work, building the models. Each one of them needs some amount of GPUs in order to do its work. But when you deploy these algorithms, these are deployed among huge amounts of cars. So in order to optimize the performance, you're more concerned about what type of hardware you'll need to fit into the car compared to how many GPUs you'll need to have in the cloud for your data scientists to build those models. So this is kind of the proportions. The training is kind of linearly scaled by the number of data scientists, while the inference is scaled by the amount of data that the scale of the deployment that you're going to use. And you must think about the performance when you want to go to production at scale.

Yeah. And as a practitioner, you mentioned thinking about inference up front, maybe even before training, thinking about what models you might even be able to use in production as you're working with different companies and different teams of data scientists. Do you see people running into this challenge a lot where they're really happy with their model, that they spent a lot of time and effort training, and they just are completely blocked in taking the model to production? And how often do you see that? And what are the main reasons, like why they're not able to do inference with that model? Is it a latency thing? Is it a memory constraint thing? What are some of the challenges you see?

So it is really divided into several problems, but there is a simple principle that you can always take a larger model and get better accuracy. And it's easy to get better accuracy with larger models, but this is only happens if you don't need to think about production. When you think about production, you understand that those models are not production ready. Let's say, for example, if you want to deploy them in the edge, you have several constraints like the memory and the latency that you want to run, probably in real time, or giving some reasonable response time for the model. But in the cloud, you're more concerned about the cloud cost of serving this at scale, the amount of hardware that you will need to orchestrate in order to serve your users or your demand for inference. And this is kind of the things that you need to take into account while you develop your model. Because as I said at the beginning, going larger is easier to get accuracy, but it's not easier to get deployed. So we see companies all the time getting some huge models, trying to fit them into small devices for edge inference, or companies that just understand that they took the largest model that they can have and get the right accuracy, but then they're starting to scale and see the costs of running these inference workloads and then trying to think about, okay, how can I reduce those GPUs costs or the CPU cost in running production in the cloud? And this happens all the time. So our approach is think about these problems at the beginning while you start development, choosing the right architecture at the beginning instead of kind of having a trial and error iterations when you get to the accuracy, but then you understand that you need to change the model in order to get it productized.

Yeah, and I actually have a follow up on that, mainly because I only have a certain sphere of experience and I only know what my organization does and what some other organizations do. Just for my own curiosity, how often are you seeing people use GPUs on the inference side versus just CPUs? Do you see that being done more and more? Or you know, what's the majority of inference use cases that you're seeing? Is it inference on GPUs or inference on CPUs? Maybe specifically thinking about cloud deployments, Edge deployments might be somewhat differently. But thinking about cloud deployments at organizations that you work with, is it often inference on the GPU or is it often inference on a cpu?

I think that it really depends on the task that you are trying to do, the inference and what are you trying to achieve? I think that if we were talking about video analytics workloads, you must use gpu. You have a lot of data, you want to process the data, the images in higher resolution. You need a GPU performance. If you're talking about having some queries of NLP model, you usually find those deployed on CPUs, but it doesn't matter because both of them are getting expensive when you get to scale. So if you look about, for example, prices on the cloud for having a four core CPU and a T4 GPU, it's approximately the same. So it's not like the problem is only the prices of gpu. Also, the computer of the CPU is getting expensive when you have to run in large workloads with large clusters, with multiple nodes and cores.

Could you talk a little bit as we've kind of touched on Edge specifically, could you touch on what some of the challenges you see about deployment to the Edge and the inference that's associated with that are even beyond just the GPU CPU consideration. That's certainly something that I'm involved a lot in. And more and more people are now having to deploy to Edge in all sorts of different use cases. And I think we're pretty accustomed at this point to thinking about cloud based deployment because that's matured a lot faster. But as more and more organizations are involved in edge deployments, I think they're trying to explore their way through that. Do you have any guidance for that?

So first of all, there's kind of a jungle of edge hardware Types. There are a lot of types of hardware that you can use at the edge, and it really depends on the application. It could be a mobile phone that if you deploy a mobile app with deep learning in it, you will find out that your users are spread across something like 10 or more types of hardware from iOS with the A and E of Apple and Samsung devices with the Qualcomm, Snapdragon and hardware types like that. So first of all you need to understand what hardware your users or you are going to deploy on. That's the first task. Then you need to understand kind of what is the software stack that is best to use for that type of hardware. If we're talking about Apple devices and iOS, we have the core ML, but most of the other types of hardware will probably be better running with TF Lite or those frameworks that are optimized for edge inference. And above all that you need to understand kind of what is the limitations of the hardware that you're going to use. For example, memory constraints, memory bottleneck of loading the weights, loading the data and stuff like this and the performance. What are you going to get? If you will run, for example, an object detection model on a Jetson, how many frames per second are you going to get? How many video streams can you put on that Jetson? And this is kind of something that if you'll get to the accuracy, you'll finish building the model and only then will measure it on the device that you're looking to deploy on. You'll get back to square one to redesign the model in order to get the sla, the latency that you're looking for at the edge. And those are things that you need kind of to have a holistic approach and to see how you solve all of them together. It's kind of a multi constraint optimization where the accuracy, the latency and the model size are things that you need to consider together when building an edge AI application.

And that maybe leads me into another question about that. I do want to get into the specific like methods and technology that you've been involved in developing. But it sounds like you're also sort of suggesting a different kind of workflow that people can have in their mind where as you contrasted to like I'm going to build my model and the environments in which I'm training my model and testing it are totally different from those where I'm going to deploy it. How can people adjust their workflow to maybe is it a matter of always making sure that you have a testing cycle where you're testing on the hardware that you are targeting in the end or how can that be integrated into data scientists workflows in a better way?

So we are kind of pushing to hardware in a loop development approach. When you are taking the inference hardware in the development stage very early in the model selection stage, where you considering some models usually based on some open source repositories and academic papers, and you have to measure them at the beginning, this is kind of the first step in understanding if you're going in the right way. After that you need to understand or think how can you bring that model that meets the SLA to the accuracy that you're looking for. So this is kind of an opposite approach then first reaching the accuracy and then reaching the latency. And this is for edge applications with constraints about the latency and the model size that those need to be considered at the beginning and not at the end.

Makes sense. I am wondering maybe if you can just give like a broad stroke sketch of what people like. Let's say that they get to a point where they have the model that they want and it's not quite optimized for the hardware target that they have in mind. Maybe there's too much latency or they need to shrink the model or something. What just sort of generally are people trying out there in terms of methods for optimizing their models for inference on certain hardware there's something we call the.

Inference tech where at the bottom we consider the hardware itself. On top of that we have a layer that we call it the drivers and the graph compilers. So for example the OpenVINO for Intel CPU or the TensorRT for GPUs. On top of that we have like open source methods like pruning and quantization. Pruning is a method to reduce and eliminate unneeded neurons and connections in the network. And quantization is representing the weights and the activations of the network in lower bit representation like 8bit or something like that. So those all are kind of open source and public methods and techniques in order to build more efficient inference. On top of that what we are doing and specializing is kind of the model design approach where you need to select the right model that is optimized both for the data to reach the accuracy and also for the hardware to reach the latency. And what we understand today is that different hardware type prefer different types of model. I will give a simple example. If you think about GPU which has parallelism capabilities, it will prefer large wider layers with fewer layers in the network because it can parallelize the layer itself, but it cannot parallelize between layers. In contrast, in CPU there's low parallelism capabilities and you will probably prefer having narrow layers with less neurons, but you can have more layers. And this is kind of a general idea how the inference speed could be affected by the model structure. And by optimizing the model specifically for the inference hardware that you're looking to use, you can get significant boost in the performance compared to having kind of an open source model or off the shelf model that you just took from an academic paper or from GitHub or something like that.

Signal Wire is real time video tech to help you create interactive video experiences previously not possible. It gives you access to broadcast quality ultra low latency video that's proven and trusted by Amazon, Ring Doorbell Zoom and others. See why the future of video communication is being built on SignalWire. They have easy to deploy APIs, SDKs for the most popular programming languages, and expert Support from the OGS of Software Defined Telecom Tech. Try it today@signalwire.com and use code AI for $25 in developer credit, just visit signalwire.com that's signalwire.com and use code AI to receive that 25 bucks. Once again, that's signalwire.com code AI.

So Jonathan, I want to maybe get into some of those design aspects, aspects that you were talking of, and I find it fascinating that you're sort of diving into this area of thinking about what sorts of layers run best on what sorts of hardware. Has that research around that topic, has that been going on for some time and you were able to sort of build on that? I'm curious because hardware is progressing so rapidly even in the terms of accelerators like GPUs and VPUs and TPUs and all sorts of used. All of that's advancing very rapidly. So it sounds like you have some intuition around what runs best on what hardware, but is that rules that you're encoding into your methods in order to optimize models for certain hardware or what's going on there? I know you've got this sort of architecture search concept.

Yeah, there are many types of hardware and we cannot have rules for every type of hardware. Designing neural architecture is very complicated because it's kind of a composition of multiple layers with multiple types and sizes. So searching among neural architecture is a large surf space with a very complicated search. So we can't have kind of rule of thumb for this hardware use that operation, and for this hardware use this operation. What we are doing, we are employing Neural architecture search that is hardware aware in the sense that it's connected to the hardware and optimizes the structure of the network to both have better accuracy on the data and and better latency on the data on the hardware. And this is something that we must connect to the specific hardware in order to get that. Because you can't really model the hardware and understand how a specific neural network will behave on that hardware. So you can think about it as a hardware in the loop neural architecture search, where we have an automatic search algorithm that search among hundreds of thousands of neural architecture and find the best one that kind of sits on the sweet spot between the accuracy and the latency on the given hardware. While searching on that space is very complicated because the latency can be measured or estimated by using the hardware. But in order to understand the accuracy of a given architecture, you must connect the data and train it. Candidate architectures could be very expensive. So the trick here, or the secret sauce here is how can we do it efficiently? How can we scan hundred thousands of neural architectures and find the best one for your operational point in terms of latency accuracy trade off. And this is kind of what we do is our proprietary neural architecture search algorithm that is called autonac Automatic Neural architecture construction.

Could you take us through maybe kind of an example just to make it very tangible for the practitioners that are listening to this in terms of how the practitioner is going to use neural architecture search to do that, maybe ideally with an edge use case, just to give a sense of how you're approaching it and what that feels like if you're targeting inference on the edge. Ultimately, yeah.

So we have a collaboration with intel that we recently published in MLPerf last September or something like that about a performance booster that we've made to image classification model resonant 50 for a MacBook laptop, for example. We've done that for several types of hardware. But let's take for example the edge use case with the MacBook Pro. So we're having the baseline model. We have three inputs to the algorithm of the neural architecture search. We're having the baseline model which is for in this example was ResNet50. We're having the hardware itself and the data. The data was Imagenet in the MLPerf benchmark and the Ultra NAK algorithm kind of searching what types of changes or how can it change the original architecture of ResNet50 in order to get a better architecture that preserves the accuracy of 76% of accuracy, for example, and minimize the latency or maximize the throughput and it is very interesting to see that first of all, it replaces some of the operations from dense convolution layers to depthwise and some other variants of convolutional layers that are, let's say, less memory bounded because the cache memory also limit the inference speed of the network. Second thing that we can learn from that idea is that some layers are more important than others. So for example, if you think about what size to have on each layer in the network, we have some understanding that the initial layers that are doing the feature extraction in the image classification example are more important important than the later layers in the network. So putting kind of most of the computation at the beginning of the network seems to have better accuracy preserving properties. This is kind of what we see when we observe on the results of that run of Autonak on this example. And we end up with a model that is faster in something like 3x compared to the baseline and having the same accuracy. So this is kind of how we take an algorithm that is fully automatic and try to do a postmortem to understand what happened there, why the output of these algorithms look like that, why the initial layers haven't changed or almost haven't changed, but the later layer have changed significantly, replaced to other types of layers, their size was smaller and some other changes that we observed on the result of the algorithm.

So I want to actually get back to that later because it's really fascinating that you can sort of use these tools to learn more about the types of things that work well architecture design wise on certain hardware. That's really interesting. But first, just to sort of bring that example into focus, it sounds like you had this base RESNET model. I'm just thinking about the sort of inputs outputs of the automatic network architecture search. Like if I'm a practitioner and maybe I have my own custom model for object detection or a custom model for speech recognition or whatever it is, and I've trained that. Then in terms of doing this automatic neural architecture search, I'm assuming one input to that is the sort of serialized version of my model in whatever format it's in. You also mentioned that a data set was input to that. Maybe you could give some details on that. So why is the data set input? Is that so that you can make sure that you're not optimizing just for the hardware, but you're also optimizing to make sure that performance doesn't degrade in terms of prediction performance? And then also do I need to have access to like the specific hardware that I'm targeting. So I need like this data set and then do I need like the specific hardware that I'm targeting and run the automated neural architecture search on on that hardware or how does that work?

Yeah, so you got it right. Let's formulate the equation that the neural architecture search is trying to solve. We are talking about minimize the latency of the model on a given hardware, subject to getting an accuracy that is above the given threshold. So the latency of the architecture or the model can be measured without training in most of the cases on the hardware. So we don't need the data to understand what is going to be the latency of ResNet50 on a CPU of Intel. But the accuracy is data dependent. And if we want to put that constraint, and obviously we want to put that, we need to have the data and verify that the model that is selected by the minimization problem of the latency still meets the accuracy requirements. Because if we'll not put the accuracy constraint, we'll end up with a model with one neuron that predicting nothing. So this is kind of the composition between the latency that is measured on the hardware and the accuracy that is measured on the data.

Could you talk a little bit more about kind of the output path on that in the sense of if, let's say that you have a model that's doing object detection, or it could be anything really, and you want to put it on maybe a platform that's on the edge that has a bunch of sensors, maybe a bunch of cameras pulling in. When you're going through the training process, how are you accounting for kind of that variability out there? You know, kind of going back to if you're not targeting the hardware in the architecture search that you're going to deploy, deploy to, how do you all account for that? How do you say, ah, there's a very unique configuration for my output target, my deployment target. How do you approach that?

Are you asking what happens when we don't know the hardware in production?

Well, yeah, I guess just like if you're targeting a particular environment that may be customized fairly significantly for deployment. As more and more practitioners are now kind of getting out of the data center and they are putting things on drones and they're putting things into automotive being a big one, obviously anything like that that's out on the edge and kind of has a custom environment, what does that look like from a practitioner perspective? Kind of out of the theory and into the hands on.

So we prefer to connect to the actual hardware that the model is going to be deployed on. So for example, if it's Jetson, we connect the exact Jetson model to the newer architecture search. If we don't know that, we need to have some proxies. So a good proxy might be the number of floating point operation, but we know that this is not such a good proxy. And some other methods like pruning target this metric. But this metric correlates to latency only on CPUs, so having proxies there, it's not the right approach. In our perspective, measuring the metric that really matters on the actual device is the way to go. In my perspective, like measuring the actual latency, measuring the actual throughput on the device is the way to go in order to understand the exact performance that you are going to see in production. Because I can show examples where I cut the flops in factor of 2 and the latency is getting slower on GPU, on Jetson, on some types of devices. So these proxies are very problematic today. And I think that this is kind of the interesting part of doing hardware where neural architecture search compared to other compression techniques, like I mentioned, there's pruning that reduces the number of flops or the number of parameters or any proxy for the size or the complexity of the network.

So, Jonathan, as you're working in this space, one of the big things that I always start thinking about when I think of optimizing networks in this way is that there's just so many different types of layers and custom layers that people are using and new stuff coming out all the time. So what has it been like sort of maintaining your search space over time and growing that space to sort of include new things as they're coming out? How do you approach that as an organization and figure out how to expand that search space and what to include and what not to include?

So that's a good question because the research field of deep learning is progressing very fast. And we have a team of researchers that's sitting on the latest academic papers that propose all those new layers and those new operators and kind of reproduce all these models to understand which type of layers, which types of techniques are worth adding to the search space of the neural architecture search and which are not. And actually the result is very interesting in most cases. For example, if we'll take the computer vision domain, the basic operators that are well known for the last five years or something like that are performing the best. So some of the tricks that we see now are not improving so much compared to using those blocks and operators that build ResNet and MobileNet and EfficientNet and those networks. So having the right composition of operators is more crucial than having all those fancy tricks that showed up in the last two or three years. This is something that we see, it's quite general claim. There are some cases that we see things that worth adding to the search space. But in general I would say that it's not so easy to beat a resonant model that is quantized and used a graph compiler like TensorRT or something like that. And you need to work hard in order to build it. So this is kind of how we see all the advancement. Of course we have other advancements in other fields like training tricks, optimizers and stuff like this. And we have to be on the front line on all of those. And this is something that we are working really hard in order to reproduce. All state of the art, all the types of model, all those new models that just announced and having their results and their operators in our sales space.

And a follow up on that, I guess which is related to that approach is on the one side you have all of these different types of architectures, on the other side you have all of these different tasks being solved with deep learning models and AI models. So I'm curious, as you've experienced this over a number of years now, is it harder to optimize the inference of certain tasks versus other tasks? So maybe like NLP tasks versus computer vision tasks versus audio tasks versus maybe it's time series modeling or something. Are there certain domains of AI tasks that are harder to optimize than others?

So at the moment we are mostly focused on computer vision and nlp. And in those domains we see that the principles that we are using that are machine learning based are working across the board. Yes, I can tell that there are some tasks in a domain that are a little bit more complex. For example, semantic segmentation network are more complex than classification networks and they have to preserve the information along the network in order to do kind of image to image tasks. But also on those type of neural architectures we can optimize them. And the principle is very simple. Most of the networks and the new fancy algorithms are kind of built on top of three components. One of is the stem is the few layers connected to the input. The second component is the backbone and the third component is the prediction block. In most cases 80 to 90% of the compute is happening in the backbone. And usually the backbone are just a bunch of convolutional layers. In the case of computer vision and by optimizing that significant part of the network, which is similar across classification, semantic segmentation and object detection, you can get that boost of performance that we are looking to have in all tasks.

As we're talking about this, I'm just visualizing everything in my head as you get to the output and you've targeted a particular deployment target and accounted for the hardware and what the capabilities are. I'm curious, how does your platform integrate in with whatever DevOps pipeline a practitioner might have put into place? What is a typical scenario for actually pushing the deployment deployment out to the hardware that it's going to run on for inference look like?

So we look at our platform as an end to end platform from development to production. We developed two production tools, one of them called Inferior and the second is called rtic. Inferior is lightweight edge inference engine that could be integrated to a monolight application easily and RTIC is a containerized inference server. So if you take that solution, it could be easily deployed by a DevOps with the model inside fetched from the model repository that we provide as part of our SaaS offering and kind of serve the model in a standardized API that contains all the packages, libraries, environment details that you need in order to go over kubernetes for inference and scale. Sometimes we see companies that already have their own infrastructure and don't want to change their existing infrastructure and we provide them with the specific model, just the exact model that ran the optimization, the model after the optimized model in their format. We support all the types of formats from onnx to tensorflow, Pytorch, Keras and all of these frameworks that you can run inference on. So this is kind of the two ways to get DESI optimized model to a production environment, either by our deployment tools or getting the model model and using your existing stack.

I'm just kind of browsing around on some of the information about DESI and super fascinating. One of the things that I see is this idea of DESI nets which you share about and share some of the sort of successes that you've had taking this approach in various domains for various types of models. Could you just share a few success stories in terms of what you've been able to achieve performance wise with this approach?

Yeah, so Destiny is a good example for that. We took a few well known tasks like image classification, object detection and semantic segmentation and we've taken the most famous open Source data set, for example ImageNet, Coco and data sets like that and we built kind of a catalog of Pre optimized model for each and every hardware. And what we are doing, we are kind of plotting what we call an efficient frontier chart with the latency on the x axis and the accuracy on the y axis and plotting all the models that we know, the resnet, the YOLO and those models on those charts and putting those decks or decidaths for detection and kind of seeing what we call the efficient frontier, how those models reach better accuracy and better latency and dominate this trade off between accuracy and latency. And now we provide those designats for data scientists in order to try feed them to their specific data, fine tune them for their specific data and having kind of a pre optimized results of autonag that is ready for use immediately.

And I'm wondering, as you plot out this landscape, I love how you term that like the efficiency landscape. That's a really cool way to think about space as you plot this out and explore that. So space yourself. I'm wondering one way to think about what you're doing and how you've expressed it is I'm a data scientist, I've trained my model, now I run it through auto neural architecture search and get out my, you know, better model faster and more efficient for the architecture while still performing well for prediction. But I'm wondering if this sort of cycle, as you do that more and more you start sort of building some intuition as a data scientist or a practitioner to like start with a better model in the first place. Like if I look at your plots of the efficiency landscape, can I learn some things about maybe I start with better models in the first place rather than sort of relying as much on the neural architecture search. Do you think there's that sort of feedback and that learning that can happen from what architectures are learned by the automatic neural architecture search?

Yeah, absolutely. We share some information intuition behind those architectures that found for a given hardware, for example, one of the things that we see that those architectures are faster but having more parameters, for example. So this is kind of an intuition that we see around these models that can be used to understand that we not always look to smaller models, but for more faster model and accurate model. We provide those models as a starting point. So for example, if you're considering taking ResNet54RI or EfficientNet B0, you can take the corresponding destiny model and start from that and tweak that for your application, whether you're doing object detection, classification or anything like that. And this is kind of giving the ability to use a NAS produced model compared to having a model that is off the shelf for general use. So for example, Efficient NATS is a result by Google from two years ago or something like that that are supposed to be efficient. But it appears that EfficientNet are not so efficient for GPUs even when they are over quantization and graph compiler like TensorRT. So having a pre optimized model for the given hardware that you're going to use is very crucial. And in our example of DESI nets we show that you can have a model that is something like three times faster than efficient at P0 for Jetson GPU while having even better accuracy. So this is kind of a result that you can take off the shelf instead of using efficient and B0 you can take that model and train it to your application, build on top of that some other prediction heads, some other tasks that you want to solve with that backbone and get a autonomous result without running all the neural architecture search for that specific task.

That's very cool. This is so interesting. As you were talking about model optimization and the things that DESE has done with it, what are you envisioning as you guys are looking into the future at this point? What kinds of things are aspirational for DESE in terms of where you want to take the platform and what you envision will be the next thing in model optimization that you'd like to implement?

I think that we feel at a good point in the model optimization space that now we are seeing that we need to expand to the whole development life cycle. So after you have the data, we look on controlling all the training, optimization and deployment of deep learning model on our platform. Whether you're using those destinets or using some off the shelf models and kind of having a full workflow of development, optimization and deployment based on our platform? That's because we understand that there is kind of a triangle that we can draw that is on one edge we have the model, on the other one we have the data and on the last one we have the hardware. And this is kind of a combined optimization that every data scientist need to understand how they solve that. And we are kind of providing the tools to solve this, optimize this triangle. And for now we are mostly focused on the model side. But in the future we'll be also focused on the data and the hardware side in terms of not having them fixed, but having some techniques for data enrichment, data augmentation, self supervised learning, having some hardware recommendation system, maybe having some FPGA capabilities and having our hardware that is optimized for the given model. And this is kind of the far future about how I see optimization in.

Its full that's awesome. Well, I'm really excited that we got to talk through this because I know I learned a lot about this neural architecture search and the things that you're doing. So really impressed with where you're headed with this and appreciate you taking time to join us and chat with us about it.

Sure. Thank you very much. It was great talking to you and I look forward to the episode.

Thank you for listening to Practical AI. We have a bundle of awesome podcasts for you@changelog.com including our brand new show Ship it with Gerhard Lazou, a podcast about getting your best ideas into the world and seeing what happens. It's about the code, the ops, the infra, and the people that make it happen. Yes, we focus on the people because everything else is an implementation detail. Subscribe now@changelog.com shipit or simply search for ShipIt in your favorite podcast app. You'll find it. Of course, the Galaxy Brain move is to subscribe to our Master Feed. It's all changelog podcasts, including Practical AI and shipit in one place. Search Changelog Master Feed or head to changelog.com master and subscribe today, Practical AI is hosted by Daniel Whitenack and Chris Benson with music by Brickmaster Cylinder. We're brought to you by Fast. Lastly, launch Darkly and Linode. That's all for now. We'll talk to you again next week. La.