Deploying models (to tractors 🚜)

It's all about intuition, I guess. These days, everything is changing so fast, it's very hard to actually try to predict something. And what we do is an iterative process. We try every time and then we refine. And that's exactly how we look at the field itself. So if we see a company or a specific field where we feel that what they need to do in order to solve the problem they're trying to approach is actually an iterative process where you have a model and you're constantly refining, rebuilding a better model, then this is a Great fit for MLOps. If you have enough automation, you can really accelerate the process.

Hey, Jared, here. One of the things we can count on in the software industry is change. The state of the art changes so fast, in fact, that keeping up can feel like a whole other job on top of your actual job. That's why we created Changelog Weekly. It's our totally free newsletter that we drop in your inbox each and every Sunday. We link to the latest news, the best articles, and the most interesting projects that you should be aware of. We also add a little commentary from us saying why something's important, pointing you to other instances of a trend, or just making a dorky joke to keep it lively. So if you haven't yet, I recommend subscribing to Changelog Weekly and help us help you keep up with the Latest. Head to changelog.com weekly and sign up today. Again. It's totally free and we never spam you.

Yuck.

One last time. That's changelog.com weekly. Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join us at PracticalAI FM community and follow the show on Twitter. We're RacticalAI FM. Thank you to our partners at Fastly for shipping our pods super fast all around the world. Check them out@fastly.com.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a Dave scientist with SIL International. I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

I am doing well, Daniel. How are you today?

Doing well. Yeah. It's been an interesting couple months and lots of new projects kicking off, so keeping busy. But we just came back from a little bit of vacation last week, which was nice.

You got to tell everyone where you went now. Now that you've actually brought that up.

Okay.

Yeah.

So we drove down to Warmer Weather. So we live in the Midwest of the United States and drove down to Alabama, did some hiking, kind of back up through Mammoth Cave, which I learned is the world's largest cave network system thing. I don't know the proper terms, but yeah, that was fun. So it went underground for a bit and yeah, it was a good time. Yeah. And I'm pretty psyched for a couple reasons for this show, Chris, because of course I always enjoy talking with you, but we've also got some familiar technology because I don't know if you remember, a while back we had an episode, I think at the time it was called Allegro AI, which is MLOps. They've since rebranded to ClearML. And we've got Moses Gutman, who is ClearML's CEO and co founder, but we've also got one of their partners, Green Eye. We've got Alon Klein Orbach, who is the CTO and co founder at Greeneye, which is an agricultural AI company. So this is going to be fun. We're going to Talk about Agriculture AI, MLOps and get to chat with some old friends as well. So welcome Alon and Moses.

Thank you. Thank you for having us.

Pleasure to be here.

Yeah, it's great to revisit MLOps with ClearML. We had that show previously and of course ML Ops is sort of like even since we had that show just kind of exploding as a topic that's on people's mind. What has that been like, Moses, in terms of just like this sort of meteoric rise of people caring about ML Ops and how they actually practically do machine learning.

So I think the market really matured in the last two years. I guess it's probably Covid accelerating the process where everyone is working remotely so you have to have automated processes and you got to log everything because you cannot call your colleagues every minute or so. And I think that the problems that we kind of discussed in theory two years ago became very day to day practical problems that companies and individuals run into on a daily basis. And now it's become a problem that if before it was nice to have, now it became like a must for most companies. Back then only a few understood the benefits and kind of the need for this very comprehensive approach where everything is streamlined. And I think now it's kind of common knowledge, probably not that common to actually implement, but at least the understanding is there.

Yeah. And Alon, in terms of your company, which is sort of like has customers who are in the agriculture vertical but is very much like at its core an AI company from My understanding, I don't know the full history of Greeneye, so maybe you could give a little bit of that. But did you sort of do a bunch of AI as Green Eye and then kind of come to the MLOps problems or from the beginning was that something that you kind of needed and was problematic for you, so you kind of started with that early on?

It's a good question. We actually started from different goals or different objectives of Greeneye and until we establish what Greeneye is about. So we started hacking with the laptop and training on the GPU on the laptop and stuff like that and really understand there is no scaling it in many, many, many ways and even had a server there and we needed to install CUDA and Cudan and using no app to keep the script running and really all the best, worst practice that you can ever that you can do.

There's no shame. We've all.

Yeah, so we started with no MLOps at all. Really. No MLOps. I think a few years ago my brother, one of them told me, hey, try Docker. And I said, ah, no, no, no. And the rest is history. Three years later, today we are completely dockerized from end to end, Kubernetes all the way cloud and edge. And this has changed everything because then we realized we can do a lot of mlops all around and move stuff. Yeah.

And maybe just stepping back in terms of Green Eye, I think, Chris, I don't know if you remember this, but I forget in what episode it was. I think it was one of our fully connected episodes. We were using the example of spraying in crop fields as an example of this sort of scale between completely human, manual process up to automation and how that's changed. It struck me, Chris, that neither of us are farmers and really know much about that process. So alon, this is very much like the world you live in. So maybe you could just step back and let us know a little bit about like AI in agriculture before we kind of talk about some of the other ML ops things that your company is doing along with ClearML. What is AI and agriculture look like generally and how has that developed over time?

Sure. So I must be honest, I'm not a farmer as well. I'm from the technical side. But I like what we are doing. So in the big image I think you can divide the interest into two. The ones that gives tools to all the farmers to get more information and more details about the field, the crop, the yield or anything like that. And you can divide the adult group is the tools that make the decision by themselves. So you get a lot of cool companies like in Israel, Tyrannis, Prospera, other cool that doing a lot of intelligence in the field. They collect data, they analyze it and show the farmers insight. You get drones there, you get satellites there, you get pivots and cameras and etc. And from the other end you get decision making tools that driving the tractors, autonomous sprayers like the same domains that we are Blue river and John Deere acquisition that was I think four years from now.

As a follow up to that, I'm curious, we're so used to on the show kind of talking about these very technical topics and yet the clientele that your company is serving is one that is getting into technology as we talked about. But if you look at the broad history hasn't really, really been something that we associate with high tech and all. What is the merger of something as cutting edge as deep learning, ML ops on one side with farming on the other, where that's making this massive transition. What's it like being in that space where you're presumably kind of tying together two very, very different worlds?

We really like it. We are really a very divisible company. We have chemistry, we have agronomics here, we have data science, we have real time and we have everything for everything. Cloud and envelopes and business sizes, sprayer operators and being this spot that everything is connected, the technology is related to the field is we really like it. You need to have the business to run and to make sense from monetized perspective. But it's also nice to do something nice.

Yeah, it's nice to apply AI I'm sure to a problem that we all have, which is we all need food. So I saw in one of the videos on your site this sort of like. And this is where my knowledge of all the machinery and stuff. But I'm assuming this is like a spraying machine. I don't know if it's specifically for spraying, but it sprays and then you've kind of got cameras. So I don't know if you could maybe just describe this sort of machine and like the arms of the machine and just so people have a visual of where your technology fits in.

So maybe I restarted to understand the problem that we are solving. So imagine you have a garden and you grow vegetable there and what you do in your free time, you are weeding, you are taking the weed out because they compete with your vegetable about resource, sun, water and et cetera. And when you are getting bigger, you are starting to use mechanical tools. And when you get really big, like farmers, the Midwest, Nebraska and Iowa and all of this area, you started to put chemicals because you cannot control the size of farming, you put chemicals and you want to do it fast. So you don't put with a small tractor, you have dedicated sprayer, self propelled sprayer, 66 meter long boom. And it's a monster. You can walk underneath the without need to bend under the sprayer. So it's a really big monster that you just drive to do as much accurate as possible in short time. And what we are doing in Greeneye, instead of assuming the worst case scenario that every spot in the field has weeds there. So instead of doing that, we are putting sensors, cameras and our computers and nozzles and everything else we are using to spray only when you need to spray. So Instead of putting 100% of chemicals over 5,000 acres field, you are putting 10% food. And so you're saving money and the world saving chemicals. And it's a win win situation. So about your question. Sorry, I forgot. So we have a big sprayer, we have cameras each like 3 meters long and we are just filming the entire boom. We are putting the cameras looking a bit ahead so we have time to process. Everything is done in real time. No connectivity at all to the cloud.

So Moses, you know, as I'm listening to Alon talk about his story and as we're doing this, I'm thinking back to kind of starting with these cutting edge mlops. When you're looking at the landscape on your side, as someone who's bringing this technology to bear in the marketplace, how are you evaluating different opportunities in industry? I mean, is it just that everything is open? Do you have a way of looking and saying I see an opportunity where this technology in a particular industry is going to be very useful? How do you make those kind of judgment calls on how to engage?

Good question. So first it's all about intuition, I guess. These days everything is changing so fast that's very hard to actually try to predict something, if I'm referring back to machine learning. And what we do is an iterative process and we try every time and then we were fine. And that's exactly how we look at the field itself. So if we see a company or a specific field where we feel that what they need to do in order to solve the problem they're trying to approach is actually an iterative process where you have a model and you're constantly refining, rebuilding a better model, then this is a great fit for MLOps because that basically means that if you have enough automation, you can really accelerate the process. If not, obviously you have to do the same process only manually. Which time to market wise really increases the time from the research phase to actually something that is working where you have some alpha in the middle. And when you see a process where you can say, you know what, with a bit of automation, this model will really work not 90% of the time, which means 1 out of 10 you fail. Which is not like in theory, 90% looks sounds fine, but in practicality, this is not something you can actually sell. You think to yourself, okay, the only thing that I need is a bit for more information from the field itself and then I can just refine the model, rerun it, get a better performance, and then just repeat the process. Then basically I'm golden. I can take it to different scenarios and get my model up and running. And every time we see one of those scenarios, then that's just kind of a moment where we hear the playing. Okay, this is a perfect fit for automation, for mlops as kind of a holistic approach.

We are going to ship in 3, 2, 1.

I'm Gerhard Le Su, host of Shipit, a show with weekly episodes about getting your best ideas into the world and seeing what happens. We talk about code ops infrastructure and the people that make it happen. Like Charity Majors from Honeycomb.

We act like great engineers, make great teams, when it's exactly the opposite. In fact, it is great teams that make great engineers.

And Dave Farley, one of the founders of Continuous Delivery.

Start off assuming that we're wrong rather.

Than assuming that we're right.

Test our ideas, try and falsify our ideas. Those are better ways of doing work. And it doesn't really matter what work it is that you're doing. That stuff just works better.

We even experiment on our own open source podcasting platform so that you can see how we implement specific tools and services within changelog.com what works and what fails.

It's like there's a brand new hammer and we grab hold of it and everyone gathers around, we put our hand out and we. We strike it right on our thumb. And then everybody knows that hammer really hurts when you strike on your thumb. I'm glad those guys did it. I've learned something instead.

Yeah, I think that's a very interesting perspective, but I don't see that way.

Okay, so an amazing analogy, but I'm not sure that applies here.

Listen to an episode that seems interesting or helpful and if you like it, subscribe today. We'd Love to have you with us.

So Alon, following up on what Moses was just talking about with sort of where the value of MLOps really comes in with automation. You gave the example, sort of the concrete example of the spraying, detecting places to spray within a field with this massive machine that has the sensors or the cameras on it. So how does the automation or the retraining of models that you're using, where does that come in, how often and what sorts of things are you automating.

In practicality, I think I can spend hours to answer fully of this question. But I must say that the things about envelopes and MLE in general, if you compare to Anderson Field in the industry, like coding, deploying servers and etc. There is no best practice yet. The entire industry, everyone invent something on his own and doing something else. We start getting to some main path, but there is no best practice that everyone is agreeing. So I think from our side we had a lot of challenges. We have challenges of data and controlling the data. We have massive amount of data, more than millions of images over the field. And one of the challenges is getting a model that was trained to the tractor. And before we had any automation we did okay, we trained the model and then we freezed it. And we are using TensorRT and OnnX to TensorRT and stuff like that. And. And we did it manually, like each step at a time. I think like we got 10 models a year to the tractor when we got automation, we have thousands of models, tractors and rerun. And when a researcher finished training there is a click and everything is done automatically is convert, is get metrics there, tooth griller, mail and what else is done is checking the converting if we didn't miss anything by the converter, like performance and we got the end of the file there to the tractor by a single click. So I think this is one example of automation that MLOps really changed.

Do I remember correctly that you're also running kubernetes on the embedded devices?

Yes, yes, we are running kubernetes on the embedded devices. This is completely game change in our end because this is also a conversation different that I can spe about it.

And that's when you say on the embedded devices. We're talking about like on the tractor.

On the tractor, yeah, yeah, we have a few devices there and they are running K3s is the light version of Kubernetes. It's really nice because we can best practice from the cloud and we get best practice to the edge. And for example, if we want to use Clear ML from the edge or from the cloud. It doesn't make really didn't make us any different to us. So we know how to press the move, the secrets and how to use it and we just do it in.

Detail so you have full visibility to the tractors inside the same dashboard that you're developing in and you have the entire cycle streamlined.

So something like this good setup.

I like it.

Rancher and I can connect any running tractor around the world that is online now and just doing even SSH to the machine I can view it all. But this is more bigger one than the mls, the ML Ops. This is like we call it Special Ops.

Special Ops. I like that.

We actually have dedicated team that doing research around special Ops. So we have MLOps and DevOps and IoT Ops. Now there is FinOps. This is the team that moves one step ahead and doing a lot of tricks. I try to find when we started to use Clear ML. I search ClearMet and then I understand I need to search Allegro and I didn't find the right point. How we got to know Moses and his guys, so I'm not really sure but before we got to know him we used kubeflow pipeline to do the runs and to do the metrics and we got there to a big wall of complexity and we shift the training from Kubeflow Pipelines to ClearML.

I'd like to ask a follow up question about something you were saying a moment ago and it's something kind of close to what I'm doing when I'm not podcasting and that is when you talked about having kubernetes in all the places in this case on the tractor and you talked about K3s. Can you tell us a little bit about. I'm a big advocate in kubernetes in all the places at various scales as a setup. And so since in your use case you have done that, I'd love to hear how you arrived at that and what benefit you think it's given you. Why do that? Because most people don't think about putting Kubernetes in all the places they're running kind of in the cloud they don't have. They're not out on the edge yet the way you are where you're way out on the edge. And as someone who also in my day job works out on the edge, I'm curious what your thoughts are about how Kubernetes in all the places is a good model going forward.

So Chris, you don't think people when they think of the ideal deployment target for Kubernetes. They don't think immediately tractor.

I'm glad that he does.

I'll say that should be right next to it.

We are in Sprayer Vault. It's a good question. When we started containers we started with no Kubernetes and we did our own deployment system and we fast like in months. We had the wall of keep them alive versioning and everything. And then we understand, okay, someone solved this problem. It's no way. And then we got to Kubernetes on the cloud and at the edge we still use containers but with different Orchestrator. We had Azure IoT if you know it. And we used it for a while and then we got another role because Kubernetes has this concept of separation virtual co separation of pods and this is amazing idea. And the other architecture didn't add it. And we understand okay, we need to change. And it was not easy because nothing is really ready to arm 64 bit. You'd be surprised that most of the libraries maybe today there are some that are more but most of the libraries back one year, two years from now no version for ARM to 24 bits. And it was like there is no way that we are on the edge of the edge as a young company.

Do you think that Nvidia having bought ARM recently will have any impact on that?

Not buying arm? Failing to buy arm failing to buy Army.

Yeah, I think the regulation issues might affect it.

I forgot about that. That's a good point.

Yeah. But definitely I think we'll be here for the edge and kubernetes. Back to your question help us a lot in for example, when we have a new researcher or a new programmer or whatever it does not do anything on his laptop. The laptop is only the gate to the pod that runs or on the tractor or on the the cloud. And this is the workspace. So if we get to the tractor or in our subject the the research a research come to the company a new research. We just hired a new one and she's really good one and she's got. Just get. Okay, you get access to. We still use kubeflow for notebooks server. So if she needs a workspace she just click and get a new notebook and she can use our tool. We use PyCharm for remote interpreter and you just connect to the pods there and you get all the data and you can play from your pycharm you can play from your notebooks and this is possible. You can get it in different ways. But this is mainly possible all the games of sharing and forwarding with kubernetes is much easier and this is true for the Edge and for the clouds. So no one installing any dependency. I don't know if you play with Java Home or something like that or NPM install or no one installing anything on his computer besides the ide. Our philosophy in general, everything containerized, no workspace to be installed in any way. And this is really up specifically in MLOP environment.

So Alon, in terms of the automation that you were talking about before, you mentioned this sort of information about you're able to sort of do more than like thousands of models a year now versus like a sort of order of magnitude lower now that you didn't have automation. I'm wondering if you could talk a little bit more about for your case and I'm trying to think of maybe weeds sort of look similar so what kind of needs to be updated so much throughout the sort of year for these vision models or whatever models you're running. And then could you describe maybe a little bit more? You mentioned kind of bringing in new data, training the model and then having the pipeline to push it out to the to the tractor and deciding when and when not to do that. It'd be interesting to hear about the sort of when and when not question in terms of like what you test within like your ML ops to determine when you push something out and how you, how you do that.

Yeah, it's a good question. I will start with the first one about the models and the number of models. I think in the end, well from my experience I might be wrong, but having one model to world them all or something like that, it's not enough. You got a version of YOLO, RetinaNet or any other model and you need to put more effort to solve a real world problem because you have a lot of variables, a lot of variants in the real world and you need to combine classical vision and, and deep learning one. So I think in the end we have metrics on the cloud for the models, for the big models and so on. But at the end we want to have metrics on the devices, on the trucks of the runs themselves. So we keep constantly testing ourselves on the real environment, how we are doing also in the trims of performance. Not only metrics, performance speed wise, cycle clock, how fast we can go today we can go about 6 meter per second. You guys speaking miles per hour. It's about 12 around this area miles per hour. So this is a really big factor for us. Besides that we have different crops, different geography and everything. Change the landscape, change the weather, change the sunlight, change. It's completely different game to play. In Israel for example, and the Midwest, the farmers change the. Some use tilt and some in Midwest they mostly stop using tilt. They just keep the crops, the old crops there and letting the ground do their magics and just sitting above the old crops. So everything is changed and we need to react to those changes. And this is for the first month, the second one, I think that every tractor run that we are doing or any other way that we are getting data, we try to get as much as variant as possible. So retraining our model by one or two or even, I don't know, 500 images, it's not the game, it won't change a lot for the model. We have a lot of them. So we try to understand when few models don't agree with each other or something like that, or tricks like that to understand when there is information that is interesting to rerun a train on it.

So are you logging your data pre processing and data set like creation and your training runs in the sort of ML ops and kind of building training off of certain, certain triggers or something that you have set up or how does that work?

One of the challenges of mlops is reproducibility. I think this is a really hard one to get right. You get code versioning and then you get dependency, let's say you solve that with Git and Docker, but then you get data versioning and then in all of that you need some system that will take everything from every place you need and then you need to push it and just click play and rerun it. So reproducibility is really hard. And if you did like, I don't know, half a year ago, you did something good and you want to go back to it, it's really hard. So we try to log as much as possible from the system perspective and from the training and the risk perspective. What's nice about ClearML that we are using it not only from MLOps, we are using Dashboard in general. So we're just pushing everything that we want to use as metrics and show stuff there. So from this perspective we just log everything possible and if it's visible we can use ClearML for it. But also we want to push our limits to run faster and faster. And if we run faster we can do even more stuff. Today we detect weed, but our mission is to spray less, grow more. So we want to do fungicides and pesticides and fertilizer and etc. So we need more compute power or to be better in what we are doing and saving computer power for different tasks. So we try to log everything and be better in what we are doing.

Quick question. You mentioned retraining models. Do you have like a model per tractor or scene?

No, no, no, it's not tractor per scene, but we retrain a model like okay, there's different reasons why to do it fix. Maybe we want to improve it in different variance of appearance of the backgrounds or anything like that. Or we want to make sure that we got, I don't know, a new width that we don't know. We are not familiar. So instead of doing a zero shot that we are kind of doing or one shot that we are doing so we can stop the system for doing this for a specific width that became more common. But in general we are playing with with a lot of tools. We are trying to get in line with the best practice in the industry and we are also experimenting. It's not just for the experiment, but it's for the research to be better. So in this sense we might be running the same train or if we want to verify that we got the same result, like a real research more. Not really, it's not academic, but it descents.

Okay, well, given the fact Alon that you're training so many models, you're updating a lot of models, it sounds like there's a lot of training scenarios that you're encountering. You're doing this at scale and you've been partnering with Moses and his team to do this. I'm curious actually from Moses, from your perspective, looking back on the things you've been trying to enable with ClearML and seeing someone use it at a larger scale like this, what are some of the things that you kind of like you thought were going to be important and ended up being important in terms of the things you're tracking or the features that you've enabled and maybe what are some things that maybe you didn't expect and now you're thinking about differently than when you started things out As.

A follow on to that, what insights are growing in your head.

I'll try to cover everything. I'll probably forget, so just remind me. Okay, I'll start where. So I think that Alons team were the first to say guys, we want better connectivity with Kubernetes. And the reason I remember is we'll start this discussion. A lot of our features are actually driven by the community and Alon and his team started from the open source and kind of graduated in a way. Basically, they just said, we're sick and tired of maintaining our own servers, plural. They had many. And they just said, it's not worth it. Just go do that for us. And we had multiple conversations even before. So we try to keep a very active stack channel and GitHub. So this is how we develop features, right? Basically, people would say, hey, I want to build something. And then it's just a crazy idea. And then you try to think about, okay, maybe this is doable, it kind of makes sense. And if it does, then we try to figure out first how to hack it so someone can continue with their day job and kind of build on top of it and then try to realize, is there a way to actually structure it into the platform itself? And if there is, we try to figure out a way to actually put it in there and see if there's traction. And one of the things that I remember that Alonsi were the first to do was to better connect the Orchestrator with the Kubernetes cluster. Basically, when we started developing it, it was like, I don't know, a long time ago. Kubernetes was not a thing. So containers were, but Kubernetes was not. It was just before Google just released Kubernetes as an open source solution, before it kind of killed Docker. So we started with Docker as kind of a bare metal. So we said, okay, fine, we'll have the Orchestrator that will just pull jobs, set up the container, and then run everything inside the container. And it did that, and it's great. But the resource management or allocation of Kubernetes is terrific. So these guys came and they said, look, guys, we have a Kubernetes cluster and we like the idea of your Orchestrator. So basically, clearmill Orchestrator will do. Think of it as a dynamic Dockerfile In a way, I'm oversimplifying. Base Docker image with the ability to kind of control the Dockerfile you need to do in runtime, but also introduce some caching. Bottom line, you do not have to have a container per job just to accelerate. Because when you streamline a process, you cannot have every step containerized. You end up with thousands of containers that no one will know who is using and no one will delete because someone might be using. And basically, yeah, you get the idea. Anyhow, so they said, okay, we love Kubernetes because it allows us to schedule resources very easily, but then when the resource is scheduled, we want this dynamic approach and obviously visibility, which is obviously hard with Kubernetes. We also don't want our users, like the data scientist developers, who have actual access to the Kubernetes cluster, because, well, no. So I think that was the first time we developed what we now call the Kubernetes glue, which basically kind of converts a job from ClearML into a Kubernetes job. Basically trying to figure out whether this job can actually be executed on Kubernetes, give you kind of better visibility into the cluster itself. So users can basically push jobs into what we call a queue, which is, think of it as in Kubernetes terminology, it's basically like a template YAML that you'll be using for that specific job, only you have a priority queue on top of it. So it kind of implicitly holds the setup itself, which is kind of resources, et cetera, but also priority on top, and then use that in order to schedule. Use Kubernetes as basically your resource scheduling, which it is terrific for, but it's lacking the scheduler itself, like order and priority, et cetera. This is exactly what the glue itself adds. And obviously it solves for the problem of making sure that the end users, meaning the data scientists, will have access to the Kubernetes cluster.

Right.

So that was a first feature that we added just because of them. And this is how we heard on. So you guys are running Kubernetes on the edge device, and we were amazed that someone is trying to do that.

So I feel completely obligated to throw yet another buzzword and just ask you about if you have an opinion. And that is don't go blockchain. Chris well, that's a. Yet another one right there you throw out. Daniel no, I'm just going to stick with kind of microservice architecture. When you were talking about all the containers out there and managing that, and as people kind of are moving more and more into microservice architecture over time and segregating off all their functions and yet trying to keep them together, does ClearML as a platform have an opinion on that in any way, or do you not care? Are you agnostic about, you know, where people end up? So if I throw the word microservices at you, what do you say?

Okay, that's terrific. I love it. Because microservices reason Kubernetes was invented basically to manage them. But the idea behind the microservice, it's alive, it's production ready, and it has to be stable, like the Default of Kubernetes is if it fails, restart it because you had a good reason to put it there. This is not what is going on with MLOps. If it failed, then it'll continue failing. Like just drop it. That's the default. That's the total opposite of microservices. And that's basically our approach. Our approach is use Kubernetes for what it's good for. So you probably have another cluster doing whatever microservices that you're running, which is terrific. But for the MLOps perspective, use Kubernetes as a resource scheduler more than anything else. It's basically the opposite of the default of Kubernetes. And then I guess the bridge is serving models, which is actually a microservice. But you want that elasticity because you still want to be able to change it without building new dockers all the time. You actually want that to be in flight. Model upgrades, canary, et cetera, that you probably want to control from outside, from an ELB perspective. So this is kind of the bridge between the two, at least from our perspective.

So if I'm getting sort of just like stepping back and thinking, I'm kind of trying to connect some of the things. Elan, you've said in terms of how things are working on your end. You still, you sort of have data coming in off of the tractors, coming into various, maybe data processing jobs which might be queued up on a queue which runs through clearml. That might sort of lead then into model training jobs, which I agree. I love your illustration, Moses, about you expect a lot of model training jobs to fail. So kubernetes spinning up a service in Kubernetes to run a training is sort of like the opposite of a lot of what they had in mind. But anyway, so you sort of spin up these jobs in a queue. So from the data scientist perspective, you're basically just saying, hey, I want to use this data to train a model, put it in a queue, it runs the training and finishes or not. But then that model then, which has a version, it's tied to the data, then sort of gets shipped out. Does it get in your case, does it get shipped out? Kind of like Moses is saying to a service that's running in your edge K3s cluster as a rest service for what's going on on the tractor or how does that piece work?

It's a good question. I think we can see two paths to start the training. One piece is from the data, as you spoke, and the other place is for the researchers who want to test the code or new experiment, new model or etc. In one to fire a training. So when we started, let's take the example for. For the researcher, he just go in his IDE and he just connect to the remote workspace that is on Kubernetes that runs kubeflow, Jupyter and so on. And it's just playing there. And he's just click remote executed. This runs to Moses servers and tells the clear ML system to speak to log everything like okay, I want to use this container and this is the changes I did. Like Moses says, it does not create new containers for every step. So it just keeps the changes. Moses, if I'm saying something wrong here, feel free to fix me. And then it goes to Moza servers and then from there it's going back to our server. So as our agent there that Moses spoke about, this is the glue and this start the training. So this is the training and the training continuously reports to ClearML to the main server what's the status and what's the metrics. And once we have metrics we can decide one of the two if we are free a new version for testing on the edge or we are stopped there and just keep this track and moving on.

And the inference you're running on the Kubernetes cluster as a service or as like on the K3s as a rust service.

So this is tricky one we have something in the box that is not in production like when I'm saying production, like internal production research that we are planning to have K3s inside a pod to have a special environment for the developers that is separate like a cluster in this cluster, something like that. But let's keep this aside. We are using at the moment KF serving to serve the model entailment. So our metric system just when we get a new model, we just serve it and the metrics just doing API rest service locally. And this is something like a microservice that they're using. One is responsible entirely. It does not know anything about the metric itself. You can just probe it and to get results for each one for what you want to do. And there was another tool that is doing the metrics and it just probe the service and accumulate and then reports metrics to the ClearML service. So we have three different services. ClearML is not micro, but the other two are. And I think this is the cycle. We have a human decision in between. So we don't do the entire cycle for every model and not for every data.

So I have a final question that's addressed to both of you in turn. I want to start with Elon. I'm actually letting Moses cheat and hear what Elon's answer is.

Yay.

So Moses, I'm throwing you a bone on that one. So here's the question as you are thinking about kind of these amazing uses of technology, both from the technology creator and from the technology implementer's perspectives, and you were thinking about what's next? What are you wanting to do next? You've made it this far, you've had tremendous success and you've got to have something that when you go to bed at night you're going, maybe I could do that. Alon, I'd first like to hear for you as the implementer in a very specific use case what you're thinking. And then Moses, hint, hint, now that you've heard his question, you can answer yours as well. So Alon, I'll throw it to you first.

So I have many things in my mind, a lot of imagination before I go to sleep. Sometimes I see bounding boxes and just seeing them and see them and see them. Sometimes I see different stuff. But yeah, in a larger scale I think still in green eye perspective we can do, and we are about to do the boss that I said in the industry about decision making and helping the farmer because we are already scanning the field and we have byproduct of lots of data and high quantity of data, high resolution data from the field. So our goal is to do both and we have lots of ideas in the pipeline to also one to get to make the decision ourselves and the other one is to help the farmer get better decision for different stuff that's not related to Sprayer at all.

That was a good answer. Moses, how about you?

So two things I can expose here. So one, we are working now, this is public in one of the repositories on the new version of the serving solution. Basically we're not fans of KF serving as an infrastructure because it's very hard if you have a single model. If you're not changing it, that's fine, but if you're constantly changing them, it's not easy. Adding pre processing is not easy. Metrics, everything is hard. So together with a lot of people from the community, we redesigned the clearmodel serving. So now it's internal testing and it's very, very nice. Basically you can add the pre processing without even code deploy it. You have it auto scale on your Kubernetes cluster. So Kubernetes does. It's basically building a serving service that is flexible and that you can change online, which is terrific. This is what we want from these type of services. So this is something that we're working on and I'm hoping that we will be able to release before the end of next month. So I think we have a talk in gtc. So before the talk, that's a deadline, basically, we will have to release it, which is it's always good to have a deadline. And the other thing that we're working on, and this is really research, we're sort of trying to wrap our heads around how to actually solve it. And that's coming directly from Alon, actually. So he was one of the guys that said, oh, I really want that. And he's not the only one. So we always want to make sure that we're not building for a very specific problem, that it's actually a widespread problem. And that problem is I have a lot of data stuck there, there in the backend of Clear ML, which basically means multiple databases. I just want to be able to query more deeply, create a dashboard. The data is there. I know I put it there. Now I want to have better interface for the database without actually accessing the database. This is doable, but probably kind of tourist. So we're trying to think how to hack together like a dashboard solution on top of it to allow you to better create better visibility for an entire process. Because the entire idea is to make this entire MLOps holistic approach basically means that if you have data, you should be able to use it in whatever step you're along the way of developing your product. And this is something that is amazing, but will find its way out there very soon, I'm hoping.

That's awesome. I'm super excited to explore those things. I'm a fan, so pretty excited to hear about the serving and the other things coming along. And I appreciate both of you being willing to kind of talk through. First of all, give us an update on what is now clearml, which previously we talked about as Allegro, and all the great things you're doing, but also in the context of this use case with Green Eye. So I'm really impressed with what each of you are doing. And yeah, thank you both for joining.

Thank you. Thank you.

All right, that is practical AI for this week. If this is your first time listening, subscribe now at PracticalAI FM or just search for Practical AI in your favorite podcast app. We're in there. And if you're a longtime listener. Please do share the show with your friends. It's is the best way you can help Practical AI succeed. Thanks again to Fastly for shipping our shows super fast all around the world to break master cylinder for the beats. And to you for listening. We appreciate you. That's all for this week. We'll talk to you again next time, Sam.