Simplifying the AI/ML to Production Pipeline with Görkem Ercan

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, do you want to know the top five trends shaping or reshaping the developer experience and more about mlops and more? Well, you're in for a treat because we talk about gherkham, all about an awesome solution and awesome ways to address these new trends. If you don't know Gurkham is the CTO of jozu, a project led for the Open Source Kit Ops ML project specializing in enhancing the speed and safety of building, testing and managing AI ML models in production, which I think is more important now than ever. And he has over 25 years of experience in the developer tools and DevOps software space, including the creation of widely acclaimed tools like the Eclipse GDT Language server and VS Code Java extension. So he really knows his stuff. Gerkham and his team are committed to tackling enterprise challenges, especially with mlops, and the goal is to enable efficiency and secure development and deployment of customized and market leading solutions. You don't want to miss this episode. Check it out. Are you ready to level up your DevOps game and crush those quality challenges? Whether you're dealing with flaky tests, scaling automation or trying to integrate security into your pipelines, we got something just for you. Introducing the DevOps Quality Testing Playbook from Test Guild. It isn't just another PDF, it's your go to guide packed with actual insights, best practices and strategies to help you create a bulletproof DevOps toolchain. It's built specifically for engineers, testers and DevOps teams want to optimize their workflow and drive continuous quality throughout their pipelines. The best part? It's free and ready to download, so don't miss it. Head on over to TestGuild me DevOpsbook and grab your copy today. Stay ahead of the game, optimize your pipelines and let's crush those quality challenges together. Hey Gerkham, welcome to the Guild.

Hi, thanks for having me.

I guess. Before we get into it, I'm always excited to know about founders, why you contribute to open source, why josu, what does it do, and why do you think it's so important nowadays?

So our story actually started with something that we were trying to do. We were initially thinking about doing business analytics kind of solution and one of the things that we discovered very early is when we were thinking about that problem space was, you know, you need to be able to do machine learning and AI better than anyone else at that. It's a very competitive space. So while doing that, because, you know, our backgrounds are our developer tools and DevOps, we said, okay, our competitive advantage would be that we would have a kickass DevOps platform that would actually be faster and better, allow us to be faster and better than any, any other competition. And we started building that for our AIML stuff. But one thing, the first thing that we discovered is it's a very fragmented space. So and none of the usual things that we are used to on DevOps are exist on MLOps. When things exist, there's plenty of them, but none of them actually talks to each other. So you're basically spending most of your time trying to stitch together discrete solutions that has been put out there, usually for data scientists. But what we were trying to do is to be able to deploy on mark, right. And one thing that we discovered was, oh, you know what, there are all these solutions, but none of them actually use the existing DevOps solutions. The DevOps engineer and the SRE and the platform engineers are completely forgotten on any of these solutions essentially. So we started building our own thing, which eventually became a part of it. It has become the KitOps project. What Kitaps project does is it changes how you store your AIML artifacts and allows you to store your AI ML artifacts as OCI artifacts. Therefore, you have your, you have all the benefits of oci, which means that you can now use your container image registries for storing your AIML artifacts. Those are like the model weights and data sets and configurations. Whatever you need at that point, you can put that into what we call a model kit, which is in the container world. It would be a container image, a model kit, and then you can store that in an OCI registry. The big advantages that in there is this is like an immutable artifact, therefore you know for sure that it's not tampered with. And then because it goes into an OCI registry, you have all the existing security, accessibility. And so for benefits of the image registries in place, and you are able to use pretty much every tool out there that exists for DevOps with these. Because every tool out there, if you think about it, you want to attach an SBoM, you want to sign a OCI artifact, those tools exist, you can already do these, or you can move these OCI objects between repositories, mirror them and so on. And so forth. So all that infrastructure is all of a sudden open to you for your OCI artifacts, for your AIML projects.

Nice. So KitOps sounds like it's a package and versioning system for AIML projects. So where does JOSU come in then?

All right, Josu is our product side. So in Josu we are launching what we call a Josu Hub. Jozu Hub is in some ways it's your run of the mill image container registry, just like Docker Hub or anything else. But what it does differently is it is built specifically for model kit. So it brings in a model kit experience rather than a container image experience. So that's part of it. The other thing that we have done, and this is something that we have discovered after we started getting a few trusted friends and family into the hub is they like to have, they like the fact that we are actually signing our model kits. We have a provenance in there. We are actually putting sboms that tells them where it is coming from and because of that they can actually verify what they have received. So those companies that are using base models, the open source models today, they get them from hugging face or somewhere else and there is a ton of them and there are a ton of variations and they can't really like for. Especially for the industries that have compliancy requirements, that's not very nice. Right. They can't prove that that's where they got it from. So what we are providing right now, or will be soon providing with jojohub is you will be able to get images that have been vetted and signed and that includes an SBOM and all the provenance things that you can actually digitally digital signature through. When you need compliance requirements, you need to meet compliance requirements.

Awesome. Yeah. You used to work for a large healthcare company and this is definitely something they probably would would need.

Yeah.

So I guess this leads us into. You've been around for a bit, you've done a lot of projects, so you obviously don't invest in things that you don't see a need for. So this leads us to maybe the five trends currently reshaping the developer experience and I assume MLOps is one of them, but maybe I'm wrong. So is ML one of the things you see as something that developers need to learn more about or you think they already know more about and now you need solutions to handle, you know, the DevOps experience for them to handle.

So I think MLOps is something definitely developers should learn about. I put the speed of penetration to of the ML Ops into our daily lives is going to differ from industry to industry, company to company, like I am. As you said, I've been around enough. So at one time there was talk about digital transformation or the paperless office, right. If you recall. So it's like that. It was like the digital transformation is a transformation, didn't happen overnight. Even when I was at Nokia, when the mobile first, the trend started. So mobile first didn't happen overnight. Right. It's like at first many companies did regular websites and then a week later they thought, oh yeah, we should actually have a mobile website as well, which didn't work. And then the app industry, the app market came and so on and so forth, but those things never happened overnight. I think the ML ops and the penetration of AI and ML is going to start happening gradually as well. And it has to because AI and ML needs to diffuse into your existing solutions. It's not a solution by itself. I mean, you can see there are solutions by itself. It's like that's your chatgpt, right? It's like you have a chat platform or you give a chat platform for your customers and which they can ask some of the questions and they may get some reasonable answers. That's a solution that stands by its own, but it's not actually diffused into your existing solutions. Therefore it's, in most cases it's an improvement, in some cases it doesn't matter, it's just gimmick. But the next wave of applications will start to come and we start to see that, right? It's like there are applications out there that are, for instance, doing sentiment analysis of support cases and saying, oh, you need to call this person back because that person is pissed. So there are things like that that are, you know, that are coming in and applied in companies in very successful ways to either retain customers, gain more customers and so on. So, but won't happen overnight, it will come in chunks and bits and therefore it's going to be very important for people who actually know the business and know the existing solutions to be able to grasp what is coming in through AI and ML.

Yeah, I totally agree. I know, like you said, digital transformation, mobile first, it's like a long drawn out process, but then all of a sudden after the pandemic, it almost seemed like everyone was on board, something you couldn't predict. But it seems like with AI, with ChatGPT, it seems to really have ignited things and AI has been around for a while as well. So I guess when people think of AI, maybe I'm Wrong from developers. They just think of generative AI. Is there more to AI and ML ops that you see coming down the road? That's way beyond, oh, it's just generative AI. We got this, not a problem.

No, it's not just generative AI. And of course generative AI is great and it solves and inspires a lot of the solution. And it's as you said, it was the ignition beyond behind the current, let's call it AI hype, right? But there are other ML solutions that are out there that are getting benefits from this wave of AI innovation. So you can get a lot of base models that you weren't able to get. There are these generic base models which can do categorization for you, which can do object detection and so on and so forth. And these are really powerful AI models. So in five years back, you would actually try to train those models with your data scientists. Like my previous job was with Red Hat for one of our products. We actually tried to train an AI model for figuring out what kind of GitHub project the project, the GitHub repository was so that we could recommend correct tooling and so on and so forth. And it was a nightmare. It just didn't work out for us. But today, like I think this is like five, six years ago today, if we wanted to do that, there are enough generic categorization models that we can start with that will give us much, much better results in a much, much quicker turnaround. So I think there are solutions like that that are getting benefit of the AI innovation wave that we are in. And I think we will continue to see that as well in other places. And generative AIs and LLMs, one of the things they are very good at is if you are able to turn a project into natural language project or even a code project problem, they are able to give you very good results. So like, one of the things that I see more often happening is there are these LLM models that has been fine tuned with creating all kinds of graphics. So you can actually say, hey, you know what? That take this data, this is the data that, and this is how I describe this data and so on and so forth. And give me a graphic that shows X and Y, right, kind of prompts essentially. So I think that we are going to see more and more of that as well. And some of that will be built into the IT systems and some of them will be essentially prompt engineering in an enterprise.

So I assume with mlops, you know, you created Kit Ops and Jovu to handle these. Josu to handle these. So I'm just curious to know, are there any other tools and platforms developers should start investigating to handle? Maybe different aspects of dealing with mlops?

Yeah, like one of the things about AI and ML is it's indeterministic, right? That means that you have to do when, just to give an example, if you have a piece of code and unless you change the compilation parameters, you will always get the same binary out of a compiler. With AI ML, with the same data set with the same code, you are not going to get the same model after a training. So there is always what we call an experimentation in AI ML. So if you are training your data, if you are fine tuning your own models, if you are training base images, then a good platform is MLflow. They do experiment tracking. It's an open source project by databricks. So they do experiment tracking, which means that you are able to do training and then train your model, run some tests, record the results, and then train your model with a different data set or a subset and record the results and able to compare the results between your earlier experiment or next experiment and so on and so forth so that you can find the optimal result that works better with your, with the solution that you're seeking. Because in some cases the training, the data that you are feeding into the model during inference actually changes its performance as well. So that's one of the first ones that I can think of right now that if you're doing training and experimentation with AI and ML, then MLflow is a very good tool.

All right, so you mentioned, you know, a lot of this is not deterministic almost. How do you test that? I come from a testing background. How do you test ML and AI? If you're using AI and ML, you don't necessarily know what you're going to get back.

Yeah, that's a. With the traditional software we have these unit tests and the result is you can predict, before you know exactly what the result will be, if the result is going to be red, you can just say, hey, check if the result is going to be red. But with AI ML, you can't do that. The thing that you can do is you can check statistically if the result is within the expectations, within your expectations. So the testing becomes a little bit harder because now you need to establish a range or metric so that your test results should meet and you need to record your results and then be able to statistically compare them. So there are actually tools for that as well. And one of the MLflow actually has some of those or works with some of those tools as well that allows you to record them and so on so forth. But to as the short answer is yeah, you compare them statistically essentially.

So along those lines then you know, we didn't mention sometimes AI is really hyped right now. And I don't know how many developers and testers really believe in the hype, but it seems like with these generative AI and copilot, the ability to have to really know this language syntax inside of O inside an O has gone away because you can understand coding, but it can generate it for you, probably optimized. So I know it's kind of weird for a developer. Is coding going to be less of a need as a developer and more like like is one of the. What are one of the trends then? Should someone really focus in on the ins and out of a syntax to become a good developer or is it more now like you said, statistics or some other skill?

Yeah, that's a hard one to predict. And I'm a coder, I have, I coded and I have written language tools on a big portion of my career. So to be honest, I think it's going to be less and less critical if you are very efficient with what the compiler is going to create. Right. Or turn to because you will be able to get AI assisted reviews on demand at any point or AI assisted computer code completions and so on and so forth. And in some manners they are going to be more efficient. The problem is AI needs to know the context, otherwise it won't be able to generate the correct code. And that is the harder problem to solve right now because a context is sometimes is what you have in a single file and in those cases AI is going to work very well and sometimes it's what you have in a code repository, but for an enterprise, most of the times the context is everything else as well, like your business, other systems that you interact with and so on and so forth. So AI will maybe at some point is going to have that context, but that is not a solved problem today and that is that I don't think see that being solved very quickly as well. It's a large problem. It's one of those problems where the enterprise actually gradually needs to move into. So I think for the developers, I think knowing that context, knowing that business, knowing the the systems is going to be needed. Of course you still need to be able to understand what AI suggests to you makes sense or not so and the only way to do that is to go through your rookie years, let's call it, and learn to code very well and learn to understand the code very well. I mean, one of the things that I always said, and that was true for 20 years as well, is learn to read code very well. Like when I got involved with Open Source projects 20 years ago, my first involvement was you're not going to be able to do a code contribution to like the first project that I was trying to contribute to was tomcat, which wasn't even called Tomcat at the time. So like you won't be able to do that. You're like, I'm one or two years out of college, barely writing Java to survive. So one of the things I did was I read a lot of code. Like I read the code for Ant, if anyone remembers the build system, the I read the code for Tomcat, you know, whatever that I can get my, I can get from CBS essentially at the time. So. But the thing that I learned a lot is how things can be done and how things cannot be done. Right. You can actually say, see and usually you get really good code on the open search projects. You can really see the difference between, oh, this is how good this problem could be solved and this is another way of solving. Right? And you get used to the idea or the ideas of what works and what doesn't work. Right. And of course I'm not talking about the little code tricks and so on and so forth, but more of general solutions. So I think with AI it's going to be very similar as well. There will be some code generated and someone needs to assess that's actually a good solution or not, because that's. It's not going to be always AI solution.

Absolutely. So as we become more and more ingrained with AI ML, are there teams that are dedicated to this or usually is it developers? If it is a separate team, especially at enterprise, how do they communicate? How can you get that communication working? Because I think that'd probably be a bigger problem as this starts to grow.

Oh, it is a huge problem, right? Because these teams are usually are separate. They are, the AI ML teams are usually coming from data science background, the data scientists, you know, they take their data, turn that into ML models, test them, get some results and so on and so forth. But usually these teams have no clue how to bring anything to production. They have to learn it. And usually like five years ago, for instance, I had to collaborate with a data science team for the project that I mentioned earlier. And one of the first things that questions that we got was, oh, where should we put our code? Like for a regular application project that would have been never been asked, right? It's the code repository, it's git. So but for data sciences, that's how new that was, that concept. And of course we didn't know much about data science and AIML at the time. And it was like, yeah, your code is on Git repository, but that doesn't help us a lot at that point because as we said, you don't really build your AIML artifacts out of code every time. The data scientist does that, does the experimentation and gives you a champion model which you actually deploy to production. So we didn't know that five years ago when we first experimented with it, but we do know it right now and we realized that, you know what, this is not really how you do this. So one of the things that we wanted to help with KitOps was to allow the artifact, the model artifact, the model kit, to be friendly to both data scientists. Because once they are done with their model experimentation, they can package that experimentation, tag it as champion, and push that into an image repository, a model kit repository, let's call it that. And then your DevOps team can just pick it up from there. And if there are validation tests that they need to run, they run the validation tests and then deploy that to their infrastructure. So their existing pipelines and tools is going to work with it. And for an ML project is essentially not very different in that sense that you. It's an artifact that you need to test and deploy and monitor. For the DevOps engineer and for the data scientists, they're job or deployment is done when they are able to determine a champion. So that actually changes the dynamic between the data Scientists and the DevOps SRE and platform engineers a little bit. Makes it a little bit easier for them to actually communicate between the data science teams and DevOps teams.

Gotcha. Because it is all software development in the long run. You have the software development life cycle they need to be integrating into this. So it's just a way of trying how to hook them into the pipeline so that they're in sync with all the other teams. And what they're doing it sounds like.

Exactly, exactly. And they won't have the knowledge, they won't have the experience of integrating with your DevOps tools. So you will be able to tell them, run this command and there's an artifact that you can work with and you can actually, you can go even further. Like they can run the command and the command creates your attestations, provenance, S BOMs and so on and so forth. If you have compliance needs and then you can move them to your regular or your DevOps pipelines, which actually are able to understand all these provenance and attestations and SBOMs and, and deploy the artifact.

So this might be a dumb question. Now developers, really, they usually have to focus in on the user experience, user what they're doing, what to expect, make sure not only the developing code, but developing code that meets really what developers want. If you have an AI ML team that's separate, do you have to worry about them being able to know what the business goals are or the objectives or how your customers are going to use it? Does that ever come into play with them?

Yeah, it does. Yeah, they need to. But the thing about data science teams is my observation so far in the last five, 10 years is they can understand the business very well because they have access to all the data and they know how to read data better than the developers. So sometimes I would say the data scientists actually know the business better than the CDO because they get a better view of the data and they know how to read the data.

Ah, right.

So, and it's not a surprise that in many large organizations there are data science teams that hold their whole job is to read that data and give meaningful results and reports to the management.

So it's almost like they should be telling the developers, communicating with developers, hey, we found this data that you may not have feature four or something like do you know your user are trying to use it this way when you don't have something to help them with that? Yeah.

There is no evidence that your feature X actually matters.

Yeah, right.

It's like that I actually heard this sentence from a data scientist. So yes, like there is no evidence that supports that feature X actually matters.

So wild. All right. So you know, I think coming from, I worked for insurance companies, healthcare companies, those type of environments, they're really regulated. So I would think with AI and ML, I know just going agile, they were like really, they put a lot of guardrails in place because it was a regulatory environment, it kind of closed down innovation. So you know, in those type of environments, how can you make sure that, you know, you're using the proper place things in place to help because it is a regulatory environment, but without stifling, say innovation.

Yeah. So and that's a big concern and it's an area that is getting a lot of attention and research essentially. So first of all, there are these things called the Guardrails, the AIML guard rails that will, that you can put in place for your models and so on and so forth to restrict your AI to what it needs to be able to do. Right. This can be outputs, this can be privacy data, this can be anything. Right. That you want to restrict. Some of these solutions are working very well, but for things like privacy, I think they are working very well and some of them still need improvement. Right. So that's one area where I would say at its current state nobody really knows how to solve the problem very well. There are solutions, pockets of solutions that solve very specific problems on the enterprises for their needs. I think that's where the state of things are at the moment. But it is getting a lot of attention, as you said, the regulated industries especially need that and even unregulated ones need that for many other reasons. So it's an area that is getting a lot of attention, a lot of open source projects getting started on that area as well. Where you're trying to put guardrails to your model so that the outputs are within the parameters that you want.

Absolutely. And probably one other thing that comes to mind is a transparency, especially with aimlab, within the governance of IT in these type of environments. So how can developers make sure that. I guess there is transparency?

Yeah, and this is a big concern. There was a news I think yesterday where I think it was OpenAI and Cloud AI. They agreed to submit their models to a subdivision of NIST for testing. So that means that there is some sort of government, but it's voluntary at the moment in that is applied by nist. That is telling that you know what that AI is doing. I think that there are similar regulations that are coming in in EU as well in European Union as well, where you need to keep records and of, of where your data came from and where model is deployed on or how your model is deployed and so on and so forth. There are regulations in some of the US states. I think California has a new regulation on AI deployments. So yeah, the regulations are coming in and some of them are a lot of record keeping. For instance, the EU regulation, if I remember correctly, asks you to make the documentation related to an AI model available for the next seven years. So you need really good record keeping. And actually one of the reasons why model kits are getting more attention on EU than US at the moment. And then the other thing is the, I think California regulation has similar things, but not that long. So you need to make the documentation available and so on and so forth. And the versions available and so on and so forth. So there's regulations that are coming in when it comes to the provenance of the models essentially. So you want to know what your model is trained with, how it is trained, and so on and so forth. And all that data I think will soon have to be shared with the public. Which brings us to the other question is like if you go to model repositories, there are these thousands of variations of the same model, right. You have a base model, let's say llama 3. And then you have a thousand other llama 3s that are claiming to do something wonderful because they have been fine tuned or retrained or something else. And of course a good portion of those claims are true and the owners of those models or the trainers of their models are legitimate. And then there are some where the claims are not true and they're just there to lure you into using their models so that they can create backdoors and traps and so on and so forth in your business. Right. So it becomes really, really important to get your AI ML models from known sources where you like. If you're using a base model like llama3, you need make sure that the model is actually the one that meta released. Right. So, and that's going to be the one that has the provenance and so on and so forth. So the, I think the governments and the industry is starting to move into more and more into state of transparency. But as with any new technology, I think it, it just happens slower than anyone wants.

Absolutely. I know there's been a push for secure code, developers have security, but this sounds like even more drastic because like I guess it's even harder to find a backdoor in AI ML than it is in like a standard open source project. Maybe I'm wrong, but I think it's even more difficult.

It is probably right because it's like you need, you have trained your AI and fine tuned it and then at some point you said, okay, if you hear this kind of question, give me these answers. And think of it this way as well. Usually your AI ML models have access to an excessive amount of your data, right? Yes, because that's what they are good at. So if you can actually backdoor to an AI ML model, that is not a good thing. Applications are more isolated when it comes.

To data, so that's a trend. You know, anytime you're dealing with data PI, you're probably going to see governments and states getting involved. Probably. Definitely a trend on the horizon. Cool. Okay, Garak, before We go Is there one piece of actual advice you can give to someone to help them with their AI ML DevOps efforts and what's the best way to find contact you or learn more about your company?

Jovu so one piece of advice is like AI and ML can be very intimidating at first because for someone coming from application development and DevOps background, because there's a lot of materials out there that talk about the data science part of it, don't be intimidated. Try to learn how to use AI ML models. There are a lot of base models out there that you can use. There are a lot of good rag tutorials. I think that's a very good place for developers to start and it gives you a very good introduction to AI concepts, especially generative AI concepts. So don't get intimidated and try to learn about it because if not this year, you're going to have to learn it somewhere in the future. And for anyone who's trying to reach to me or the project, we are@kitaps ML. So if you go to the KitOps ML website you can reach the open Search project. The company address is josu.com and my name is Gurkham, so my email is gurkhem.com and for links of everything of.

Value we've covered in this DevOps Toolchain show, head on over to test guild.com p166 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain.

Awesomest.

As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation, software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges. We're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.