QE AI Code Quality in DevOps with Itamar Friedman

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, AI is all the rage, but how can it help you with code quality? That's what we'll be talking all about today, where Itamar, who is the CEO and co founder of Kodu, previously Codium AI, and before founding Kodu, he actually was the founder of Visual Lead, which was acquired by Alibaba Group, which probably everyone knows about. And he's worked for the Alibaba Group for four years as a director of machine vision, which I really excited to dive into as well to see what that's all about, and really excited to hear more about this new technology. It can help DevOps engineers, testers, and everyone with code quality, which I think is a good use of AI. You don't want to miss this episode. Check it out. Hey, welcome to the guild.

Yeah, thank you so much for having me here.

Great to have you. I guess before we get into it, is anything in your bio that I missed that you want the Guild to know more about?

Yeah, I think that was good enough. I think, like, throughout my career, I worked a lot on quality stuff. Whether, like, I can add to what you said, that I worked on a chip certification and Mellanox and also, like system verification, et cetera. And then, you know, when you're doing machine learning, I have a master and bachelor's degree in machine learning. You're dealing a lot with benchmarks. So it's not a surprise that eventually, when we came into disrupting the software development world, we care a lot about quality and testing and reviewing. And that's also part of the reason why we're renaming the company Codeium AI, although I love the name to Coto with a Q in order to emphasize the quality of development. That's what's behind the Codo name.

Nice. So, you know, back in the day, when I started, Quality was relegated to a group of testers. And I'm not saying developers didn't think about quality, but it was. Wasn't the top of their list, I don't think, because they had a group that handled it. So obviously there's now a shift where developers are more and more responsible for quality. So is that why you see more of a need for tooling to help them not only create excellent code, but make sure it's of high quality.

Yeah, that's one reason. But there's another reason that AI is already writing more and more of our code and probably in the future is going to like, has even automation level of code writing by eyes going to increase. There could be like lines of code, maybe many of them that a human would not review. So we have to have the counterpart. We don't want to have those outages where like half of the world is can't work because of blue screen or whatever. And referencing to one famous outages outage from last, last few months last month. So in order to enable that level of automation of AI writing more and more code, we have to have the counterpart that focused on the quality. You can ask yourself, by the way, okay, if the same AI is like writing code, why it shouldn't also check its own code. The thing is that I'll keep it short to start with first. It's different techniques. Yeah, maybe both cases like LLM as being one of the core components both in code generation and code quality. By the way, sometimes we call it code integrity. This field I can explain. But so it could be like, could have some mutual building blocks like LLM, but still the overall system looks very different, the UI looks different. And also you don't want to let the cat make sure that he's guarding his own milk. Right?

Right. Yeah. So that was my follow up. How can you trust AI written code being tested by AI? So where does the human come in at this point?

Yeah.

First let me like answer your first part. Like it's like saying, how could you trust, you know, code written by human if it's tested by another human? Don't you think like there is some equivalent and like both of them are human. So the thing is that they use very different tools. In like, in a platform like Codo, we have tools that integrate mutation testing as part of its we, we have a statical analysis that is focused on understanding behavior of the actual usage. We are focused on analyzing best practices of our organization and then making sure that they're being applied. So the techniques and the user interface and the method, by the way, I'm going to share like a news that unfortunately, like it's only a few days from now that we're going to share the actual numbers. But we took for example 01, the new models by OpenAI and they released that the number that they share, that they competed with 001 on a competition called Codeforces and supposedly 01 syncs by itself. You don't need to put it as part of an agent. It could do all the steps that think by itself about the problem. Okay. It happened to be that we developed an additional agent. Not additional, sorry, like an independent agent called alphacodium. We call it like that because to Give credits to AlphaCode by DeepMind. An Alpha Codium can take any model and also compete on that programming competition. But this time also applying testing technique, code review technique, spec matching technique. Surprisingly or not, we O when you put it inside Alpha Codium. Alpha Codium improves the results of O1 by 55, 0% better. Okay, why? Because it exploits the techniques that O1 cannot like around testing and code review and then the overall outcome of the agent Alpha Codium that is using a model like O1 that can supposedly think is much better. Okay, so it's like why AI? Why should we trust it to check other AI? Because they're not exact same entity, similar to how a QAER is not exact same entity and not using the same tool as a developer.

All right, so how does that work? Can you break it down for me? Like is it do you don't prompt it? It just knows like this code does this. So therefore I'm going to test these inputs and things that I'm going to crawl it automatically and be able to determine states that it should and should not be in.

Actually great question and a paper we released, it's actually very. And by the way, what I'm sharing is open source. It got accepted by many like even Karpathi tweeted and related to it, etc. So basically we call it from prompt engineering to flow engineering. So your question is like spot on. It's not just using the right prompt. Actually it's almost not a lot at all about that. It's actually developing a flow where as part of the software generation of the code generation, we don't only use LLM to generate code, we also use LLM to generate test. We also use other techniques that are not related to AI to run the test, debug the code, verify that the specification meets the actual development. So why. That's why we call it flow engineering. Because think about it more like as an algorithm that runs a step by step running different LLM calls, different prompting, different tool usage until it generates a code that was actually reviewed in different ways. And O by itself cannot do that. Like it doesn't have the framework to do that, doesn't have the tools that's connected to it. Two things that are important, the framework and the tools all Right, So once.

Again, how does this work then? Like, how do you envision this working in a modern software development company? Then do they just say, here's what we want? They hand it off to an AI that then codes and creates it and then calls an agent that then tests it and then lets you know whether or not they're going to ship it into your production. What does that flow look like?

Yeah, great. So I'll start with five years from now and then I'll back propagate it, derive it back.

All right, awesome. All right, cool. Even better.

So think about like in five years from now. I think many of us believe that there's going to be some technical product manager. Notice I didn't mention a developer, but I did mention technical writing some feature. Let's imagine that like I am a technical product manager in one of the biggest E commerce and now I'm dictating a feature to my AI. Hey, every time one of my users is adding a second item to the cart where it somehow relates to the first cart, suggests the color that will fit the first one. And then the AI generates like 5 different like flows for that, including the actual code underneath it. And then the technical product manager chooses according to different criteria. I'm not going to touch on that. One of them. And then click push to production. That's the part that we need to stop for a second. And there have to be that additional dashboard. Okay. That shows here all the tests that were run to verify this code here all the tests that were run to verify other flows that are relevant to that. Nothing was broken. Here's what we checked that this Codo platform checked the that maintainability of the code and according to the company's best practices and even according to the company's values. Let's not touch that for a second. And by the way, we noticed one piece of code that we're not sure about and you should send it to your tech, your dev lead. Okay? So I think how I imagine it is that there's going to be like intelligent coding system, intelligent coding solution that is in charge of writing the best code it could. It will probably work quite fast. I'm relating here to thinking fast and slow on the generation part. You probably work as fast as you can. But now that you understand that this is probably what you mean, now it's time to think slow and think how do I want to refactor verify test, etc. Before I push to production. And that's a different technique and a different platform. So now deriving back to today. In order to do that in real world enterprise software, there's so many things that needs to be attached, checked, mocked. So it will take time to get to that future. And I think that this future will be achieved by many mini agents. Each one of them is doing something relatively, you know, compartmentalized, like really specific. Like this one is very good in verifying that you're not breaking databases. This one is very good in verifying on unit testing, this one is good on egox testing, like end to end etc. And then each one of these agents needs to run once you generated some piece of code and give feedback back and get the feedback back in order to give a green light or something that needs to be improved. And now at Coto and other companies as well, each one is building a different building blockchain. Eventually someone is going to put his hands on all of these building blocks and then we'll be able to enable the next generation of AI developer and enterprise, which just doesn't exist today.

All right, so I've worked for healthcare companies, insurance companies, completely different domains. But you needed to understand all these regulations and rules within each domain. So an AI I think would be generally trained. How would it know your system or your domain eventually will there be an LLM for healthcare, an LLM for insurance? Like I mean obviously you can have one that writes code and writes and test it, but how does it know the context of your company or your software or your business domain? I don't know if that makes sense.

Amazing question, makes a lot of sense. That's actually very related to my point. I think that. Why do I claim that there's going to be many agents on the integrity part? Because I think that each agent will have like a set of guardrails and data that is related to its task. Like one is like was well engineered, that's why we call it flow engineering our paper. One was well engineered to deal with the compliance and security and different like best practices related to what you expect in software and healthcare. And specifically would also know how to attach itself to specific code bases of specific organization and it will verify that the co generation part of the system actually adheres to this. So and I think like to create each part of this specific agent requires a lot of expertise and time invested in it. And yes like it would be different guardrails and guidelines, giving in a health and or security or education, et cetera, like markets. Again I'm talking about like few years from now. I think we're still building Those building blocks and capabilities.

So do you see like a role of like an AI God, rail engineer type deal.

As a possibility?

Let's say, you know, it's like a sub. It's like you have front end, back end, et cetera. Yeah, so I definitely seeing you have security experts, et cetera. I definitely seeing that type of sub role.

All right. I mean if someone's listened to this and they're a developer, a tester, they might be like, you know, this is just disruption at its highest and I think it's going to happen. A lot of people think it's hype. How do you know what's hype and what's real though? Obviously you've been in AI game and machine learning before. You know, it really started taking off the past few years. So how would you persuade someone to say, hey look, this isn't just hype, it actually works.

Okay, so first I want to relate to these people that think this way that I actually can understand them because I think right now there is like people tend to overestimate the capabilities of AI today. I think like eventually there is no real solution right now to create like software end to end. If it's not. Despite all the amazing demos that you see out there, they're usually like five pages website that you can probably just do yourself by choosing some GitHub repo now. You know, it's still amazing that you need to go and search the GitHub repo which is better and actually make modification, etc. Might take you a few hours then and I'll take you 20 minutes. It's amazing. But we're far from actually, you know, having AI automating like creation of real world software that's run by millions. The gap between this five pages website to a real like software is huge and, and it will take time to actually fully disrupt that part. So I can understand people that are actually like professionals and looking what we have today and saying, oh my God, everybody are exaggerating. At the same time I think they might be underestimating the power of like using this technology and slowly or quickly building system like comprehensive system around them like the one that I just described in the last 15 minutes. Once we actually integrate AI, not just like hey spit like this website for me, but we integrated to apply, verify best practice. We integrated to check 100 different like it will take time to engineer that, but we make it help us verify 100 different types of possible issues and bugs or groups of them and another one that is in charge of like verifying that we're not ruining or like databases, etc. I'm focusing right now on the integrity part. Then suddenly there could be an opportunity of a level of automation that is really, really high and eventually it will even exceed our expectation. I think that happened with real, let's say dark or magic technology like electricity, right. Or resist like chips and things like that, transistors, et cetera. That originally people imagine like AI when transistor came out, but it took much more time. But eventually it came the same thing here. Eventually it will come. So that's what I have to say to those people that do. There's full systems that are being built that will be more than what you see and they are the one that going to like provide the promise of automation that we're talking about. But still I think people will be involved by the way, like developers going to, like they're going to profession going to change, but they're going to be a lot of developers.

All right, so there's going to be a lot of developers. So how much do developers need to know about coding then? Or language syntax and testers need to know about testing. At that point it almost sounds like you're going to have to be an expert developer and tester. Not a low level one. But if you have AI doing it, how do you get to be an expert with AI doing all the heavy lifting?

Yeah, I very much agree with your second part. I'll explain. I think that the majority of people that are working right now are like there's like a Gaussian, right? Like you have quite a few, you know, junior, but the majority is like, you know, somewhere between junior to mid and mid to senior and then you have the seniors and the principals. And I actually think that what's going to happen, or maybe the more accurate saying like being junior about coding, like not the amount of years of experience, how good you are like in actual software development. So that's like a Gaussian. Gaussian. I think we're going to see two Gaussians. Like actually there's going to be much more. Like many people like like 10x or 100x more people like Canva by the way, 100x more people can do design or outputs of designs of design. So it's going to be like 100x more people can do software development because they need to write technical, technical English. At the same time, I think that with all the respect to AI and relate to what I said before, it's going to take many, many Many years until you don't need the developers to verify, to review, to deal with the hard problems. And actually there's going to be so much code written by AI, so there's going to be a lot of code that developers needs to, you know, tackle, hustle with, debug and steer like other AI to work together with AI to solve. We will be able to solve much more complex problems. I'm talking about flying to Mars, but also like software problems and with that developers very, very like professional, when with AI would be able to solve things. And then considering this world, how do we get to a point where there's many more like developers that are professional than before? That's an open question. I don't think I have a good answer. But to some extent I think AI could push people that do want to learn, actually learn faster. Like I can tell you myself that with AI I'm learning faster if this, if I care about it. Right. If I just like I could do copy paste from anyway so I can do even copy paste faster now with AI, but if I care about the learning, actually I can ask many questions that will take me a lot of time. Okay, why did you give me this code? Why it could work, why it could not, what are alternatives? Things that will take me a lot of time, might help, like actually push to even a bigger portion of professional developers.

Yeah, that reminds me of saying developers won't be replaced by AI, but developers with AI will replace developers. I don't know if that makes. But you know, it's going to be an assistant almost sounds like.

Yeah. And the same thing for testers, by the way. So I related to developers, but I, I believe the same thing that I said, copy paste almost to testers.

Yeah, yeah. So what can AI not do? I can't think of the algorithm that I always read about when I started university about. I think it was a halting problem or something that like no matter what computing power you have, it would never be, it can never, you can never get over that hurdle. So there has to be something with AI that it's not going to be able to do or is that indetermined because we don't know what we don't know.

Yeah, I think that I'm not necessarily going to answer that. You expected not a mathematical one or so despite like promising websites or chatbots that are actually replacing people, replacing professional. I don't know, like, like people are talking to bots more and more. I just don't want to mention names, but I think you Know what I'm talking about then I still think that even 10 or 15, sorry, 18 months or 36 months or even five years from now. I don't believe that despite the fact that AI could talk like a human, it will actually replace the human interaction. So I know that's not what you meant, but I think that actually AI won't replace the interaction between the product manager to the business owner to the developer. A lot about creating the intent of what we want to do is going to be augmented by AI. But I still think like decision of what we actually like want to do and is going to be by humans. Like again I'm saying that, I'm not saying that it's not going to be involved by AI. We're going to brainstorm with AI and AI would give us like recommendation and even statistics like hey, this has a chance of 51 success chances to succeed because of this and that when we want to make a decision. But still I think like the bigger part of decision and communication which includes like, like brainstorming the creativity, it's going to happen by, by, by people. So like developers, like, like we said before is going to shift more to a technical product managers than they were before.

All right, so we touched on a lot of topics here. I don't think we really dived in and we touched on a little bit what actually you all do at Kodu Codium. So maybe you mentioned mutation testing. So like what do you all do? Like how does it help? I know you've touched on it, but maybe like a little bit more.

Yeah, sure, yeah. So by the way, just elaborate a bit, a bit more about what my previous question and then it actually fits well this one. So you might have thought that maybe like things like architecture would be hard for AI. And I think I see a lot of people talking about that. Like AI would have a hard time about software architecture and things like that. So in my opinion one day it will come as well. Like I can learn old AWS and actually maybe can even read logs and other things that for us is hard to do. But I still think that the eventual architecture decision, it will be made by human. Just the suggestions would be made by AI, which are not. There is no tool like that today. And that leads me to your question. I think like for example, in order for AI to suggest architecture decision copilot can't really do that and LLM cannot really do right now, you would need to integrate like other techniques into such kind of a solution. I'm not only talking about knowledge that it would train, but actually like thinking processes and thinking frameworks. What we do at kodo. Yeah, it's the first time, by the way, like in a podcast I'm mentioning this name. So I hear that sometime. Codo, I heard you saying, but it makes sense. Like we're Q O D. Okay.

Yeah, yeah.

But KODU is actually also written. You write it the same. Right. Qodo. So KODO is like our solutions. Yeah, it's the first time and you're the first person probably to hear this change. Like we're announcing it in a few days. So the thing is that we have different agents, right. We have for example, what we call the COVID agent. This agent is in charge of taking an existing test suite, for example, even one test and increasing the test suite coverage, for example, from 10% to 80%. It is right now doing only regression tests. But it's a huge problem. Like every time you want to add a new feature, you're afraid you're breaking something. One of the ways to reduce that is like create a men's testing suite. Like reaching 80, 90% code coverage regression one. And then you know that if you're developing a new feature, you shouldn't break the regression test. So what it does it takes for you, like your one or two tests you have in a test suite and enhances it for 10, 20. Really depends. Like how are the criteria to stop at a certain code coverage? Now you know, code coverage, that's the criteria. Where to stop is actually many would consider that as a proxy metric, even a vanity metric. I heard one. I have one friend that told me that he's a VPRND. His teams are they must reach to 80% code coverage. One day he checked one of the repos that reaches 85. He was so like proud. And he noticed that there's no asserts. Not bad ones, no asserts. So it means that the code coverage like almost not reliable at all. Because you know there's some value for it. Because if the test crash. But it's not really checking logic. So what we think is that you need to apply and by the way, AI many times might do that. We notice it. You call even very good LLM and it might create you like bad or not complete or good asserts. Now with mutation tests, it's a specific technique. What it does, it creates mutants in your code, actually changing the logic. And it wants to check that the test suite actually kills them. For example, if there is no assert in your test, probably it won't kill, won't Fail the mutant. So it means that despite having a high code coverage, you're actually having a bad test suite. So this is like an example of a technique that usually most code generation tools LLM would not exploit. And if you're focusing on quality like we do at Code qo then then we do, we do do that. Yeah.

Awesome. Yeah. Code quality, especially with the coverage metrics I, I've seen teams gamify it where they, they just put in checks that are meaningless and they get to high code 80, 90% and but it's not doing anything. So that's awesome.

Yeah.

All right, so we talked a lot about development and using AI. How about production? Do you see, you know with open telemetry and things like that that once you can actually use AI in production as well that yes the code has been tested but once it goes in the wild you don't necessarily know third party services it's using or whatever it's using in the background because it's in the wild, it's not in your bare metal type of environment. So there's a lot of unknowns, juicy AI being used now to say okay there's an issue because it's open telemetry, I know the logs, let me fix it without having a human involved in that process.

Yeah, totally. So I think when we started the company then we thought what do we need to do like philosophy level to reach 99.99 code integrity for us code integrity is like means that it's well tested, the code is according to your specification, the performance is as you expected. There is no even. We can even inject like security in it but it's maintainable. That's what we call code integrity. I mentioned I'm going to get to it and yeah it happened to explain what I mean by code integrity. And now when we imagine how can we get to 99.9 whatever the thing we imagine is what are the sources of data that this such system would need. And we actually marked seven data sources. I'm not going to talk about all but definitely one of them is that the software, the running like all the data relates to the actual run of the software in production. You mentioned opentelemetry. It could be like a very good glimpse into that. Like it's not like as EBPF or whatever that gives you up down to the stack like the function level stack in most cases but you can like output like very important parts as far as I know of, you know of the data and that runs through your software and the event that's happening in your software, etc. So I definitely think that we must have this kind of information to eventually verify the integrity. I'll give an example. The same method, like even a sort method. Think about a sort method might work or not work, might be good or bad depending on the distribution of data that run through it. Right. So and usually you know the distribution of data that runs through that sort function only when you actually run it in production. I gave a simple example, but now take it a bit more complicated. Just also like to more complicated example.

Love it. So you mentioned security a few times. I would think, doesn't this make security even harder? Because you know, even using third party open source libraries you're like, it has vulnerabilities. How do you know like bad actors aren't going to almost like seed an AI And I would think it would make it even harder to detect these type of things. Is that something people could be concerned about? Is AI going to be able to help? Do you see regulations saying hey, we can't use AI for this because we're worried about security? Is that something that'll be worked out over the years if it comes to that?

Actually I think it could go to two directions to be frank. One, one future is that you know, you were gonna get have like 5x10x more line of code generated every year. So it means that the surface risk surface actually grows security. One, two, it could be like bugs which we saw like also could be really meaningful S3, data leaks, et cetera. So the risk surface grows as we use AI more and more. And that's one future. And that's why you know, it's very important to have more and more tools. And this is roughly speaking how we imagine the world's going to. Because I think like people would, would be very biased towards generating really fast. And then usually when you think very fast, even if you're an AI, you will do mistakes. And the security part and the verification part, et cetera, would think slower, let's say like that. And there's another future where hey, maybe we can train during the training part, can train AI so well, including managing to catch like injections that people try to put in the data that AI was about to train on, or we trained so well to manage to recognize that it's going to spit like a security problem, that it will help us to actually, you know, to start with, have less problems. That's the second future in my opinion. We're going to see a mix of both. So we're not going to see the risk surface, explode, totally explode. Like although lines of code and automation is going to explode, it's just going to grow linearly, let's say like that. And at the same time it's still going to grow and we will have to have those tools if we want to have our world keep working because the software, our world runs on software.

All right, so we touched a lot on AI for development and we did talk about testing, but I'm just curious to know a lot of my audience. This is for the DevOps show, but I also have a one for automation testing. Where do you see AI with testing specifically or the future of AI in testing if someone is listening in and they see themselves more as a tester?

Yeah, I love the question. Thank you. And I tell you why, because I think like many people hear the word testing and it sounds like one thing actually testing is such a huge like field. You have the triangle of testing bottom up like unit testing, component testing, integration testing, system testing, end to end testing, et cetera, UI testing, even UX testing. I would claim there is different flavor like regression testing or smoke testing or like testing for new features, testing for regression like I mentioned, et cetera. It's such a huge world and fortunately and unfortunately to check software you probably need them all. So asking like AI for testing is like asking like work and all. This entire like field, like big field, where can we gain the most? So we at Codo, we think about it quite a lot like and I think like each testing has different UX UI and different techniques. So we started with unit testing and I think that's like the closest to co generation to some extent. But I think like AI could really help for example a lot with UX testing. Like maybe we'll call it end to end testing. Like testing for example flows good and bad flows on your, your website. Think about it, it's like very intuitive. You can describe what you're doing in natural language. There's even the cucumber or whatever like, like different languages, playwright, et cetera that are almost English on how you want to like. It's not like that far fetched to think that AI could help there. I saw a few tries, I don't think anyone nailed it. And again shameless plug. Be aware if we are going to release something in that field very soon. But I still think like there's potential about that. So bottom line, my suggestion around it and my intuition and insight is that like testing is one of the more most complicating Part of software development, just what I mentioned and I think it will take time for actually I to integrate there because it's such like a complex world but it's going to happen. And as a tester, look at tools, try them, be that futuristic tester and as a, as a manager, think about what's your biggest, you know, pains around testing and check it out. Maybe you'll find something that is different than just autocomplete that can really like meaningfully boost your quality and productivity.

With modal AI, it could do visuals now. So where do you see the future of end to end testing then? Is it going to be all image based where you just click, click, click because you don't need to do, you don't need to code it it seems like because it understands the visuals of a flow, if that makes sense. It knows what a button is, it knows what a text box is and you just say enter like just using visuals and then you can take that and run it on a mobile device, all these devices because it's just going by image based or visual cues rather than text prompts. Does that make sense? Is that what you see?

Yeah. So I think there's a difference between what's possible to what I think would actually be implemented. I will explain. I think like it is possible, even better. Yeah, exactly. I think it will be possible to supposedly create testing agent that only focuses on your visual instructions or maybe even explores your website or mobile app by itself as a human only through the visual. The thing is that I think different than us human and developers, sorry, human and maybe testers that it's hard to us for us to have the capacity to look on the code and visual simultaneously. There is no reason that actually AI won't do that simultaneously because there is still a lot of knowledge, sorry and information that can be instructed from the code code itself by the way also in order to, you know, to do test healing or to get some hints about like the next steps etc. And by the way also in order to try to find ways to break your your flows maybe there's going to be hints, not individual but actually hints in the code. So although there is possible in my opinion like in the next, within even next year to build a testing agent that is focused on the visual itself. I believe at least that's what we're doing at Coto to mixing between the two modalities. Now last reason why because I think like for some things it's actually much better and easier to check on the visual itself because Some things are very dependent on your Chrome version or things like that with all the Z indexing, what's the actual thing that humans are looking but at the same time in the code there's a lot of hints in the information that you can get to try to find good flows or bad flow that could break or make things awesome.

Okay, to mark before I go, is there one piece of actual advice you can give to someone to help them with the AI DevOps efforts and what's the best way to find contact you or learn more about coding?

So about my advice might be a bit generic, but I would say using AI is like a skill. It's like using a new programming language, et cetera. Don't let it be perfect before you start checking it. Exploring different don't only use AI for code completion, use it for There are so many other options you know to embrace AI or exploit AI. For example Shameless Plug. If you go to kodoai Qodo AI, you'll find that we have various tool that helps you around things around quality that are not necessarily code code generation. So that's my advice like try out like develop that skill and become like that. You know the future software developer about reaching us? Yeah, we just renamed our company from Codeium AI to Coto Codo AI so you can find us on social Qodo on our website from there like probably the best way to land and you can find me and Itamar Itamar usually like Friedman and you'll find me or Itamar Marmar again you'll find me on X on Twitter as well.

And for links of everything of value we covered in this DevOps Toolchain show, head on over to testguild.com p165 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate, visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres. The bards began their song A tune of knowledge a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.