Generative AI for DevOps with Shani Shoham

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe and welcome to another episode of the DevOps Toolchain podcast. Today we'll be talking with Shani, all about generative AI for DevOps. If you don't know, Shani has been on the show multiple times on the automation podcast. I think this is second time on what used to be the performance podcast, now the DevOps toolchain. He's been a CEO, he's been a COO, he's been a CRO, at least five different dev tool companies, including companies like Testicem IO, which I think got acquired by Tricentis, 21 Labs, 21 Labs, which got acquired by Perfecto and he's now the Chief Revenue Officer of Cubia A. I believe I'm saying it right. Alfaido, a chatgpt for DevOps and really excited to have him back. And so it's a really cool smash up of DevOps with AI and a lot of cool things that you're going to need to know about. So you want to stay all the way to the end and I think you really get a lot of value from it. So check it out. This episode is brought to you by SmartBear. As businesses demand more and more from software, the jobs of development teams gets harder and harder. They're expected to deliver fast and flawlessly, but too often they're missing the vital context to connect each step of the process. That's how SmartBear helps. Wherever you are in the testing life cycle, they have a tool to give you a new level of visibility and automation so you can unleash your development team's full awesomeness. They offer free trials for all the tools, no credit cards are required, and they even back it up with a responsive, award winning support team. Showing your path to great software. Learn more@smartbeer.com today. Hey Shani, welcome back to the Guild.

Hey Joe, how are you doing? How's the chickens?

Chickens are awesome. Actually we lost a few of them so we just got some baby chicks. Now try to make the flock bigger again. It's kind of tough when you have a hobby farm. You know, animals die unfortunately. But yeah, it's good other than that for sure. So yeah, as I was saying in the pre show, I always follow You, Shani, because I almost see seems like you have a pulse on what's going on in the industry. And it seems like whenever something's about to take off, you seem to be right there in the midst of it, ahead of everyone else. And so I've seen you on social media recently talking about this new technology called Cubia and I just want to get you on to talk a little bit more about what it is and a little bit more about generative AI and how it helps DevOps.

Yeah, happy to do that and thanks for the compliment, I guess I'm only, It's, you know, it's pure luck. So yeah, let's talk a little bit about Cubia. First of all, you passed the first test which is saying the company's name, Cubia is essentially comes from Kubernetes and initially we're thinking of doing something specifically for Kubernetes. And then the other thing is Cubia is a cube in Hebrew. So but you know, in a summary, what Cobia does is with Chat GPT, like for DevOps, ChatGPT is actually more attention grabber. The experience is like ChatGPT, but we don't really use a lot of ChatGPT. The idea is making complex DevOps processes accessible to the rest of the engineering team using simple conversations. Again, anyone can just go over the chat tool and ask something that's infrastructure related and Kubia will understand the context and is able to have a bidirectional conversation that's contextualized to your engineering platform. So again, knowing your kubernetes namespaces, your deployments, your terraform models, your S3 buckets et cetera with proper guardrail that is set by the DevOps team and I'm sure we'll talk a little bit more about proper guardrails. In a nutshell, the problem that we're trying to solve is the DevOps toil that the DevOps teams are experiencing today. When I talk to DevOps teams, you know, the ratio is somewhere between 1 to 20 in a good case to 1 to 60. And you know, single DevOps person supporting, whether it's 20 developers or 60 developers with relative requests, dozens of them coming in the middle of the day, many times in odd hours in many of them require context switching, a lot of them require back and forth. And so if there was a self service experience that you know, doesn't require developers to know ansible and terraform and helm charts and Kubernetes and a bunch of other things and again has proper guardrail so that no one can just write something and suddenly your entire production environment is down. Then you know, that would significantly reduce the toll on DevOps. And then they would, it would free up their time to spend more on, you know, innovation and building, better monitoring, better alert systems, observability and other things.

So that was actually one of the questions I wanted to ask is why was this created? What problem does it solve? So it's mainly for toil. Is there anything else that was created to help alleviate and is it for just developers? Like who was it created for?

Yeah, great question. So first of all, it's not just for developers. I mean, we even have customers where, you know, it's actually used more by the ops teams. And again, making infrastructure or complex processes accessible to solution engineers, to sales teams, even around like spinning up demo environment to program managers and others. There's two sides of the equation. So the first part is the engineering team. And again, it's not just developers, it's also QA that want to trigger a new job or want to pull logs from failed pipeline or, or data scientists that need some kind of proper instance with seeding. And then on the other hand is the DevOps. So today when a developer, as a data scientist, a QA engineer wants something infrastructure related, you know, there's two solutions. For most companies, it's open a JIRA ticket and then a few hours later someone from the DevOps team comes back to you with a bunch of questions and those back and forth and maybe it requires an approval of a manager or someone in finance and before you know it, it's been three days. And that's the solution that most companies have. The other solution is you expose some kind of API to the developers, which, or to the engineering team, which requires a lot of maintenance, which requires the developers to know those APIs, remember them. Those APIs are typically very rigid. And so either way it's not a great experience. And again, what we're trying to do is we're trying to first of all leverage the interface that you spend most of your day in, which is, you know, Slack, Microsoft Teams, Facebook workspace, and again, just have a conversation, which is natural. I mean, the way that you do that today without Copia is you get in touch with someone on the DevOps team and you just talk to them and ask them for something. Same way with Kubia. So there's literally zero learning curve. There's, it's as natural as, you know, as it can be great.

And it's all in one place and it's in the DevOps or the developer's workflow almost, so they don't have to get up, find the right person, track them down. It's all from where they're working. So it helps speed things up as well, I would think.

Right, yeah. I mean, I'll give you an example. So let's say, you know, I'm a developer and I want an expensive EC2 instance and so I would open a JIRA ticket. And again, there's probably going to be some back and forth and at some point someone in management will need to approve that. And given that they're not like JIR is not, you know, in front of them all day long, then there's gonna be delays even just related to like the handoff process. And again, if you're on Slack all day long and the manager gets a message on Slack and you know, it shows a notification next to his app and in a minute or two, even during the call, he can just go and approve that or go and have conversation with the developer that asked for that instance to get more context. It's inside your communication tool. So we save a lot of that handoff and you know, in a nutshell, we take a process that might take, you know, a few days and we narrow it down to literally two or three minutes at most.

Nice. So, you know, once again, I'm just thinking of what people are already familiar with. They're probably already familiar with things like chatbots or rpa and this sounds a little bit like that. So are you just taking advantage of AI and slapping AI on top of this, or how is this different than a chatbot or rpa?

Yeah, it's a long answer. The short version is, first of all, you know, chatbots and Slack bots are very rigid in terms of the experience. You click on a slash, you get a list of options, and that's about it. So it's not like you and me having a conversation, you know, me saying I want to get logs. And you know, the system understands that you want to get logs from a kubernetes namespace. So that's one. The second thing again, going back to that widget experience, two developers are different. So one might be a senior developer that knows exactly what he wants. And you know, I want to pull logs out of this namespace and that deployment and another one might not know what he wants. And so he knows he wants to get logs and he needs something to workflow in order to. And a little bit of back and forth and some questions in order to have what, what he want to get done. And so in a Slack bot, the experience for both developers would be the same and they would go for that same rigid experience. A third difference is on the knowledge management side and I talked a lot about kind of triggering DevOps actions, but the other challenge that they're also facing is many times, you know, I would say 20, 25% of the time when I talk to DevOps, the time is consumed by, you know, how do I do this, how do I do that? And there's a document somewhere but nobody bothers like looking for that document because that document is not easily searchable. And one of the nice things that Kubia can do is you can just forward a bunch of docs and it will learn from those docs and you can ask a question and Cubia will construct an answer on the fly with pointers to those documents. And so again, this is a very flexible and open ended experience. And then two final things that I think makes it different and is also important when you use AI in an organization is to have the proper guardrails and the proper control. So again, two people on the engineering team might be part of different teams. Different teams require doing different things. So one might be a data scientist and he needs access to GPU instances. The other one might be a developer that needs access to like simple, you know, EC2 compute instances, nano, micro, etc. And a Slack bot is, I mean you can probably build that experience, but it's going to be very complex and again, a very, you know, boxed experience, if you will. And then on top of that you need the ability to request temporary access, you need the audit. Well, you also want, the DevOps team also want to be able to kind of define the experience for the developer. So a good example I like to give is you can just list all the instances, but maybe certain teams, you want to filter out certain instances or maybe if they choose certain instances, you want to ask for manager's approval or maybe don't want to list the instances because the instances mean nothing. For the data science team, you want to ask them what do they need the instance for and then based on that, templatize the experience and we even start talking about like GitOps and Terraform and Ansible. So those are just some examples of like how, you know, a bot experience is very different than a Cubia experience.

So yeah, I'm still trying to visualize it. It almost sounds like Siri for DevOps. You just say, hey Cubia, do a deployment for me or something. So it's almost like a conversational deployment type deal.

I. You know, honestly, we should hire you to the marketing team. The.

Cubia. Before Cubia started a little over a year ago and ChatGPT wasn't around then, so nobody like, it was very hard to visualize the experience. So we would call ourselves Siri for DevOps because that was the closest thing. Funny enough, we j'd in Kubecon last year, which was around October, and it was right around the time that OpenAI, you know, started making a lot of noise. So, yeah, we. We kind of got lucky that we literally ga'd around that time. You're right. I mean, essentially you just go over slack in the same way that two people would have a conversation. You either have a conversation with Kubia, by the way, you can have a conversation with someone and take Kubia in the middle of the conversation and ask it a question and it will deliver the answer as well. But yeah, I mean, the idea is you literally just have a conversation under the hood. You know, we talked a little bit about Guardrail. So the workflows and triggering action is something that the DevOps team can control. And that's also, you know, pretty cool. We use generative AI to create those workflows. So, you know, a DevOps person can just go and say, I want to create a workflow that pulls logs out of kubernetes namespace and deployment, and Cubia will be able to do that. So again, a lot of usage of LLMs and embeddings and all kinds of algorithms to deliver that experience.

Nice. So you mentioned this a few times. This was developed before. I mean, ChatGPT has been around for a little bit, but it seems like before ChatGPT really took off. So is this using ChatGPT and OpenAI? And if not, you know, is there a difference? What are you all using? What have you seen? Yeah, in the real world, the use of this compared to if you're not using chatgpt, whatever you're using.

Yeah. So first of all, chatgpt is a very broad term. You have, you know, GPT, which is essentially a large language model you have, which is powered by OpenAI. And then ChatGPT is just kind of the, you know, the experience of it. So we use, at the end of the day, you know, engineering platforms hold very sensitive information. It's like the core of, you know, your technology. So definitely you don't want to just expose that to like some public LLM. And that's also the reason, I think, that, you know, it's going to take a while for things like ChatGPT to get into an organization. We use our own LLMs. We use four different models of LLMs. We have a separate LLM for each organization that lives in a separate namespace. The reason being is, you know, a term like a virtual machine might mean different things in different organizations. And so we don't want, you know, if for one organization it's an EC2 and for another it's a VPC. We definitely, you know, we want to keep it separate. The vector database is also separate for each and every organization. And we use some form of GPT 3.5, but again, it's in a contained environment that we control. And it's not like the public ChatGPT. The only place where you can use ChatGPT is we have a plugin to ChatGPT. And you can, if you choose to, you can create a workflow, for example, that takes, you know, Jenkins logs, which, you know, everybody knows it's a nightmare to wait for Jenkins logs and essentially send it to ChatGPT and it will give you a summary of, you know, the root cause. But it's under your control whether you use that. We don't use any public chatgpt. I think it would be a big security concern for, you know, for all of our customers if we would.

Yeah. So I'm just reminded of another conversation I had. A lot of companies when they get software they want on premise software because they're worried about security. And he's saying he doesn't see that happen because AI is so expensive. But it sounds like you have a way kind of around that. And what's your solution?

So the Cobiaz architecture is actually a hybrid architecture. So most of the internal engineering platforms, we actually trigger the actions for a lightweight operator that sits inside their network, typically on some kind of EC2 or Kubernetes or something. And then they have full control over that operator. So if tomorrow they, you know, they have a security bridge and they need to investigate or something, they can literally just cut the, they can literally just turn off that operator also then they have control over, you know, which clusters that operator is part of. You can also install different operators on different clusters and then you can separate, you know, one operator for production clusters and one operator for development clusters. And as I mentioned, the other part is essentially a vector database that holds the embeddings and the LLM. Each organization again has its own separate namespace. We can, I mean, if organizations choose to, they can even hold the Database, the vector database inside their premise. And so again, there's full control over the data, over the information, et cetera.

Nice. So what other special sauce do you all have? How does Cubia learn? I know you mentioned documentation. You can feed a documentation learns on that. But does it learn over time based on conversations between your team?

Yeah.

Where it may answer something one way and they're like, well, that's not really right. Here's the real answer. And then it knows going forward, if anyone asks that question, does that make sense? How does that work?

I think we're going to hire you, we're going to transition from the marketing team to the engineering team. So yes, spot on. So actually I think I kind of also alluded to that the each organization is different. So actually the answers could be our leverages, which reinforcement learning with human feedback. And so we actually learn from the experience of our users. And so for example, if a user asks for something and you know, we serve a list of workflows in a certain order and it chooses the third one after a few users will request the same thing, or even that user will request the same thing multiple times, we're actually going to change the order by which we deliver the workflows. Or if we delivered something and the user said no and we made a few other suggestions and it chooses one of those other suggestions, then over time we'll actually bump up that workflow, that knowledge entry to be delivered first. So you are, it actually does learn from the interaction with users, which is very, very powerful. The other thing, the other piece of technology is, you know, as I mentioned, we, the DevOps team creates those workflows that make them accessible for their end users so that they have some control and some guardrails of what's being delivered. So we actually take those workflows. We, you know, when you publish a workflow, we'll take the workflow, we'll create a description for that using, using LLM and essentially kind of mapping the schema in into a description. From that description, we'll create embeddings. Those embeddings will then live inside the vector database. And you know, when someone has to quote question, we actually compare that to the embeddings and figure out if it's a workflow and knowledge entry, et cetera.

So the guardrails also, I guess has policies where it's not going to give you sensitive information that you shouldn't have. Like how do you know if it doesn't get a hold of sensitive information that I'm Just like a junior programmer and all of a sudden I'm getting access to, I don't know, some sort of company secrets or things like that.

Yeah, great question. I think that's the other important piece generally, by the way, around AI, we had an in person forum with like 100 engineering leaders last night here in the Bay Area. And the conversation around AI, everybody will interested in talking about AI. We had a roundtable around that and what, you know, I think my main takeaway from that is that data and some level of control over what AI does is what everybody wants. The other thing is organism, you know, contextual. The responses need to be contextualized to the organization. We can talk about that later on. So there's a couple of ways by which we box the experience such that, you know, developer cannot just go and, you know, delete the entire production environment. First of all we have access control. And so you can define per person or per group who has access to do what. So it's not just at the resource level, it's also at the action level because like I said, I mean, data engineering teams require different instances than your typical developers. So that's one level of control. By the way, when you have access control, sometimes you need to give people temporary access because there's a production issue and someone you know is on call and needs to do something. So you also need to be able to, you know, create some kind of approval flow with AuditRail and managing TTL, et cetera. The other level of guardrail is kind of how you deliver the experience and what can a developer do and what he cannot. So simple example would be again, I list all the namespaces and I want to filter out some of the namespaces or I want to filter out, like I said, you can actually take that operator and install different operators. And maybe you want to give developers only access to the operator that has, you know, access to the developer clusters and not to the production ones. So you can filter out some of the entries. You can, you know, I'll take again the example of an EC2. Maybe an EC, the EC2 instances doesn't mean anything to most of the developers on the team. And maybe instead you just want to ask, what do you want? What do you plan to use the instance or the walker for? And based on that provision, the right instance instead of giving them the ability to create any instance that they want. Same thing for the security group. So maybe one asks, do you need a walker for an internal application or an external application? And then based on that Provision the right security group. So again, there needs to be some kind of contained experience that assures that you're not just going to use AI and you know it's going to create some kind of code and you're just going to deploy that code and without knowing what it actually does. Also one final thing, one final thought around that GitHub. So lots of organizations today use Terraform or Ansible or hand charts. And so again, maybe you want to provision those instances using Boto 3 or the CDK. Or maybe you want to use Terraform and actually allow them to change, you know, just a few parameters and before the Terraform model gets deployed. So again, those are just some of the ways that you want to control the experience.

Awesome. Would a site benefit be it almost has an audit trail as well. Joe's the one that did this deploy or asked this deploy did these three steps before and all timestamped and you can trace it back directly?

Yes, totally.

When and how something may have broke.

Super important is audit well and the ability. First of all, any action has audit well on top of that, I mentioned for example, an approval flow. So I'm asking to provision a C3 large instance and Joe needs to approve that. Again, we want Auditrail with IDs and maybe even a few details around why did Cheney ask for that instance and why did Joe approve that? So again, all of that has proper audit rail.

Yes, this might sound like a dumb question. How good of a question do you need to be able to ask in order for Kubia to give you the right answer? I mean, you know, sometimes with ChatGPT I know I could phrase it a certain way and there's certain prompts you can give it to get better results. Do you have any of those issues when dealing with Kubia?

Yes and no. So I'll start with a no. So what we do is contain to a specific domain which is DevOps. And so you know, it's easier for the LLM to kind of understand what you're asking because again, it contains its there's automatic association to the DevOps domain. So you know kubernetes jobs. Like if you ask to trigger a job, I know that you want to trigger, you know, a CI job and not some publisher job on LinkedIn. So I think the experience it's easier on the one hand to build a conversational agent that is contained to a specific domain. Naturally, the more information that you give it, the more accurate it's going to be. But you know, we also build the experience such that if you ask something and it gives you a list of options and it's not one of those options, you can actually say that's not what I meant. And it's going to try, it's going to iterate and try to give you other suggestion. And again, at the end of the day, since we're talking about, I don't know, 500 workflows, a thousand workflows and some knowledge entries, then it should. I haven't seen a case where at the end of the day I wasn't able to do what needed to get done.

Nice. So generative AI is kind of becoming a buzzword now, but you've actually have it built in to build workflows. Any feedback from your customers, how you see it working, or percentages of how using generative AI has helped the DevOps process? I don't know, reduce toil by 10% or anything like that.

Yeah, I mean I can tell you that companies that have deployed Cubia literally double the capacity of the DevOps team. I mean think about what percentage of your day is like, you know, these day to day request and, and again 20, 30% of the time, you know, those requests are really just about giving someone access to a doc. So yeah, I mean we've seen organizations transition from the point of no automation to 80% of these requests being fulfilled by Cobra. Somewhere between four to six weeks a lot of the workflows come out of the box. So again if you want to do things on Kubernetes, for example, scaling up, scaling, getting pod information, killing pods, restart, rollout, changing images, those sort of things, all of that come out of the box. So there's like, it's very easy to get started even without creating a workflow. By the way, quick story, a few weeks ago we had an issue in production and you know, we had to roll out the deployment. Our CTO was on driving south to a vacation, he was literally on his car, he stopped on the side, you know, open slack up and on using his phone in pretty much rollback production. So yeah, so very cool. So that's a little bit in terms of the impact. You know, I'll just throw out one other thing that comes in mind. We have a media company where like if they want to send a certain file to the customers, then you know, the file has to go through a couple of processes. And those processes today are manual and those processes take a while and so you might be looking at two, three days until they get something done. And the nice thing is with Cubia, they were essentially able to provide the information at the beginning of the workflow and then Cubia essentially handles the handover from, you know, one system to the other. One of the powerful things is you can create workflows that combine multiple systems. So in the case of a developer, it might be, you know, you merge a branch and you close the JIRA ticket or you move it to another stage or we just got approval for something, you move it from pending approval to something else. And so the ability going back to that use case, I mean, we took a process that took a few days and involved a lot of handover and you know, someone from it spinning up an EC2 walker and someone on the product management, on the management team waiting for that worker, then doing whatever he needs to get done. And by the way, many times that worker stays idle for a while afterwards. Again, we're able to cut that process to, you know, literally a few hours of an automated process plus, you know, tearing down, terminating the worker at the end of the job. So we saved both time and cloud resources.

So what's on the roadmap? Is there anything. The roadmap, you know, AI has been, it seems like you're just starting off almost with generative AI. Is there anything you see the future, how you could see built in on the roadmap to make it even better or any other plans? You see AI playing role in DevOps that people may not even be aware of yet?

Yeah, so there's a couple of things coming down the pipe. So first of all, support for a wider range of chat tools to, you know, one documentation. There's a lot of use cases. I, you know, I just talked to very large companies, everybody knows I'm not going to mention the name, but they said our docs are crap. There's, you know, they're old, they're outdated, et cetera. And so when you ask Obi question actually already today, you can give some feedback on the answer. And so if you think that the, you know, the document is out of date, you can actually give it a thumbs down and we can provide that feedback back to the team. But the next level is to actually take a conversation that's happening and you say, oh wow, this would have been a great, you know, doc on conference or notion and you can actually take Cubia and Kubia will be able to summarize that conversation and put it into a doc so that the next time you don't have to manually answer that question. Yeah, I mean there's more stuff around, you know, we're constantly expanding the integrations. We're actually using some cool generative AI to do that as well. So essentially mapping the APIs and the SDKs into something that Kubia can use literally in a few minutes and trying to think what else is cool and coming down the pipe, the ability to discover your network, your architecture and your resources. Yeah, I mean, something that we do today, but we also plan to enhance that. The ability like already today you can ask BR on the cost of your cloud resources, for example. So finops, the next level is actually the ability to provide graphs directly through your chat tool and, and again provide a much more rich experience. Yeah, I mean, those are just some of the things that's coming down the pipe.

Awesome. Okay, Shani, before we go, is there one piece of actual advice you can give to someone to help them with their AI DevOps efforts? And what's the best way to find contact you or learn more about Cubia?

So I'll start with the canned answer. And the canned answer is use Kubia Cubia AI Kubi ya AI or reach out to mention Cubia AI. You know, as I alluded to earlier today, I think the topic of AI is like, everybody feels that there's some kind of disruption. It's like the same to me. It's the same experience as, I don't know, 10, 15 years ago when cloud started. And you know, I read an article a few days ago that says that it's as a disruption as the iPhone. So everybody's trying to figure out what to do around AI. Literally, I get on calls five, six times a day where people say, hey, you know, we're investigating AI and they don't necessarily have a specific use case, but they feel that they need to do something. So I think, you know, I, I agree that it's a disruptor and you know, I think it's going to create a lot of new use cases. At the same time, I think there's still a lot to figure out, especially in terms of security and private information and also in terms of the context. I mean, a lot of the AI that we're seeing today is general purpose AI. And I think the AI will really become much more powerful when it has the content, the organizational context. And so a good example, I spoke to a VP Engineering yesterday at, you know, kind of the event that we had and they tried to use Copilot and they said it's, you know, it doesn't really save us a lot of time and we didn't find it valuable. But if it could learn from our repos and provide recommendations that are specific to my organization, then that would be powerful. And again, I think that's part of what we're doing at QBI is getting the contextual, the context of the organization, the namespaces, the architecture, the services, the buckets, et cetera. So again, I think that's the point, that AI is going to be much more powerful. And that's kind of what I hear from organizations when it comes to AI. And you know, again, we're the core fit. So we're definitely, you know, having a lot of these conversations day in and day out.

And for links of everything of value we covered in this DevOps toolchain show, head on over to test guild.com P115 and while you're there, make sure to click on the SmartBear link and learn all about SmartBear's awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed, creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.