DevOps Crime Scenes: Using AI-Driven Failure Diagnostics

When a QA person comes into office tomorrow morning and the nightly tests are done, it's like he's coming to a crime scene. There are like 500 dead bodies there, right? And we are trying to provide as much forensics data to this guy or girl to help them sort of figure out who's the killer. Like, what's the problem?

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, today we'll be talking with Harpreet Singh all about intelligent test failure diagnosis and why it's going to help you accelerate delivery in your DevOps pipelines. If you don't know, Hapreet is the co founder and co CEO of Launchable, which is a solution that leverages AI machine learning to enhance software testing by providing intelligent diagnosis for test failures. It has a bunch of other features as well. The platform really helps DevOps engineering teams to quickly identify and classify test failures, reducing the cognitive load and speeding up the process of finding and fixing underlining issues. So if you want to know how to remove bottlenecks in delivery, optimize your tests, filter out noise, analyze problems, and track fixes to help you release software with confidence, you really don't want to miss this episode. Check it out.

Hey, if your app is slow, it could be worse than an error. It could be frustrating. And one thing I've learned over my 25 years in industry is that frustrated users don't last long. But since slow performance isn't sudden, it's hard for standard error monitoring tools to catch. That's why I think you should check out bugsnag, an all in one observability solution that has a way to automatically watch for these issues. Real User monitoring. It checks and reports real user performance data in real time so you can quickly identify lags. Plus you can get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to Bugsnag.com and try for free. No credit card required. Check it out. Let me know what you think.

Hey Apreet, welcome to the Guild.

Thank you so much for having me here. Really appreciate that.

Awesome. Great to have you. So before we get into that, I'm always fascinated by founders on why they created the solution. You have a great background with a Bunch of different things, a lot of DevOps. So why launchable? Why did you create Launchable? What inspired you and why An AI driven test failure diagnosis platform to begin with.

Right, so there's a bit of a long story there. My background started with Jenkins. Sure you've heard about that tool. Interestingly enough, it was started by my co founder. He created Jenkins and he was sort of, sort of hunched over in a cubicle at Sun. They had offices and he coded that and I was around and I was like, I was an engineer who just transitioned to product management. And like I was like, this is pretty interesting. Seems like it is going to have a massive impact on. On software delivery cut short. I spent maybe eight years helping build a company called CloudBees which takes Jenkins to market. I build that business and as me and my co founder Kosuke were part of this company, he's the creator of Jenkins, we came to this place where we were sort of helping a lot of people do DevOps transformations. You know, Jenkins is the sort of backbone over which software is delivered. And we realized that despite sort of helping people kind of do that, it was still taking them ages to kind of ship software. And that didn't seem great. And we started talking and we were like, okay, where is the bottleneck in software delivery? And the answer popped up to us. It's really in the test. What happens is you write all this, you know, you create all these pipelines. And what pipelines are, are a series of test suites connected one after the other, giving you greater confidence software in the software that you deliver. And that's, you know, the tests kind of keep growing, they take time. And every time you push a change, no matter how smooth your pipeline is, you're going to spend time doing that. So we sort of honed in on that as a way of saying, like, there has to be a better way of delivering software and cutting this huge cost out from that delivery process. And that's essentially what got us started with the company.

Awesome. So I was listening to another podcast you were on. I think you got into AI maybe 20 years ago or more, and you said you were too early, but yet I think you launched a bull about four years ago, which was still, I mean, the AI was around obviously, but it's before the chatgpts and generative AI really took off. So it seems like you were right at the sweet spot of that taking off. So how do you know it was the right time to launch this solution with AI then?

I think I was still too early actually to Be honest, starting the company. But, you know, as we looked at this problem and we went like, there has to be a better way. It cannot be the same old ways of doing things. And in my past lives, I had sort of tried to communicate to my leadership that we should be sort of doing something with AI. And it was always seemed like, you're too early for that. And when me and Kosuke sat down and we go like, there's so much data that is coming out from the exhaust pipeline of the software delivery process, and it just seems to be thrown away. Right. Nobody is using it. And now this is four years back. So, like, we could look at this, all this exhaust, and we could, you know, throw this data to machine learning and see what comes out at the other end. Right. And that was sort of the second leg of, you know, our company hypothesis. The first one was like, we can go in and, like, look at tests and optimize tests and that process. That seems to be a problem. And the second is the way we would do that is use machine learning and AI and go solve that.

Love it. So, you know, you mentioned that obviously tests are often the bottlenecks in your software delivery pipeline. So how does Launchable specifically address that issue? How does it, like, enhance the delivery speed?

Yeah. So core changes that are coming in and we can predict what tests are likely to fail based on these code changes. We look at a bunch of other signals as well, about 40 or 50 other ones. And what that lets us do is instead of, you know, waiting, say an hour for your test suite to go finish, and then, you know, minute 50, some tests fail. We can actually straight up go to the minute 50 test, bring that up, and execute that first. And so you get the feedback right away that your build is going to fail. So rather than kind of wait through for the signal to come back to devs, you can almost get that, like, in the first few minutes. Right. I say, like, about 80% of your tests are pointless. You just don't know which ones. Right. And we can find which ones are the most important ones, run them up front, and that feedback goes to your dev, and that can drastically reduce the turnaround loop for dev teams in qa.

So rather than waiting to production, it's early and you shift it more left when they check in code, it runs. The more important tests that, you know, could fail or tend to fail more often.

Yes, the tests that tend to fail based on the changes that are coming in, the test that could fail or tend to fail more often would be in that mix as well. That's where we started off. And it's a risk based approach to testing. The one way we thought about this is like, you know, SREs are fairly risk, you know, they look at the risks in the system. So we are bringing that approach here. So you kind of go in and say like, well, my risk to reward threshold is x. I want 90% confidence in my test results and we'll give you a subset of tests that will match that. I want 10% confidence and we'll give you the test that match time. That's one part of the solution that we built and that's been running in production for a few years.

All right, so I love this because usually, you know, I used to work at Healthcare, we'd get like a hot fix we'd have to release right away. And so they check in code and we don't know what test do we run. I don't run all of them. And some would fail and they had nothing to do with the check in. But we had no confidence in knowing what caused that issue. So does this help you with that as well then like, you know, like if I check in code A, that test Z and F will cover that check in. How do you know that if it does that?

Yeah, absolutely. That's really the basic premise of that, that piece of software. So the way we do that is over a period of time we look at the code changes that are coming in and we look at the tests that are being run against them and we build a machine learning model out of that. And the model now after a point can look at changes coming in and say, okay, if you did this, it's likely that test 1, 5, 15 are going to fail. And the model then predicts that and gives you 1, 5, 15 and you execute 1, 5, 15 and you get that signal back. To make it a little more sort of simpler and intuitive. Like let's say Harpreet was an engineer, he no longer is. It's been a while since he's coded and suddenly one fine day he pushes changes in and the model says like, well, he just broke the build. And then Harpreet comes back again and pushes changes in. Fairly obvious that it's likely that Harpreet's going to break the build again. So you're kind of looking at those kinds of signals that are coming out from your test runs and your code and building this model and predicting that.

That's awesome because once again I worked with like eight to 10 sprint teams. There was always like two of them that were always breaking the build. So it sounds like this would know. Okay, Team Sopranos just checked in code. Better run these tests that usually have to do with their check ins.

Yes. So it kind of goes down to the dev dev levels and to the level of a developer and you can kind of come in and predict what tests need to run. It does really well with sort of mapping regression tests and integration tests, which is where a lot of complexity comes in. And like lots of people are checking in on a non trivial project. Hundreds of people are checking things in running integration tests and it's always a problem trying to figure this out. Right. So your main line is always red and we can actually come in and say like, well you can intelligently shift these tests left. That's one way that we get used. So let's just say let's contrast this. Like people put together smoke tests, right. And somebody has cherry picked a bunch of tests that they think are important and they are running regardless of what's happening. So let's say the authentication infrastructure is being run and there's one test there, but you're still running your database connection tests and so on, so forth. Now what this approach lets you do is sort of build a dynamic smoke test. So what happens is, oh, you made a bunch of changes to the authentication infrastructure and looks like the more authentication related tests need to be run versus database kind of tests need to be done. So we can actually, the model can actually automatically pick those and help you execute those. And so you can now take these and like shift these tests much earlier in the lifecycle if you wanted to. Right. And so it's intelligently shifting these tests left and you're catching issues much, much earlier in the cycle.

Right, nice. So I know when we talk about tests a lot of times people think of functional automation, but I think you mentioned integration testing. So it's all types of testing, unit integration. Could it be performance security, does it matter? Is it what do you consider test? I guess that it can handle.

It doesn't matter actually. Really the reason for actually choosing machine learning was that we could be applicable to any type of tests and test suites wherever they are. We don't do performance tests. That's the only exception what we look for. So this is like a, since this is a machine learning model is essentially classifying whether the test would fail or pass. Fail or pass. Performance tends have, you know, tests have multiple outputs, so we don't do that. But otherwise it doesn't quite matter what tests you're using.

Awesome. So how hard is this to integrate into your CICD pipelines then? Is it, is it like an agent that's like a sniffer that knows where you checked in and how does it know for production? Is it learning from production to know what is risk based? Like, how does that all work?

It's actually very simple to integrate. So the way we integrate it is it ends up in your CI tool and there is like a Python script, Python program that we've written that gets integrated into your CI script. So once you push some changes in, you point us to your source code repository and we know what changes have been done there. And then once you run the test, you send us the test results so we know which tests were run and how did they perform. And then the model builds out. So these are like about two lines of code changes in your CI script. Takes about like 30 minutes to run. I often joke that it takes me two weeks to find a person in a customer's account to kind of come up and do this, but it takes about 30 to 40 minutes to do that. And then when you're about to execute the test, you just call launchable and say, hey, launchable, I have a 10 minute threshold, can you find me? Kind of going back to your earlier problem. Okay, a hot fix is coming in, I have 10 minutes to check. So launchable, give me all the tests that you think are likely to fail in this 10 minute window, like find and package and squeeze as much juice as you can and give me the highest probability of failing, you know, failing test. And we return that to you, three lines or four lines of changes in your CI script. And then the other thing that we did was we actually don't look at the code, we look at the metadata from the code. So your code remains with you, your test remains with you. We are looking at code and test metadata and when the results come back from us, which is, hey, watchable, give me XYZ test. Those tests execute in your environment. So really nothing changes on your end. Right? The test. We are not asking you to migrate your test to us. They continue to run wherever they are. All you see is like, suddenly you are getting failures way, way faster than you did in the past.

Awesome. So, I mean, when I think of machine learning, I'm very a novice. I assume it takes some time to really get better and tweak itself to learn over time. Once someone implements this, how long until you get results that you could be fairly confident in?

So the model builds right away and it starts predicting right away. But we tend to ask customers to wait up two weeks before we tell them to turn this on. And from there on it's fairly good to go. And the beauty of the approach as well is the model always keeps learning. So any changes that are coming in today you are in, say the authentication piece of the code base, tomorrow you might be somewhere else. It kind of follows you pretty well and then it can predict very well. So that's really the machine learning piece of that takes about two weeks.

All right, so this might be a dumb question. What happens if you don't have a test for a risk area? Does it also bring a bubble up insights? Hey, you're running a bunch of tests but the check in has no coverage. Does that ever happen?

No, that's actually, it's not a dumb question. It's an often asked question. We don't cater to that because we were like, hey, you know, there are enough test coverage tools out there that we don't need to reinvent this problem. And people have test coverage tooling already set up. And so we don't cater to that. So where we work with is like, if you have a reasonable size code base, reasonable size test suite, these tests are automated. We come in and kind of help you there.

All right, so obviously you're saving time because you're shifting things early in the process. And rather than running a ton of tests, you're targeting just the tests that relate to the code check in. Can you also assume then that obviously if people are using containers and ephemeral environments, that their cost is going to go down because they're not running all these tests as well? Is that another plus to this type of solution?

Yeah, that's a great insight. Actually, that's one common use case that people have, which is, well, I was running all, you know, our test execution budget is testing. If you kind of latch on to the 80% of your tests are pointless, intentionally inflammatory comment. But if you latch onto that, you're like, I don't need to run this all the time, so why don't I just, you know, not do that, right? So that's one way where people reduce costs. Another, you know, counterintuitive insight that happens is, is they go like, okay, now I'm running like 5% of my tests, right? I don't need to run my entire test suite. But now I can run these 5% tests all the time, right? Instead of waiting for my nightly test to run, I can run like a 5 minute run or a 10 minute run every hour. And I'll find and catch issues every hour instead of the nightly build. Right. And I might actually be executing more tests because of this approach in the same costs that I was earlier spending to run that one, you know, massive test suite.

All right, that's a great point. Because we used to run our test at night. The whole test suite would come in, we'd get failures. We assign it to people this automatically. Oh, does it. I guess it also. Does it alert the right people then that it does. It knows, hey, not only did the test fail, but you're the one that's responsible for this.

Yes. You're actually latching onto something we built called Slack Notifications, where we are sort of using this data and notifying individuals right away saying like, hey, you broke the build so you don't have to go and hold your CI server to see did I break this build and did I break this build? So that's one piece that we've done. The other, you didn't ask this, but like the other piece that we have been spending a lot of time lately building is this notion of triaging issues. You ran the builds, you came in the next morning and the first thing you see is a data dump of like 500 test suites. Right. So what we discovered with our customers is they were using the test to get this feedback much earlier. But really what they are trying to do is get a handle on like their triaging activities. So as we saw that, one of the things that we. So what we're doing here is I call this intelligent test failure diagnostics is all right, I have 500 failures. Are they all one issue? Somebody disconnected the database. Are there 500 underlying issues? Can we collect them? And this is where we are throwing this to LLMs and we can come back and classify these failures for you and say like, all right, Instead of these 50 failures are this one underlying issue and we've up leveled that issue component. And so you can go look at that particular issue and if you fix that, it's going to fix the 50 test failures. So a lot of the cognitive load that happens when you're debugging these kinds of failures is like you start looking at the log of test one and you look at the log of test two and then by fifth you've kind of given up and you go like, all right, these seem to be related, let me go fix those. Right. So we are sort of bringing in AI to augment that process.

I mean, that's a big game.

So that's another piece.

Yeah, that's a huge piece, I think, because when you come in and you have a thousand tests, 500 failed. Rather than go through each of them knowing, hey, if I fix this one thing, it'll fix 499 of these is huge. Because, you know, someone's checking in code and their tests are kicking off and you need to do it really quickly. So that really helps get to that.

Yeah, absolutely. Yes. I think it's going to be a game changer in terms of helping QA and engineers get to the bottom of issues. And as part of that story, we are also sort of helping bubble these issues build a dashboard. And then our customers are asking ways for them to assign to particular team members or teams. And this is where your question started off with. So we're building stuff that says, all right, Harpreet and Joe on Team X have caused this problem, so let me assign it to them and they can go figure it out and fix it from there.

Very nice. So this might be another weird question is when you started, you said you used to try to pitch AI to a company. You're working on it. It's too early. How hard is it now then to you have a solution that's AI based, it's AI driven, How hard is it to cut through the noise? Because everyone's saying their AI, their machine learning, Is that still an issue? Like, to educate people, hey, this is real and this can help. Is that something you struggle?

Suddenly there's. I'll give you a very interesting story. How the dynamics have changed. Right. The first two years I pitched the product, people would look at me and go, like, AI, this is not real. Right. I actually know somebody who came on a call from one Bay Area company and he said, like, well, I don't believe in AI and I think I can build something like this over the weekend. All right, good for you. Right? And then suddenly, like, people are now coming in and they have a mandate from their office. We need to do something with AI. Go figure. And now they're going in. Every tool is AI. And so it's quite a different change. And now what's working is people are leaning in. You know, you can't be that guy saying that I don't believe in AI. So that has changed. But then it sort of becomes very interesting talking to people. And as they're trying to cut through the noise of AI, one of the things that I have found is, like, people have a very hard time understanding sort of things statistically. Right. And so when you talk to them about AI and You talk about like, okay, this is what the model is doing and you show them the statistics and how you evaluate models and these curves, they don't understand that. Right. They want to know like, oh, but in this particular instance, what did your AI do? It's like, no, no, that's not how AI works. Right. It's not like I can go in and explain every individual decision. So there is to your question, people are trying to get to understand like, okay, what's AI about you? Right. And that's. And I think they're doing that across various, various tools.

Nice. So you've mentioned a bunch of different features, what it's doing, how you're adding onto it. It's been four years, so I mean, are there anything else that it helps with that we may have missed or that you're working on, like the future of AI and DevOps? Any new features you could see coming into play soon on your roadmap?

So one of the things that we have worked on and it's available in our product is this notion of flagging unhealthy tests and test suite to people. So most test suites, when they get big enough they have bunch of flaky tests, right. That's causing friction. So we can, it's not AI, just to be clear, it's like statistics. We can come back and say like these tests are the most flaky and this is how much time it's costing you. These tests are the ones that most fail. So we are trying to give engineers the data to go back to their product management and say like, you know, we need to fix these tests and structure of building the net new features that you're asking. That's one component that we are working on, we have worked on. But like where we are headed towards is that triaging process. Right. So the example that I gave you on the triaging is really where we are taking the company. So the thing that I led with, which is helping you auto classify these failures, that's something that we land in, you know, somewhere in the next quarter or so. But as we've sort of brought this to our customers, they're coming back and saying like, okay, you know what, I actually would love it if you sort of can tell me whether these tests are, you know, whether these failures are caused by code or caused by a human or caused by tests. And so like there's a lot that can be done there to kind of make it intelligent. So one thing that we today do when we launch it is like an anecdote about this is like when a QA person comes into office tomorrow morning and the nightly tests are done, it's like he's coming to a crime scene. There are like 500 dead bodies there. Right. And we are trying to provide as much forensics data to this guy or girl to help them sort of figure out who's the killer, like, what's the problem? So there's a lot of forensics data that we can keep adding to it. So a straight up example of this is we can sort of on your test failures. When we create these issues, we can tell you, hey, you know, these tests are flaky. They are flaky. I don't really need to look at it. Maybe I just need to rerun it. But hey, this test is never failed. If that never failed, then I should be really going deep into it and so on, so forth. So that's how we are helping that person kind of get a handle on failures.

Love it. And back in the day when I had to do this, I used to have to tag things as a team meme and quarantine this because I noticed it runs three times. But you're coming at from a statistics perspective, it sounds like where even that part is not manual anymore. Am I understanding that correctly? Where I don't have to then start tagging everything?

Yeah, we can just auto classify and tell you. Right? So to make it really clear, it's like, okay, here's the top 10 flaky tests, and here's how much time each flaky test has cost you in your organization. And like, here's, you know, a prioritized rating where you can go and look at it and then you can click on, say, the top, you know, flaky test and kind of go into the forensics of it and go like, okay, this showed up in test session 1 1520. And it's been kind of showing up in the last five tests. So it's really. Something's gone wrong here. The flakiness has kind of increased and so on. We can graph that out for you and just make it really intelligent. Make the AI do the. I don't know what's the best way to put it? Just make it intelligent so that you can just focus on the problem versus digging through the under the nooks and crannies to see what's going on.

All right, so a lot of people listening to this probably. This sounds great. Maybe a little too good to be true. You have been around for a while now. So do you have any case studies or anything that like of actual statistics of like you've saved X amount when you've implemented your solution or anything like that.

Yeah. So you can go to launchableinc.com customers and there are a bunch of case studies out there. And so you know, we have customers like BMW using us for a while now and using that and as part of their delivery process and there's a bunch of others out there. So yeah.

Awesome. Okay, before we go, is there one piece of actual advice you can give to someone to help them with their AI DevOps efforts and what's the best way to find or contact you?

All right, so let's start with the best way to contact us is launchable inc.com come in. There's plenty of write ups out there that tells you how AI can be used. We have a very targeted approach. I remember I was watching one of your earlier podcasts in which the person was giving a very nice advice, which is take one problem that's causing a lot of friction, you know, and just pick that and go and solve that rather than kind of boiling the ocean. So my biased answer is most organizations, you know, tests are a huge problem, executing them. And so you might be, if that's an area, then you have a targeted approach to kind of going and looking at that. Right. So that would be my advice.

Remember, latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and how to fix it with Bugsnag from SmartBeer. See it for yourself. Go to Bugsnag.com and try for free. No credit card required. Check it out, let me know what you think.

And for links of everything of value we covered in this DevOps Toolchain show, head on over to testguild.com P149 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed, creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'D love to hear from you. Head on over to test guild.info and let's make it happen.