AI Testing Is Breaking Your Pipeline. Fix Quality Before It’s Too Late

Hey, what's up everyone? Joe Colantonio here and welcome to another episode of the TestGuild Test Talks Automation Podcast. Let me ask you this: if AI is making developers faster than ever, why are so many teams dealing with more failures, more rollbacks, and more late-night firefighting? Because while AI coding assistants are exploding productivity, there's a growing gap happening downstream in testing in pipelines, and in production stability. And the data shows we may actually be trading speed for quality without realizing it. To help break this all down, we have Eric Minick with us today. Eric is a DevOps veteran and co-author of O'Reilly Books' "AI-Native Software Delivery." He's been digging deep into real-world data on how teams are actually performing after adopting AI coding tools, and what he found might surprise you. So we're gonna unpack what he's calling the velocity paradox, why more code doesn't mean more value, and more importantly, what you could do right now to fix your pipelines before things stop breaking at scale. You don't wanna miss it. Check it out. Hey Eric, welcome to the Guild.

Joe, great to be here.

Awesome to have you. So I came across a report you created for harness.io. I covered it on my new show. I'm like, oh my gosh, there's so much great data here. I'd like to have you on the show to dive in. I guess before we get into Eric, maybe a little bit more about, you know, where you see the, where we are right now, especially with we're shipping faster and are we just breaking things as we go faster with AI or is AI helping making us ship faster, better? I don't know if that makes sense.

Yeah.

Yes.

A little bit of each. A little bit of each. So we're certainly shipping faster. Like, that's coming through, but I think we are breaking things more, or at least more people are breaking things than they're doing better. We did see some evidence that there are, you know, some folks who are having some more success, but overall, it looks like we're going faster and lighting things on fire as we do it.

For sure. I think in your report you called it the velocity paradox.

Absolutely. So, you know, you see this thing where we know that coding assistants are a wonderful productivity tool for developers, right? I'm able to crank out so much more code, right? That's really simple to see. But when we looked at the data ourselves, when teams like DORA were looking at the data, let's say a year ago, what they saw was no more software was actually being delivered, right? And so that was really weird, right? Where you had a lot more code, or like developers like, I'm a lot more productive. And then you ask teams, "How much software are you delivering?" And the answer was, "Not anymore." And that was odd. So that was the first hint that there was some sort of a paradox going on. And then what we've seen over the last year is teams have said, "Oh, I know. I know how to ship more. I can get this out the door. We can just sacrifice quality." And I don't know if that's a conscious decision. I'm being a little facetious here. But in the data, that's what we're seeing. The speed is caught up, but the quality signals are all on fire.

So why is that? Because if you can automate the coding, why can't you automate the quality aspects of it? Um, are people, like you said, just not aware? Or like, are they trying to catch up? Are they just going for velocity and like not even coming across like, hey, maybe I should have better security baked in here as an automated test before it goes out the door or something?

So you can, and, you know, we asked people, where are you using AI? And something like 95% of them said, we're using it in coding at least once a week. And half of them were saying, like, daily. Right. So like that's darn near ubiquitous, right? But as we start going down the pipeline, we're saying, well, are you using it in testing? Right. Then it was more like 60, 70%. Right. And even there, I imagine that's like, well, did my vendor say there's AI in my tool? Yeah, they did. Yeah. Okay. So I imagine that some of that's even just some checkboxy stuff, right? I think the maturity of a lot of the downstream tools in AI isn't where we have it with coding. And then the other side of it is just like the maturity of the pipelines themselves, right? So, we're seeing, you know, people are putting things in production more often, but their ability to respond to it when it goes wrong isn't great. So, you have teams that are tired, right? We've got all sorts of signals that people are working more nights and weekends, that meantime to recovery is going up. That the likelihood that you put something in production, you have to roll it back is going up. So there's a lot of stuff that just doesn't look healthy, right? I think it was something like that for the cohort that was doing that daily use of AI, 22% of deployments to production either caused an incident, needed to be rolled back or required a hotfix. Right. So they were broken.

So where's the hype then? Because you've, because it sounds like AI is going to make things easier where we work less, but that's why it was one of the numbers that popped out to me. People still work in weekends. So like, where's the, the, uh, where's the gap? Like what's going on? Like you said, I know maybe the downstream systems in the pipeline aren't as strong with AI. But is it just making like, it's actually not gonna replace us, it's gonna make us create more code and therefore we'll have to test more of it and be busier than normal rather than it's gonna create more code for us to do it all for us and then we'll work less.

Yeah, I, I, at least that second one. So if, if you imagine, you know, we go back a year, right? And we say, well, for some amount of code, right? I have to do, some testing, and I've got some chunk of that automated and some of it's not automated. And I've got some amount of build and deployment and all of that stuff, right, to do my delivery. And a bunch of that's automated, but some of it's not, right? I go into meetings and checking stuff and all of that. Well, if I've made the coding part much, much faster, but I haven't improved all the other things, then I'm doing all of the other things more, and all of the manual stuff becomes a much bigger portion of my day.

Hmm.

Because I've gone from, you know, maybe releasing 3 times a week to 5 times a week or 10 times a week. And so if I, if I'm releasing twice or 3 times as often, now I have twice as much manual toil. And even if I've started to automate some of it and I've eliminated 20% of my manual toil, but I've got 3 times the releases, I still have more. And so we've seen this increase in the amount of toil that engineers are reporting. A year ago, we ran a similar survey. It wasn't the exact same question, so I can't say this is scientifically perfect, right? But the toil response was about like 33%, right? Which is bad. And this time we got it back, and I think it was in our AI cohort. I had my notes here, 38% for the most frequent AI users. And for the less intense users, it was down at 32%. So, we're seeing the more aggressively we use AI, the more manual work we have, which is very counterintuitive. But I think it is this, like, I still had all of this stuff, but now I have to do it more often.

Yeah. It's interesting because it's, that's not what I'm hearing online though. Companies like Block say we can get rid of 40% of our organization because AI replaces it all. But is it because people aren't measuring manual toil so they maybe mistake the output for progress? Where's the disconnect between these organizations then?

So I think, I mean, there are probably some jobs that are being automated out of existence. And, you know, I think you have some organizations that are probably better at this than others. Absolutely. And then I think there's probably also, you know, if you're doing a mass layoff, you have to tell the street these days you're doing it because AI. Right.

That's true. Yeah. Right, right.

Yeah. Yeah. That's just the PR play. Like you, you do that and your stock goes up. If you do it and you say, well, because business is bad, your stock goes down. Goes down. So if I'm a little jaded here, I think that's probably some of what's going on out there.

Earlier you were talking about the deployment and that, you know, things are breaking after you create the code. And I think you had a stat there, some of like 69% of heavy AI users say AI-generated cause deployment issues. And I see that myself when I'm vibe coding, just simple TypeScript issues. Is it because people are using AI incorrectly or is that just the nature of where we're at right now? And maybe it'll get better and then these things are just going to go away over time, so we don't need to really worry about it.

I'm hopeful the answer is they're using it incorrectly. And we saw some data that supports that. And that wasn't covered very deeply in the report we published. Just, we didn't have room for all the subtleties in the data. But I think one thing that we shared is that across the quality metrics, it was really consistent, right? It was about 50% of people said, since I started using AI, performance got worse, or performance problems were more frequent, or security problems were more frequent, or quality problems were more frequent. Quality issues were more frequent, right? The interesting thing that I noticed is about 30% of people said it was less frequent. And when we looked at the cohort who said, oh, I'm not using AI that much, the good news is they said things haven't changed very much for me. So it wasn't random that people were picking high or low, right? The people who said I'm not using it very much said things aren't changing. The people who are using things a lot said things are changing. More said they're getting worse than better, but there was a big split. There is a sizable minority who are saying, "As I use this, I'm having fewer security problems. I'm having fewer performance problems. I'm having better code quality." And that suggests to me that the organizations that have the best practices around AI coding are able to have some better outcomes. And I would guess that that's where we see a little bit more rigor. We're seeing, well, before I just start saying, give me a bunch of code, I'm saying, okay, let's get a spec together. Let's Let's generate the test first. Let's do TDD, right? And I'm saying let's, right? Because this is a pair programming sort of situation. So we're going back to XP practices in a sense, but let's find these kind of best engineering practices and the rigor and the strength of good engineering. And I think where we see good engineering done with AI people are having better outcomes. That said, broadly, things are on fire and people are under more pressure.

Um, and I, that, that would make it harder for implementing best practices cuz their organizations are probably pressuring them to create more code with AI. So it's kind of hard to push back and say, maybe we should put these practices in, may slow us down initially, but in the long run it'll be beneficial.

Yeah, we've seen this before though, right?

Oh yeah, yeah.

When we put in place the new thing and it helps us go faster, but the quality goes down, the bad stories start coming out, and then the things get rolled back, and then we lose a year or two in rolling out the new practice, and we get the bad stories. And like, we've seen this before. We know how the story goes. We know the answer. The answer is quality first. And quality, quality is what gets you speed in the medium and long term.

So who's responsible for quality nowadays in the age of AI? Is everyone becoming a software engineer and it's like everything's morphing into one role that has to do everything? But do you still see a role for like a, a quality group or a tester? As part of a software development lifecycle?

I think that's interesting because I think we're seeing the same questions around, you know, design pros, DevOps folks, etc. So I think there's always a role for that skilled professional, right? I don't think they can be off in their own little corner doing their own thing. I think they have to be more consultative because I think the tests, because the design is gonna change so quickly and because the code is gonna change so quickly, you don't know what you're gonna be testing very often. In a lot of these teams. And so, to be able to go sit with the engineer and say, okay, let's make sure that we've got all of these things in place. Are you doing TDD with your bot? Have we thought through these negative cases? Have we thought through these corner cases? Are we in a good place, or are we going to light things on fire? I think having somebody with that right mindset is outrageously powerful. And a great complement to the team. And that person may be writing some tests on their own sometimes, might be consulting with the engineers on the team sometimes, but probably not off on their own too much. So QA center of excellence, I think, is dicier. Professional testers working on teams, Yes. And then I think, frankly, if coding isn't that hard, but we're seeing things get lit on fire, the key skill might be testing and being able to think through, how do I know whether this is shippable or not? Um, so this, this might be the revenge of QA, right? Because the, the best, the best developer right now may be the QA engineer. Um, I, I think this, this might be the time that's the revenge of QA.

Yeah, and your numbers prove it in the survey. I think you asked 70%— 70% of teams said that their pipelines are plagued by flaky tests and failures So AI is making it worse, but maybe because developers don't know how to test, so therefore, uh, that's the issue. Why can't I write a, an AI agent that says, not only are you a coding expert, but you're a testing expert as well. So anytime you create this code before you check it in, make sure you also test it. Like what's, is that unrealistic or is, why is that not being done? Or is it being done and it's just a different testing issue where they're not doing exploratory testing or, or like it's not smart enough to know about surface and security things that are exposed that it can't figure out.

Yeah. So I think that the best teams are doing that, at least for their kind of unit level testing. Right. I don't think they all are, but I think the, the best teams are saying, okay, let's write a spec first. Great. All right. Claude, or whoever my coding assistant of choice is. Now let's write the tests for this spec. Now let's start, now go write the code for it, I'll be back in an hour. Now does the code match this? And then we were working that way. And so we do have good unit tests for our code coming out of it, and we know that our code passes those tests. At the same time, we can be working in an environment where any of the dependencies that are being pulled in by our AI are going through a dependency firewall, right? Those things exist. We can put those things in our environments. And then we've got security tools in our local environments. That's a really normal thing to have. And then after we commit, we're running through a pipeline that's running more rigorous tests that give us resilience or performance sort of checks that are giving us another level of security scanning because some developer is not going to do all the things that we were supposed to do earlier. And, and, and, and if our development practices are strong and our pipelines are strong, and once we go into production, our observability is strong, and if something goes wrong, we can roll back. They're going to be in a really safe place. And if you're safe, you can fly. So, I think that's the play, right? But that requires a combination of serious engineering for the developers up front, as well as a platform team, probably, to make sure that you got rigorous pipelines set up.

Yeah, and I think in the report itself you mentioned, you know, the need for more observability and feature flags. It's no longer nice to have. Been talking about feature flags and observability for a while now. So like you said, maybe because AI is only being used or mostly being used in the coding aspect, they're missing out on applying it to these other areas that probably need in place, especially as it goes to production, be able to figure out what happened and what broke, right?

Yeah, and I mean, I think, you know, observability, there's been some level of AI for a long time, right? We've talked AIOps, right? I think it's a 10-year-old term at this point. But making sure that that's actually connected into the deployment pipelines, that's not normal at this point, to make sure that when I deploy to production, we're doing— that's connected to the observability tool. So, if there starts being smoke, there's an automatic rollback. I think in most organizations, if there's smoke, an alert is fired, and someone goes and looks at that alert, then tries to figure out what changed. It's like, oh, maybe there was a deployment, and then I go contact the DevOps team or something, and maybe they roll it back. That's disconnected and not that great, right? These tools should know about each other and be automated. Like, we see, I mean, certainly within our own tools, we'll hook those together so that it's an automatic rollback. You know, our deployment knows about the observability tools. We're starting to see this within Argo CD, has some concepts for this to have hooks into the observability tools. but most teams are not doing that today from what I'm seeing. So that's, there's some AI in there, but some of this is just, you've got two tools that are adjacent to each other in the logical pipeline and most people don't have them talking to each other. Typically they're owned by different people. And the person who's setting up the deployment tool doesn't know the metrics in the observability tool, if they have access to it, to look for. And you would need the SRE person to sit with the DevOps person for an hour to set it up. And somehow that's just too big of a lift. So this should be getting fixed. We're starting to do things like have our AI reach out to the AI of the observability tool and negotiate this configuration, which I feel like is how this should be at this point, right? The AI DevOps engineer and the AI SRE sit down because the actual DevOps engineer and the actual SRE won't sit down together. So I think that sort of thing will become more normal and hopefully we'll start resolving this over the next few years. But we're behind on this as an industry. Right? Like I'm saying, this is what's starting to happen.

Right.

Meanwhile, everyone is doing AI coding.

Yes. Yes.

Like that's the imbalance here.

Interesting. So if someone's looking for opportunities, they're afraid of the job. Isn't this a great place to say, okay, they're focusing on coding, they're missing out on really the benefit here now. Do you see people should be a little more open to expanding their roles or being more leaders in organizations? Say, yes, all right, we generate code, but we need to focus in on the pipeline here. It's— we're kind of not paying attention to it.

I, I think so, absolutely. I, I think there's this huge opportunity for individuals as well as leaders to take a serious look and improve the pipeline. It needs to be more seamless. We need to drive quality through it. It's not a small task. At the same time, I'm seeing, you know, when we see we're having non-compliance issues and we're having security issues and we're having all these issues, Right. Non-compliance plus security. Someone's going to get sued. Right? Right. When you have a major data breach and you were also non-compliant with your own policies and with, you know, whatever other compliance you're supposed to be under for your industry, Like, that's going to get very expensive financially and reputationally very quickly. And there's a bit of a YOLO attitude going on right now. It just scares the daylights out of me.

So is it because people are overconfident? Like, I think once again, going back to the, which we'll have a link for down below. For the study you had, the report, 95% of teams said that their DevOps was good. But everything we talked about, rise of incidents, burnout, security issues, non-compliance, it's not good. And if you're an enterprise, like you said, especially regulated insurance, anything like that, healthcare, you're in trouble. Like, what's going on? Confidence versus reality.

That made me so angry. When I find out, I'm just like, Yeah, so the findings, 95% of people said their DevOps was good or very good, and it was about half and half between the two. I'd have to check. Meanwhile, they're all saying we're working nights and weekends and everything's on fire and all of that, but they were also saying that we're going to production multiple times a day. That was about a third. And we're going, or once a day or multiple times a day was about a third, a little more. Multiple times a week was a little more than another third. So I think there is like, if I'm going fast, I'm doing good. That's the only thing I can think. And then I think there is some pride of ownership, right? And many, I'm going to imagine, and this is where I'm getting out of data and into my imagination, that this same cohort, right, they're going fast, right? They've adopted AI really aggressively. They've said we've got really good training and policies around AI, right? All of this stuff. They've done a lot of really good tech stuff, right? It might be at the edge of breaking right now, or maybe just past the edge of breaking. But they've set up DevOps pipelines where I can do a commit and that thing can get to production in the same day. Is it perfect? No. But do I feel pretty proud of that? Yeah, I do. I think that's what's going on with the good and very good. I think the trick is, in a world where I'm generating so much code, like, that is no longer good enough, right? And that, that's what we're seeing, right? We're, oh, now I'm working nights and weekends, and 1 in 5 of these are crashing, and, and all of that. Um, what was good last year is just no longer good. So, there's, I think, some justified pride in what was created, but it's turned into complacency.

All right. So, we've gone almost 30 minutes. I'm just curious to know, Ed and Asher, what does Harness do? Can Harness help people in this situation? What do you all do? Why did you create this report, I guess, is the main question.

Yeah. I mean, it's a marketing report, right? So, we're a DevOps platform. So, we do AI stuff for everything after code. So we're not competing with any of the coding assistants, right? But we care deeply about the things that happen afterwards. And yeah, our thesis really was that, you know, until people straighten things out downstream, they're not going to be able to deliver quickly. We've been very surprised over the last year. Right. As people have chosen to sacrifice quality and deliver quickly. So that's, that's been an interesting thing to change. But yeah, so, you know, we got our start really in CI/CD and have built out around that to be a kind of full-fledged DevOps platform with everything from that to, you know, artifact registries, infrastructure as code. Feature flags, all the way out to cloud cost management, just to try to keep everything nice and efficient. So yeah, that's where we play.

Love it, love it. All right, Eric, I'm going to ask you some rapid-fire questions. Okay. All right, so do you think AI coding is a net positive or a net risk right now?

Yes, but a net positive, net positive.

More tests or better tests?

Better tests.

Okay. Feature flags or test coverage, which matters more now? And you can add a little context around this too.

I'm gonna go feature flags. I like testing in production.

Thanks. So AI-generated tests, then, are they useful or dangerous? For what you've seen? Yes.

It's all code. And so it's all about code quality. And we need to have our eyes on things.

And one metric you think every team should use in an AI-native software development lifecycle?

Change failure rate.

Why?

Because we are lighting production on fire way too much, and we need to stop.

All right, Eric, before we go, is there one piece of actual advice you can give to someone to help them with their AI software development lifecycle initiatives, DevOps efforts, and what's the best way to find or contact you?

Sure. I, I would say get serious about, uh, your engineering top to bottom. So test-driven development up front, solid pipelines all the way through. Nothing can be shaky, right? And if everything's solid, you can go fast, safely. And it's all the stuff that we've known we should be doing for the last 10, 15 years. We got to do it. You can find me probably these days LinkedIn as much as anything. Marek Minik. I'm also on, you know, BlueSky and X and all of those. But I don't know, that doesn't seem to be where much of the tech chatter is these days.

All right, that's for sure. And we find links to the study and all their links down below, so check it out and let us know your thoughts. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com/a585. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the TestGuild Automation Podcast. I'm Joe, and my mission is to help you succeed with creating end-to-end full-stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a troop of Provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. Oh, the Test Guild Automation Testing Podcast. With Lutz and Laios, the The bards began their song, a tune of knowledge, a melody of code. Through the air it spread like wildfire, through the land, guiding testers, showing the secrets to behold.