The Rise of Agentic AI: Transforming DevOps and Testing with Bridges Smith

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, what is the future of AI and DevOps? That's what we'll be talking all about in this episode with Bridges Smith. If you don't know, Bridges is the VP of Product Management for AI at SmartBear and so really knows his stuff. He knows how to accelerate AI innovation. He's done it across Smart Bear's portfolio. So if you went to Smart Bear, you'd see all the awesomeness they've been doing with AI with all their tooling. Really excited to see that. He also has over 25 years of leadership experience in product development, machine learning and AI. If you want to learn about the real deal about AI and DevOps and tooling and testing, you don't want to miss this episode. Check it out. Hey, before we get into it, I'm going to give a quick shout out to this week's sponsor, SmartBeer. SmartBeer is empowering teams to build, test and deliver great software faster. Explore how Smart beer is bringing AI to the forefront of quality engineering, helping organizations to accelerate test creation, optimize automation, and drive smarter decision making across the sdlc. Visit the AI hub to see how AI is shaping the future of software testing using the special link down below. Hey, Bridges, welcome to the Guild.

Thanks, Joe. Really appreciate you having me.

Awesome. So you have a lot of experience with AI, so I'm just curious to know, is this where you saw AI going when you started 25 years or so ago?

I have to be honest and say no, absolutely not. Definitely. A lot of times in my career where I thought things were going one way and they went another. So it's been an interesting ride for me. And I think one of the things that is really most important is to start to see the momentum of where things are shifting. So from early on when I was working with speech recognition, early systems, and we were excited just to get simple menus going and then natural language and virtual assistants. I think similarly, we're seeing some real shifts in AI capabilities as things start moving from simple generation into agentic AI.

All right, so let's talk a little bit about that shift. What do you mean by moving from, like, chatgpt to agenc AI? How is that different? How do you see that as a as almost an evolution of where we're going.

Yeah, absolutely. So the simplistic version of generative AI was really about just, you know, as it's predicting individual words. Phrases to generative AI got to be very good quickly at coming up with text documents, explaining things, using language as a way to explain. And that is really just predictive. There's not really a lot of intelligence in there per se. It doesn't know what it's saying. That's why we really saw a lot of hallucination early on. But some of the shift that's going on now as we move to more agentic AI is to really enable decision making and to have some of these predictive models get better and better at really fully automating by being able to take a broader view of not just, you know, what are the next few words, but to be able to go through criteria and evaluation and to start to do more really human like things.

All right, but it's still based on prediction. Correct. It's not really thinking, it's using an algorithm to know or guess over training, I don't know, huge amounts of data to almost guess what the next thing should be. Am I wrong? Does agentic AI take it even further or is it just it's been trained on more data now?

Yeah, you're absolutely right. It is still just prediction. And in fact we're really still learning. One of the things that came out fairly recently from Anthropic is they've done some studies to actually understand a bit more about how that decision making process is happening. Because really up until even just a few months ago, we didn't really know or we threw a lot of data, did a lot of training, but we didn't really know how this gen AI was making decisions. And one of the things that was I found particularly interesting, even the way that that AI is doing math is not like we do math and it's actually doing things like looking at the least significant digit and just kind of figuring out specifically what that should be and then doing a number of estimations. And so again, I certainly plug some of that. It's pretty interesting if you're into that kind of research side of things. But starting to get a view of how that is working is part of what's enabling us to take better advantage of the technology and to use it in more effective ways.

Nice. So I've been reading how we really don't know what it's doing and I guess at some point we're going to get to a point where if they're talking to one another, I mean, how do we keep in control? I guess within the age of agentic AI?

Yeah. So I think one of the really key things is to make sure that you're instrumenting your solutions in such a way that you're able to keep task on the outcomes and the sort of setting some expected behaviors and parameters to make sure that you're taking incremental steps towards your agentic solutions to try out bits and pieces before you start stringing them together and start to look at those decisions with again, take some confidence, sort of threshold way of approaching to see at what point can you start taking the training wheels off and letting make larger and larger decisions again with a real emphasis on observing the outcomes and keeping track.

So it sounds almost like you still need a human in the loop approach and do you see that changing anytime soon?

Yeah, I do. So I'd say that right now still would want to keep a human in the loop some level. But I would imagine that by the end of this year we're going to start to take some of those training wheels off a little bit more. And certainly as we get into early 2026, I think that technology is just evolving so fast. It's one of the things that, you know, we at SmartBear are really trying to do is make sure we're staying on top of that. Because while the technology isn't quite all the way there yet, it's going to be soon. And while I can't necessarily predict how exactly some of these solutions are coming, I can with confidence looking at the trajectory and the acceleration say that it is coming very soon.

All right, so how do you approach this? You've been, I guess it sounds like you're in charge of knowing where to put AI in the right place and all the different products SmartBear has. But the consumers of the product sound like the developers and testers, yet it sounds like this AI is almost replacing developers and testers. Is that a fine line? Am I misunderstanding the end goal of this?

So really, as we look at some of these things, we're trying to look at the elements that are the most mundane, the most repetitive, and those are the areas we're focusing. So looking at, for example, how can a juntic AI help keep documentation up to date throughout the process, whether you're talking about an OAS definition, whether you're talking about a portal to surface, that those are some of the least favorite parts of a developer's job. You know, how can we take some of Those sorts of things. I know as we look at the test process to sort of ensuring that the coverage is where it needs to be and running some of the most basic things, I'd say that we're really trying to free developers up to do the more interesting coding side of things, to allow the UA teams to really focus on the most interesting elements and not the sort of, okay, I've tested this a thousand times before, let me do it again. And it's just the parameters have changed a little bit.

Nice. And I know on SmartBear's site when I go there, you know you're using AI strategically and I think you emphasize it's to help teams deliver software faster, just like you mentioned. So you mentioned a few use cases, but are there any specific examples of how it's leveraging? It's taking care of maybe the busy work and then allowing a developer tester to do what then?

Yeah, so some of the things that we're doing now are, for example, being able to take and generate initial documentation, OAS spec, something we've come up with recently. So that you could say, given that you've got some descriptions and if you don't have an OAS doc yet, to be able to generate that, to be able to make updates and keep them consistent with your organization's rules, the spectral rules you might have in place. We're able to take advantage of AI to do that again pretty squarely in some of the gen AI technology. So, you know, works, works quite well. Similarly, we've got some abilities to do self healing for tests as you look at coming up with tests for mobile applications or web applications. Historically you had to be hard coding particular names of things in the doms. If that changed, then you're kind of back to oh, let me go fix a hundred tests that now no longer run in my automated suite. We're now able to do more with AI there to say, okay, well if you describe the button, it can find where the button is, for example.

Nice. So this sounds great. Are there any blockers? Say if someone's listening, oh, this sounds good. How do I actually integrate it? Is there anything anyone needs to do to be able to be successful when they're integrating AI based tooling into maybe the DevOps toolchain?

Yeah. So I think one of the important things is just to start with a real visibility to your current workflow to make sure that the tools you're adopting are going to not completely break everything that you've done and how do you fit into that? So it's one of the things that we really care a lot about because we are helping throughout the life cycle, whether it's from the design and into the testing and observability. So I'd say that's one of the first things, is just to make sure that you're pretty clear on how that comes in. I think now there are enough kinds of solutions, whether you're working on a desktop application that is 30 years old to be able to make sure that your AI is going to fit into that paradigm or you've got a new mobile app. Either way, you want to make sure that you're looking at a solution that has, that fits into that and follows the sort of testing methodology that you're going to need to go through.

Nice. So obviously AI makes a lot of big claims, especially the tooling. Are there any metrics or measurements someone can use, like before AI and then after AI to actually show, okay, this really did do what the vendor said or what I thought it was going to do. So we are getting this win or something like that?

Absolutely. And I think what's really important before you look at any solution is thinking about the problem you're going to solve. Definitely. If you look out there, there are some times where I'd say there's AI for AI's sake and not necessarily focused. So I would say that if you're looking at something that's going to help drive efficiency, for example, make sure that you're looking at your before and after times. Whether it's, how long did it take me to set up and run a test suite? Before and after, you know, how quickly am I getting through my test cycle? If you're looking at tools to help you with deduplication of tests, for example. So I think that you can absolutely look at before and after in terms of time spent. If you're looking at efficiency, you can look at coverage. You can start to track whether defects found is going down, which really they should be if you're starting to look at improved test coverage, test automation. I think absolutely that the most important thing is to look at those key benchmark metrics and then to really track them before and after. Way nice.

So, you know, I know just my personal experience, AI always does better with context. So training on a bunch of documents that I know what's in it and things like that. How does someone do that? Maybe like in a, in an enterprise like health care or insurance or finance, Fintech or something? Is AI better at like more generic things or can you train it on your specific domain. So not only is it just like to know that trusting not only is it doing tests, but it's doing tests that pertain to, you know, insurance or healthcare or regulations and things like that.

Yeah, it's definitely very important to give as much context as possible because AI always does better if it knows more about your company, your space. One of the reasons why is we look at a number of the sort of generation solutions we put together from SmartBear that we're able to say, you know, make sure that we've got the context of for example the spectral rules that you have for your organizations, that these are what your APIs need to follow and so that those can be taken care of automatically. We provide the ability to pass in references if you have certain API style guidelines or you say I want it to look like this, I want to generate new portal documentation and this is how, you know, my voice is to make sure that you're replicating that. So being able to pass in those sorts of examples can be really important. And even just having the ability to pass in placeholders of business requirements, here's the scope of what we're trying to accomplish. So absolutely being able to take those sorts of things in as inputs are going to only give you better outcomes. And then the last thing I'd say there is again one of the other things that, that everyone gets a bit sensitive about and well for good reasons given some of the history. But you want to make sure that other people aren't taking your data and training and kind of taking advantage of your models. So definitely you want to check the fine print in these solutions to make sure that they're not taking your data. You own your data, don't let somebody else build models and keep them. So that's definitely something to watch out for.

Nice. So now I've been reading a book, it's a AI driven leader or something and they talk about how it's dangerous to outsource your thinking to an AI. So but it sounds like almost you said with the EIH it's going to do better thinking. I mean how do you, I don't know, find a balance I guess between that.

Yeah. And I would also say that there's definitely a path, I don't think we're right to the point where we're going to see complete 100% automation. So this again comes back to the ensuring guardrails, make sure that you're setting expected, you see what you're expecting and that you're coming up with the right sort of checkpoints that things aren't going off the rails. I mean, you can think of the same thing for a calculator, right? Calculators are great at doing math, but if you just blindly punch in those numbers and you hit enter and you don't have some. Wait a second. I was expecting it to be about like this. You could then use that really great tool and have it completely go sideways on you. I think in some ways AI isn't much different. You need to know the sorts of expected outcomes and keep an eye on those and make sure that you're seeing them within the range. So absolutely, I would say that part of what makes us human and I don't see getting replaced at any point in the near future is that ability for us to really apply critical thinking, that sort of problem solving, looking across domains to understand. I mean, look at a lot of innovation comes from and taking something that worked really in an interesting way in healthcare and saying, okay, great, how does that apply in sports? And I can come up with a different model there. That kind of thing is. Again, I wouldn't outsource the strategic thinking to AI at any time soon. But if you're looking at. I want to make sure this documentation is up to date. I want to look across a sea of 10,000 test cases and figure out which are the duplicates, which are no longer relevant, where are the gaps? That's something that. Where I think that we will see some real benefit from agentic AI in the near future.

So I just got a comment on my, one of my YouTube channels, hey, this guy is trying to put us out of business. It was a different tool and someone else saying it's already happening. I guess my question is, and maybe I'm wrong. I don't know why people don't get this point. If AI is going to be smart enough to replace a tester and what testers bring, which I think is thinking in different scenarios, or even DevOps could then replace anyone. It could replace developers, it could replace managers. We'll all be out of a job, replaced by an AI. So is that being too optimistic? Do you see a breaking point where okay, this is the type of job we're going to have, we're going to be, I don't know, I was even thinking like electricians would be a good job, but then I heard about robots being integrated with AI and then it be taken over. So maybe we'll get a little off topic. But any thoughts around that particular scenario?

Yeah, this comes back to Something that you asked me earlier of, you know, how good am I at seeing the future? And I will be the first to say that I'm not great at that. But I think the thing that's interesting is if you look at previous historical disruptions in technology, you look when spreadsheets were first brought in, people said, oh my gosh, accountants are all going to lose their jobs. This is going to be the worst thing ever. You know, the whole industry is going to get ruined. And then what did we see? Well, actually, accountants embrace spreadsheets. We got more accountants than ever. So I don't think that necessarily we're going to see disruption in the way we expect. Similarly, when speech recognition first became a thing, there was real concerns about, oh my gosh, or, you know, anybody who works in a call center is no longer going to have a job because we're going to have automation that replaces it. And actually it was a bit more of the, okay, well, people want a quick status on something. It's actually really, in some ways great. You don't have to sit on hold. You get the quick answer. But still, if you really have something complex you have to work through, you want a human on their end of the line. So my expectation, again, you know, maybe there's some displacement, maybe there's some refocusing, but I think that there's still good to be. I don't see a mass displacement based on AI. Again, I think, yeah, there are going to be some kinds of tasks, but I think that's going to just then enable, focus on other things, build out the other aspects of products faster, moving down the roadmap. I know as a product manager, I've always got a wish list that's longer than we can get to. May we start to execute a little faster. And it's not necessarily that those people aren't around anymore, but they're, you know, able to scale a little bit better because they don't have to do quite the same number of mundane tasks.

Yeah, no, I agree. I guess maybe I'm pessimistic. I don't see this as magic. I would see, even if it did, that you're going to need someone that is almost an AI integrator that's like, you know, in the weeds and not everyone's going to be able to do that. Maybe that's. You're not a tester per se. You're maybe an AI validator or something like that. Verifier. I don't know.

Yeah, and I think that's going to start to develop. I think one of the skill sets that particularly is, you know, students are coming out into the market. You need to know how to work with AI. I think these are tools that are not going away in any way, shape or form. But yeah, I think you're right. The idea of being, having a human who's now orchestrating perhaps a few AI agents and knowing how to do that, how do you give the right guidance? How do you make sure that it's staying down the path that you're getting the results you expect? But yeah, I think I'm with you that I don't see it being a massive. Yeah, it's not like we're all going to go home and AI is going to do our work for us, I.

Think, which would be nice. I guess that's the end goal, right? Everyone would be in the same predicament, all chill.

So you know, I glossed over AI agents, autonomous AI agents. Let me how I understand and I could be completely wrong. So when I think of autonomous agents, I think like you have a functional test, you'll say like test complete. Does a, say a functional test and then it's able to call, say SOAP ui. It does an API test and then call. All right, now do a performance test. Be able to talk to one another and say okay, this looks like it's good to go. Let's add it to the CICD pipeline and something like that. I don't know, is that what I guess what's autonomous AI agents? Is that a realistic scenario? Am I on the right track?

Yeah, I think so. And I might even go a little bit broader than that. One of the other things we've been looking at is, you know, how can you have an autonomous crawler that goes and discovers what that application can do, go through the various paths, push the various buttons, the various fields have it read some of the documentation to understand what that, what are some of the expected capabilities. So I do think that there's, you know, it will go somewhat beyond but I think as we look at this there's sort of that, you know, grand vision of, you know, eventually it'll be able to do all this stuff. But again kind of the crawl, walk, run. I think there's a lot of value to be had by solving individual pieces of that what will eventually become an agentic flow really well. So how do you get really good at the self healing part of an application so that even if the screen changes, the button name changes, the DOM code references are changing, that we still know what that is some of those sorts of things, I think, you know, make sure that we're, you know, generating really good tests that are kind of completely well written out, well thought, well structured, so that, you know, anyone could look at those and understand clearly what's being tested. But yeah, and then as we start to string those things together, it's just like you were describing, to be able to say, okay, great, you know, how do we go from that sort of initial inputs to validating the outputs? And then how do you look at, you know, between two different systems and the contract testing? There's, you know, how are those interactions? Because everything's AI driven or API driven these days. So, you know, clearly you need to be able to understand both sides of contracts that aren't necessarily entirely owned by the company that's doing the testing.

Nice. And when we're talking about DevOps specifically, do you see it working in production where it's learning? So a lot of times we make assumptions, we go, oh, it's going to behave this way, put in production. And then we kind of look at all these logs, we don't know what we're looking at, we make more assumptions, we put a shift left, but it's not really intelligent. I mean, we're trying to make the best assumptions we can with the data we have, the ability we have. Do you see AI being able to then also work in production and feed your pipeline, your DevOps pipeline for what it's learning as well?

Yeah, definitely. I think that's one of the places that I think has not been explored as much yet. And we've got a lot more coming. Is really on that observability side of things to see. Are the paths that you were expecting to get exercise being exercised or the features getting used that you thought were being used, Meaning maybe the code works great, but your users aren't, aren't going down certain paths, the application or maybe they're really going down a certain area and they're hammering it. And so suddenly performance is becoming an issue. I think one of the things that AI is good at is crunching large amounts of data, processing a lot of things and looking for and finding patterns. So I do think there's a lot that we can benefit from given that, you know, as much as I like to think that product managers are always seeing into the future in terms of what users want, hey, that's not always the case. So how can we to see, okay, well, you know, had the wrong set of expectations up front based on what's on happening in the real world, how do we feed that back and say, okay, well we actually need more processing power here. We might be running out of memory there, there parts of the application that aren't being used at all. So maybe we can retire some code and shrink the code base.

Nice. So you do work for Smart Bear. I'm just curious to know, do you have a unique approach to AI? Like a lot of times I get people saying why can't I use just use chatgpt directly or even like now MAN s or operator. I just tell it and it does it and it crawls and does everything I need for me. What's the secret sauce then for SmartBearer's AI approach? Maybe?

Yeah. So I'd say there are two things. So one is that we've got a lot of context based on what you're doing for your particular applications. We can know about your organization's rules that need to be followed and that maybe somebody who's going into ChatGPT doesn't think to say, oh yeah, you know, we have these hundred spectral rules we're supposed to be following. We can know about that automatically and make sure that we're applying those consistently, proactively. So I think that's one thing, that context you talked about being so critical. We can take advantage of more context by being within you're the space where you're doing all of your current workflow. That's the second thing that I'd say is pretty different is because we are already in the workflow, we're not taking something, copying it, pasting it into your web browser, going into ChatGPT, copying it and bringing it back. We're looking to really plug this in so that we can then again get more and more proactive to start alerting teams. Hey, by the way, this API just was updated. You're probably going to need to update the portal documentation for your external developers or hey, you're going to need to update some tests. That sort of proactivity only comes from having everything within that existing workflow and that awareness again, I think it's part of the special sauce that we can bring and generally also I think as you look at AI solutions, anything that is just sort of a simple wrapper or a clever prompt for an LLM is going to be obsolete in a short term. So really you have to be thinking about what are the worst workflow challenges, problems, opportunities that can be solved and not just if you can just do it by dropping it into ChatGPT that in and of itself then isn't very valuable.

Absolutely. So you know, I have two ends of the spectrum of people I talk to the either either all in gloom and doom or they're like does not this is doing anything. This isn't going to replace anything. They don't even try these tools. So that's why I love vendors. Vendors talk to a lot of customers. I don't think you'd be in existence or have this functionality if it's not helping anyone. So do you have any insights on how it may have helped AI driven insights that may have helped a customer catch something maybe that might have otherwise been missed or maybe saved a bunch of time for them, maybe triaging issues or anything like that?

Yeah, definitely. One of the things we've seen some customers come back and talk about how some of the self healing capabilities have helped them reduce their testing cycle times by not having to go back and make large scale changes manually. That's definitely one of the pain points we've seen again just creating a large volumes of tests generating whether it's generating manual tests from requirements, whether it's taking manual tests and turn them into automation. Again areas that we've seen some real time savings getting back to the make sure you're measuring before and after those are the areas that we've had some really positive feedback and as we start to look at innovation and one of the other things that that as sort of our general approach to AI is to look at how can we get early prototypes, early capabilities into our customer hands to try. Because it's one thing to talk about something and it's another thing to actually try it within your workflow. So we introduced something we call Haloai Labs which is a way for us to get the early beta capabilities into our customers hands who are interested in being at that cutting edge and being able to give feedback. So in one of those reviews we had somebody who talked about some of the work we're doing around generating portal documentation said wow, this could save me days if you're able to be proactive about telling me these changes that are coming that you can help me, you know, go and recommend a starting place based on how we structure our documentation. So I think those sorts of things are the places where people start to get really excited and we're looking to really lean into them. And the other great thing about this sort of labs kind of environment is it also lets us sort through the hey, what are the ideas that sounded good but aren't really that wonderful? Because again, as much as I would love to be right all the time. I'm not. And so it's really want our customers.

So what are some of those. What's. What's one idea that we're like, well, we missed on that.

So let's see. I'd say that one of the recent things we looked at, I'm trying to come up with a good example or.

Even an example of maybe where you thought it would do something and not only did it do that, but it did something else you weren't even thinking of.

Yeah. So that maybe that's a better. I can come up with one of those. So. So an example of something where kind of our expectations were somewhat wrong. We were looking at how can you use natural language to make updates to OAS definitions, the API documentation. And we thought, great, it's going to be really good at. You can use natural language, you can make changes that are repeated through that are not just simple find, replace, but changing structures to data elements, that kind of thing throughout the API. And because we were providing in context about spectral rules, one of the things we found was that people would request to make a change and sometimes some of those rule violations would also get fixed at the same time. So while they were asking for, hey, I want to update the structure of this thing, the contact information, it's not just a username and a password, but there's an email in there and there's a URL and that's stored for each of them. But then at the same time it realized that there was some structure, structural validations of naming conventions that then got fixed automatically. So I think there's sort of unintended consequence of having additional context that that was an unexpected benefit.

Awesome. So I have three questions I want to ask. I've been starting to ask people these three questions, just like a lightning round. So just a first thing that comes to mind really quick first one is what are some practical ways? You've already mentioned some of these, but what are some practical ways of seeing ways? You're seeing how DevOps or testers can use AI effectively, like right now, today.

So I'd say that for right now, today, make sure that you're using the generative capabilities to create and update and deduplicate tests. That's something that is kind of solidly working and well proven. If you're not taking advantage of those sorts of things. I'd say definitely that's an easy lift.

Awesome. So what skills or habits then do you think developers or testers need to develop to stay relevant in an AI driven future.

So I'd say one thing is just to make sure you're staying reading up on the things that are coming. Watch great podcasts like Test Guild. So I think, you know, really there's things are happening so fast that it's really helpful to find sources for people who are reading and watching and keeping their finger on things that are evolving. Because things are moving so fast, it's really difficult to do it if it's not your full time focus.

Absolutely. And the last one is, what's a common misconception or mistake teams make when trying to adopt AI in DevOps or testing?

Yeah, I'd say that a common problem is trying to push too much change to a workflow. If you're asking people to kind of scrap completely the way they do things and adopt something different, that tends not to go very well. Change is hard. Organizational change is very difficult. So I'd say that look at solutions that are going to seamlessly fit into that process and help you accelerate through all right, bridges.

Before we go, tell us a little bit more how we can find out more about Halo AI or any other AI solutions by SmartBeer.

Yeah, absolutely. So actually I can take this as a opportunity to plug. We've got a webinar coming up on May 22nd where I'm going to be hosting with Diego Lo Judice, VP and Principal Analyst at Forrester Research. So that's a place we're going to be talking about Forrester's approach on how AI can help optimize agile development. And we'll talk about AI and automation. So that's one way that I would certainly plug for looking at things that are changing. And then again, watch our website. We've got a lot of webinars, we've got updates coming through. Things are changing fast, so definitely encourage you to reach out.

Awesome. We'll have a link to all these in the comments down below. And for links of everything of value we covered in this DevOps Toolchain show, head on over to testguild.com p186 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet. Join our vibrant community@test guild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showed up the secrets to behold.