AI in Testing: Innovation or Just Noise? with Fitz Nowlan

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, is all the buzz about AI and DevOps testing innovation or just noise? Well, depending on what you think. You're in for a treat because we have a special guest joining us to talk all about how AI is solving today's testing challenges, myths and fact. Joining us today we have Fitz. Fitz is the VP of AI and Architecture at SmartBeer and leads all AI innovation across various software testing products and services. Prior to SmartBeer, he was a software engineer and founder and earned his PhD in CS from Yale University. So he really knows his stuff, really is in the weeds of AI. What it can do and what it can't do. You don't want to miss this episode. Check it out. Hey, before we get into it, this episode is sponsored by Smart Bear, makers of Insight Hub. If you're curious about open telemetry but find it a little bit overwhelming, Smart Bear has you covered. They just released a free ebook, what Is Open Telemetry? A straightforward guide that I think is a must read. It breaks down exactly what is open telemetry and why it matters for application monitoring and how you can start using it right right now without any confusion. Grab your free copy now at Test Guild ME telemetryguide or click the link in the comment down below. Hey Fitz, welcome back to the Guild.

Hey Joe, thanks so much for having me. It's good to be back. It's been a couple times there.

Yeah, always excited to have you. You always seem to have your pulse on what's going on. It really seems like, you know, you work for a company that's evolved a lot in AI as well, so you really have a front row seat almost to AI. The evolution of what's going on, what's working for your customers, what's not. So I want to look at maybe some recent AI launches in software testing and beyond and see what stands out as truly impactful versus what just feels like marketing fluff. So the first thing that hits me right now, and I'll get your opinion on what is hot and what's not is MCP servers has been all over the place. So what's the deal with that? Is that something you see as fluff or hype? Or is it the real deal?

I think it's the real deal. But I think the impact of them is still very early days. So the idea behind an MCP server is that you want to allow your customers at runtime to extend the functionality of your AI driven application. And so they can plug in their own data sources like their code base or internal files, internal databases, any kind of a data source that they would have, that they want to make available to your application as a vendor, that they can then use it within your application. So I think the concept is straightforward, it makes sense, and I think it will gain a ton of traction, but I think it's still kind of early days where people need to identify what is that killer application, where I need my special data to be plugged into it. You know, I think historically all these cloud apps have been just the vendor's data residing in the application. And now finally we're realizing, oh, with some of these AI applications, we want the customer's data to be injected at runtime.

So is it almost like a wrapper that creates an API for your particular application, then you then feed a third party LLM on your data so it knows context? Is that how it works?

Kind of.

It's.

So imagine you have a vendor that you purchase and that vendor has introduced AI or LLM powered features into their application. But that LLM, one of the things that it needs is the customer's data. And so MCP allows a customer, or really the vendor, to add the customer's data as a data source to the LLM. And so basically making data available at runtime to the LLM that wasn't available to the vendor when they were building their application. So imagine you have, imagine your Salesforce and you're working with some really large customer of yours. And that large customer has a database full of historical interactions with a bunch of their opportunities and their leads. Salesforce may not have that entire database or that whole data warehouse. It may be massive. However, the customer can plug into Salesforce through an MCP server to make the customer's data warehouse available to Salesforce's AI at runtime.

Gotcha. So. And like I said, it gives more context then, because the AI is only trained on what it's trained on, public data. But this would be more private data on your specific situation. So the results in theory would be hopefully better, I assume that's right.

Yeah. And it could be two aspects or two ways you could work it. You could make it so that the MCP server allows the vendor to train on your customer data and therefore get an even you know, a finer tuned LLM. But you could also use it more on a rag style approach, you know, retrieval, augmented generation, where there's no fine tuning or training occurring, but they're incorporating the data into their prompts at runtime. So it's still going to be more personalized, but it wouldn't necessarily be a full on trained model. So there are kind of two approaches there.

Gotcha. So I know like Playwright and Selenium both released MCP servers. So what's the benefit of that then? Being able to interact with the browser that way or an API for a browser?

Yeah, you know, I'm not, I haven't used either of their MCP servers, like, you know, their extensions through mcp. But I could imagine that the benefits there are exposing, you know, either a few things, either exposing a new data source that's internal to the customer who's building the Playwright or the Selenium tests, or exposing things like source code or other parts of the application that aren't accessible at the browser level that you need to plug in an extra data source to make that available.

Gotcha. So at SmartBear, I guess like I said, everyone's releasing these MCPs. Are you doing anything like with any of your solutions that actually going to take advantage of this where you're going to do your Smart Bear MCP servers?

We don't have any plans to do like a Smart Bear branded MCP server. However, we do have two applications where we're already kind of talking about how do we want to expose MCP capabilities to our customers. So we're trying to, in other words, we're paying attention to the latest and greatest, but we're not just rushing out and just getting something out just to say, hey, we do MCP2. We're really trying to focus on the customer use case and understand where is it that they have a data source that's rich and valuable in the context of one of our applications and then lean into that. So I actually have an MCP meeting tomorrow to talk about it, but we don't have any concrete plans right now to say, you know, this or that product has an MCP server capability.

How do you evaluate whether an AI feature or tool is meaningful or has meaningful purpose or it's not just noise, you're just doing it just to, like you said, keep up with what everyone else is doing. And do you have like a framework you use to go through with your team? Okay, does this actually make sense or are we just doing it because it's cool? And it would be nice to have.

Yeah, it's a great question. I mean the most basic baseline is always build something that customers want, you know, build something that people will pay for. And so that's where a lot of our, I shouldn't say a lot. I mean, that's a big part of where our AI features are rooted. When we talk to customers, you know, we want to do something like this, is this possible? And so we have our Halo AI Labs is sort of this beta program that we have available to all of our customers across our products where they get first access to our new AI features. And so we have AI powered features that are in that. And sometimes when we give them to customers there, if we find great usage, then we're going to look to promote those to production for everyone. And other times if we have, if we build a feature where customers aren't so enthused by it or it's just not so practical or useful, then we can just quickly unpublish it, basically. So the Halo AI Labs is a great way for us to get that early feedback from customers. The other part of it, how do we know what's noise and what's not? On the AI team that I run at SmartBear, we focus on trying to be cutting edge and trying to be in touch with both the SmartBear products and the SmartBear customers, but also kind of where the AI is going. So some of it is intuition, to put it plainly, so that we, we try to lean in where we feel like there's value here, where there's a real differentiator. So it's kind of a mix of those two things, you know, customer feedback and intuition.

Nice. So I've spoken with a few companies and they're kind of now pulling back on AI powered type terminology. Like I've been seeing a lot of AI powered test case generation tools. So in your experience, do you think these tools fundamentally are really changing workflows? Are they still more AI assistant? So do you see people starting to pull away from AI and just using the word like intelligence? They don't necessarily care how you get the end result as long as the end result is correct.

Yeah, that's a good point. I don't have a strong opinion. I think this is kind of related to your question and this is kind of coming to my head. So the thought that I have here is I wonder, with the rise of agentic AI, I think that organizations will start to see that as a true step change, a true increase in value being brought to their customers. And so it's possible that the LLM wrappers that most of the AI features in the world today are will kind of fall back into sort of, they'll be simpler or more baseline intelligence as opposed to full on AI powered xyz. I wonder if, in other words, I wonder if AI powered is kind of being reserved for agentic AI features in the future. Whereas you know, those LLM wrappers which rapidly lose their value over time as the models get better and better. I wonder if they're kind of falling down into more like intelligent test case authoring as opposed to full on AI powered authoring. But that just kind of an off the cuff thought there.

Yeah, makes sense. So where do you actually see innovation then happening in AI for testing? Like real. I know you create a really great solution that was acquired by SmartBear because I think it really took really good advantage of AI and did it in a way that made sense and it wasn't just hype. So where do you see real innovation then happening in AI for testing? Specifically?

I do think that it's the agentic AI. So we're working at SmartBear on a full on, not just a QA tester, but a full on like QA department abstraction. So the idea that with agents now you can have an army or a fleet of agents that perform all the different functions that you might find in your whole QA department, not just a single, you know, manual QA tester. And so I think with the, the ability of these reasoning models to really kind of churn in a loop towards a single objective or a single goal and then being able to pick, call it five to 10 different goals that a QA department might perform, you can really start to focus in, in an expensive way currently with all the tokens, but focus in on solving different individual problems that are all related toward, you know, in QA and testing. And so that's a research project that we have around agentic qa. So I think that's there's real value there now in terms of like what's in the market today. I think we're just kind of scratching the surface of what a lot of different providers are doing. One that comes to mind is cognitions, dev and AI. That's obviously around software development rather than software testing, but it is an agentic flow and it is kind of a higher level of abstraction from some of these other LLM wrappers we've seen that are doing things like, again, to use the software development analogy, code completion or Code reviews. Those are kind of single insertions of AI into the software development process today, as opposed to a full junior developer. Abstraction is a level above that in terms of abstraction, right? That's an A, that's a. It's an AI that's doing a whole role of the junior developer rather than just one little task within the software development cycle. So I think we'll see similar things on the QA side as well. And you know, we hope to be at the forefront of that. One other technology that's new that is worth mentioning is this computer use model from OpenAI and Anthropic's computer use model. These are really different because these are models that are trained to interact with UIs. And I know this very well because we were interacting with UIs two years ago using GPT4, the original GPT4 and 3.5 and we had to do a very complex process for interacting with UIs. Now today, with these models that are trained for producing XYZ pixel coordinates, they can really drive interfaces, they can really drive whole UIs with AI. And that is not something that the previous models could do. And I know we tried to get them to spit out correct locations and coordinates for interfaces before. So this is really powerful and this is very revolutionary and we are investing a lot in our use of these computer use models.

So what I'm confused about this is SmartBear has had like test complete forever. How's this different than test complete? Is it just now catching up to what you've all been doing for like 20 years or all these tools? Or is it different? Like how is it different than like a Windrunner back in the day or a Silk or something like that?

So I hate to Pre announce but TestComplete will be using these approaches to drive what we call self healing. So testcomplete for years, right, was using locator based approaches and some visual approaches to identify elements as part of a test script. But when that interface would change, we didn't have any intelligent or dynamically intelligent at runtime ability to find the same element that was there logically, but looked different or was in a different location. And so now with AI and some of these computer use models in this particular feature, we're not using the computer use there just yet, but we are now able to bring LLMs into the fold to make that decision for where some logical element is now residing or what it now looks like. So the idea is that we can look at an interface like a human would and say, oh well, that doesn't have the exact underlying locator that it used to have deep in the code. But logically it's the same thing. So that's the element we should interact with and test complete. Will be rolling this out, you know, I don't know, in Q2 maybe. And we'll call it self healing. It's AI driven self healing. And that's the difference that in the past it was static, it was code based, it was a locator. Today it's logical and it's intent based.

All right, so once again, I don't want to be bogged down on this because I'm just trying to wrap my head around it. Once again, used to have ability to do like an image compare this image versus that image. It's different feel, but this is more sophisticated I assume because now you're using actual AI rather than just a generic algorithm to figure out, yeah, the image is a little different, but it's still, I can still understand it because I have context and I'm able to reason within my model that this is still the same image or something or the same element.

Yeah, you're spot on. And your knowledge of testcomplete is awesome because I appreciate it. So yeah, you're spot on. So basically the previous ones would do visual image compare and they, you, they were trained using a computer vision model and so they would identify objects that look slightly differently but are basically the same. This is now interpreting it using gen AI or an LLM to say, you know, this object is logically the same as this other object, even if visually they are different. We'll say, so it's the next level kind of intelligence to that self healing approach.

Nice. All right, so I just want to go back to some of those abstractions to just put a little more legs on them. So when you were talking about abstractions, so you talking about like having like a model in a gentic AI that was like a security Sally or any accessibility where you say, here's my application, I want to test for security. So I have a Persona, a security Sally. Say run it. Run any type of test you normally would as a security expert.

Yeah, I think we haven't decided if, you know, if that is exactly the future for us, but I do think that type of approach will emerge. I think there's a benefit to specializing your agents in AI just like there is and having your roles at your organization, you know, your human roles, specialized. It allows you to, you know, if nothing else, like a really basic example where this can help is it allows you to regression test the performance of your agents better. If they're supposed to do kind of one thing and you have a class of challenges or problems or tasks that are all related in that same vein, then it's easier to validate the performance of your agent, if this agent is security Sally, for example. So, yeah, I think that we will see that type of specialization in agents. And then of course, it gives customers the flexibility to turn certain things on or off, you know, to use certain agents at certain times, you know, for their specific needs. So. So, yeah, so I think you will see specialization in these QA agents in the same way that we see it in your actual, you know, QA workforce.

Anytime anyone hears of AI, especially if you start thinking about abstracting your QA team or your testing team, they start thinking, losing my job. So where do you see, you know, meet people, actual flesh when it comes to roles or jobs in the future? Is it.

Yeah, so much of it, I think, is on the oversight of security Sally, for example, it's in defining the requirements for what security means for your particular application. You know, I think over time. Right. Progress always changes workforces at organizations. Technological innovation or, you know, for years, changes the makeup of your workforce and changes the needs and the requirements and the tasks and the job. So that's not going to change going forward. There will always be a dynamic, you know, some level of fluidity to the roles at organizations. So there definitely will be more disruption in that vein. It's not going to happen overnight and the disruption won't be. It's not going to be 180 degrees different. It's going to be slight changes. You know, instead of performing that manual test that you had to do, I now need you to oversee the execution of some manual tests that you authored potentially, or that you oversaw or that you guided by a fleet of agents. Another example, you know, maybe rather than writing that Playwright script, we need you to write code for build our product instead. In other words, you know, it's a small shift of perspective in these roles, but, you know, I don't think they go away entirely and certainly not overnight.

Gotcha. It's hard to gauge though, because we've been talking for two years and all of a sudden we have mcp, we have the new computer use and they have a new model and ChatGPT with the image and like, like a thousand times better you've been posting on LinkedIn. Like, how do you even know as a company, like, how do you keep up with all this?

Yeah, it's super challenging. And I think one of the best parts about coming on with you is you give me some of these tidbits that you picked up from your sensors and filters. It's really hard. And even internally we talk about it how, you know, one person or one team even can't keep up on everything. Some things are hype and noise and some things are substantive and real. You know, I think like anything, you can't keep up with all of it. So you kind of have to pick your spots, apply a filter, and if you feel intuitive, like there's something there, then dig into that. But don't feel pressured to read every single article because it can't be done. And so, you know, I think back to your, the point behind your, like, your question. We have been talking for two years and the landscape is still rapidly changing. But I would also say, like, I don't know if there have been full on industries that have died overnight as a result of these things. So I think that kind of gives me confidence that things won't change overnight and a lot of times too. The other thing is we live in this tech bubble and this AI bubble. If you go out and you talk to people at Thanksgiving or at a holiday and you're talking about things and their lives aren't impacted, there's a big wide world out there that's not necessarily going to change overnight. And so I think, you know, that there's some perspective that can be maintained that way.

That's true. My wife thinks I'm nuts. I take pictures of calories on food and say, which one should I eat? Which one's better? Yeah, I use it for all weird things. She's like, she would never do that. So that's so true. Adoption?

Well, she's like, how about just eat.

The vegetables and you know, on the carbs for sure. So, you know, we talk a lot about success, but I'm curious to know if you could share any failures or maybe not. Have you seen any AI applications and testing you thought would really be impressive but end up maybe causing more complexity than value?

It's a good question. So in testing, maybe. So one of the, not one that I've personally experienced, but one that I have heard about is, you know, so for years, right, in automated web testing, you use Playwright or you use Selenium, or you use Cypress. So there's this idea now to use AI to generate selenium and Cypress and Playwright tests and then run those tests. But I think that fundamentally misses one of the main challenges of those tests, which is that they are brittle and they break because the application changes. And those tests are just like the locators in the testcomplete case. They're specific to that implementation, to that or the underlying HTML or the underlying structure of the application. And so while you can use AI to update those tests, there's still required, you're still required to do a little bit of human in the loop to tweak the tests and make them work. So I feel like that's kind of a leaky abstraction in the sense that you're using AI to generate tests, but these are tests that you are only generating to test the quality of your app. Instead of generating those tests with AI, you should test the app with AI and basically rise to a level above using agentic AI, you know, using a QA agent. So that's what we're investing in. We're hoping to kind of skip that process of improving your code based tests with AI and instead let's get rid of the code based tests entirely.

No, that's a beautiful point. A lot of people say, oh, we're developers, you know, we need. They spend so much time coding these tests, thinking they're like they're hardcore developers and they're missing the point of actually testing the application as a real user, which would be more visual, which it seems like we're going back to more visual based testing, which was, would be better, I would think, not worse.

Yeah, well, I think that's exactly, I think it's a great, it's a great point. The point there is, right, that the user has no idea whether the underlying code of the application is a button or a label, for example. So why would your test need to rely on that fact? Because your application ultimately is just trying to allow the user to click on something. And if that's communicated visually, then you want your test to assert that intent, not to assert the way it was coded originally, 100%.

Now, one of the biggest issues I've seen with tests being flaky is test data. And for some reason every time I talk to a vendor, it's like, you know, we have something, but not really. Do you see that as an issue that AI in the future will be able to tackle better? Because it seems like for some reason after all these years, people still have problems with test data generation. You have solutions out there, but they're, they're kind of expensive, they're hard to integrate into functional tests or in DevOps.

Yeah, I like this area of investment a lot and we smartberry within testcomplete actually has the intelligent iq. Add on I think is the name of it where they use an LLM to generate test data. So you basically give it your instructions, you know, your plain text instructions. I want first names and last names, I want zip codes, I want a sentence or a description, a message related to this or that requirement and then the LLM will generate that data for you. And we've been live in production with it for, with that for about a year and we have some decent usage from customers, you know, repeat usage using that particular feature and purchasing that add on more broadly, we're investing in that for API tests creation. That's a case where you need test data for your API tests, but the data itself is fairly well defined. You know, these APIs have shapes, they have types on a lot of these fields that you're going to use in your API test. So it's a good area for LLM data generation where you want something realistic and authentic, but you don't need the bar for correct is easily measured. You don't, you know, you don't need to generate values in the entire expanse of all possible values. You just need a number between 1 and 10 and that will work. And so translating those we'll say API definitions from open API specification docs into actual data can now be done by LLMs. And previously it would have been very hard to get a general purpose binding from your open API spec into data using just regexes, for example. So. So that's a long winded way of saying yeah, I think LLMs will definitely help with data generation. We've you know, done a little bit of that at SmartBear and we're doing more of that now in development now.

Nice. So you know, we talked a little about Tuscaplee. It's a very mature product. I'm curious enough people are trying to implement AI into their more legacy type applications. Is it hard to integrate AI? Is that something you're like, oh, it's only for newer greenfield applications. How are you able to build in a way to be more AI ready for whatever innovations might come down the road in a year or two?

I'll say it probably is harder to integrate into legacy applications. Probably two reasons. One is just the UI or the data model. The structure of the application may not lend itself to great to sort of modern LLM concepts. The second challenge would be probably the implementation of it. If the implementation doesn't have a clean, what we would call in code like a narrow Waist for where you can hook in at the code level. It's going to be challenging to comprehensively or to like coherently call out to AI in that application. So I would say it probably is more challenging. But I would also say that just because it's challenging doesn't mean that you shouldn't do it. And of course, in the test complete case, you know, it's a large, you know, long running product, it's done very well for us and I think it still has a lot of room to grow. So just because something's hard doesn't mean you shouldn't do it. You know, it just means that you've got to be really intentional about it. But we are investing in AI in that product because we think there's a lot of benefits that it can bring.

So Anoa, I'm thinking, is there any AI power features at SmartBear that you think are currently underrated or misunderstood by your users?

Oh, that's a really good question. Yeah. So I think the one that probably comes to mind is in Qmetry, which was a recent SmartBear acquisition, they have a host of AI features there that I think are underrated. So they can generate full manual test plans. So not just individual test cases, but a whole group of test cases that are related from a requirement or a user story. So, you know, basically you describe your application or maybe you even give it your functional spec that a PM wrote and they can come up with the full test plan for that and load it into your test management software, which is Q metry in this case, and then update those as the application requirements change. So based on a prompt, you know, based on natural language, they can update your test cases and then also they can deduplicate test cases that are the same that you maybe have authored or that different people have authored over time that are really sort of functionally testing the same thing. So those are, I think are some underappreciated AI features that we have in the SmartPear portfolio today.

Nice. So we talked about MCP, which I've been hearing a lot about agentic AI before MCP was the hot thing. Is there anything else you see on the horizon that's about to blow up?

So the one thing I really I wanted to kind of highlight or talk about coming in was I think a lot of the initial AI features were and still are to some extent LLM wrappers. Your wrapper around the LLM where you're, you're providing a small jump over a gap in a process or a Workflow using AI. And that's again the case of like auto genning your playwright tests using an LLM. There's a small wrapper around the LLM. We say, here's the current code, here's the application, generate me a test. And it's going to be great at that. And I think those wrappers lose their value over time. What I'm starting to see now, which I think is, you know, is this next big trend that's worth paying attention to is this notion of agentic reasoning where you don't guide the LLM into producing an output for that specific problem or that specific gap in your workflow, but you instead just give the AI a ton of tokens and you say in a loop, keep working and keep working towards that objective. And here's what you said previously to the AI. You know, here's what I know, here's a little bit of a guide, a little bit of a nudge. Here's what you said previously, keep going. And basically this idea of if you had a big enough budget, the latest and greatest models will eventually produce something looking like a solution. And maybe it won't be quite as good or quite as coherent as the human would do today. But I think we're getting the kind of, the key insight here is I think we're getting to the point where you don't need to impose the structure or the guardrails on the LLM. You can actually just let the LLM talk to itself in the same way that you would give a human a task and tell them to go work on something. I think we're getting to that point now where the O3 reasoning models, for example, you know, is pretty, pretty close to being able to solve some of these base level tasks with a big enough budget. So that has an impact for application builders like us, because I think a lot of our AI integrations today are very focused on solving a specific task with guardrails and wrappers around the exact type of output that the LLM can produce. And I think if you can, if we get to the point where we can more generally ask the AI to solve a problem that unlocks, you know, a whole bunch of new application use cases.

So what's holding it back? Is it the power, the like token power, the money? So once that drops, it's going to be more and more common.

I think it's the token cost. It just needs tons and tons of tokens, I think, I think that's one piece of it also. I just, I don't Think the models are quite, quote unquote, smart enough yet I think that they, I don't. In other words, I don't think you could just have GPT4O in a loop and like let it run forever with infinite tokens. I still won't get to the answer for anything other than a basic problem, but I think as the models get a little bit better, you know, we just saw Llama 4 came out in the last couple weeks. And so I think the models will keep getting better, I think the token costs will come down and I think those two things together will make it so that you could solve baseline tasks. You know, not everything, but like, we'll see the point where you don't need to direct the LLM to choose one of these four options. Basically you could just sort of give it an open ended question and it will produce what looks like, you know, an answer that a human might come up with.

I know it's in my head. It's like a thousand monkeys banging on a keyboard would eventually write Shakespeare with enough time.

That's right. I think it is that. But I think if you can compress time down that you'll get that output, you'll get that Shakespearean output in a reasonable amount of time. And to kind of flip it around, when someone asks a human a question that they have a thousand neurons, millions and billions of neurons firing away banging on that keyboard to produce that output. It's just that it happens so fast that, you know, we don't know all the options they've discarded.

Ah, true. So switching of gears here, I don't know. Why do you seem like think we have it backwards a lot where we start from requirements using AI? Wouldn't it make more sense if you had already had an application to use AI in production to listen to what your users are doing using AI that automatically then would be able to visually see what they're doing, create those tests for you, which then would become your requirements that you would then start when you shift left. Does that make any sense?

Absolutely. So I'm smiling because we just had a conversation like, I don't know, it was a couple months ago internally about how it's a continuum, the shift left was on, on a line, you know, left to right, but it's actually a circle and the circle includes software development, software release, monitoring, observation, and then it cycles back around to testing and then, you know, feeding into your, to your software development. So if you imagine that you start with an idea for an application you want, those are your requirements Right. That has to come first. There has to be some need, customer need that needs to be met, or a personal need that you need met to build an application. So you have requirements and you define what that application should be, should look like. Maybe it's in plain text, maybe you just draw a mock. It doesn't matter how you represent it, but you have an idea for what an application, what the application should be. You know what should exist from there. You can then now use LLMs to generate that code. You can generate the code and then build the artifact. You can use the artifact using AI. You can then build tests from that original requirement document. You could publish it live and you could see how users use the application. And from that usage, you could then inform improvements, et cetera. So it's all a continuum. And I think the point is that the AI acts as the glue across those, those points. So there used to be the requirements definition point, there was the code implementation point, there was the test implementation point, and then there was the live monitoring. Those were all sort of concrete points. They're no longer on a line, they're on a circle. And the glue between all of those points in that circle on the, you know, the edge of that circle is LLM.

Crazy thought just popped in my head when you were talking there. So you're starting a greenfield application. How far away are we from recording, say a radiologist doing their job and they're talking through what they're doing. You don't have an application yet. And then using that video and that text to train an LLM that then create the application for you. So it's like really shift left or shift right. Like before you even write code, you are like, you're kind of like not spying on them, but you're actually watching them do the video and have the AI do the reasoning to actually create what they really need, rather than you assuming you know what they need by creating all these requirements by someone and then going back and forth.

I hope we're far away. Cause I'm toast when that happens. But I mean, concretely speaking, we have that for toy applications. You know, you see now with the all the vibe coding stuff, the developers basically, they say, I'm not going to write a single line of code. I'm just going to describe what I want, I'm going to keep describing, keep using. And they can build things. Like the one I saw was a recipe application that works. But you also see that in the vibe coding disasters where there's no security and someone made, you know, started making a million dollars a month or whatever it was, and then they get hacked because there was no security built in. So I think that for toy applications we're probably pretty close for real, you know, legitimate B2B applications. I think we're still a ways away from what you're describing of like taking a video of a radiologist or a really complex process and observing what they're doing and then building the agent that does that. I think we're probably years away from that or more. And so I think, I think there's a spectrum there, you know, a gradient of the applications that can be built. But certainly for like the Toya applications with Bolt New or there's a bunch of them, those app builders, just based on your prompts, I think, you know, you can, you could get a working application to run on your local machine in very short time.

Okay, Fetz, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps or animation testing efforts? And what scan what's the best way to find or contact you?

Yeah, the piece of advice I think is to, to stay current. And I think I kind of said something like this last time. Be aware of what's an automatable process and where you are really making a value added decision and try to automate the stuff you can and try to preserve your brain power for the things that really matter. And ways to reach us would just be fitznolinmartbear.com and you can find me online and on LinkedIn as well.

And we have links to everything value we talked about in the comments down below. And for links of everything of value we covered in this DevOps Toolchain show, head on over to test guild.com p185. So that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres. The bards began their song A tune of knowledge A melody of code through the end There it spread like wildfire through the land Guiding tester showing the secrets to behold.