Revolutionizing QA with GenAI: A New Era in Software Testing with Fitz Nowlan

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way O the Test Illumination Testing Podcast Guiding testers with our nation awesomeness from ancient realms to modern days they lead the way. Oh, the testicle of ancient Testing Podcast.

Hey, welcome to another episode of the Test Skill Automation Podcast. If you want to learn more about how QA and gen can merge together to help you with testing, especially in the future, you're in for a treat because we have Fitz joining us who is a software engineer and founder. He earned a PhD in CS from Yale University and most recently leads architecture and AI integrations. Or as you know, smopier. So really excited to have him back on the show. It's been a few months in april on episode 142. So great to have him on back because things are always changing with AI to get a new perspective on where we're heading, especially as we head into the New Year. You don't want to miss it. Check it out. Hey Fitz, welcome back to the Guild.

Hey Joe, thanks so much for having me. It's great to be back. It's been about half a year now.

Half a year seems like forever in AI years. What's new? And great because it seems like Genai now seems like it's been around forever and everyone's talking about it. So why, why this topic at this point?

It's a good question, I think. You know, as you said, things are changing a ton. It feels like everyone, the initial wave has kind of crested. Everyone realizes kind of the baseline power of Gen AI and now I think everyone's gotten past the basic chatbot experience and they're looking for a little more functionality, a little more actual value creation from their gen AI. And so that's where I think, you know, we're starting to enter a new phase of Gen AI where we're actually doing things for users instead of just sort of making it easier to find information, if that makes sense. Kind of beyond the search engine experience.

All right, so I've been hearing a lot about AGENC AI. Is that what we're talking about? Is this something different?

Yeah, kind of. Not in that exact term, but I think that's kind of all these experiences that you'll now see in 2025 from vendors will be more agentic in spirit. They may not call it that, but they're basically trying to do the functions for you that you would have done previously rather than just exposing information to you.

So you have a lot of hands on experience of what you're Working with. Just curious to know how you see Genai transforming the role of qa. I think in the pre show you were talking about how going from. It's going to help going from tactical to strategic.

Yeah, exactly, Joe. So if you think about today in your QA process, historically it's made up of a lot of tactical steps, a lot of tasks that you perform. You check a box off a list and then you record your results and then you say go. No go. Gen AI is going to be able to turn a lot of those tasks into automatable tasks. Whereas before the only thing that was automatable was what you could actually get an engineer to code up for you in a script. Now a human can get a script out of that, can get an automated task or process out of a task they used to do themselves using Gen AI. And so that moves them from a tactical sort of operator to a strategic thinker. And you know, that's where I think the true value is created with Gen AI in qa.

All right, so I just got off a webinar this morning with these two AI automation uber thought leaders. And the idea was brought up that it's almost like we're ending the entering the golden age of testers. Because back in my day you had to be a tester, then you had to learn automation, become a developer. Now it sounds like AI is replacing automation in the sense and now we need to become testers again. Is that what you're seeing? Is that what this is what it sounds like to me?

Yeah, exactly. So the strategic thinker is basically the expert in your application. It's the expert in what the application should do. And you know, when the quality bar is met, that's a much wider pool of people than just the people who could code up automation before or the people who were narrowly assigned to do these repetitive tasks. So it's an operating at a higher level of abstraction, which, you know, I think is a topic we talked about last time when I was on the call. This is our firm belief that Gen AI raises the bar on where the human operates in the QA process.

All right, I'm assuming a lot of testers don't like to hear this because they're pretty much secure in what they do. They know the tools, they know how to code now. So what's the main difference between like traditional code based testing that everyone seems to be, that they finally embrace, but now they're like, now I have to go Gen AI driven testing, how does that even differ? And like what do I need to look out for if I do go with that approach.

Yeah, it's a really good question. I think it shouldn't be something scary for someone in QA in the big picture, basically broadens the pool of the things that they can now do. So previously you had a narrow set of tasks that could have been written, automated with the script and code, and then also a narrow pool of people who could do that automation for you. So the pool has broadened both for what is automatable, because Genai can automate some of these repetitive tasks by controlling you or by giving you sort of higher level scripts that you can work in that don't actually require you to code, to write Python code or type trip code or Java code. And then the pool of people who can actually complete that are now the people who are the experts in your application. So it's again like a higher level of abstraction. If you are familiar with the application and you know how it should perform, you are now capable of testing it. So it shouldn't be scary. It actually is an empowering thing. And I also think that it's all the more important that a human remains in the loop because the humans are the final decision makers for what the application should do and whether it's meeting the quality bar. They set that quality bar. In other words, here's a great example. Genai could tell you that your application performs generally under 300 milliseconds of latency. It returns a result within 300 milliseconds. But Genai doesn't know if that's a suitable metric for your application. In some applications that might matter, in other applications it might not. The humans are the final decision makers in that sense.

Yeah, I 100% agree. But it also almost sounds like when I first started in the late 90s, they had keyword driven approaches to automation and then they had like BDD now and it was all in the spirit of let's get non technical people involved. And so everyone can contribute to QA and testing never happened. How is Gen AI different? Like if someone's listening to us they're like great, get my whole team now to do this. Because now they don't need to be an uber coder. And in the end it's like, how do you get people then to actually embrace it?

I guess, yeah, I 100% agree with that point. That like that it never really happened, that everyone got involved. I think what's really different about this is Genai takes us back to the source of truth for an application and the source of truth for an application is the user story. It's the requirements, it's something human authored for what should be built. And what's the purpose of the application? That expression, that document, that, those wireframes, whatever, those artifacts that are now produced for the applications we want to build, those are your inputs to Gen AI on like the code building side of things, and they also serve as your source of truth for the testing on the QA side of things. So in other words, it's not so much that we have to get everybody involved, it's that everyone is going to be involved by virtue of the qua of the power of Gen AI operating from these source of truth artifacts that we have just for software building, if that makes sense. We're kind of backdooring our way into Gen AI, you know, impacting QA and software building by virtue of the fact that it can speak our source of truth artifacts, it can speak that same language.

All right, so now that it's back during its way in, how can teams prepare for. I think it's actually, it's going to be inevitable that people are going to have to embrace some sort of AI. How do we start working it into maybe our current workflows? I know I talked to a lot of testers that they're playing around with a lot of these AI tools. Is anything they need to know in order to be able to get started or prime the workflows to really embrace the Gen AI approach?

Yeah, for me, I think the, the best way to think about this is to divide your work into what is an automatable task or a repetitive task, sort of a procedure or a function that you're always doing and what is a logical or creative decision that you're making? On the one hand, Genai, you think, oh, Jennai can create tons of content and it certainly can. But there's still something fundamentally different, at least today, between the human created content or the human decision that's being made about whether the quality bar is met or about whether the application is performing to spec or why we should build this application in the first place or who we want to serve. Those are all human decisions. So you want to separate your work, whether it's on the software creation side or on the software testing side, along the lines of what does not really require me, you know, what's an automatable task, what's a procedure that can be performed versus, you know, where am I creating value? Where am I making a difference as a human decision maker? And then obviously you want to maximize your, your impact on the human side of things and prepare for the future where those automatable tasks, those procedures can be performed by AI.

All right, so it sounds just like automation. You want to find out repetitive tasks and then replace those with AI for the critical logic decision making. Couldn't I create like a Joe LLM that I train on my decision making that then would act on the AI and what it's doing?

Yes.

At some point is that far fetched?

No, no, no, no. I think that's reasonable. I think at least from everything I've seen, you still want a human signing off on the final decision. So you could make those decisions, you could prep those decisions, you could seed an idea for a decision, but you still want human sign off. I think for most of your tasks solely due to hallucinations at this point, you know, if Genai won't solve that based on its, its like mathematical construction, it's still always going to be sampling from a distribution and so that's not going to change. I think you still will always need some kind of a human in the loop for that final sign off.

All right, so you know, you work for a vendor, I'm sure you talk to a lot of companies upfront. How many people are thinking hey, I could replace my whole developing testing team now with the AI?

We actually don't hear that a ton. We don't hear that a lot. What we actually hear is we can go faster, you know, we can ship faster with higher quality. And I think obviously I'm an engineer by training on the software creation side we're under just as much of a quote unquote threat from Gen AI as on the testing side, maybe even more on the creation side. But on the testing side we have the benefit of all the terrible code that's about to be written by Gen AI for the next five years. Right. Gen AI is going to infiltrate on the code creation side that's going to produce a whole massive amount of bugs and opportunity for vendors like SmartBear to come in and, and test the quality of that code and give people confidence in what they've written. So I don't think it goes away. I think it, the landscape changes a bit but doesn't go away.

Right. So it seems like as we mentioned, a lot of companies that aren't necessarily testing companies are coming in with these type of solutions that can automate tasks, not necessarily tests, but could be used for telesight to anthropic just came out with one called Computer Vision. And obviously have like millions and millions of dollars, billions of dollars probably behind them. OpenAI is supposed to come out with one later on. How do I know what tool to get behind, especially when these larger companies eventually almost seem like they're going to be monopolies that are going to have, you know, able to do all the features that a lot of other companies have right now that are testing companies. I don't know if that makes sense.

But yeah, it's a really good question. The computer use tool from Anthropic, it's both like, straightforward in the one sense. Humans interact with computers in a finite number of inputs. There's a mouse, there's a keyboard, and then there's a screen that we, you know, read and react to. So if you could just manipulate those on those three axes, you could certainly look like a user of a computer. So on the one hand it's sort of straightforward. On the other hand, it's obviously a massive amount of engineering and billions and billions of dollars, like you said, have gone into making that actually look like a real human using a computer. So very, very impressive in that sense in terms of like getting behind a vendor. I'm a big fan that over time I do think the open source models will eventually catch up. If you look at how far back they were two years ago and how far back they are today, they're closer than ever. And at some point we have this question of do we run out of training data? You know, do we run out of can OpenAI crawl enough data in the whole Internet to, to get a better model? So I think we're at a point where I think the, the paid models are still ahead. The paid vendors, you know, the computer use from Anthropic, the, the new version from AI, they're going to be ahead, but long term, I actually don't think you need to pick a bender. I think the open source models will proliferate and you'll have your choice in the future of what's the right model for the right job.

All right, so if that's the case, if you have a vendor tool as you do, do you then decide behind the scenes what model you're going to use or like how, how would a vendor then still be relevant? I know they'll be relevant, but I'm just curious to know like, how, how does a vendor exist in that type of scenario?

Yeah, good question. So we as a, as smartbear, we don't actually train our own models there. We use the foundation models and then we hone them and basically fine tune them into our use cases. And so the vendors are always going to be relevant because the user's not just buying the model, they're buying the experience that they get from the vendor with that model. That, that expertise, you know, of using the model for a specific purpose with all the guardrails and all of the reporting and the notifications and the user experience, all that comes together for an application experience that customers will want to pay for. So I think vendors remain very relevant into the future even as the models get very powerful because you still ultimately, you know, when you drive a car, for example, you have a familiar set of controls. When you fly a plane, you have a different set of controls. You know, when you do a certain task, you want to fit into the look and feel that you're used to. There's simply a better look and feel for a different task than another. And so that always remains. So vendors remain relevant as these models get. Very good. But my point is that I don't think the foundation models necessarily will be ahead of the open source models forever. So that's good for vendors and it's good for customers because vendors can choose a cheaper model and customers can, can get those benefits passed on to that.

Absolutely. So I've been kind of surprised at the adoption of a lot of these technologies and I know like with things like self driving cars, they have laws, you can't just let it do make the decisions for you. Do you foresee that being an issue? Any ethical considerations like laws being passed, like you can't have Jenny I making all the decision makings for you. You need a person in the middle to like audit it or be accountable? I don't know if, I mean, are those the type of things that are way out ahead or something we need to get ahead of before, you know, laws get into place?

Yeah, it's a good question. I personally just maybe like the, just kind of the litigiousness that is personally the litigiousness of this society today makes me think that there will be laws for everything in just a couple of years. Like the law landscape will change rapidly and dramatically going forward. So yeah, there is a sense you have to keep in touch, stay in touch with that. I think in terms of like how you bring AI as a vendor to market, I think that there are, there are some pillars that you can establish that let your customer know that you've thought about the right way to bring AI into the market and, and you're communicating with them about when AI is in use. So for us we have kind of these pillars around transparency and accountability. Tell your customer when you're using AI for something. We have data privacy and security. So tell them if you're training a model or not. You know, we don't train models at SmartBear but we take great care with our customers data. Another is this idea of ongoing assessments. So we should have, we have a golden set, basically a regression set that we test our own AI integrations on to make sure that when we move to a new model we're still delivering the same quality of service to our customer. And then the last piece of last pillar of our framework is legal compliance, which you kind of alluded to. We have to stay on top of the changing legal landscape for AI and I think your example of self driving cars is a perfect one. We may see in the future cases where you can't use AI without telling your customer exactly how you're using it. And we would obviously welcome that. I think that's a great development.

Absolutely. So you know, I'm going back to where we talked about earlier, how genai can do the repetitive task and you need a human in the loop. But couldn't I just train a model like once again I said like Joe AI but I could point it to all the documents from my application, all the requirements, all the discussions from my team and say okay, it probably could make better decisions than me. All I need to do is prompt it with the right questions. Is that being realistic? Yeah.

So you're saying on like the software creation side of things.

Yeah, like minimizing the human even more because not only I'm not only a domain expert anymore, I, the thing I could train in on all the documents that I read is probably a better domain expert than I'm going to be.

Yeah, it's certainly possible. I don't know when maybe it's not even not a question of if, it's just a question of when. But yeah, I think we kind of see a future where software is going to be created from your application requirements, from your wireframes, from your user stories, from your feedback from your customers, from your product marketing research sessions. All that content comes together and then the genai can already generate, you know, valid code or I should say compilable code. And so I do think that will change rapidly change the landscape for software development. However, what we also are seeing is that gen AI still struggles with refactoring, with deduplication of code throughout a large code base. And so what you may end up having is the same function repeated multiple times throughout your code base, which is poorly factored, which basically like exponentially increases the maintenance burden, which then also leads to bugs. And so that's why we're happy to be on the software testing side of things using Gen AI as opposed to the software development side, because we think we're kind of we're coming up on a wave of AI authored code that will need really good qa.

Absolutely. So I always get asked also, how can I upskill myself or my team members to be AI proficient? I know this ChatGPT, but like anything specific other than that.

Yeah, there's not like there's not a silver bullet for this. I think you have to be naturally curious and do your homework. Stay abreast of all the latest changes in Gen AI. For example, if you see anthropic rollout computer use, download and use it and go become familiar with it and the concepts of it. I don't think you need to know how LLMs work at a fundamental level, but you should know, for example, properties or facts about them, such as they're statistically sampling tokens from basically a world of all possible tokens and you know, you should have a little bit of an understanding of how they're producing content, stuff like that. Apart from that, I think it gets back to if you're on the QA side of things, knowing what is a task where you're a differentiator as a human in the loop and where you're not. For example, if you're doing the same seven clicks, the sequence of same seven clicks every day when you're testing something, that's probably something that could be automated in the future. So that's not something where you're creating value.

Is there a place you would never want to use Gen AI?

Oof, good question. I think definitely there are things today that are not such a burden and they're important enough that I don't even want a smidgen of risk, for example, like paying my bills. I don't have that many bills or I don't mind going in and queuing up the payment once a month over time, right. As AI gets better and maybe if that amount of time that I have to sync paying bills increases dramatically, maybe then the trade off starts to fall into favor of using AI. But for me, the rule is kind of, you know, how much of an inconvenience is it multiplied by the risk of Degen AI hallucinating or making a mistake, you know, and that's kind of there still are things that fall under that threshold for me. Yeah.

Awesome. So last we spoke, you had been acquired by Smart Beer and you started to integrate your Reflect with other product lines that sound alike with Smart Beer. I'm just curious to know, I don't know if you could talk to this. I think we said something about Zephyr, how you integrating the. No code.

Yeah.

Automation to Zephyr. Do you have like, how did that go? Any results from that? Anything? Any lessons learned?

Absolutely, yeah. I'm glad you asked. So we are. We're totally live. We got live in July. And when I say live, what I mean is Zephyr Scale, which is a smart pairs product that's in the Atlassian marketplace, or end to end test case management, now has the ability to take your plain text descriptions, you know, your test cases written in text, and execute them in the Reflect Automation test execution engine. And so that's released. That was in July. And so all customers in Zephyr Scale now have access to Reflect, so they can take their tests and they can run their tests in an automated fashion. They can also use record and Play if they don't want to, you know, use AI description or plain text description to use our AI. But that's live and we have hundreds of customers using that every month. And it continues to grow. Zephyr Scale itself is sort of a growing product, so that continues to grow the exposure to our automation, but it's going really well. And then on the execution side of things, on the Reflect side of things, we're rolling out the ability to test native mobile applications using AI instructions. So if you think about like today, the state of the world for testing native mobile applications is you connect to a grid, you have a device, you know, physically residing somewhere, and you point and you click and you test to make sure things work. You'll now be able to express those tests in instructions. So, you know, open the application, enter a new record, click submit, login, verify that the application does xyz. And so that's going into beta, I think, actually just went into beta last week. So with one or two customers, we're growing it, you know, growing the beta throughout the rest of this year and then into next year. So it's been going really well. The fit has been really good. I think the blend of Reflect automation and AI with SmartBears distribution, the Zephyr Scale, you know, customer base has been great and we're excited for the future.

Yeah, really excited. Sounds really cool. I know it's hard for people to visualize this when it's just a talking head audio. So I'll have a link down below so they can actually see like a demo video of it in action to get a feel for it. All right, so, you know, I know you're always working on new things. I know you just mentioned another new release that just came out, but, you know, we're about to enter 2025. Any predictions for 2025 testing WISE or AI WISE? Last time we did it, your partner there, he mentioned multimodal AI before it came out, and then it became a thing in that year. So I'm looking for one of those type of insights that you might have.

So I think the big prediction for 2025 is that AI will, as I alluded to before, it'll go from that level of exposing your data in this chat like interface to actually doing things before you. So workflows. And so you mentioned agentic workflows. And I think definitely we'll see that, you know, whether people call it explicitly that. Some will, some won't. You know, we basically are doing agentic workflow on the back end, but we don't necessarily call it that. But these specialized workflows driven by AI, potentially constructed by AI, and then reported on by AI, all overseen by a, you know, human operator or a final decision maker. In other words, it's not, it's not a revolutionary prediction. I guess what I'm saying is I think it'll just, it'll reach adolescence maybe, you know, it'll come out of that early stage of this is possible. There's this capability here and I can talk about your data to now I can do things for you. And it comes back to you like a kid and says, like, hey, how did I do? You know, was this good, was this bad? You know, what should I change? How could I improve? I think that's kind of the next stage. So it's more of an evolution instead of a revolution, I think.

And why, why are you hesitant to call it agentic? Is it because agentic's a little different than the workflow that you foresee?

No, no, no. Just, just. Sorry, it's purely just not, not really hesitating to call it that. I guess just since I've been doing this for, I guess a couple years, I've been doing it without that term on it. That's all.

Gotcha. Right, right.

So really, no, no reason that it's not. I think you're ahead of me in the terminology on a couple, a couple months by like A half a year or whatever? No, no, a hundred percent. It's agentic. I just don't know if like, in other words, some vendors will expose their agents as things for their customers to control, and then other vendors will use the concept of agentic workflow internally, but they won't necessarily sell to their customer an agentic workflow. And that's all. I think we're in that latter category where we're structuring our workflows in an agentic way, but our customer won't know that we're we're doing that. And they wouldn't think to themselves, oh, this is an agentic AI integration. This is just an AI powered feature. Whereas you do see some vendors where they say, like, you can use our agents to do this and this and this. And so I think I'm just sort of drawing a distinction between the public perception to the customer.

Makes sense. Okay, Fitz, before we go, is there one piece of actual advice you can give to someone to help them with their AI automation testing efforts and what's the best way to find or contact you?

The best advice is know exactly what your application is supposed to do. The number one thing that doesn't matter whether you're using AI or not. If you don't know exactly what your application does and who you're serving, who your customer is, then you know you're not going to have success no matter what you do. But if you know those things, then you will be able to deploy AI in a really useful, powerful, impactful way because you will stay focused, laser focused on delivering for your customer. So I think when you're building applications, when you're testing applications, the number one thing is what should my application be doing and who am I serving? If you can remember that, I think you'll be successful with whatever vendor or tool that you use. In terms of how to get in touch. I'm Fitz Nolan. I'm at SmartBar. You can do Fitz. Fitznolinmartbear.com Feel free to shoot me an email.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com a526 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating, end to end, full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation and software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.