AI's Role in Test Automation and Collaboration with Mark Creamer

I think Genai is really good at making smart people smarter and more efficient. It's not good at making novices brilliant because we've heard of Genai hallucinations. Well, that'll probably contribute to that level of feedback you get. But Genai in a sense is working with somebody that gives you suggestion and you know, not all the people you work with are brilliant.

In a land of testers far and widely journey seeking and succeeding skills seeking a better way O the Test Skill Innovation Testing podcast Guiding testers with automation awesomeness from ancient realms to modern days may lead the way. Oh, the test in all of ancient testing.

Hey. Today we're going to dive into the exciting world of Gen AI and AI in general in test automation. Joining us we have expert Mark Creamer who's going to help uncover how Genai streamlines test processes. At the push of a button, you'll discover how symbolic AI transforms user stories into optimized test cases, achieving better functional test coverage. You're going to learn how AI can boost collaboration in Sprint teams and explore the benefits and challenges of AI integration and testing. Plus all other kinds of awesome insights from Mark on AI's impact in testing. If you don't know, Mark is currently the President and CEO of Conform iq, which focuses on improving the quality and efficiency of software testing. With over 40 years of industry experience, he focuses on taking software testing to the next level. And in this episode he provides AI insights that you probably haven't heard anywhere else you want to listen all over the end. Don't miss it. Check it out. Before we get into it, are you ready to jumpstart your testing process? Meet Conform IQ, the leader in AI powered test design automation. With Conform IQ's automated test design process, you could generate faster and better tests without scripting. So experience seamless script scriptless automation and join the smart AI LED testing revolution today. See how Conform IQ can transform your testing process by requesting a free demo. Now support the show, head on over to testguild me aidemo and see it for yourself. Hey Mark, welcome to the guild.

Hey Joe, thanks for the intro and happy to be here.

Awesome. I think before we get into it, maybe you could talk a little bit about your journey, how you got into testing.

Well, I got into testing when I joined Conformi a little over 12 years ago and been involved in a variety of different software companies. But ConformIQ has been focused on delivering high technology advanced testing Solutions for over 20 years and I joined 12 years ago.

Nice. So what's interesting is I believe you know Conform IQ has been using AI for a while now, but recently with the rise of Gen AI, I'm just curious to get your opinion on where we were before and maybe how gen, how it changes things or how it doesn't change things or how it might be different from what we've seen in the past.

Yeah, interesting question. Something I've been contemplating for a while now. Yeah, certainly Gen AI is.

The proud new baby in the AI world. But interestingly, AI has been around forever. Even when I was working on my MBA in the early 80s, I did a project on AI. So the technology's been around a long time. The question is why is it so much more prevalent and so much more interesting today versus you know, what it was in, in years past? So but interestingly, you know, AI has been used in testing and it has actually it's been used in Conformiq's products for over 20 years. The question is, or the point is what type of AI and you know, the kind of AI that I think people are familiar with but you know, don't kind of consider as part of the new revolution of Gen AI are things like, you know, natural language processing, you know, things like vision recognition systems. And interesting one is symbolic AI. Symbolic AI is actually kind of a technology that was most visibly deployed probably in the late, I think 90s when IBM built a product called Deep Blue, which was the system that beat Garry Kasparov in chess. And you know, that's a technology that we've had in our product, you know, for years. And you know, the way I look at it versus the older technology or the older AI, you know, versus the new AI is, you know, certainly the, the older AI was more buried into the solutions that were delivered and not kind of the leading, hey, everything is AI type.

Recognition that Genai has in its product. And the way I look at it, we can figure out how to leverage the value that AI has been delivering for years. And I can talk about that, how I think it compares to Genais and then talk about the combination of the two and how we can deliver even more value to our customers that need to do efficient, high reliability testing.

Nice. So you make a great point. AI has been involved in a lot of these testing tools for years. So what made Genai put the spotlight on it? Is it because it's easier to use and see than what people may not have known? Like, hey, this application was already using AI because they didn't see it though, they didn't know the power of it?

Yeah, no, I think the whole ease of interaction and the availability of Genai today is something that is, you know, kind of consumed and infatuated the world where you know, before the usability and ease of access for AI was more buried into products like natural language processing and symbolic AI. It, it's used in a platform or an infrastructure. Now Genai is its own platform and interestingly can do a lot of the things that the AI that's been around for a while can do. But on the other hand it also presents some risks and some variabilities. You know, that the predictable, highly reliable know AI that's been used for years that you know, anybody who wants to embrace it needs to be aware of. You know, symbolic AI is really more of an embedded technology, you know, so that's why symbolic AI doesn't have the recognition that Gen has because Geni is, hey, it's a user inter, you know, you know, interaction and easily accessible to anybody who wants to use it. Where symbolic AI is, you know, a technology that's embedded into your model based testing framework or a test case generation framework, you know, so you know, the discussions we had up front where you say, oh, I'll use symbolic AI versus Gen AI, well it's not really that way. So I think we need to be, you know, clear that I don't choose one or the other. It's just that symbolic AI is embedded and you know, the natural language processing stuff is embedded in these test case generation solutions where you may not know you're using AI. That's why one is prevalent.

Or one is less prevalent. And Genai is really prevalent because you're all about using AI versus getting a problem solved.

And once again we're not saying one's better than the other. I guess it's different application. But is one safer or more private than the other? Like if someone's concerned about Gen AI leaking things I guess or training models, is there a one to one, is an embedded symbolic AI better for that or more private or safer?

Yeah, that's a good question. Symbolic AI is an embedded technology that is controlled and what are they? You know, the accentures and the IBMs of the world call it responsible AI, you know, is it's very well defined and doesn't have the security issues and exposures that a gen AI can have. So you really, you know, symbolic is controlled and well defined. You know, if you look at Genai and how it's implemented internally, if you are deploying that technology within your organization, you have to ensure that the models you're using, the LLM the large language models are constrained and implemented in a way that they don't impact customer data and those kind of things because if you don't do it intelligently, you are potentially exposed there. And symbolic AI does not have that issue.

All right, so that's a good point. When you see discussions around AI and testing, a lot of times people pick on Gen AI for, for things like hallucinating. And I think they equate all types of AI with gen AI and it sounds like Gen is just a sliver of the whole AI pie.

Yes.

And that you've been using it for years reliably within the product, it sounds like. Is that correct?

That's correct, yes.

Nice. So. So maybe some examples then of how you were using it, like for testing specifically before Genai, you may not have known.

Yeah, so you can relate the two. But we've been using it for automated test case creation for years. Genai does that. Yeah, interesting. The other thing is the automatic script scripting generation for going to automated test execution. You know, we've been using, you know, nlp, natural language processing to get to that, you know, in an automated way for years. And if you look at the test case generation that we do with the symbolic AI, it has some unique characteristics. You know, one is it creates a deterministic set of test cases. And what does that mean? You know, it's, it's predictable. You present the, you know, the problem, the fundamentally the requirements, you know, into, you know, our environment and based on those requirements, it'll provide you 100% coverage of those requirements for functional testing and do that deterministically. So what that means is I have the same problem set, I get the same result. With Genai.

You get basically a different answer one day versus the other. Hopefully no hallucinations in there as part of the process. But in the symbolic AI environment, you get a predictable and our algorithms and approaches actually deliver an optimized set of test cases. So not just test cases, but we pretty much guarantee the least number of test cases to provide 100% coverage of the testing requirement that the test designer has. You can't get that with Gen AI.

So I would think you're kind of in a pickle because a lot of these tools that come up with test case generation are Gen A and they, they give a bad experience. But you've been doing it for years using symbolic AI, which sounds like it's built better for that use case. How do you explain that to someone? I mean, you just explained it great right now, but Is there like an example you give, like, hey, here's using symbolic AI compared to Gen AI and here are the differences.

Interesting question. So yeah, I think you have to step back and say, okay, what do the two approaches do? Test case generation. Okay. And you know, what's your goal with test case generation? Because if your goal is just test case generation, Genai is a perfect solution. You know, if your goal is to have an optimized set of test cases or a very focused set of test cases based on, you know, your particular objectives, Genai can't do that. You know, that's where the symbolic AI delivers, you know, significant value. The other thing that's important is the predictability and the repeatability of Genai. So if you have like compliance requirements or financial.

Testing requirements that present a significant amount of risk associated with the application you're developing and you really need to do some very focused, intense QA and regression testing, you can't necessarily use Genai to target specific areas like that. Where with the symbolic AI you have the ability to focus and target and optimize the type of testing that you're doing.

Love it. So I like to give people some tips if they're listening. One thing is if they're looking at AI solutions, they want to do test case generation. Especially if you're in a regulated environment, they should ask, I guess, is it using symbolic AI or gen AI? And if it's using just gen AI, they probably would want to be cautious, I would think.

My suggestion is they just be aware of what they get. With Gen AI versus symbolic AI, the goal is to have the repeatability and the definability of what you're testing and how efficiently you're testing it. You don't get that with Genai if you have a low risk website type application. And I just want to do some exploratory testing. Gen is great. You know, you don't, you don't need to deal with that. But you know, look at where does Gen deliver value and it can deliver value complementary to symbolic AI, which is, you know, something that I think everybody who wants to use AI should be aware of is, you know, where do I get the most value? And you know, frankly there are some challenges with using symbolic AI. You have to provide a well defined environment and a model based environment to do that, where, hey, I can use Genai to help me create those models and make things a lot more faster and efficient and copilot and assist the people to get to the value that the symbolic AI delivers versus oh, let Me use Gen AI to generate test cases? No, let me use Gen AI to help me be more efficient and create taking advantage of symbolic AI as part of the testing process. That makes sense.

It does, absolutely. So what is Gen good at then? Like, like you said, is it just give you ideas then that you know or create a model that you then use to train the symbolic AI like any. How does that work?

I think Genai is really good at making smart people smarter and more efficient. It's not good at making novices brilliant because we've heard of Genai hallucinations. Well, that'll probably contribute to that level of feedback you get. But Genai in a sense is working with somebody that gives you suggestion and not all the people you work with are brilliant. So you have to assess and really be aware of what it is you're getting from interacting with a gen AI framework like ChatGPT where with a symbolic AI and a structured approach your output is predictable and well defined and you know, reusable. Where the quality of the product in a using a symbolic AI is as I mentioned, well defined. Where it's not so much with Genai.

And it sounds like a lot of people when they, like you said, if you just want test case generation for the sake of test case generation, you don't care. Genai is just going to spit it out. But if you have like areas of risk and things you want just the, like you said, the optimal amount of test cases, symbolic AI is the way to go.

That would be our recommendation.

Perfect. All right, so I think we have the gist of the differences. I'm just curious to know, like how else can AI help us? I mean test case generation is big, but are there any other areas of testing that you find AI is particularly really good at helping people with? Yeah.

So the way we look at it at conformaq is we have a platform that we know works and we've been delivering for years and how can we make the usability of our platform more efficient? And we see a number of opportunities to use Genai to make existing high valued test case generation and script generation and intelligent.

Automated testing frameworks much more efficient. And how do we do that? So we've got some products that are focused on capturing requirements and using those requirements to build tests and so on. And then how to get from a BDD behavior driven development testing process to simplified automation process, you know, automated execution. Well, you can use Gen AI, you know, to fundamentally fill out the forms needed to go from hey, I know what I want to build I know what I how to test it, but I'm not an automation expert. But I can use Gen to solve that test automation process and with a push of a button, integrate the application under test with the tests that are the result of the behavior driven development and get to automated execution on my own desktop. That's really cool. So that's where Genai complements testing in behavior driven development. But now you think about behavior driven development, it's really kind of focused on scenarios and user stories and, and really doesn't take into account the entire system. You go from end to end with all the options and all the interactions and possible interactions that.

And these high intensity compliant and or financial testing areas like banks and so on. It doesn't test the end to end scenarios. And how do I get to that and do that efficiently and effectively? Well, what if I could take those user stories and kind of combine them together into a system level model and then take that system level model and apply symbolic AI on top of it to generate an optimized set of test cases that allows you to test your entire system, you know, and that's where Gen can contribute, you know, significantly to getting from the user story or scenario level to the system level modeling. And that's where we at conformiq are looking at providing those kind of solutions as well. So you take the user stories that you are testing, you know, at the unit level and combine those to create a system architecture and use the intelligence of figuring out how all those user stories go together to define the system level model. And that's where Genai can help a bunch. And then you have the environment that now gives you the test case generation and the clarity that you need to do very efficient end to end testing.

All right, So I just want to explore a little bit more just to get some clarity around it. So someone has a bunch of BDD feature files and they may be caught up in like maybe their particular feature they're releasing for that Sprint. Maybe kind of forget about the overall how does it fit with what's there already, what's going down the road. So based on that, symbolic AI is smart enough to take it and come up with a system level model, then it can maybe show them maybe areas that they missed or areas that over covered or under covered.

Yeah, so I even call it basically the symbolic AI framework because it's the application of symbolic AI that allows you to do this. But.

You think about, hey, I'm working on a Sprint and I need to test the functionality for this new loan that I'm going to, you know, process, well, that new loan fits into a loan portfolio. There's credit checks and there's all, all these other, you know, components of the end to end application that need to be considered before you actually release this new product. You know, so I do my user story and I qualify this person for the loan and, you know, we got them signed in and whatever. But that needs to be tested in the context of the overall system, which are a bunch of other user stories. And that user story needs to be integrated into the big picture. And the testing of that new functionality needs to be tested not only in its own environment or level, it needs to be tested in the context of, of the overall system. And that's what we're talking about here, is leveraging the work that's been done at the sprint level to drive the automated creation of test cases in an optimal way for the entire system. So we can use those user stories and Gherkin descriptions and so on to create that system model.

So then does the system model then create its own test automatically without you.

Having to write the features? Yeah, there you go. Yeah. So you're testing out the system at the unit level and then the system model creates an optimized set of test cases for testing the entire system, not just the component or the user story or scenario that you're integrating into that system environment.

Nice. So, you know, a lot of times when we talk about testing, a lot of people talk about shift left, shift right. It sounds like the system level model can help address both areas almost.

Yeah, I kind of take advantage of that description of shifting left. It's more about starting right, you know, and starting correctly, where you're capturing, you know, what it is you're building properly with the context. And that's where Jen and I can help a lot with. You can use the natural language processing capabilities of Genai to automatically generate the BDD scripts and the BDD tests and the BDD descriptions in Gherkin. So you can use Genai to do that. And by doing it and starting properly with the requirements of testers and the developers working together, you can fundamentally drive the testing process much more efficiently. Because. Because you have a well defined description to help drive the end to end testing as well as the agile testing that's required as you build with the product along the way. So the symbolic model based approach really addresses the efficiency of the regression testing that needs to happen at the system level. And the BDD approach provides the agility and the testing that needs to happen. At the unit level.

So I don't know why this idea just came to my head. I could be completely off base. Let's see. So if I have a system level model that understands the system, if someone checks in code that changes a feature that touches code B and Z, a lot of times people like, just run all the bd, run all the tests, we don't care. Does this help narrow it down? Like, hey, this was checked in. You only need to run these two test cases. You have a hot fix, run these two test cases, you're covered. Am I off base with that?

You're in the ballpark.

All right?

But you know what, you do it at the system level and you have the flexibility to say, oh, okay, I just updated, you know, a certain set of code and put that into the system. So I don't need to and test. Test the entire system. But, you know, I maybe need to, you know, see how the new code works with a couple of other components. So I can take a piece and specify a piece of that system, generate the test cases just for that new code and the environment that it specifically interacts with, and just target the testing, you know, for that particular area, you know, so you can, you know, test that new code at the unit level. But, hey, I really need to make sure it works in the context of the environment that it interacts with at the system level. But I don't want to run, you know, a thousand test cases. I just want to run, you know, 30, which tests the area that it's particularly impacted. So, yeah, you're in the ballpark.

What I love about this, though, is kind of a hidden feature. It sounds like, you know, I work for a large enterprise. We had eight Sprint teams, eight people on a team. It was chaos. It almost seems like the system level model. Once again, I could be off phase. Almost helps with collaboration or communication, where you actually have something you can point to, get people talking, thinking, innovating. Is that another, like, maybe not offhand, but another undocumented feature of this approach?

You know, interestingly, the collaboration is a key component I see at both ends of the, you know, unit and system level testing. Because collaboration AI doesn't really drive that. So you have to do collaboration. One, two. One of the unique components that we deliver as part of our BDD solution is not only a textual description that you get through Gurkha and coincidentally creates a model, so you have a picture and the verbal view of what it is you're testing and what it is you're building. And you get the same value at the system level. You have the visual model of how everything connects and flows. And there's. You talk about collaboration. Well, one of the real values of having a picture, being able to look at how things are connected and how things flow really facilitates the collaboration. And we've gotten that from our customers at both levels. At the BDD level and the system level, having that picture in the model delivers real value. So, yeah, you hit the nail on that one.

All right, so another one then. Once again, random thought. You go into like a sprint planning. You have like eight teams using a visual model. You could probably say, okay, my story actually has a dependency on this team. Is that something it could help with collaboration or help planning?

Oh, yeah, absolutely. And having the model view of how things work together helps significantly. And not only at the front end, but at the back end where you verify and validate that, hey, I built this and I did these things and I made sure that we expected this information from this group or this function that was being built. So, yeah, absolutely. The visual model provides a significant value in the collaboration side.

I think the past few questions you kind of addressed how AI isn't really replacing people, it's replacing tasks that are kind of grunt work and kind of making it where collaboration is becoming now at the forefront, which it always should be, rather than the grunt work. That's the message I'm getting almost.

Yeah, so, absolutely. I don't know that it replaces all the grunt work, but I think it more makes smart people smarter. But you got to have collaboration regardless of whether or not you have AI or not. And God forbid, AI provide the innovation function at some point, hopefully that'll also come from real intelligence versus artificial intelligence. But collaboration has to happen regardless, and the frameworks and tools that we provide really enable that collaboration.

Mark, I'm just jumping into the mind of a listener. They're probably saying, oh, this sounds great, but it won't work for my application. If someone wanted to check out conformiq, is there a way they can actually see how it works and how it could apply to their particular tech stack? Maybe.

Yeah. It's interesting. Depending on where you're coming from, you know, if you're looking at the system level model, you know, the way we engage, there is more defining a proof of concept, you know, at the BDD level, you know, it's just, you know, go to the Elastic Marketplace and download us and, you know, start using it, you know, so depending on the approach and the problem you're trying to solve, there's a lot of, you know, ways that you can get access and you know, if you're interested, you know, we have a lot of videos and nothing better than a real time demo to take a look and see how all the approaches work together and have a discussion about where to get started depending on where you're coming from.

Okay Mark, before we go, is there one piece of actual advice you can give to someone to help them with their AI automation testing efforts? And what's the best way to find or contact you?

Well, hey, check the link below.

But no, let's have a discussion. Let's engage. Reach out to conformiq.com and we'll set up a demo and discussion to talk about how this best applies to your environment.

Thanks again for your automation awesomeness. The links to evident value we covered in this episode and head on over to testguild.com a506 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.