AI-Assisted Testing with Mark Winteringham

In a land of testers far and widely journey, seeking answers, seeking skills, seeking a better way. O the Test Illumination Testing Podcast Guiding testers with automation awesomeness from ancient realms to modern days, they lead the way. Oh, the Testicle Automation Testing Podcast.

Hey, if you want to learn more about Generative AI and software testing, you've come to the right place. Today we talk with Mark Winningham all about his new book Software Testing with Generative AI. You probably know him, but just in case if you don't, Mark is a quality Engineer, course Director and author of the book Software testing with generative AI and the book testing web APIs. He has over 10 years of experience providing testing expertise on award winning projects across a bunch of different technology sectors. He's also an advocate for modding risk based testing practices, holistic based automation strategies, behavior driven development and exploratory testing techniques. You can find them all over the socials. We'll have a link for in the show notes, but you definitely want to listen all the way to the end to get the most of how you can leverage Generative AI for real with testing. No bs. You don't want to miss it.

Check it out.

Hey Test automation engineers, are you tired of maintaining in house grids? Are you struggling with limited device coverage? Say hello to BrowserStack Automate your ticket to effortless cross browser testing. Get instant access to over 20,000 real devices and browsers and you can run Selenium, Cypress, Playwright and more with even better minimal latency. So get crystal clear test summaries and powerful debugging tools at your fingertips. You also get interactive debugging, web performance reports and support for advanced use cases like two FA and payment workflows. Scale your testing without the headaches. Browser Stack Automate because your apps deserve the best rated Supercharger testing. Try BrowserStack onto me for yourself today. Head on over to TestGuild me BrowserStack and check it out for yourself. Hey Mark, welcome to Back to the Guild.

Hi Joe. Yeah, it's good to be back again. Third time now. Fourth time.

Yeah, at least it's been three or four. So it's great to have you back.

I'm actually wearing my Selenium Conference T shirt.

Oh, nice.

From the last time we did this when we did the livestream. So perfect.

Perfect. Yep.

In honor.

Sweet. So you know Mark, you're always busy and I'm always curious. I asked you this last time, why write another book? I know you took like a long time to write your previous book and I thought, oh, there's no way he's going to do it again. And a year later you are a full time writer. What's going on?

Yeah, I find myself asking myself the same question quite a lot at the moment. As I near the end, the light at the end of the tunnel is almost there, so it's sort of that final push. So you do start asking yourself the same sort of questions. I think ultimately it is just down to the fact that I enjoy writing, I enjoy the process of putting this sort of material together. I enjoy teaching, I've always enjoyed teaching. But what's been kind of different for this book compared to the last one? So with testing web APIs, I kind of went in with an idea of what the book was going to be like. I'd been teaching web API testing for quite a while, so I felt confident with what that was going to be. Whereas this one's been much more of a collaboration with the publisher and focusing on generative AI. Although the concept's been around for a while, the actual sort of day to day public usage of it's only been obviously in the last couple of years with the rise of things like ChatGPT. So it's kind of a greenfield sort of space to be exploring, which makes it exciting. And it's really been fascinating finding out about sort of researching the tools and how they work, but also as well, it can be a little scary. But fortunately a lot of the stuff I've done in the past around my attitudes towards automation kind of infuse in this book as well. So that's sort of given me a start for 10 as well.

That's what I'm going to say. What skills allow you to do this? Because as you mentioned, it's not like, oh, it's not API testing, everyone knows it, but at least it was a known kind of thing that you've done for years and years. Now you're dipping your water in something that seems to be changing every month, every how do you create a book or how do you research it to make it so it's going to be when it finally is released that it's on point and it's still relevant.

So I think first of all you accept that that's not going to happen, which is, you know, it seems like a bit of a cop out, but yeah, things keep changing and keep evolving. I would say this time last year it felt faster than it does this time this year. I think things are kind of sort of starting to coalesce and slow down a little bit. Enough For I think with the broad strokes to sort of take stock of how generative AI could be used in a testing context. In terms of research, it was a lot of conversations with people, people within the testing community. A lot of tool providers opened access and let me have conversations with them and pick their brains about AI. And it was sort of setting a lot of sort of passive ways of sort of learning about stuff. So lots of newsletters, lots of blog post reading, using things like ChatGPT as well to sort of dig into some of the meanings behind some of the tools and stuff like that. So sort of a combination of all those things. I've always been kind of an outcome driven person in terms of how I teach and how I write. So I always have a clear idea of what I want the reader or the person attending my workshop to do at a certain point. So although the kind of the information and maybe links to things change, you know, Google really are giving me run for my money. They probably will rename Gemini to something else just before the book goes out. Just have to go through and change it. From Bard. Those things will happen. But what I hope is that the core themes around the relationship between us as individuals with these tools will stay roughly the same. Because as much as the tools change, I don't think that the way that we engage with tools changes that much. There's a lot to talk about in that space as well.

So is there anything that did change? You said it's kind of slowed down from when you started till now that made you have to. Besides name changes, anything big? Has there been a greater adoption to it than you thought it would be at this point in time? Or is there anything different than your assumption was a year ago to where it is now with the AI, with testing?

Yeah, I thought I would see more stuff in the tool vending space around it. I thought we'd seen more of that sort of stuff. There was that big gold rush last year for sort of AI driven products everywhere and you've seen little bits and pieces appear in tools. And I was lucky enough to see some demonstrations of some of the things that tool vendors were working on. But actually it hasn't pushed that far forward. And I think that partially that's because there's the sort of we're in the trough or heading down into the trough of dissolution. But what I hope as well in a more positive way is I think that the people creating a lot of testing tools are starting to appreciate that this is not the center of your product. This is something that can be added to sort of elevate certain aspects of the product, which is similar to what I'm talking about in the book. It's not about replacing what you do, it's about picking those individual opportunities and then using these tools to help with that. So yeah, I would say I take that as a positive really that there's not been such a sort of, kind of cynical sort of land grab in terms of how these tools are used. But ultimately I think a lot of the other stuff kind of feels the same. You know, people have sort of engaged with these sort of tools in a shallow way. They've come up with a few foot thin things. It hasn't really worked out for them and then they've kind of either sort of dismissed it or moved on. Whereas yeah, I wanted to spend a little bit more, bit more time with it, a bit of a longer burn to sort of, to understand how these tools could really sort of be integrated into our work.

So what could someone expect to get out of this book if they purchase it?

So I break the book into three parts. So I've got a quite clear mental model in my head these days of how we approach it. So I always sort of say that value with generative AI is rooted in three principles, which is mindset, context and technique. So mindset is very important and that's rooted in the mindset of what the value of testing is and how we do testing. So testing isn't a monoculture of test cases, it's a holistic whole of lots of different testing activities from automation to exploratory testing. Then technique is things like understanding how to use things like prompt engineering. And also in the mindset thing as well is how do these large language models work and appreciating that they don't think like us, they are probabilistic machines and that they have their limitations as well. So in the mindset space then in the technique it's yeah, things like prompt engineering, it's understanding things like AI agents as well and learning how to. Like saying this today is like how do you sort of knock that needle towards, you know, like in your favor? So you know, if they're probabilistic machines, how do we sort of game the system so the probability is on your side in terms of the value you get. So lots of techniques that can be learned. There's and then context as well. So large language models are very general based. They're trained on such a massive corpus of data that it's kind of a bit of A garbage in, garbage out situation. So if you ask it to create test cases for an upload feature, it will just generically come up with a bunch of test cases because it wants to answer your questions for you. Whereas if we do something which is much more sort of test and upload feature, this is how it works. Here's some details about its technical specifications, here's some business rules that are in place, here's a bit of the code or the HTML, that sort of idea, you get a better response back. So we need to bake context in. So that's looking at that as a principle, but also tools like retrieval, augmented generation and fine tuning, because those are the tools that can help you with baking the context in. So yeah, it's exploring all of those and again, like, you know, demonstrating as well, like clear situations in which you could use these tools to sort of, you know, get the point across about how they can be useful in certain places.

Perfect. I guess before we go any quicker, any further, if someone's. Most people might know this, but just in case, how do you define what is a large language model?

So in my mind it is a neural network that has some sort of transformer model that's applied to it and then it's been trained with like, you know, billions, if not trillions of files of data, of text or images or audio files. And then inside that large language model, basically what you end up with is lots of individual nodes or parameters that have strong and weak connections to one another. So that's where the probability comes in that help determine based on the word that it's generating. It will determine what's the next word to generate, what's the most probabilistic one. So happy birthday is more probable than happy happy. I've run over your dog.

Day.

Terrible analogy, but you see what I mean. It's very unlikely to say that, especially on a podcast.

But that's ultimately it's probabilistic model, lots of parameters that connect it to each other in that way. I would say my understanding of these is shallow because I don't think we need to know the absolute intricacies of these tools because we are users of them. We are not people who are necessarily building them. But appreciating at least that they are probabilistic and appreciating that we should not be anthropomorphizing them. They are not us. They don't think like us, they are not creative like us. That is not a criticism, that is just what they are. And that is of value in certain places. And I think once you get that appreciation, you get a lot more value out of it. And it's not changed with any tools, whether it's AI or not.

I love that you addressed this pretty early in chapter two, I think with LLM and prompt engineering where you set up kind of like, you know, what is prompt engineering? How to get the most out of it. And also you talk about hallucinations. A lot of times people you seem pretty even center where some people are like, oh, it's just all garbage anyway, it's going to hallucinate. You can't trust it. And on the right you have. It's 100% true. You seem to come down the middle, especially in chapter two. Can you talk a little bit more about that? Like what people, what the expectations you're going to set, like maybe what skills they need for prompt engineering and then what things they need to worry about with hallucination. How common is it? Is it?

Yeah.

And is it all due to how well you do prompt engineering?

Yes. Yeah, I think it's fair. I do sort of, kind of sit in the center ground because yeah, it is kind of rooted in prompt engineering. They are the techniques that you can use, but really it's about slicing and it's also understanding what these models are very good at and what they're not good at. So some of the examples I've seen shared that sort of demonstrate its failings, I think are valid demonstrations of its failings. But don't use it that way. It doesn't mean it doesn't negate the fact that it can be useful in other ways, but equally as well. Again, it's that kind of garbage in, garbage out sort of situation of more like generic in, generic out. So if you are approaching your problem regardless of your tools, if you are approaching your problem from a very broad, high level, abstract way, then it's going to be hard to find a tool that solves that problem. Whereas I think if we get the slicing right and we have. So like, for example, like, you know, you see a lot of people sharing examples of how you can use them to generate automated tests. I still think that's to generalize. I want to use it to generate my page objects. I want to use it to help me build my utility code. I want to build prompts, unit tests as prompts with tools like Copilot to help me build my production code. I'm getting quite very specific slices where I think that it can be of use to me. So I Think that's kind of why I sit in that middle ground is because I think if you do try to go too generalized, you, you are either going to end up in the land of hallucinations because these models want to give you some sort of kind of response, some sort of kind of answer. And also, I just don't think it doesn't do the craft of testing any service because it is assuming that basically it is an inherently algorithmic activity when testing is not. It's both, it's algorithmic and heuristic. So yeah, that's kind of why I sit that sort of in that middle ground. So prompt engineering as a technique, that's like the sort of specific craft. But if you're not, if you don't have the right sort of kind of mindset, the right sort of eye to sort of apply things like task analysis to the things that you're doing to break them down into specific chunks, then you are going to run into problems like hallucinations. You are going to have issues where it becomes less value to you or you end up over trusting these models and they lead you down the garden path the wrong way in your testing. Absolutely.

I love how you have to understand what the large language model you're dealing with does. I know a lot of people were like, like chat GPT 3.5 is, is bad with like math and figures like that. And so someone would do things with it and say, look, see, you can't figure this out and say, well, it's dumb. Rather than realizing what it's good for and show an example of what it's good for. It's almost like they set up the straw man for it. So I don't know why I put on that rant, but understanding what your, what you, what it is is going to help you, I think, be better with it, I guess.

Yeah, yeah. And it's again like to play the devil's advocate, that isn't an invalid way to prove it. So there are a lot of papers out there that research these models and demonstrate how they're not very good at planning. And there's a real debate in that sort of space about whether these models are actually creative, can actually help you plan. But really, again, if you don't think of them as planning tools, but think of them as assistants and you're doing the planning. So I think of it as like an area of effective model. I'm planning, I'm making the decisions and then I use these tools to kind of enhance the different aspects of the plan that I'm implementing, then they can be really valuable.

So how do you know what it can and cannot do then? What's realistic? Because the opening of my show was created by AI. All I did was say write a song about automation, Guild, boop, spit it out. Like things like creative that you thought wouldn't be replaced are being replaced. So is that creative? Is it garbage? Like, how do you know what to use it for? I guess. Does that make sense?

Yeah, I think it depends on what you're. So I think something I've been thinking about a little bit is like, where do quality characteristics fit into this? So for you, that thing that you've generated, you know, I'm going to assume is ideal for you because it's something that serves the larger product, which is your podcasts and the conversations and things that are going on there. Whereas if maybe it was sort of, you know, this is a band that's trying to get their first album out and they've just literally just got AI to generate all of it for them, then there's less sincerity there. Like the quality characteristics change and shift. You're looking for a band, you're looking for musicians who have channeled some sort of artistic sort of impression through their instruments. But that's all completely, completely gone. It's why I think, for example, in genres. So I've mentioned this before in the past, Rick Beato, the music producer, he's got big music producer on YouTube, does all these sort of videos and stuff like that. He talks a lot about how AI has slowly crept into the music industry. So things like auto tune are algorithmic based. And what it's done is it's nudged us ever, ever, ever closer to just fully AI generated. But I think that if you try to do that in other genres of music, you probably get more pushback or you'd probably get people noticing it more. So I do think like quality characteristics and you know, we'll just talk about music here, but anything like this, you can start to see that quality characteristics, context matters in terms of how you react to what's being produced. I slowed for a second there because I was reminded as well. A great example is the inverse. So someone would create a piece of work, a written piece of work, they were writing part of their novel, they fed it in and they said do the next two chapters and then deliberately didn't write it like that. So they use it like as anti creative. Because they were like, well, what it's coming out with is formulaic. So I can use it as a guide to go somewhere else. So again, it's that sort of how you approach it. We still have a say, we still have an impression. We are interpreting whether something is art or not, whether it's useful or not. The challenge again is making sure that we have a clear picture of what that is. And slicing can help because if it's not a broad, broad thing that we're like, I don't know, could work, might not work. Whereas if you have clear, distinct ideas of what's right and what's wrong, it can be easier to react that way, make decisions, makes sense.

So I think you've covered a lot of this in the. Like you said the first part. It's all about like mindset type of things. When you get to context, then can you talk a little bit about maybe the things that can help you with. Like from some examples of the chapters I think are really good, is rapid data creation. I know a lot of people struggle with data creation. I was soon to be really good with data creation. But maybe some examples of what you think it's good for. Because you mentioned you don't think it's really valuable for creating all the tests for you, but you're almost using it for more specific slices of activities.

Yeah. So the data one, I think is one of my favorite chapters because it does clearly show how data management is one of the hardest things in testing at all. But actually focusing very thinly sliced in that space, you can get a lot of value from LLM. What's interesting like on the context side is things like one of the challenges of data is complexity and structure, for example, or the fact that you want to do things like data masking, or you just want to generate some data, but it hits a lot of different tables. So things like retrieval, augmented generation or RAG can be useful because you can actually allow it access to your database and maybe say generate me a bunch of records based on these rules and it will go and extract some example data from your database, pull that into your prompt to give it that much more context, and then you'll get some sort of response back that's much more useful for you. If you care about certain domain words and want relationships between keywords and domains, then things like fine tuning can work because you're basically pushing those weights and balances towards your context. You don't teach it knowledge, but again, you push that needle towards your context. So yeah, ultimately that's what context is. It's not just being aware that you need to add some information, but it's also knowing what's the right information to add as well. There's a real interesting conversation as well with things like Gemini 1.5. You know, it's got a context window of a million tokens. So you could literally put a book in there and then say, you know, I need to know a specific piece of information in that book. Is that good? I don't know. It may be. It might not be. Would it be more effective if I just put the chapter in or the paragraph in? I don't know. So you kind of got to experiment with those sort of things. So, yeah, so context is about being aware of adding it in, but also having the ability to detect what's right and what's wrong. And you can use tools for that or you can kind of use your own sort of judgment.

So great chapter. People definitely need to get the book to get their hands on that. Another chapter that stuck out to me was you have one called Assisted Exploratory Testing with AI, which I was kind of surprised by. Maybe a little controversial if you just looked at the title. So maybe give a little insight around that.

So, yeah, again, it's the same. The same things like exploratory testing. I don't want an LLM to be an exploratory tester, but if I have some AI agents or some prompts that are available to me, and as I'm exploring and I'm like, so we talked about the data example. So you're, oh, I just need 20 records quickly set up, but they need to be unique. You can use a prompt to generate all that for you and throw it in. You're writing your test report notes, and you need to convert them into something that's a bit more legible or something that has like, some sort of kind of sentiment analysis to sort of say whether the quality is increased or decreased, and just sort of add that into your report, then those sort of things can be used as well. So the whole of that chapter, again, is this slicing mindset of. And I do a case study of where I do an exploratory testing session and I demonstrate where I'm using these prompts. So, yeah, I like that chapter because it's kind of building on some of the stuff that we've learned earlier on because we're pulling from, like, the data side. We're pulling from the. Oh, suggest me some ideas. Here's some heuristics, here's some things I've done. Suggest me some new ideas. Break me out the mold of My testing, it's sort of kind of combining those things together and like using tools during testing sessions is a big passion of mine. I love a good session where you use lots of different tools. So, yeah, that's what I'm trying to sort of kind of convey in that chapter of. Yeah, there are lots of different ways in which you can call an LLM and maybe like I said, you just have a load of prompts saved somewhere or you can use agents that can help you with these sort of things as well.

Great. And before that, you also have a chapter on UI automation, how to improve UI automation with AI. Can you just give a few more examples? I know you said like, help you write page objects and like small slices like that. Does it help with anything else that people may not be familiar with? A lot of people just think of like, it'll help me find a flaky locator. But are there any other use cases?

Yeah, so it's. So I think, yeah, that was one of the key examples that I put in that chapter. And then again, kind of going back to data, so integrating with things like open API platforms to call a model that's sitting on a platform somewhere and get some sort of retrievable data that way. There's some of the stuff that I didn't really kind of, because I just didn't feel like there was too much to go into, but I think is interesting is kind of using it for analysis, for results as well and giving you sort of interpretations there. And as well it's sort of, you know, it's a combination of that, but it's also kind of combination of like leveraging things like GitHub, Copilot, those sort of tools as well. So using. So this is the thing as well. Like we tend to talk about LLMs as singular things, whereas I think some of the successful teams, whether you're using it to assist your own work or building products, you use different models for different purposes. So I might use something like ChatGPT to like say, generate that page object model, but once I've got that in place, then I can use GitHub Copilot to quickly generate the code within my IDE. As long as I've got my page object open in my ide, then I can then fly through and then I can start baking in the business rules. Because again, goes back to context. Because if copilot starts going, oh, hey, you want to enter a username? Right? Here's my suggested username. Oh, hang on, you've given me an email address and we actually do employee IDs or we do usernames. It doesn't know those sort of things. So using different models to help me get certain ways, but I'm still jumping in at different points to kind of sort of break the rules in there as well.

Nice, nice. So, you know, you probably get asked this all the time, besides prompt engineering, what skills do testers need to know about to work with AI? I would think they'd have to become better testers because then AI isn't going to replace testers. But maybe I'm nuts there. And also, like, do you see people hate this term manual testing, but do you see that going away in five to 10 years? Where do you see how can people use AI better? What skills do they need and will they eventually be replaced by the AI?

Yeah. So I really like the idea. I think I've mentioned this nearly on every situation where I've been interviewed. I tend to be talking about Nicholas Carr's book the Glass Cage, and he talks about the principles of algorithmic problems and heuristic problems. So when you talk about manual testing, I think about it more in that sort of heuristic based testing with elements of algorithmic tasks within it. And sometimes the needle may be moved towards algorithmic more, sometimes less, depending if you're doing exploratory testing versus test cases. So again, it's that sort of being able to identify when you're in a situation where you're doing something that's algorithmic, so clear steps can be explicitly described. You can kind of tell when you're not following the path versus the heuristic stuff where it's sort of more creative, more freewheeling. So I think we need to get better at being attuned to that. And I think that's kind of one of the things I've been talking about for a long time is that that sort of idea of using any sort of kind of test, automation, it's great in the algorithmic parts, but it's not so good in the heuristic parts. So, yeah, that kind of mindset, you know, there is the potential. I've been thinking a little bit about this, about how it could open up automation more to new testers or like to those who have sort of shied away from the sort of more technical aspects because, you know, you can basically provide it with your instructions and then it turns the syntax, you know, converts it into the syntax that you need, so you don't necessarily know what the special incantations are. To make the script work. But you can at least plan it out in your head and describe it so that a tool can kind of convert it. And it's the same things for code reviews as well, like being able to provide blobs of code, assuming you're allowed to do that, but providing blobs of code to give you feedback that way. So I think that there's more benefits than there are risks to testing roles. But I think again, it goes back to mindset. We need to be clear about where they have a purpose, where they have a use and where they don't. And try not to get sort of too caught up in the hyperbole, regardless of perspective.

Nice. All right, so we covered part one and part two of the book. You need to grab it to really find more. But like part three, let's wrap it up here. At the time of this recording, it was called customizing LLMs for testing context. I assume it still is, who knows?

But when people hear LLMs, they're like, well, it doesn't work with my company because it's not trained on my data, my situation. So therefore it's useless. You have like, like three, three chapters here. Like one is customizing LLMs to help you better with testing and how to fine tune LLMs with business domain knowledge. Maybe can you like sum up the third part a little bit more as a teaser so people can get their hands on the book to learn more what you're talking about here?

Yeah, so the first chapter kind of puts forward the argument of context is needed, all the things I've kind of talked about. And then it sort of gently introduces the two approaches that we can apply, which is retrieval, augmented generation, so we can add more context to the prompt and we can do that programmatically. So we can either connect it to log files, databases, things like vector databases that we hear a lot about when talking about sort of rag. So it gives an introduction to how that works and gives you the opportunity to build your own. It's not going to make you some sort of rag expert by the end of it, but it's enough for you to appreciate when that approach can be of use, to open up context and things like that. And then the final chapter, yeah, goes into fine tuning and it's the same sort of thing going through the basics of how fine tuning works. Letting you have the opportunity to fine tune something, but not necessarily to make you an AI engineer or a machine learning engineer. Just enough that you appreciate the complexity of fine tuning, but also that you don't teach a large language model through fine tuning. You just move the weights and balances in favor towards your context. So yeah, that's kind of what kind of part three is. It's sort of taking it to that next level as well.

Love it. So this is a must get your hands on resource. We'll have a link for it in the show notes. But Mark, before we go, is there one piece of actual advice you can give to someone to help them with their AI testing efforts? And what's the best way to get our hands on your new book Software testing with generative AI?

You can get a copy of my book on Manning.com it's still in early access, so that means that if you purchase a copy now, you get access to the digital copy now. And then once the book is finished you will get a print copy sent to you if you choose that option. But what's really useful as well is that you can actually read the book, drop comments in, give me feedback. And I do read it and I do react to it and I factor it into the book. Single bit of advice is basically don't be afraid to experiment with them. Don't be afraid to sort of try out different techniques and different approaches to and tweaking prompts. You know, it's where my journey started with this was just, you know, I was trying to get it to generate some SQL for me and it went horribly wrong. But it's from there, it's from that experimentation you learn more and you can sort of practice and get more engaged. And it's not something that's for the technically minded only. It is accessible to everyone and that's why it's so popular.

Thanks again for your automation awesomeness. The links to evident value we covered in this episode head on over to testguild.com a504 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or so a problem for the Guild community. I'd love to hear from you. Head ON over to testguild.info and let's make it happen.