Testing AI's Promises with Michael Bolton

In a land of testers far and widely journey, seeking answers, seeking skills, seeking a better way O the Test Illumination Testing Podcast Guiding testers with our nation awesomeness. From ancient realms to modern days, they lead the way. Oh, the Testicle Ovation Testing Podcast.

Hey, in today's episode, I continue my conversation with legendary software testing expert Michael Bolton as we explore the world of artificial intelligence and its impact on testing. Now, I know I tend to have a lot of real pro AI testing experts on the show. A lot of people say hey Joe, enough is enough with AI, so I want to give another point of view to balance things out. And I can't think of anyone better or more perfect to do that than than Michael Bolton. So is AI truly the revolution it's marketed to be? Well, Michael cuts through all the hype and reveals what he thinks these tools really are. What are they? Well, you need to listen to the show to find out. You'll also discover when AI can be genuinely helpful in testing and when it's likely to lead us astray. Michael also shares some fascinating examples of AI inconsistencies and explains why the overconfidence these system projects can be dangerously misleading, so whether you're concerned about AI replacing testers or looking to incorporate these tools into your workflows responsibly, this episode provides the critical thinking frameworks you need. So stay tuned for part two of my conversation with Michael Bolton. Hey, before we get into it, quick word from our awesome sponsor for this episode. Hey, let's face it. Test automation with traditional code based automation tools require high coding proficiency, excluding non technical team members from the testing process. Steep learning curves, high test maintenance, and the need for skilled resources and infrastructure setup are other significant challenges. As a result, QA teams are forced to limit automation to critical applications, leaving significant gaps in automation coverage. So what do you do well? Introducing BrowserStack Low code automation It's a powerful end to end low code testing platform built upon best in class browser stack capabilities that lets anyone create automated tests in minutes without coding expertise. With their intuitive test recorder, you can create your first automated test with ease. Simply interact with your application and capture complex actions like hovering, scrolling file uploads, et cetera, and generate test steps with meaningful human readable descriptions. Timeouts are intelligently configured for each test step, ensuring stability without manual intervention. Visual validation ensures specific UI elements appear correctly, improving accuracy and test stability. But here's the interesting piece Their low code automation tool is powered by AI. It uses natural language to create test steps and simplify automation of complex user journeys. When UI changes occur, Self Healing technology automatically adopts your tests to prevent failures. You can easily scale low code testing with automation best practices while bringing modularity, reusability and extensibility into your low code tests with advanced features like modules, variables, Data driven testing, and many more. Join over 50,000 customers, including some Fortune 500 businesses who trust Browser Stack low code automation and take your automation testing to the next level using the special link down below. I'm just curious to know, you know, with AI coming aboard, how much does it muck all this up? We talked what is testing? We talked about automation, but now I've been seeing AI testing AI automation like it's like everything now has AI. And everyone says well this is all fine and good and dandy, but AI is going to solve all these issues now with all these misunderstandings.

Well, what's our history? What's our history with claims of X is going to solve everything or X is going to do this or that? I mean look at self driving cars. My favorite example, because it's so evocative, seems that in 2014 there were lots of people making claims that by 2024 there will be no more human driven cars on the road. I mean people were making that claim and here's the problem with it.

There is a kind of Carol Cadwallader had a wonderful word for it broligarchy.

That sees a problem solves 20% of it the easy part of it in 5% of the time. Okay, well problem solving doesn't quite work by projecting the time it took to solve the easy problem into the time it takes to solve the hard problems. The hard problem is with self driving cars is that self driving cars or driving is not the business of keeping the car in between two lines on the road. That's the easy part to solve the problem is not keeping the car from hitting a car immediately in front of it when the car immediately in front of it stops. That's a problem and that's got to be solved. And enormous steps were taken for quite a while in solving that problem. But what people who are fascinated by technology often forget is that technology is always an extension of ourselves, an extension of us. And what is driving? Driving is not a mechanical activity. Driving is a cognitive and social activity wherein lots of stuff depends on communication between drivers. Lots of stuff depends on things that drivers do that involve things other than operating the steering wheel and the brake pedal, maintaining situational awareness, communicating via both behavior and gesture, and so on with other users on the road, working out as we're incredibly capable of doing. Humans are just amazing this way. And machinery is terrible at it, working out people's intentions. So what they focused on was, once again, as Harry Collins would point out, they're focused on the behaviors of driving and not the actions of it. Right. The behavior plus intentions. When it comes to AI, there's a bunch of different problems that come up. First is.

AI is a marketing term, not an engineering term, because any kind of reasonably sophisticated software can be marketed as AI. If we think of intelligence in terms of something that takes in some kind of a sensible. That sensible data, that data that we can take in with our senses and then processes it, somehow applies decision rules to it and then produces an action. So the example I like to use is Shazam, you know, shazam, the music software. You hold it up to the speaker in the pub and what's that song? And it tells you what the song is, and you think, wow, that's amazing. How clever it is, how smart it is. Well, you can market that as artificial intelligence if you wanted to. It'd be nobody stopping you. In 2018, I was working on a particular kind of artificial intelligence. Pattern recognition in brain waves for people who are unable to speak or unable to gesture. What this thing did was it through a.

A band that you wore around your head and sensors attached to the band. It would read your brain waves and try to figure out which of six or eight blinky lights that you were looking at on a tablet or a phone screen. And then a bunch of techniques would be applied to that data collection and through programming that was selected, rather than programming that was intentionally designed. It's how machine learning works. Essentially. You throw a zillion different possible candidate programs or candidate algorithms at something, and then you choose the one that fits the data the best. That kind of machine learning is something that people have classified as artificial intelligence, presumably because the process of algorithm development and selection of the appropriate algorithm that's been abstracted out by algorithms. And so we get a candidate, a machine learning model that does something and that has the feeling of intelligence about it. And then we come up with these things that generate long strings of text.

And from that, we look at these things which are models of, literally models of how human speech would go. And then we look at that and we say, wow, that looks a lot like human speech. Our brains say it must be intelligent. But our brains are wrong, because we know how these things actually generate text. They generate text probabilistically based on training data. And sometimes that gets refined a little Bit by various kinds of tuning and feedback, assessment or assay of this stuff that users supply. Is it what you were looking for? You know, thumbs up or not thumbs down. And stuff gets refined that way, but it's still just spitting out text. One of the things that we have worried about is that risk. The risk that what we see, which is very legible once again, and which is really seductive is not right.

Let's think about the problems associated with AI and what makes GPTs particularly problematic. The first thing is that they're algorithmically obscure. They're not written by intentionally and socially aware people. A machine learning model is generated and selected by various kinds of algorithms and we don't have the source code available so that we don't. If there's a. If it's producing unhappy output, we don't know exactly why. It's really hard to trace back through layers and layers of back propagation and data upon which it's acting. That interaction is really hard to trace and by and large we're unable to do that in any kind of reliable way. So what this means is that the product isn't easily, certainly not easily checkable. There's an obscure relationship between the input and the output. We don't know about the details of the process by which that's happening. So that's one kind of problem. Another kind of problem is that when we retrain the model or try to fix it in some way, we don't know the effects that retraining is going to have by changing the weights. In a large language model, we don't know how the change of one weight is going to affect the changes in all the other weights because of the phenomenally complex interactions between these things. Where we need huge farms of servers to process all the linear math that goes into those sets of relationships that spit out a string.

So we can't fix them, we don't know how they work on a low level and therefore we can't fix them. And therefore they're vulnerable to regression kinds of bugs. Bugs where it gets better in one way, yeah, but it gets worse in another way. On a social level, one of the problems we're facing is that there are claims of these things. Thinking like humans, there's very great excitement about how good the output looks and that causes people to turn what might otherwise be reasonable desires into wishes and fantasies. So anything that works great in one context or appears to work in one context is vulnerable. So we have to experiment with it really thoroughly. Another thing that is challenging about these things is that we are inserting them into places where humans once fit.

We're trying to put them into the human social order. But these things are not themselves social agents. They're not brought up like people. They don't have responsibility like people. They don't feel responsibility. They are not social agents, but they look like they are. And so we're. We anthro. Anthropomorphize. Not only do they look like they. They are, by the way, they'll say all kinds of stuff to seduce us into believing that they care for us. Apologies for any confusion. Right. Or. Sure, I can help you with that. Well, that's very nice, but it pulls psychologically on us in ways that cause us to read stuff in that's not.

There or builds trust where it shouldn't be.

Well, yeah, yeah. The important thing to remember about that is it's not the thing that's building the trust. Right. GPT is not building the trust.

Yeah, exactly. We are. Exactly. So we've got to be responsible for that. Now, there's a wonderful blog post by a guy named Baldur Bjarnason called the Large Language Mentalist. And the subtitle is something like How Large Language Models Reproduce the psychic Person's con. The psychic's con. Because psychics use the same kind of approaches. They'll give you an answer that looks really good or that sounds really good, but that isn't based on any kind of knowledge or understanding of you or what you want. But you. The. What do you call it? The psyche. You know, the person who is the client of the psychic, you fill in the details.

Right?

And when you fill in the details, all of a sudden the psychic looks masterful and brilliant and insightful when what's actually happened is they've said something that, for instance, fits the Barnum effect. You know, you are a reasonably diligent and thoughtful person, but every now and again your discipline slips and you do something careless and so on and so forth. Well, find a person on Earth that doesn't apply to. So these things are very sophisticated on that basis. They're really clever at making us feel good and it's a mistake, or at least a very risky to put them in places where a human wants fit. And we all know about this from our interaction with chatbots. When we're trying to get a problem solved or we're trying to get customer service, these things are enormously frustrating because they're not great with specific details and they're not great with dealing with our frustration with them. And we got lots of experiences and lots of accounts of that. But there's another aspect about it which is really bad, really bad. These days it's starting to dampen a little bit as more and more people are getting experience with these things and realizing that our, the things that we've been warning about for the last two and a half years have come to pass. There's lots of fear missing out. There's lots of investment, lots of money riding on this stuff and there's lots of weird, weird boardroom stuff that says AI is the next big thing. We got to get on it. Right? We gotta, we must apply it. Well, that's not engineering.

I've heard companies have a mandate. Thou shall use AI in your day to day work to save time for companies.

Yeah, I tell them to, you know, take a hike. As a professional, I'll use the tools that I believe help me discharge my service to you. And appropriate ways thou shalt use AI is just, is dumb advice in my point of view. It's like saying thou shalt use a marker instead of a ballpoint pen. You know, it's just why, you know. Now of course, if you are developing machine learning models, I guess you're going to be using them in your work. But the social aggressiveness and the corporate defensiveness about these things. Well, we spent a lot of money on this. We've got to, we've got to get that money back. It really buys into the sunk cost fallacy.

So I here's cases in which I actually fairly regularly use chat GPT.

I use it as a thesaurus. Sometimes, sometimes I ask the bot, can you this because this is something it should be really great at, right? It's a large language model. It's got thousands and thousands of instances of people using this stuff. So I'll ask it for a synonym for something. When I came, I'm trying to write something, I can't get just the right word. So I'll ask it for a synonym of something. And the results just are kind of underwhelming, unfortunately. And part of that is because it's based on stuff that people have said before already in the past and it's, you know, probabilistic in that way. So the, you know, the competition between a large language model and a thesaurus.

It'S.

Best 50, 50. Usually looking something up in a thesaurus is a wiser approach. If I'm doing something in testing work that requires me to write a very short body of very simple code that is easily analyzable and where risk is low. I remember one thing recently where I wanted to make a change to a whole bunch of PowerPoint slides and I've got code somewhere that iterates through a body of these things, but I was too lazy to, to look it up and to revise it. So I asked Chat GPT for it and on the fourth or fifth time it got it right, you know, but it's a actually a wash to determine whether I would have done better just going back to the code that I'd already written to do this stuff and reapplying that, taking out the special casing that was in it or something.

So yeah, I just want to make sure though, a lot of times people think you're against automation and you use it in the right place when it makes sense. So when can you use AI? Use AI all the time. I use it to take pictures of the calories of ingredients, of things, say which one is healthier? Or even if I go to a restaurant, I don't know what the dishes are, I take a picture of it and say, hey, what does this mean? And it gives me good results. So as a tester, are there any cases that you could see that it can help you?

Okay, well that's one nice example. One of the things that I actually an AI based application that I really liked, I think it's called Picture this. When you take a shot, a photo of a flower, a particular kind of flower in a botanical garden, which my wife is especially into. I remember using this thing for the first time in Butchart Gardens in near Vancouver. Near Vancouver, near Victoria bc, not Vancouver. And it was pretty good, so it seemed at recognizing certain kinds of flowers and identifying them. Here's the thing though, nothing in particular was on the line there, right? There's no risk involved. So I want to show you. First of all, I'll give you a set of things whereby I say, ah, go ahead. Use large language models or use machine learning models if you like, in these kinds of circumstances. One, I could do this as a build. Would you prefer that I do it as a build or instead of just blasting a list at you?

Yeah, that'd be better.

Okay, well, let's do.

Doesn't matter that much. Let's do it this way.

Okay.

Start and then change this all to on click and there we go. So when can AI in one form or another be okay? Well first of all, if you use the output for inquiry rather than control, we said the same thing about metrics for years and years. If you're Using it to help you frame better questions, to give you something where. Let's think about this in a slightly different way, then it's pretty safe. You know, it's okay for that when the output is used for discovery and analysis rather than for pushing aside responsibility. If it's used, you know, for that, for the purpose of delegating a decision to machinery, not playing. But when it's used to help us, you know, as a tool to help us discover things or analyze things, that might be okay. The unreliability turns into a problem with that, as we shall shortly see. In general, the simpler the model and the more constrained the set of features. That is the. If you like the. Not the hyper parameters, but the actual parameters, the things that it is processing when that's, relatively speaking, constrained or low, you're probably on safer ground there. And there's a wonderful pair of authors, Narayana Kapoor, who have written a book on this, and this is one of their essays on that subject. When risk is low, you know, when we're taking pictures of ingredients or taking pictures of flowers and using this as a classifier and we're saying, well, that appears to be an amaryllis, you know, nothing, nothing really at stake there. I'd want to be a lot more careful about it. When I'm taking pictures of mushrooms.

Right?

Right. And I'm considering eating them. That's the sort of thing where I want somebody with expertise to let me know whether this thing is poisonous or not. So anytime when there's no risk of loss or harm or wasted time or diminished value or bad feelings, opportunity costs, societal consequences, then we're probably not too bad. When the volume of output is relatively low and it's easy to scrutinize the output, again, no really serious problem there. When the risk is elevated, when the possibility of trouble is higher, but we are very careful with the output and we have it examined and scrutinized and vetted by people with the necessary expertise to make an expert judgment on it. Then again, not so bad. Could be okay. When variability is tolerable or even welcome. When we're helping it, you know, we're getting it to help us to inspire something or create something. Although I must say my experience is the level of creativity from these things is low. And it's by definition subject, because something that looks inspiring to me is basically the output of something that's been trained on a lot of artists, proprietary work. And my wife, who's a graphic artist, has a pretty strong objection to that. Various kinds of Societies and businesses have tried their very best all the way along to make sure that artists don't get any money for anything. So this really weaponizes that thing when the actual creativity isn't the point, but just getting jiggled is right. When you do the creative bit, that that's okay. When variation is all right because of our capacity to repair, to fill in the blanks, to fill in the gaps, to recognize the difference between what the machine does and what a person would do in a similar kind of circumstance.

Here is what I believe is actually the most powerful and interesting use of large language models. When we use them as a mirror or a lens, when we use them as a reflection or a focusing mechanism on what people are like. Now. Go ahead.

Are you going to give an example what that means?

Sure. The most famous one that comes to mind at the moment is the case in which Amazon in the 2017, 2018 period, somewhere in there might have been a little earlier, decided that it was going to apply machine learning algorithms to selecting who should get hired at Amazon. And you probably remember the outcome of that. The outcome of that was the ideal Amazon employee would be a Programmer dude between 25 and 35 years old, always a guy and typically a white guy. Well, what. This is not good advice. What it is, it's a mirror on how Amazon actually hired people in that era.

Right. So when we look at the output from any kind of machine learning algorithm, what it reminds me of that line from the movie the Russia House, where one of the characters watches another character go into. He's talking to the Russians and he's got a list of things that the, the spies on our side want to know. And one of the spies on our side is not really happy with this idea. He says, I don't like lists. Says they tell you too much about the people who made them. Well, in, in this case, I think one of the more powerful uses of AI is to reflect on what we're like and what our world is like. That is super powerful. Now when it gets. Here's the instance of when I think it's problematic. Let's have a look here. I was asked to review a book.

And.

I'd asked to review that. I was asked to review this book and in the book there's a. An example of how to use an LLM to generate test data and, or to generate a. Yeah, an object. So that's the assignment, your JSON to SQL transformer. Convert the JSON object delimited by triple hashes into an SQL statement That will create a table to insert the transformed records into and then create insert statements to add each record to a database.

We think this is a perfect example of using AI. So I'm curious to know what your results are.

Well, this is the result from the book. Now, for people who are only listening, it's going to be hard to see this, but basically it comes up with three records for people and there's a data associated with the the person. And so that's one thing in the JSON one, it's, it's an array. So there's an array of three people and two of them have additional needs.

So I took that exact prompt and what I saw was three different. There's one from the book and I did two trials of this using the exact same prompt. And what we see are some pretty interestingly significant differences between each one. For instance, in the book the first name and the last name are characterized as varkar and of 255 characters each. In my first trial it's 100 characters. In my second trial it's 50 characters for each of first name and last name. The total price field in this JSON in the book and in my second trial these are decimals with two decimal points. In my first trial though, it came back as an integer.

In the third instance of a problem here. The additional needs fields in the book and in my first trial were Varchars of 255 characters and in my second trial Varchars of 100. Now this is sort of interesting because when we get down to the actual insertion of the records, this is what happens in the book. It's, it provides the data to this total price field as a decimal with two digits in the second. In my first trial that's what happened in the book. In my first trial it provides an INT value into an int field. In the my second trial it puts an INT value into a decimal field.

Interesting.

Now, is this a problem or not? My SQL is a little rusty, so I don't know how it would accept that or if it would filter it and reject it. But my more important point here is the level of inconsistency we're getting in the test data. Our experiences with machine learning models in general suggest that if your output is of a certain length, it starts to go bad. It starts to get worse and worse, more variable, more consistent. And by the way, that doesn't improve in any kind of consistent way between one model and the next, even when those models are supposedly incremental versions of the same thing. So 01 and 03 in the ChatGPT family and the OpenAI family of model models, those things have varying degrees of success and accuracy. But one consistent thing we've noticed is that when your output gets longer, the error rate gets higher.

Isn't this a great example though of AI assistance where you do the thinking, you don't trust the output. You look at the output looks good and you fix it like you did and say, okay, it gave me, got me started like it's a starter. And then I applied my thinking, knowing that it's not a thinking output. I don't know if that makes sense, but.

Well, that would be great if we did it thoroughly and reliably and consistently and from an informed perspective, it would be really easy for me, as somebody rusty in SQL.

Yeah. To get right. Right for us to fall asleep and these things that it. If they're nothing else, they tend to be very reassuring.

Yeah, right, right.

They tend not to be very. To prevent their own skepticism. And there's a report that. I saw it, I guess it was a LinkedIn post today, but it's based on a report from a week or two ago.

Where one AI company was looking at, I think, the O3 model and noting that a lot of what it said was simply untrue in terms of. Well, what did you do to get this result? Well, it says I created a Python program to create this result. Did you run it? Oh yeah, absolutely, I ran it. Okay, well, let's see the code then. So it pretty. It spits out some code and it's a. I think it's a code for factorizing a 512 digit prime number.

Or something. It's some. Something of that nature. Anyway, it's got a routine that starts, I think, with the first 10,000 prime numbers and tries to, you know, an easy way of deciding that this number is non prime is if it goes in. If one of the first 10,000 primes goes into it, then that number is non prime. Well, it turns out that it gives it this long list of the program that it ran and the output that it got. And it turns out the person who is actually testing this thing points out that this huge number is actually divisible by three.

Now it's funny because there's a really easy way to check that. Do you know it? Well, no, there's a much easier way to do it.

You look at the last three digits of a number and you sum them and if the result is zero, you sum them and divide by three and you keep doing that successively. So, for example, if the numbers are, let's see, is 999 divisible by 3? Sum it. Well, the answer is 27. And so, oh, with sum that, sum the digits in that, and the answer is nine. And divide nine by three. And yes, 999 is indeed divisible by three. So it's. Is it the last two digits or the last three digits? Last three digits, I believe.

It's embarrassing to say it this way in this context. I don't remember. But if the last. I believe it's the last three digits. If they're divisible by 3, then the number is divisible by 3 in the same way that it's easy to identify if a number is divisible by two, if the last number is even. You can do that. So anyway, Buddy points out as he's testing this that the number is indeed divisible by three. Oh, I apologize for any confusion, you know, that sort of thing. So we, I mean, when we give it a hard problem, we're setting ourselves up for it to mess up a hard problem in a way that is going to fool us through the confidence that these things are designed to exude. We're going to get fooled by its confidence. So that's one of the really hard things about testing a large language model is that the output tends to be voluminous and requires an immense degree of scrutiny by means other than a large language model. Otherwise, you know, people say, well, if something is 90% correct, and then we run it against something that's also 90% correct, then we're good. But the product of 90% correct and 90% correct is not 10%. You know, 90% applied to the 10%, it's 90% applied to the 90%. So two processes that you put together that are 90% correct give you an 81% possibility of correct output or 81% kind of reassurance of a correct output. It goes the opposite way from the way people would think. So it's devilishly expensive and difficult to test these things. And the easy way to get around the problem is not to use them when it's not a good idea to use them. And I went through a list of ways, places in which it's okay to use them. Well, if any one of those things is not okay, then it might not be a good idea to use them.

All right, so, Michael, obviously, you know, the buzz is AI testers are being burdened with having to now test AI. So are there any Traditional approaches to actually testing AI that you could recommend?

Well, one traditional approach that actually works is to recognize that our job is not to confirm that everything's okay, but to go through a process of exploration, experimentation, experiencing, with a focus on finding problems that matter. Because that's going to work. Because our job is not to confirm that everything's okay, but to highlight trouble.

One of the problems with a certain kind of traditional approach to testing, the idea of checking the output is that the output from AI is non deterministic. So there's no way of doing a unit testing in the traditional sense for a machine learning model when the output of that model is going to be generative and text based. Right? That you can't do a unit check on something unless you ask it to produce stuff in a very specific way in a JSON, then I guess you could output the unit check from the JSON. But to generate a JSON, maybe you want something a lot more normal than using a GPT to do it. A little bit of code in the scripting or, you know, a programming language of your choice. But unit testing isn't going to work for that. The kind of testing that's in what we would refer to as the discipline frame of testing the low level programmer style stuff. There's no low level programmer at work here, no conscious one. So that stuff isn't going to work. And it's not enough to show that the machinery can work because people are going to fix up, they're going to repair in their minds the output from it. So anything that's based on determinism or precision or something that's not generated in a much more free form kind of way, like LLMs generate free form text, they generate jsons too. And those jsons are checkable, but not the lower level process of the data creation that, that sort of stuff. You got to, you've got to look at the output, you've got to examine it and you've got to. If you've got a program, you've got to test that coming out of a GPT. I mean, if you've got a program coming out of a GPT, you've got to test it carefully and review it and look for problems in it. Otherwise you'll miss those problems.

What we perceive of as a magic trick, when we see a magic trick and we have this perception that something magical happened, that's not actual evidence of magic, right? We're filling in the stuff that provides a magical explanation for it. And then another general thing is that just because we don't understand its behavior, it doesn't mean it's intelligent. A lot of the time we say, well, we say somebody's intelligent. We don't know what intelligence is. We don't really understand what intelligence is. But if something appears to be intelligent, then we infer that it's intelligent. But that's actually kind of risky because of the magic trick phenomenon. Something that appears to be is not necessarily what is.

However, I guess I could say that automated checking isn't going to work very well for LLMs. But the processes of using tools to help us analyze results, to help us investigate results, to search, sort, filter, parse data, those things are all still available to us as helpful. Nonetheless, the volume of stuff that's coming from an LLM and its inherent variability is going to make that work hard, which actually is kind of fine with me. I like that kind of hard work. There's some people, though, who just want that stuff to go away. And I got sad news for them, unfortunately.

Absolutely. So, my God, it almost seems like it's going to make There's a need more for testers. A lot of people say, am I going to be replaced by AI or all this? It sounds like not only are you not going to be replaced with all this code being generated, all this gak, it's going to make testers even more important. So along those lines, if someone wants to learn how to be a tester as we defined it in this episode, how can they learn more? I know you have a bunch of stuff on your website. You do a lot of training on rapid software testing. Where can people find out more?

Well, my website is developsense.com I would also point people to James's website. He blogs a little less than I do. Satisfice.com we offer classes both online and in person. For public classes, I do the live ones, the in person ones, and James does the online versions of them and he runs those fairly frequently. So watch his site and the rapidsoftwaretesting.com website for those things for in house classes. If you're at an organization that's got a crew of testers that you want trained in the rapid software testing approach, James will do it online. I will do it online or in person, depending on what you want. I love in person work. To me, it's so much more engaging and interactive in ways that I think are really important to preserve. Being in the room with people has a certain kind of magic to it. Finally, this will come out. I imagine after this podcast emerges But James and I are working on a book currently slated for release in September of 2025 called taking testing Seriously. The Rapid Software Testing Approach. This book has been a long time in coming, and it's been. It's one of the things about writing a book is that you get to find out gaps in either your own understanding of something or your capacity to explain it. So James and I have spent untold hours arguing about what's going into the book and what's not going in and how to say it just so and just right. So it's, it's like a good meal in that way. You can't spit it out off the end of a fast food assembly line. It's. It's got to be prepared and to some degree practiced because we are trying to make a book that's going to stand the test of time. And that's tricky in a world that's moving all the time. So we're looking for things that are more or less eternal about the business of looking at products critically. Stuff that, that's going to last for a while.

Love it. So I'll have to have both you on a show to do a promotion for that book for sure. That'd be awesome.

That, that would be wonderful. And we absolutely welcome that and thank you for it.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to test guild.com a544 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges. We're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

Testing podcast with lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.