The Secret Life of Automation with Michael Bolton

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way o the Test Kilonomation Testing Podcast Guiding testers with our nation awesomeness. From ancient realms to modern days, they lead the way. Oh, the testing of Nature Testing Podcast.

Hey. Joining us today is renowned software testing expert Michael Bolton, who returns to the show after an eight year hiatus. In this part one of a two part series, Michael dives deep into software testing, reflecting on what's changed and what hasn't since our last chat. And also the impact of technologies like machine learning and AI and what it really means to do software testing in 2025. Also, make sure not to miss out on our next episode next week in which Michael takes a deeper dive into AI's role in testing. You don't want to miss it. Listen up and before we get into it, I just want to share with you a quick shout out to an event that I'm speaking at along with Michael Bolton. Hey, before we get into today's episode, I'm happy to let you know that Breakpoint 2025 by BrowserStack is almost here. What's it all about? Well, this may join thousands of developers and QA engineers from around the globe for three days packed with expert talks, interactive workshops and the latest testing trends shaping the future of technology. Hosted by BrowserStack, the leading cloud based testing platform, BreakPoint 2025 is your chance to explore cutting edge innovations in AI and testing, scaling, automation, accessibility and the real world stories behind the tools you rely on. Every day you'll hear directly from industry experts at top companies like Atlassian, Amazon, Walmart, Reddit, Glassdoor, PublicIT, Sapient plus, you'll connect with the testing community and learn how BrowserStack's platform empowers teams with instant access to 3,500 real devices and browsers to scale their testing workflows. Whether you're leading a QA team or deep in automation workflows, breakpoint is your opportunity to discover what's next in testing, connect with peers and walk away with actual takeaways that make a difference. And I'm even more excited because I'm actually going to be speaking at this event on how to become an AI driven testing leader. Best of all, it's absolutely free, fully virtual and tailored for busy professionals like you. Don't miss out on Breakpoint 2025 by browser stack Where Testing meets the future. Register now using the special link down below to secure your spot and hope to see you there. Hey Michael, welcome back to the Guild.

Thanks Very much. It's great to be here.

Great to have you. It's been a while. I just looked at my calendar. I think last time we spoke officially on the podcast was 2017, so I can't believe it's been that long, but it has, so that's crazy.

Eight years.

Eight years.

The kids in school already at this point.

Wow, that's nuts. Nuts. So I guess a lot has changed and a lot hasn't changed in software testing. I want to ease into some topics I want to get into. But like, I guess, what have you been up to in the past eight years? Anything new or exciting?

Well, there was a. There was this pandemic thing. That's true. Seem to recall lots of stuff. Obviously in 2017, or was it 2018, I was working on a project with a colleague that incorporated a certain form of machine learning, certain forms, in fact, several of them of machine learning and AI. And we could see this starting to creep up in the zeitgeist. But then In November of 22, along came this version of Chat GPT, which took the world by storm and got everybody terribly, terribly excited. And I started getting pretty seriously bored with it within a year. But the rest of the universe seems to have gone nuts for it. I've been traveling, of course, with, in the periods before and after the pandemic. Just finished a.

Trip that included Abu Dhabi in New Zealand and Istanbul and a couple of places in Romania. So the other thing that I've been interested in lately and developing with my colleague James Bach is a class. We're now teaching a new thing on testing and automation, avoiding the traps. At one point I want to reconsider that title because in terms of a little bit more in terms of what we're actually promoting and advocating in terms of tool use. But I've been having a lot of fun teaching that class and showing to people what we've learned about how to use certain kinds of tools, especially focused on data. That's been a lot of fun lately.

Love it. So I'm going to dive into some of those points, but I think probably, I think you have a unique, or maybe not unique, you have a philosophy, I think, on what is software testing. I think a lot of people nowadays don't actually have a really solid background in software testing. So I want to start on defining what is software testing and then start diving deeper into these other forms of testing. So I guess question is, how do you define what is software testing?

Well, testing is evaluating a product by learning about it through experiencing and exploring and experimenting. Now that includes a ton of other stuff. Examining, explaining. It includes making conjectures about the product, making inferences about it, modeling it, questioning it, studying it, deliberately manipulating it, generating ideas, overproducing ideas, abandoning ideas that we've overproduced, refining ideas, expanding on ideas that we've refined and then re refining ideas that we've expanded on, navigating, map making, collaborating with other people, most significantly importantly, looking at products with a critical eye, looking at products with the perspective that there's trouble there to be found and that trouble will hurt our businesses and their clients. When there are problems in the product, those are problems, problems that matter to people, then we want to be aware of them and find them so that the business can decide to address them in whatever way the business wants to. And of course resourcing and using tools and developing tools is a, is part of that in a natural kind of way. We don't put tools at the center of things. We use tools to augment our capabilities as people, as investigators, as testers. But the essence of testing is that learning about the product to find the problems that matter.

Love it. So I think this is a great starting point for the rest of our conversation. So the first one, I think there's a lot of misunderstanding. Sometimes when people hear your views on software testing is almost sounds like you're against automation, but it's not. You just mentioned there you're all for instrumentation or tooling to help augment but not replace. So is that correct?

Yeah. A big problem with automation as commonly conceived out there in the wild, so far as I can tell, is that automation refers to one kind of tool use in testing and that is operating the product via some mechanism and then every now and again making an assertion about the product's output or about its behavior somehow and reducing testing to that which I don't know if you could certainly do a certain amount of testing that way, but it's really, really limited because what that amounts to it much of the time is testing that demonstrates that the product can do something and not a probe into finding problems in the product, problems about the product that would threaten its value. Now I did a brief video on this. It's, it's nicely brief. It's only about three minutes. Where one of the things that I talk about is this weird delineation we have in the software world about functional requirements and non functional requirements.

Where just by calling it non functional we, we seem to sort of dismiss it a little bit. Seems to me, at least now it's really Important to make sure that the functions in the product do their thing. That's because all of the requirements for the product pretty much depend on a function to make something change or happen in the product. That's, you know, in software, we. We need functions to make that happen. That's how code works.

And that's important because if a function doesn't do what it's supposed to do, some requirement or another probably won't get met. Somebody intended for that function to do something to help meet a requirement, and if the function doesn't do it, the requirement doesn't get met. But there's an asymmetry here. The asymmetry is just because functions can be observed to produce correct output does not mean that the requirements will be met. So there are all kinds of requirements associated with a product, and many of them, by the way, I want to make it clear that many of those requirements are not written down. They're not specified in advance, they're not made explicit.

But everybody wants a product to be capable of doing something. Everybody wants a product to be usable by their lights, in terms of their notion of what's usable. Everybody who uses the product wants it to be usable from their perspective. People want products to be reliable. They want them to be aesthetically pleasing and solve a problem in some kind of unique or interesting or valuable way. They want the products to engage them and to entrance them to some degree. They want the products to be performant, to be secure, to be scalable, to be configurable and installable and uninstallable. And they want them to be maintainable, testable. Certain other people inside the project want the product to be supportable, maintainable, portable, localizable and testable. That was the one I was forgetting. Supportability and testability, maintainability, portability and localizability, all these things are things that can be tested. But it's difficult for me to imagine how they could be checked without experiencing those products, without interacting with them. And my colleague James has said over the years that much of what passes for test automation these days amounts to some kind of conspiracy to make sure that nobody actually interacts with the product directly. And I mean, that's a bit of a. Only a bit of overreach, so it seems to me.

So, and the problem with that is people are just relying on the tooling and the biases of the tooling rather than actually getting their hands dirty. And that's why there's a big difference.

Between checking the output from functions in a product and testing the product I think what one of the big problems is it the. It happens because programming is glamorous in our business, right? We are software development people and the, the most important people in making software happen are the developers. Arguably, because you don't have developers, you don't have a code, there's no product. So that focus on programming, I think, has sort of infested the testing world. And we've kind of forgotten about people's experiences with the product. And we've also forgotten about the necessity, I would say at least of performing experiments on the bits of the product as they're being built. And on the whole built product, there's a certain crowd of folks who think that as long as we've got a build pipeline and the product passes its automated checks, it's always automatically 100% ready to deploy. And I, as a tester, as sort of amateur epistemologist, I've got to be really careful about that because the built product has a reality that its components do not have. The built product has a reality that the last version of that built product didn't have. So sometimes, frequently it seems to me, we want to interact with the built product using human eyes and human minds and human fingertips to get experience with the product and to perform experiments on it as a something that we would do in a high risk kind of situation where we want to be careful about the difference between the bits and pieces of the product that we've tested nice and thoroughly and that we've checked really thoroughly. And the product as it's actually been built this time, we don't want to inflict that on people if there's going to be problems with it. So it goes back to what I was saying before, that much of what's going on in what people call automation, one kind of use of tools for testing, is a demonstration that everything's okay and there's something to be said for that. But as a tester and a skeptic and somebody who worries about the possibility that there could be problems hidden, lurking that we haven't noticed yet, I really believe in engagement with the product. And it doesn't have to happen every time, and it doesn't have to happen on every build. But sometimes we've got to take the built product and experiment with it while the developers go on and make other incrementally different builds with little changes in them along the way. And every now and again we pull out a complete build and really take it for a good workout.

When should we do that though? Is there Rubrics or a way of knowing, I guess risk. Is risk an indicator of. Are they messing around with the payment plan here? So I better check it out because it's important to my company or how do you know?

You never know for sure. That's the thing, right? There are heuristics you can apply. Heuristic is a fallible means of solving a problem, or as we used to say, making a decision, but making a decision, solving a problem. So there are decisions we can make about the state of the product. And some of those decisions are going to be influenced by past experiences of bugs getting past us. Some of those are going to be influenced by our reasoned belief that there are no problems in the product and no reason to believe that there might be problems. When risk is elevated, when money is on the line, when human health, safety, opportunity, liberty, when those things are on the line, it might be important to give our product some exercise before we inflict its consequences on people.

Absolutely. So do you have an automation strategy you usually recommend that people can use to approach it so that they don't get all in on just automating and not necessarily experiencing the software as they use it would.

Well, yeah, several. The first thing is that we want to look really carefully at what is being automated in the world of automated checking. What is being automated? Input to the product and operating the product that can be done algorithmically, mechanistically. Then the product operates and we apply a decision rule to the product. And that decision rule basically says if the output from the product is consistent with something pre prescribed, presumably desirable result.

Or some calculated result that comes from a table or from a, an algorithm someplace, apply that decision rule. And if the decision rule comes out favorably, then we turn on a little green light. And if it comes out unfavorably, we turn on a little red light. That's the part that's being automated in much of what people are calling automation these days. But there's a bit beforehand, as you mentioned, strategizing, deciding what we are going to choose to apply automated checks to the framing of risk. In terms of a question that we might like to ask of the product, an experiment that we might like to perform on it, the framing of that question into a yes or no question. Because computers do the binary thing, the encoding of that into a bit of code that makes the check happen. All of these things require substantial amounts of skill, which is why I'm kind of jaundiced about the, the no code or the low code people. There's always code. It's just, it's code that you write and control, or it's code that you don't write and control, but there's always code. So the no code notion to me is a non starter. But then after the check is performed, a light glows red and at that point the testing begins again.

There was testing work that was going on before the check, and now after the check there's more testing work which includes observation, analysis, investigation, discovery, learning, and then relaying what we've learned to the people who need to make some kind of change in order for that presumably desirable outcome to appear. Now that, that could involve fixing the product code, it could involve fixing the check code, it could involve realigning those two things with the environments that they're running on. Could include all kinds of investigation and rework. But that's. Rework happens when we've made errors, and that's what we're trying to trap with these automated checks. So that's the strategy associated with automated checking is let's look at the risk of not being aware of some function somewhere in the product, doing something that shouldn't, or not doing something that it should. That's a great thing to do. But there are lots of other ways to apply strategy to different kinds of tooling and more to the point, strategies for applying tooling to different aspects of testing. So we could have a look at those. One of the first things to do is to review what is available to us, what we can do. So there's a computer and people using it. So what can we do with that? Well, one of the things that we could do is we could simulate the user. Now I was having a chat with a coaching client yesterday in which I asked her what is being simulated here? And she said, well, users, user actions. And at that point I wanted to inject a little bit of caution because there's a fellow that we learned a lot from, a sociologist by the name of Harry Collins, and he points out a distinction between.

Action and behavior.

Behavior is something just a physical thing that happens in the world, something you could look at and see. So for example, here's an example of a behavior. I'll take my glasses off, but you can, you can see that behavior there. Most people would call that blinking.

But then there's a different kind of blinking which goes like this.

That's a, that's what we call a wink. And that distinction really helps to underscore this important point that an action is behavior plus intention. So when we're simulating the user, we Say that we're simulating the user in a suite of automated checks, you know, in a set of end to end checks, for instance.

We say that we're simulating user actions, but we should be careful about that because we're simulating user behaviors. The machinery doesn't know anything about the action, doesn't know about the intention, it doesn't know about the desire. It doesn't know about the user's concept of what would be happy or not happy, what would be useful versus problematic. So that it's important for us to recognize that the behaviors are what we're simulating, not the actions that gets abstracted away in the user simulation. So anyway, that's one of the things you can do. And that's what people commonly talk about when they look at, when they're talking about automation. But there's so many other things that we can do. Static analysis that's been around for a long time and it gets a bit of attention. It's sort of getting brushed aside in the excitement about the tools that automate browsers, for instance. But we can also simulate the system. We can use tooling to do that for backend simulations, mocks and aspects of the product that we want to interact with. If we're testing an API, one of the things that we can do is we can say set up. Not something that queries systems that aren't necessarily the real thing. And that's all right, that's okay. And certainly once we're testing a system that's behind an API, there may be other systems behind that which we can also simulate. So we can simulate systems. That's cool. Another aspect of tooling that we can apply is monitoring and analyzing and observing the behavior of the product in production. So that's largely associated with things like logging and log files and gathering lots and lots of data for analysis, looking at the interactions between people and product, watching the jsons, going back and forth, doing that sort of stuff. Another thing we can do, very helpful, is tool supported test design, including things like generation of data, modeling of flows, mapping various perspectives on the products in various ways.

One of the big ones, one of the ones we're kind of excited about these days, is what we would call augmented experience.

Now, James has invented and developed some tools which we're finding really, really handy. One of which is a coverage recorder which indicates various places that you've hit as you've interacted with the product and then graph and notes it. I wanted to show you an instance of that.

That'd be Cool. You have that?

Yeah, let me just. Let me do that. Here's a dopey kind of automation that I apply all the time.

It's called a batch file. And it just takes me where I want to go right away. So that, I mean, silly as it may sound, this saves me, you know, many minutes a day. So it's a way of doing that. Now that doesn't solve the problem of me not having been here in the last few days. So let's look at that. Oh, let's look at this. Granalizer. That's what we're looking at.

Whoops. CD Scranalizer. How about that?

All right.

Okay, let's see. Okay, so here we go. Thank you very much. As predicted, the coffee.

Oh, beautiful.

Thanks, love. Now.

Let'S see here.

Okay, let me take.

I want to put these. I want to see if I can put these all on the same. Yeah, I can too. Cool. All right. Okay, let's look at this.

One of the things that we can use tooling for is what we would call augmented experiential testing. So we have a little sample here. What you're seeing is a record of data gathered by the James's coverage Analyzer. And what this does is it goes through a record using this particular little tool here. And you can't see it in action at the moment, but it's just something you pop up and you associate it with a given URL.

Okay.

So it's trying to grab this one right now. And what it does is it identifies everything you've clicked on and everything that you've interacted with and captures that.

And then logs it and then graphs it. So here you can see stuff where, for instance, a bit of testing that happened on the Tricentis sample insurance app there. And on the left hand side you can see elements in the. In the DOM with which James interacted at a particular time. Okay, so what's happening here is that we're tracking a person's interaction with the product. And then if we go back up here to the thing that looks like a home. There we go. We can reset the axes and you can see what happened over this period of what I guess it was an hour and a half, 30 minutes or so of interaction with this product. You can see where it got hit. Here is something from one of James's students. This was her interaction with the product. And you can see, first of all, you can see that she took a break at a certain point. But you can also see very different patterns of what got accessed and what didn't. In fact her coverage, her visits here seem to be a bunch of places where James didn't go if you, or at least he went at a different kind of time for different amounts of time.

By the way, the tool also captures notes.

And bugs.

Oh cool.

So it's something else you can do when this window is up here. There's a ways by which you can grab the. The notes, you can do a sort of console stamp and there we go. Whoops. That doesn't work at the moment because I'm showing you the. I'm not on a product that I'm testing, I'm showing you the output from that.

Right, right, right.

That was a bad time to do that. But anyway, at this point you can drag down a, or drop down a thing and put in a note or a bug or this time the session start or the time of session end and that'll get recorded as well. Here is an instance of automation being run on exactly the same thing.

And notice that the number of things that got hit here in the automation is substantially lower than the number of things that got hit here. And it did so in a very repetitive and uniform kind of way, which for routine output checking is probably not that bad an idea. But it does show you that the. By its nature the automation doesn't explore, it doesn't investigate, it doesn't ponder, it just looks for some kind of checkable result and then off it goes. So that's one kind of augmented experiential testing. Now I'd like to show you another. This is kind of funny.

Is this a free Chrome plugin by any chance or page?

It is a Chrome extension. Let me. You want to ask me that again?

Sure. So James, is this available for anyone from the Chrome store?

That'd be Michael.

Sorry Michael. I'm sorry.

Not to worry. Give me a chance to laugh and clear this thing out of my lungs there.

You know what, you can do that as an outtake. That'd be funny.

So Michael, is this available on the Chrome store?

No, it's not. Not just yet. James and I provide it to our students after class. We're a little leery about it being ready for prime time. Some of the tools though are available. Let me give you. Let me show you an example of a.

Place where. Yeah.

Here'S an example of something that is available, I think. I hope.

You actually going to come up there, bub.

Oh dear. The whole thing is down. That's rather tragic because it's such a wonderful. Oh, there we Go. Okay. Okay. So.

Let me run something. And I'm not sure.

If the echo cancellation will make this happen. We'll allow this to happen properly, but we'll see. So this is a program called Chrome Log Watch, and it's a little way of sort of augmenting your testing as you go. What this does is it looks at the Chrome debug log.

Now the question is, is the debug log even enabled?

This is what I absolutely love about live demos, that the. For some reason, when I start the browser.

The browser.

Is not bringing up the logging. So why is that?

I believe it's because it's running. Let me do this. Let me see if this. Ha. If this works. Well, cut. This here's a tool that we use all the time. And this is not. This is something. The kind of tools that people don't talk about very much.

Do you know about Search everything?

I do not.

Oh, it is really cool, simple tool. It's free. It's a free tool. And what it does is it monitors every single file on your system in real time. Well, it models. That's not quite right. It looks at the file system in real time and indexes every single file on your system. So what I'm looking for is the link to Chrome that comes on the start menu rather than. Ah, you see, these are a bunch of things that the operating system appears to have put in against my will. So I want to take those out. I want to take those out. You can. You can do various kinds of file manipulation with this. And I want to start Chrome from this one first. The thing is, I want to open it and see. Yeah, you see, that's not coming up somewhere along the line. And this happens all the time. This is part of the secret life of automation. The secret life of automation is whatever looks good in a demo got there by painstaking rehearsal and probably by editing.

Right. Whatever looks smooth. Because this is one of the things that we talk about a lot in our classes, the secret life. Testers are very tempted to hide the secret life for lots of reasons. If you don't mind, we'll go into a bit of a detour about that from. And then we'll, you know, return to our regularly scheduled programming. There are elements of a secret life to. To automation. Now, why is that?

There are some answers.

And that isn't right either. Why is that not right? That's supposed to be this one. That's not.

See secret life. Again, we're going to address this.

Why might there be secret processes that Is the slide that I'm looking for.

Right here. Oh, it's 85. It's not 70 at all. So 85, here we go.

There's a big secret life in automation and there's a secret life in all kinds of processes. But in, in the world of automation and tool use and stuff, there are processes that are. We don't talk about very much. There's. They're kept secret. Now why is that? Well, one of the reasons is that people don't notice their process, for instance.

In an automated check. Joe, let me ask you, what specifically is being automated?

Well, you tell it to automate step by step.

So, but specific.

What specific things are being automated?

The user interface, the interaction between a browser or application under test objects.

What kind of interaction specifically?

Usually clicks, tabs, enter, minimize, maximize, close, opens.

So certain interaction via a simulated. Simulated, yeah. And that simulation is really important. There's a little video I can show you about that. Because believe it or not, hitting a real key on a real keyboard is different from a tool operating a virtual key on a virtual keyboard. When we hit click, we are not actually performing exactly the same thing as a user would. We are catching the process somewhere along the way, such that we call a click function somewhere in the browser. But we had a wonderful experience one time with a. It was a, a web based product that had an email field in it and one of the testers that this was in a class this was happening. One of the testers reported that 8, the 8 in his email address. His email address was something like, you know, bobsmith 1980mail.com so he would type Bob Smith 19 and he'd hit the eighth key and it wouldn't register. This seemed bizarre to us. So we set up a. I think at the time we were using Selenium Playwright does the same thing to just, you know, pop a string of digits in there and there it goes. 1, 2, 3, 4, 5, 6, 7, 8, 9 0. But then when we tried to do it on the actual keyboard, didn't work well it turns out. And I've got some video I could show you about that and we're jumping around quite a bit. But it turns out that there was code in the application.

That was attempting to filter out stars because stars aren't allowed in email addresses.

Yep, yep.

So the developer had trapped the downstroke of the eight key. Oh. And was filtering that out, forgetting to check whether the shift key was also down. So the consequence of this was that anybody who had an 8 anywhere in their email Address was unable to enter their email address.

Oh, my gosh, that's wild.

Yeah. Now, the key thing, the key thing to notice there is that automation would be unable to detect that problem, or at least automation of the kind that most people implement using the popular Selenium, Playwright, Cypress kinds of tools, or indeed anything that is driving the browser by interacting with stuff at the browser level like that. Now, that's not to say that these problems are necessarily common or that the developer was particularly swift. In this case, that's a whole different deal. But it is the kind of problem that automation would not alert us to. Okay, so.

Maybe a good way. Mark this. I guess maybe a good way to do this is to look into, to start again with the secret process stuff.

I'll do it this way. I guess.

There'S a secret life to work in. Any kind of work, really. A lot of what we do, we do because people are hiring us not to think about it, right? Their job. They don't want to think about it in their job, so they hire us to do it, which is great. That's fine. It's a good thing. But there's some processes that we like to talk about, and then there are other processes that we don't like to talk about so much. We prefer to keep them secret, partly because we don't want to necessarily expose people to the sausage being made, but also because sometimes there are certain things that we might not be aware of or certain things that we might be a little bit embarrassed about if they were to know about it. So here are some reasons why there might be secret processes, and we're focusing in particular on the world of automation. One thing that we hear from a lot of testers when we quiz them and kind of interrogate them on what they're doing, is that there are lots of things that they actually do that they've not noticed before that they're not taking account of. For instance, I was speaking with a tester yesterday and asking her about what she does, and she says, well, you know, I get. I get a set of checks that have been developed by other testers and I turn those into code. Okay, is that really all that happens? And she said, yeah, yeah, yeah. Well, okay, so what else happens? What else is happening in there? Let's break that down a little bit. And she said, well, you know, I start the product up and I interact with it and make sure that I can do the things that the other testers were able to do. I said, oh, okay. So there's actually some Interaction that you're going through yourself. What happens when you find a bug? I asked and she said, well, you know, if the check crashes or if the bug. If it yields a bug, well, then, you know, I investigate it and I try to figure out whether it's a bug in the check code or a bug in the product code or something about that. So her epiphany at that point was, you know, oh, wow. If we really break it down, there's lots of stuff that I've never noticed before. Lots of stuff in the process of developing and applying tools that people don't talk about because they don't think about.

Just goes below the level of their consciousness, and they're just doing work, you know, so that's a big factor. Sometimes, whoops, I'm. There we go. Sometimes there's something going on that they don't really know how to describe very well. And that's partly because lots of what we do is involved, is embedded in tacit processes, stuff that even we have not made explicit to ourselves. We don't notice it. And then when we do notice it, well, then we don't have words for it. We don't have terms for it. Part of what James and I do a lot of in our work is we try to make visible and legible and comprehensible things that are sort of below the surface of what we think about and talk about much of the time. So we work on that. We try to name heuristics that people apply. For instance, another reason that people are kind of secret about some of the work they do is that when we describe it to people, they don't get it. They don't have the lived experience that would be required for them to understand what's going on and why. Why it's important and why it makes sense. Some things are secret because we don't record them, and we don't want to record them. That feels icky to lots of people. Some processes are forgotten. Now we. One of the things we say about note taking and testing is, listen, it's perfectly okay not to keep records and not to take notes as you go. Perfectly okay to do that as long as you're cool with forgetting lots of stuff. That's my experience and your experience too. That stuff that we don't diligently record, we forget about it. And lots of the time that's perfectly okay. And sometimes it's important. Sometimes people say, well, managers actually don't care what I do. They just want results. Right? We hear about that. Or even worse, they say they just want numbers. I don't think that's true, actually. They don't want numbers one way or the other. What they want is a feeling. They want a belief, they want an understanding that, well, as John Bach, James's brother, once said to me, what do they want to know? They want to know of me as a tester. They want to know, am I on top of it. Well, that's true to some degree, but I think what managers really want to know is from the manager's perspective, am I on top of it. Managers want to be aware of the status of their product. Now there's some people who have a sort of cynical belief that managers don't actually want that. And to a certain degree that's always true. Nobody wants to hear about problems in the product, nobody wants to hear about that. It's just that pragmatic, responsible people would probably prefer to know about them so they could deal with them rather than being oblivious to them in such a way that they could cause harm. So this is one thing that afflicts me all the time, and that is I'm shy about the code that I write because among other things, I cut my teeth in a company where the programmers were assembly language level programmers. And I feel like if I showed them my code, first of all, it wouldn't be up to their exacting standards, but also it would look simplistic and, you know, but it like a newbie stuff and that sort of thing. And that in a way that's just kind of silly because these guys were a lot smarter than that. They've recognized that, hey, code is what you need to get the job done. When you do that, don't, you know, don't worry too much about it. If it's not frightfully important that it be super, super precise or super clean or super elegant, then it's not a big deal. But as humans, we kind of get shy about that. A lot of the time people are embarrassed about the, the way they test as being redundant. And there's actually something to that sometimes. I'm sure you've heard stories of people who are testing at the GUI level, things that are not only easier to test at the API level, but have been tested already at the API level. But because the, the GUI automation is dazzling, they sort of get away with that. You know, it looks good. So the opinion of the person who's not looking too closely is, well, if it looks good, it must be good. Sometimes the, the process involves recognizing that there's Maintenance stuff. Sometimes people talk, decline from talking about certain aspects of their process because.

It'S really hard to do and they can't tell when it's going to end. So they just dampen that conversation. Here's something that I actually often have a hard time talking about. I don't know sometimes how to talk about things in a general way such that other people won't come along and say, well, that doesn't apply to us. Well, that's actually true.

But if it doesn't apply, oh well, it does apply usefully in this sort of circumstance. So that's something that all of us need to get over a little bit and recognize. Hey, if we found a useful way of approaching a problem and using tools to do it in a certain situation, we can talk about how useful it is in that kind of situation and not worry, worry about the circumstances where it's not. Sometimes the code that we inherit is opaque. Sometimes we're secretive about our process. We like to tell people, oh, well, you know, I read the requirements documents and I read the specifications and I read the, the API documentation. Well, actually we don't do that. Sometimes a lot of the time you just sort of dive in and try to get to work. And again, that can be a little bit embarrassing. So lots and lots of text on this slide. But basically we're trying to help people become aware of their processes in our work. And there are certain obstacles to that with respect to knowledge and awareness of the work, the nature of the work itself. Right. This sort of higher level strategizing part of it, the relationship to the work that people have, how much agency they feel they have over it, how much they have to be accountable for it.

The, the actual data about it, the data that needs to get collected, and then the, our, our own feelings and the feelings of our clients about that. And so sometimes those processes are secret.

So the outcome of that is that a process may not be described in a legible way. Legibles are. Legibility is an important kind of idea in our work. Legibility is almost literally readability, the ability to observe something and to make sense of it because it's been presented in a way that's easy to see and easy to wrap your mind around, easy to understand. One of the things that's kind of seductive about automated GUI level output checking is that it is quite legible on a certain level. You can see stuff going by on the screen, you can see stuff that looks like user interaction, you can see browser behaviors flying by, and that that looks impressive. It looks like a superhuman doing things at superhuman speeds. But there's lots of stuff that might be going on in that product that the, the dazzling nature, the legible nature of that stuff doesn't illustrate so well. So that the, the processes that, underneath that are the processes that lead to it or the, the processes of what, what we do with the information after the checks run, that is not so legible, that's not so visible.

So for instance, when I was asking my student the other day about, well, what actually goes on in your work.

She didn't talk at all about the processes that involve investigation of what happens after a check has run red or after a check has crashed.

And she made a distinction between her work and what the other people in her crew do. She is the automation tester and they are the manual testers. Now this is a distinction that just bugs me no end.

And the reason that it bugs me is because it seems to me that neither one of those terms, manual nor automated, do very useful work in describing what we actually do, especially the manual bit. By manual, people often mean, well, let me give you a little visualization of this.

By manual people often mean, and where did that slide go?

Oh, there it is.

And that is slide number 50. So it seems.

So people talk about manual testing. Now what do they actually mean by that? For a long time it seemed to me simply that when people talked about manual testing, they meant, you know, they didn't say, I do manual testing. What they meant was, I don't write code to operate the browser. But it bugged me one day because a tester was saying, I'm worried that I'm going to get replaced because I'm a manual tester and my, my company is going whole hog into automated testing. And first of all, testing can't be automated. All those things that I mentioned earlier on about investigating, critical thinking, risk analysis, interacting with the product and that sort of stuff, interaction with the product on a sort of surface level that can be automated. But all of the thinking stuff and all of the investigative stuff and all of the analytical stuff that goes into testing that can't be automated, those are distinctly human processes. So it bugged me when she said, I'm a manual tester. I said, no, you're not a manual tester. You get.

You interact with the product, you get experience with it, you're an experiential tester. And that sort of started turning on a bunch of light bulbs for me. How can we express what testers actually do so that we recognize how kind of bankrupt. The term manual testing really is.

What do people actually mean when they say manual testing? Sometimes what they mean is they mean interactive testing. They're interacting with the product directly and they got to be there. There are some processes that go on within testing work that can indeed be done in an unattended, non interactive kind of way, right? You give a process for a machine to perform and it performs that process and you come back later and you look at the output and you look at the logs and you look at the state of the, of the application of the data afterwards and you make a some decisions about what to do next. Whereupon you're back into interactive work again. Another thing you could say though is that you interact experientially with the product. That is to say.

Your encounter with the product is practically the same as that of a particular user that you had in mind. If somebody were watching you working, they'd have a really hard time figuring out the difference between you, the tester and some contemplated user. Except for one thing maybe, and that is when the contemplated user encounters a problem, the user is going to say damn it and try to work around it. Whereas a tester is going to say, oh, that's interesting and it's going to start investigating. So, but when we talk about experiential testing, we're talking about testing where we are interacting with the product in a way that reproduces or that anticipates the way that some somebody affected by it is going to be interacting with it. So sometimes we say user, I want to put a little bit of a, I want to open the fence on that idea and say that an ops person is a user of a product. The, the person who is the client of somebody who is directly using the product is also a user. So we want to have a very expansive notion of that. Now contrast experiential testing with instrumented testing, where something gets in between the person interacting with the product and the product itself. And automated checking is like that, right? There's a medium, something in between that is going to alter or change or distort or accelerate or intensify or extend or enable or disable or limit or hyperspeed the interaction with the product. A lot of problems in automated checking are the consequence of that, that the tool is running too fast for the product to respond.

So the distortion that we're seeing in instrumented testing can be a feature. It can also be a bug. It, you know, this is not evaluated, evaluating statement that the experience is distorted. It's just changed in in some way from the naturalistic experience of the person using the product.

We could be testing. When we say manual testing, what we could be talking about is testing where the choices have not already been made for us, where we're exercising agency over what's going on and our procedures might be open rather than constrained by something. So the essence of exploratory work is making choices. And we do that in automation work all the time. We make decisions about how to write and maintain our scripts. Another way we put it is there's no script that tells you how to write a script, and there's no script that tells you how to investigate a problem that you found that your check has revealed. So that's an issue.

And then finally, sometimes this is on the list and sometimes it's not. But let's put it on the list for today that some interactions with the product that we call manual testing are sort of transactional. We don't learn anything from them. In particular, we're just walking through a routine that somebody set down beforehand and the test doesn't affect the tester. But to us, if it's going to be a test, it has to affect the tester some way. At very least, the tester needs to learn something from the experience of performing the test. Okay. People who are heavily engaged in developing and maintaining and executing automated checks are doing this kind of stuff all the time. So called test automation work involves an enormous amount of what people would call manual testing. And the trouble with manual to us is that it hides what's really going on. Because manual, after all, means using your hands. But the hands are merely an input mechanism. They're not. They're not the essence of what's going on. So we just encourage people to drop the idea of manual or automated testing and talk about testing instead, and then talk about how we would apply tools in testing.

Thanks again for your automation awesomeness. The links to Emptiness value we covered in this episode head on over to testguild.com a543 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms. And over 40,000 email subscribers who are at the forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software, test, testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

Cast with lutes and liars the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.