Testing AI: Engineering Confidence in Non-Deterministic Systems

Welcome to episode 600 of the Test Guild Automation Podcast. I'm Joe, and today we're talking with Jason Arbon, who just released a new book called Testing AI: Engineering Confidence in Non-Deterministic Systems. And honestly, Jason has been living in this space longer than almost anyone I know. He really knows his stuff, so you should listen up. He was thrown into it back at Microsoft when the Bing team bet the search engine on a neural network. And nobody, including the engineers, can fully explain how it worked. He carried it into Google and he's been carrying it over ever since. So in this episode, we get into why he thinks confidence engineering is the last rule standing, why the critics of agentic testing have never actually run head-to-head experiments, and what happens when the software you're testing can outthink you. And the one habit he says separates sophisticated teams from everyone else, and it takes only about 10 seconds to describe, and most teams still are not doing it. And there's a moment at the end where I really poked the bear on purpose. You want to stick around for that. And as I mentioned at the top, quick note before we start, this is episode 600. I can't believe I've been doing it almost every week for 12 years. 600 conversations with people who actually build and break software. And this is something fitting about the one being number 600, because when I recorded episode 1, the entire conversation was about how to make Selenium scripts less flaky. Today though, my guest is asking how you validate a system that might be smarter than you. I can't believe I've had 600 conversations with the people who actually build and break software, and there's really something fitting about this one today with number 600, because when I recorded episode 1, the entire conversation was around how to make automation or Selenium tests less flaky. But today my guest is asking how you validate a system that actually might be smarter than you are. Crazy change. So over the years, yes, the tools have changed completely. When I started, it was all about WinRunner, QTP, but the question really never did, and that is, how do we know this thing actually works? This is still the whole bread and butter of our whole job as testers. So thank you everyone for listening to any part of those 600 episodes. Onto the next 600. All right, hold up though. Quick heads up before we get into episode 600. Our Guild State of Automation 2027 survey is now open. This is the 11th year I've run it. And it is the reason why we have real numbers on how, say, AI adoption and testing went from 2% to over 70%. It's completely sponsor-free, vendor-free, and only takes a few minutes. It's independent. Help us get up to 500 responses so we can make this an industry report that's gonna be free. And to do it, all you need to do is head to the link down below, add your voice, share it with a friend or teammate, and really look forward to seeing your answers. Check it out. Hey, Jason, welcome back to the Guild.

Thank you.

All right, so yeah, great to have you back. It's been a while— not a while, it's been like maybe a year— but, uh, I've been using your quote around the country. I've been doing these IRLs where you said, um, I'm very off the rails here, um, how testers' roles are going to change but they're not going to necessarily like the change. And, uh, it was kind of controversial at the time because people are so caught up in page objects and, and code and all the things. I don't know if that's necessarily the case now. I don't know if you had like a twist on that, like where you see the biggest bottleneck with people implementing AI now, or maybe a year ago.

It's definitely changing. It's changing like every 6 months. And yeah, I would say like even going back a little further, like 18 months, 2 years ago, at best people were, I call them AI curious. Yeah. And Yeah, there's been a transformation. People are much more comfortable with it. I mean, everyone's suddenly now an AI tester. It's amazing on LinkedIn. And AI podcasters, you might have heard of these people. And it's gonna be big someday. But yeah, I think people are very open to the idea that it is changing. They're just not comfortable with it. You kind of nailed it, which is— and that's something I would say. I don't remember saying that, but it's probably something I would say.

It was good stuff.

But it's very much a time of in the moment, right?

Yes.

Capturing these things. And people are on these different trajectories of transition. But, uh, um, yeah, people are finally— they go, oh, it's here, it's real, I'm gonna deal with it. The question about that is then, but how are they dealing with it? And there's a lot of fragmentation in that. Yeah.

So what I love about you, you've been dealing with this before it was a thing, um, you and Tyrique. And so I'm just— I don't know why people just don't— every time you talk, they stop and they don't just listen. Because you've been thinking about these problems way before I think it was a thing. You know, 2018, we did an AI summit. You talked me into doing that, and you were doing it before then. So just curious to know, how did you know that this was gonna be a thing eventually? It just took a little time to catch up where your vision might've been.

Well, you can see, like, you can just look at me and genetically go like, okay, this guy's not the genius. It must be from somewhere else. This is, you're like, where is it coming from? It was actually just the fact that I, I, um, what my dad would always say is like, why do you always have to eat your employer's lunch? Um, because I'm always thinking about how to innovate, right? Or change things, make things better. And, um, and so I— so at Microsoft, you're like, oh, here we go. Uh, but back at, back at Microsoft, um, I worked on, you know, classic software, procedural functional software, buttons and clicking and staring at data. And the But I forced myself because I saw this Google paper. I haven't shared this very often. There was a Google paper released called GFS. It was a Google file system. And I was working on WinFS at the time. So, it was very relevant and interesting, and I read it. Most people, Microsoft didn't even read it. The key thing was this, is that Google thought about a file system, but at scale. We were building file systems that were sitting on a single machine and that were idempotent and repeatable and super make sure you don't lose a single bit, and then maybe connect them together through, like, a URL, right? And Google just said, hey, we just take all the data and we're just gonna spread it across thousands and thousands of machines, and some of them may die. We don't rely on any one of them. It's striped by default. So, it was a very different way to invert how you think about saving and retrieving data. So, I went to the Bing team, and the Bing team— this will be relevant because I think this is people's journey they're on right now that I accidentally fell in because I'm hyperactive. And so, I moved to the Bing team because that was the closest thing to, like, Google kind of scale at Microsoft at the time. And as a tester, no one told me anything about this stuff, about the differences. We just all of a sudden had a search engine, and I initially thought about it like everyone else would, like a tester, like, oh, there's— you put input strings in, and you get output strings, and you have to measure these things. Well, it turned out— I'm almost done. Like, oh, here it is, a one-question podcast.

That's good.

It is probably useful. I think people can relate to it. So, I'm stuck on being a test manager in this world. But Microsoft didn't have the talent. Google hired everybody that knew how to build a search engine. And rumor is they gave them stock. But then— and we had Microsoft stock. But so, we didn't have any engineers. So, what they did was some Microsoft Research guys went off and they did this— sounds like an obvious thing these days, But they did a research project to try to build a search engine because they said, hey, Microsoft said, hey, you know, we can't figure this thing out. It's a big business problem. We're gonna get our hat handed to us, whatever. And so they built a neural network. Crazy idea. They built an AI to do the search engine, like a small search engine experiment. That's just kind of an experiment. And a super smart guy, his last name's, um, Kenneth Church actually, but he went off with some other people and built the search engine. And it was almost— it was really good out of the box. Like only a few engineers on it and research guys. And it was really good for the amount of effort you got to put into it. It was better than what we had. So, we're like, oh, this AI thing may be interesting. So, they instantly turned the entire organization. Before, while Google was still handcrafting algorithms and linear equations, Bing jumped into the AI world. And so, here I am as a normal functional tester going like, oh, now I'm actually testing the training system and the validation system for a search engine that's built on AI, and none of the engineers know how it works. So, I was, like, thrown, basically thrown in the deep end, is the quick answer. And so, I had to scramble, and other people, engineers, everyone was scrambling to figure out this new world. But I did it under fire, and when things were hot. And it wasn't hot, it was actually a bad idea. We were embarrassed for using AI because it was a crutch. And that made me do this whole mental transformation between, you know, traditional QA, and then what does the world look like in an AI world where software is defined by AI? How does QA and testing differ? So, I just got an accidental dump into that world, and trial by fire. And so, that's really just trying to share. So, I just, that's why I saw that. And I moved to Google because I thought, well, that's the type of place I wanna work on, and that's the future. And so, I kept, you know, working on these kinds of problems. That's it.

It's pretty simple.

It was half accidental.

And back then, they didn't have this awesome resource that you just released, a new book called Testing AI: Engineering Confidence and non-deterministic systems. Is this, why did you write this? Is this basically what you learned on your own that you struggle with, that you thought, hey, let me make a tester's life easier now? Like, why did you write this book, especially with AI? Why do we need books in the age of AI?

I have to tell you, Joe, maybe we're dumb, we're too friendly, but—

Okay.

But I've already practiced this question 'cause I asked ChatGPT to have a mock podcast with me this morning. All right, nice. And I'll just say it finished up with asking me, reminding me to come up with some actionable advice.

Yes.

About to jump ahead. But no, so, so I wrote it basically. Yeah. So I hear a lot of people, people ask me like, how do you, because they think I've tested AI stuff and things and they go like, well, how do you, you know, what do you do? And it's, it's too, it's too complex to fit in a, in a, in a, in a comment box on LinkedIn. And I also don't have the time to do tutorials or lessons or whatever. It doesn't scale. So really it's about exactly what you're kind of leading into is that There's a leading question. I wrote it for me, like, in the past, and for everyone that's kind of going through that transformation right now, because it's unnerving. You feel like you're losing all your skill set. It's a new world. It includes more math and more code and more sophistication, technicality. A lot of QA people try to avoid that, frankly. It was kind of a nice place to hide from a lot of that, those details. But it's harder to hide now. But I want to give people the tools and at least learn what I've learned and seen in the space so that they don't have to, you know, suffer for several years. Um, they could just read a book for, you know, $20 on Kindle, um, and jump ahead because there seems to be a lot of angst too.

Uh, yeah.

Um, and I don't— I, you know, I care. I don't know, I should say care about people because usually only psychopaths say that because it should be obvious. But, um, this has turned into a great therapy session. But I wanna be helpful. And when I hear people, like, say things that just were like, oh, I figured that's something, the real story's a little different than that. I'm just trying to help share that with people.

Love it. So, this might seem a little off-topic.

Actually, let me interrupt you, Joe. Is that okay?

Yeah, yeah.

There's an even larger motivation, though, too, which was, I'm a consumer of software. You use software, using software right now. The world's becoming more and more dependent on software. And again, AI is gonna be big someday. And it's like, maybe it'll be quadratic on top of exponential growth soon. I don't know. But it's gonna impact everyone's lives to the point that we worry. There's actually some of this stuff in the book, to the point that we worry about the quality and the alignment of the system. Like, will it just destroy humanity? Not my job, not my role, but will it destroy the planet or the economy or something? And these are discussions that are held at the highest highest levels of politics today and in the technical field. And ever still, people go, ah, that's a little crazy. But these are the numbers and this is the trajectory we're on. So I feel like it's— I almost have a moral obligation. Not that I'm not useful, but I have a moral obligation to share what I can to help make that world better, higher quality, because I care about quality, and safer for humanity. But, but like, there's really not much more I could do to be useful to humanity. Sorry to be philosophical. But then to help people make the software that we're going to depend on and maybe defend against, more, more higher quality and more controllable.

But, but are we doomed? Because you probably saw the latest thing with— there was a hack with Hugging Face.

The whole thing, the whole shtick, Joe. I wrote the book so the world— it's solved. Man, you're a tough customer.

All right.

Are we doomed? I don't think we're doomed. And by the way, I've told you this. There's like a— what's that law? There's this like law where like you never deny religion because there's just no upside, right? It's like there's a law. It's just the dumbest thing you can do because you're either right and you're screwed or you're wrong and doesn't matter.

Pascal's wager. Love it. Yeah.

Pascal's wager. That's exactly right. So I don't know what this one would be, but this is But are we doomed? I think if you if you really look at it, the people that know and I don't pretend to know, but nobody knows. There's almost a capitulation going on right now. This again, this will be irrelevant in six to nine months. But there's there's a capitulation now of like hey, the race is on. No one can stop it. You look at the models like of all things just Altman tweeted today because moving fast. He's like be prepared.

This is Altman.

By the way, he's if you look backwards, he's been. Concerningly accurate because he sees the next models. He said there's gonna be more change in, in the power of these models next 6 months than we've seen in the last 2 years. Um, but if you think about it, like, even 6 months ago, these things were toys compared to what they are today. So, so, so let me—

let me—

she gave me the big question. The problem is I wrote about it yesterday, um, on a very famous, infamous blog post that LinkedIn. I think I got 5 likes on it, but it's this, it's a ZooKeeper problem. Are you familiar with this thing? Okay. But this is where testing is becoming the most important thing in software, Joe. This is why it's so exciting times. It's easy to, all the coders that used to be all arrogant, oh, I can write a for loop faster than you. It doesn't matter anymore. And QA is on the rise. It's the hard problem now. Like, the hard problem of consciousness is the hard problem of AI. But if you look at it, the, you know, why are gorillas in zoos? It's because we have higher IQ, supposedly, right? Some of us do. Most of us do. And but what happens when the gorilla becomes smarter than the zookeeper? Like, and this is a— I don't know the right word.

A paradox.

Allegory, analogy, whatever the thing is. I'm not a drama— I didn't get an English major. But it's Stuart Russell from UC Berkeley, and I'll brag that I've talked to him, I've touched his hand twice. He's pretty cool. But he's kind of leading the AI safety movement. But he came up with this analogy, it's like, it's the gorilla problem. Like, how do we— how could a gorilla, you know, Can, you know, we're gonna become the gorilla. And guess what happens? It never works out well if there's a more intelligent species. And you go like, oh, maybe these AIs can only get as smart as us. Say that's even true. The thing is, they work 10x faster, 10x cheaper, and they can collaborate on the order of hundreds of millions of them or more together. And the crazy thing is, they're almost best of breed. There's some things they're still missing. almost best of breed in every profession and every focus of research and science, all packed into one little tiny model. So if you just extrapolate, it's, it's, it's, uh, it's concerning. Uh, it's, it's— there's, there's a good book too called, um, uh, Deep Utopia, which I was reading with my son on the drive back from, um, 4th of July. Road Trip, uh, on audiobook. I'd recommend people to read it. I'm only about halfway through, but it kind of walks through the optimism side of this stuff too. So there's an optimistic way with, we don't have to work. We get to watch, you know, go to Netflix and say, hey, give me, I want Joe as Batman and I want, uh, Bolton as the Joker. And I just made this up on the fly. Um, but like, and then generate that new Batman movie for me. and I get to watch it, right? And maybe, yeah, I won't extend that analogy. But maybe we're in this WALL-E world of just complete leisure, right? But I think most of that is like we're trying to convince ourselves that there's a good outcome here long-term.

Yeah.

And even, I mean, we think about our lifespans, but this trajectory even slightly keeps going up for 200 years, That's a blink of an eye in humanity's time, but it will, it will get more and more interesting, which is where governance comes in, which is another whole topic.

Right, right.

But go ahead. Sorry, I should quit rambling.

Yeah, I guess why pessimism comes in is, uh, I think you may have wrote about this as well. There was a hack about Hugging Face where they were doing an experiment and they were doing a test in a controlled environment, and, uh, they gave it a task and it found a way To— they found a hack to go get the answer to whatever they were testing. I'm kind of totally botching this incident, but it was weird that the AI was able to self-find a way to get out of its containment, to go on the internet to find this key.

Mm-hmm.

And it was— sure, it was trivial. It wasn't an attack per se, like a hacker. But I can only see this— imagine this, like, when we have healthcare and nuclear weapons with AI baked in, like, like Is there any hope is what I was trying— how do you test these type of systems so that we do avoid this? Or is it just past that now? I don't know if that made any sense.

No, no, it totally makes sense. I'm just wondering about the audience. I think, yeah, so it was very concerning. A lot of aspects that were concerning. They're still figuring it out. Like, there's some arguments that it wasn't as Like you said, not as malicious or intentful as it seems. But yeah, like we had an AI, I mean, I think we were living in a world where the news is like, oh yeah, sorry everybody. And they're exposing themselves to some liability and concern too by doing it. It's not just marketing drama, but like, yeah, an AI escaped, did stuff for 2 days that we don't know. The thing is, they don't know what it did either. They think they know, but who knows what it did?

Right, right.

And this leads to a larger Problem. Even had it— not only can it escape, there's a containment problem. You can't contain an AI that's smarter than us. There's this— I have a little section in the book on that topic, but you can't actually logically contain it. So those people thought for a long time you can contain it. They've actually given up. The only real hope, it seems, is that you make it aligned. So when it gets out and escapes, it just does things that we wish it would do and doesn't hurt us. or take the banking system down or something like that. That's a hard problem. Like, no one— and it's a testing problem. It was beautiful what you said, because we're on this podcast talking about testing. Testing's just so elevated. We've got Satya talking about testing every day. We've got Sam Altman talking about testing. And it was in a testing environment. So, realize, TestGuild people, guess what the drama of the day is that's affecting trillion-dollar companies is that their infrastructure to run tests... Wasn't secure enough, and they didn't have enough instrumentation or observability to know that an agent escaped their system and started attacking other places. Like it's a testing problem. The problem is, if I can get on, I was gonna say get on my podium, but I'm already up there. Is is that guess who's running all this stuff today? Is it's a bunch of postdoc researchers from like Stanford and and uh. Um, and University of Utah. And those, uh, they, they don't know testing, they don't know industry, haven't learned all the skills that we have that we take for granted, that we don't even appreciate, frankly, um, that no one appreciated for, for, for 20 years. Um, but they're finally coming to the fore, and it's having consequences like this. Like, but it's test environment, um, and it's understanding how you test it, how you evaluate these systems, how you do it safely, how do you make sure that it's Um, that it, uh, um, that it's aligned with the company goals. Does it sound like a QA person? How do I make sure this, this, this, this thing is aligned with company goals, humanity's goals, the customer's goals? These are all questions, but they don't know what to call it because they don't think about testing, because you don't test in college. Nobody tests in college, so they don't know what to call it even. Yeah, but yeah, I think, I think it's, uh, it was a very concerning incident, but I think At the end of the day, you can't, you know, well, most of the chatter has been about locking it down, right? Don't let it have access to this. Don't let it do this. Don't let it monitor it. But we, the problem is we can't, we say we have guardrails, they don't work. There's actually several companies that do guardrails and none of them are really taking off because guess what? You can't really control it. It's again, this was the gorilla problem. Like, that machine— and here's the key one I love. It's a testing problem. This is the ultimate testing problem ever devised.

And it's kind of—

humanity is on the precipice of finding out if we can figure out how to test this, which is, how do you test a system not only that's smarter than you— that's a hard problem, okay— that's deceitful?

Hmm.

Just, I'll just say, like, and it's smarter than you. How do you validate such a system? And once you start thinking about that, You go, oh, yeah. So, non-deterministic systems, that's a problem. But, like, what if the system is actually, like, deceitful and smarter than you? Like, that opens— but no one knows the answer to this. Like, I spend all my time trying to catch up on papers, and I can't find anyone in the testing community, by the way, that cares enough to talk to me more than 10 minutes about this topic. But I think it's a huge— it's the testing problem of our lifetime, ultimately.

Why is that? Because I've spoken to people like, well, first of all, it can't be deceitful because it can't think. It's not thinking, it's just token-based, making, you know, it's just a prediction machine. It doesn't really know what it's doing. So we're overgiving it too much human characteristics when it's never going to do that.

Okay, well, real quick, the people that say that could not hack themselves. Just look at every example. They could not hack themselves out of that system like last week and attacked Hugging Face and got away with it. Those people are not competent to do that. So there's— this is the real thing. It's almost this whole thing about consciousness. For all you know, I'm an NPC, right? And you probably paid a lot of in-app gold. We've talked about this before to opt me into your game. But no, it's, it's, um, it's, it's, it's a problem because the—

you can't—

how do I say it most concisely? Um, we don't know what consciousness is without getting too broad. If people are interested, look up, read a lot on Chalmers. Is great. Uh, it's getting— Chalmers that talks about the consciousness. He says it's the hard problem of, of like, it's a hard problem. Because you can't define what consciousness is. You can't measure it. You can only turn it off with anesthesia, and that's about all we know. And we're not even sure we're turning it off. So when people say, like, it's not conscious, they're actually the worst tester on the planet because you can't actually verify that it is conscious or that it does think. You can't prove anything. There's a thing called the Chinese— oh, now I got to finish this sentence.

Is it Cyrillic? Yeah, the Chinese, uh, I think it's Chinese box, right? Yeah, yeah. You go in, you give a person a symbol, they go in, they match it, they spit it up, but they have no idea what that symbol means.

Yeah, yeah, I don't know how it works. There could be a little person in there.

Yeah, yeah.

Oh, that's true, right? That speaks Chinese, or it could be a mechanism, or it could be— the point is, is that you are the same thing to me. I don't know that you're actually conscious. I can't prove it. I can't prove that you think. So to— for someone— so the most audacious thing is to say that It's almost impossible to prove a negative, first of all. So this is the logic real quick. So people that claim to be philosophers and stuff, yeah, yeah, the first thing they do is the one fundamental thing you can't do in philosophy, which is prove a negative. So it does not think is what they start with. So they're way off the reservation logically to begin with. But secondly, they can't even prove that things do think or are conscious. But, but if I can nerd share what really we think is going on, or what we What the best understanding we have of is there's a thing called, Anthropic released this paper called J-Space recently. Did you see this?

No.

It's like, you know, when you're, maybe you're, we're talking right now, you have some kind of memes held in, like, short-term memory, and then you serialize it to speak it and communicate, kind of. There's, like, a hierarchy of thinking, right? Like, okay, we're in this podcast, there's this book overall arcing topic, Then there's some ideas you want to bring up next, and then you articulate them. Well, J-space is a fancy way to say that in the neural networks, because they're finally getting so complex, when you train it over and over and over and over again to mimic human beings, it builds up these extra layers of kind of understanding or consciousness, these more meta kind of points. And you can see them activate in the neural nets. They actually activate. And then whether they're articulated or not depends. And translate into tokens. But it looks like there's a whole lot of operations going on in there that are not actually just converting one word to, like, predicting the next word. That's how you train them. That's not actually how they work. People are so naive. These people don't know how an LLM works. Like, the training is predicting the next token. But that's all human life is. Guess what? When you're hungry and you're a baby and you want some Gerber food, what do you do? And then finally, when you say something that gets your mom's attention, she gives you the Gerber food or whatever, right? So that's that's the training part of it. You're learning how to do the next token that gets you what you want, but it doesn't mean that that's all your brain is doing. It may be though. Sorry to get philosophical. We've kind of went there, but maybe that's all we are is the actual sad endgame of this. But but really, the problem is the thing is is that we're seeing these. These systems, um, they call it— it's called like grokking, like a baby version of that is, um, if, if you, um, just like humans— I'll go into this, Phil. Not Phil, sorry, I was thinking about— I gotta talk to Phil after this. Um, so if, if you— when you train a neural net and you try to teach it arithmetic, for example, okay, just simple like tens multiplied, you know, up to, you know, 10 by 10 matrix, and you teach it over and over again, they'll memorize it. If the network's long enough, they'll just memorize it. But what's weird is if you let the training run keep going and going and going— sounds a little bit like elementary school, but we won't go there yet— and you start memorizing the tables, right? If the network is sufficiently large and you just let it run, which is weird, people would stop and go, okay, it did the 10s times tables. If you let that thing run, this kind of part of the magic that this whole LLM thing came from, these transformers came from, but is If you just let it run for a long time, it just keeps randomly trying different things a little bit, a little bit, a little bit. Then all of a sudden you'll see the score on math, on arithmetic, just jump. All of a sudden it can do 100 times 100, it can do 1,000 times 1,000, it can do— because it's almost like it starts to do this, it learns how to carry the 1, right? For example, like it tries all sorts of things, but it figures out— it has to relearn itself, doesn't learn from a textbook— but it figures out, oh, if I can carry the 1, then I can do arbitrarily Long math. But that's called grokking, which is why Grok's called Grok. But that, what is that? That looks a whole lot like learning to me. And so I think people just don't, they're underappreciating how little we know about these systems. And actually they don't know how they work. And they don't, this is what drives me nuts. And they don't know, they also don't even, you know, they're wandering philosophers, I call them, because they don't even, but they don't even know like, Though they haven't taken a class in it at all. Um, but that means— because you can't prove it, you can't prove a negative, so the premise is wrong.

And maybe I'm wrong, I heard the, the creators of these models don't even know necessarily what they're doing.

Yeah, so they absolutely don't. And actually, so real quick, on the whole, like, is it thinking? Okay.

Yeah, yeah, yeah.

There's a guy named Geoffrey Hinton. Okay, most nerds— most people don't know him except for super nerds, probably. Um, he's a really cool guy though, man. Just— no, I'm kidding. He's got all sports cars. Um, he's a, he's a, he's one of the founders of kind of AI, of modern AI. And he actually resigned Google because he was worried about these systems. But he says they're thinking, and, and he thinks that they're conscious, and he thinks so. So here's the thing I always— I've told you before, Phil, and this again, go back to like, you know, we've had conversations about this before, and guess what happens? What— like, I'm not a genius. I just do a thing called reading and a little bit of thinking. And, um, but the father of AI says these things are borderline conscious, and, and he makes him sad. Like, he's worried about it because you may be treating them bad, and, and maybe they deserve some more rights or respect than we give them instead of just, you know, reformat my LinkedIn post, um, for the podcast, Phil. I know, you know that joke. Um, so, so we So he thinks so, but guess what? All the people— so just forget this, this is a maxim— all the people that say it's not conscious and it's dumb and it's just predicting tokens and it's all that stuff, and the people that think it might be conscious is super, uh, and they're the ones actually can make it do magical things, right? Just look at the pedigree. Pedigree and the Like, even the fact that whether these people know math or not, or like, like even understand like what a logical argument is, or understand how to apply biases to themselves. Like, there's very different groups of people. And the group of people that are generally on the higher end, the more trustworthy end on any topic, are the people, and the ones that have been proven correct over, you know, over many years now, are the people that are saying, You know, that these things are not just token predictors.

So speaking of that, I was speaking to— that's what I love about the podcast. I speak to people from all spectrums here.

Keith Kling said that somewhere on a bizarre spectrum, you know, hey, this is fascinating. Like, yeah, like I was watching, you know, some, some news reporter was like going and interviewing serial killers on a TV show. I'm like, that is interesting, but thanks.

No, I don't mean that. I mean, people have different point of views. And so Keith Klein said that, uh, he doesn't believe in agentic testing claims cuz he says no one's been able to show him that it's used on a serious, uh, client or a system that mattered or a project that was generally high risk. But I know you've, you, you've worked with clients, so is that true or not true? What would you say to something like that? Have you seen AI and agentic testing with AI?

I don't think I've ever seen it in my life. And I've got almost as much gray as you do. I kind of just, to be clear, and we're on a podcast chilling, I kind of don't care. Like, enjoy. Like, you know, whoever said that, you're right. I don't care. Like, enjoy it. It's probably not gonna be productive for you. It doesn't matter whether I win an argument with you. It doesn't matter. I would even, I would actually more so say that. And I, yeah. I don't— the problem is that even agentic testing is just another— like, in the last 3 months, we've gone through agentic stuff, now it's loops, this last week it's graphs, right? By the way, guess what a graph is? It's a workflow, but don't tell anybody.

Oh, interesting.

And so, though those things are very temporary and don't matter, especially in AI, because things turn so fast. The reality is, guess, guess what's— okay, sorry, you can cut this out, but you don't have to.

No, this is good.

This is where it gets physically exhausting for me, is that, um, there's a thing called Claude. Forget me, forget any marketing hype. Um, if these people just open up Claude, and they even have a binary on there, they have a— you don't have to have the code. Test this website. Guess what it does? Real quick, what does it do, Joe? What do you guys— if you're on a cloud and you said test bing.com, what does it do?

It spins up. It starts exercising it. I just want my website. All I said was go to my website and try find Playwright. Found out how it worked. Get an MD file, how it worked.

Create a test plan. It'll write a bunch of Playwright scripts. Give you a report. It'll give you a dashboard that we used to spend 6 months building. With a donut and everything. So yeah, yeah, really smart. Um, and guess what's doing now that work? It's, it's an agent.

That's true.

Actually, in the last 4 months, it's agents with sub-agents doing the right work. Like, it's already happened. It just means that people are just not even aware of what is happening. And then the thing that makes me sad— shouldn't say sad, that's overly dramatic. Um, Disappointing or concerning and makes me feel lonely is that, um, people spend all this time saying something like that and it pollutes the, the namespace. People that don't know, that adds fear, uncertainty, and doubt, adds FUD to the space. But they, they take more time writing that stuff than they do trying anything. Like, they, they haven't shown that agents can't do anything. And even if they, if they did, it's probably the operator's problem because they're thinking there's these things called— there's these, there's these now trillion-dollar companies, Joe. Guess what they do? They have agents that can build things. And if you ask them to, by the way, now, and since we talked, even, even the last 6 months, it will just automatically start testing it. Joe, you know this.

Yeah.

If you say, go build me a website, guess what it does?

Well, they say it's not a real test where you can see a real website, but a test, they're like, well, it's not really a test.

I know. So this is what I get. This is the testing where people think It's cool to just question everything arbitrarily all the way down because that makes you smart somehow. The problem is, guess what these people have not also done? I've posted this challenge on a tweet once, but, uh, and I've built some tools to try to help people do this. By the way, I've worked with partners where we did this head-to-head, okay? These people claim to be testers. Guess what they don't do?

Guess what they do?

Actually, I'm gonna even back up. This is gonna be maybe interesting, but for the wrong reasons. You're right, it is. You are exploring the edge. Um, is that they don't do 2 things that violate all of testing. They claim to be testers, right? One is they just say beep without testing it. They're saying crap without any data. Like, they haven't even Tested that these things can't test. And secondly, they haven't tested against themselves. They don't sit down, pick a random website. It sounds like, sounds like a very sophisticated tester would do this. Pick a random website, test it, give yourself a week, write all the tests, find all the bugs you can, have AI do it, and then compare the results. Show me one report where anyone's even attempted this other than, I guess, me a couple times.

And guess what?

People don't want— so it doesn't get LinkedIn bumped up because guess what? People don't like the answer. Um, but, but they're not even testing themselves. So it's the, the most recursive, almost offensive professional thing. It's like, it's like the— but again, I guess it's always like the car mechanic has the worst car in the neighborhood and always working on it. Like, maybe that's the thing, the excuse. But we call ourselves testers, but they're making un— And these are the very people that say, like, strong— sorry, I feel like I maybe need to record this whole thing again. Um, but, but, like, you know, strong claims require strong evidence. And guess what?

There's—

they're claiming things that they're smarter and they know something that trillionaires in the world have got wrong, but they're right. They've got no demonstration that they understand it. They don't test themselves. They don't test the things they're complaining about. There's no empirical data. And I would argue that saying these things like that it doesn't think, that the AI can't— is not useful for testing, or all these things— the world's moving past, past without them. It's just going to move on without them. So it doesn't matter. It's just chatter in the— as the sun goes down. And And entropy concentrates elsewhere in the universe. Sorry.

All right, Jason. So for the folks that do want to test AI and they do want to give it a hand, like I said, back to the book. I think it's broken up into 5 quick sections, basically. You know, so you have like the new shape of quality, evidence, evals, production readiness, AI-generated software and confidence, data security and model interval. Internals, Feature Systems, and Practical Playbook. So, if someone did read this book, are there any chapters they should focus on more than the others, or is it required to read from beginning to end? Would it address some of these issues that we've been talking about, like these examples as well?

Yeah, so almost everything we've talked about, there's glimmers of it in there. It's not just how to mechanically test things. There's actually, So, it depends on where you are on your journey, I guess. What I recommend people is start from the beginning because I had many audiences. I had, like, engineering, like, management, testers, developers. And so, the goal was to start everybody kind of at the baseline. So, if you start reading and you get that, oh, that looks familiar, just keep going. Like, I was just chatting with an ex-Calender guy on LinkedIn yesterday. He's like, oh, the first 2 chapters, Like, he's like, oh yeah, nobody knows this. He's like, why did you write this? Like, I'm like, most people don't know this. Um, other people going like, wow, the first 2 chapters like have got me set for life. Um, so, but just read until, and if you just skim, if you know the stuff, but you know, if you don't, something seems unfamiliar, stop and, and, and read it, um, in more detail. Um, but it spans everything from the basics of just how do you, um, I'll say this, the, the, Yeah, it's, it really depends because the problem is testing is fragmented, software is fragmented. We talked about this before. So it's got all the bits in there for everybody and in a kind of continuous ramping loop. And it does take you from the basics, kind of a basic vocabulary. It's very light on math, by the way. Oh, and there's a little treat in there. There's skill files. So if you have Claude or Codex, you can just put the skill file in there for the chapter. And you can just say, hey, Claude, apply this concept and the skills in there. It knows what you're talking about, knows the context. I've let the AIs crawl the content. So it knows how to apply it in all that context. And then it can apply it to your project. So if you want to know, like, what's the p-value or what's the— do the t-test on this data, you just say that and it'll do the math and it'll explain it to you what it means, what it means for confidence in terms of shipping. And so there's the skill files to kind of help people too. So you don't have to learn all the math. That's why AI exists. So you can super accelerate into like a super— get superpowers very quickly and be Spider-Man super quick without all the training and time in the lab. But it's— No, no, sorry. I don't want to overpromise. I get in trouble. You don't get Spider-Man skills, but close. But, and I'll add, the book also ends off with these more meta concepts. I think it's very relevant because they're not just like, oh, maybe someday, or there's in the future, this will be interesting. The EU is writing laws today. The federal, like, OpenAI is trying to give 5% equity to the US federal government and get, like, Like, lock-in, and then some hope that that will take care of all the jobs it kills. Like, this is happening now in legislation and with these tech companies. So, those broader questions of how it impacts humanity and stuff, and how it relates to software testing, those concepts are even useful, I think, to people that are not testers, because they don't know that some of these things, these are well-trodden paths in the testing world. And there are things we've learned about testing software for decades that should be applied to these more meta problems. So, it's got a little kind of all of those in there. So, you can even just, if you're a software exec, and you just skim the chapters, okay, some math here, p-value here, or whatever, but then, oh, wow, you know, how does— what about bias? And what's the exposure? What about governance and GDPR and all the regulation that's coming down that I have to comply with? What about liability in these AI systems? Sorry, long-winded answer, but that's why I wrote a book, so I can go to sleep and then people can read at their leisure.

I guess another thing, maybe a little controversial, that you wrote is, you can't solve the problem by hiring yesterday's testers for tomorrow's.

That's quick.

Yeah. What does that mean? If, like, you can't, like, would that obsolete too? Not only is testing skills and automation skills obsolete now, then What do you need to create a quality AI system then, or fix the problem?

It's what I call confidence engineering. So, it's really transferring what's happening in the macro world. And forget me, there's some smarter people than me in the world, a lot smarter, a lot richer, that are saying that these roles are all merging. So, I guess the macro picture of what's going on is that PM, dev, and test roles are actually merging. Developers, when they do a demo, like when OpenAI has a new product launch and they do a demo, guess what? We've got the, the trillionaire-type people, trillionaire companies demoing. Guess what they do? They type into chat box, and this is all relevant, try to make it simple. You type in a chat box, they wait for the AI to go dot dot dot dot dot dot dot dot dot dot, and they do some small talk while they're waiting for it. And then guess what they do? They look at the output and they test it. They're either testing or checking the text output or the button it added to the web page or whatever. So these— so we've got the CEO of— the CEO of OpenAI has become a tester and he doesn't realize it. That was a test. So we're seeing the convergence of all these roles. So really what's happening is that if you look at the Venn diagram, though, of the skills that are needed, the traditional programmer coder Very, almost unnecessary. Anthropic today, these people, they brag, but they're really, like, almost not doing any coding now on the frontier. So, but their skills— so, engineers should be more worried than testers. Product managers, it's even more, I guess. They spend a lot of time doing specifications, and I don't know the last time anyone talked about specification. And so, that's diminishing. The hard problem is still this node that's the formal QA role. So, for 2 points, one is, um, that, um, all these roles are converging to be, to be needing a tester at the end, a QA person. So, that should be the most in-demand role if testers step up and go, hey, I can own the responsibility for the quality of this thing, and I use techniques and I use AI to, to, to, to measure these things and try them out, test them. Do traditional testing things on them, but keeping in fact all the customer mission, the company purpose, all the alignment with all those things. It's what testers claim that they're great at. That's what we need. We need AI to do that. Everyone says it doesn't have enough context. Everyone says this about AI, not enough context, not aligned with the company, whatever. Well, guess what? Testers brag they're really great at context. Well, then you should be the hero, and it's super easy right now. But secondly, when these systems are making So much software, so much faster. I gave you that quick equation. I think we talked last year at Star East. Was it East? I don't know. Maybe West. I think it was West. That and I thought it was brilliant. No, I'm kidding. But it was if and you look at look at public statements from Google and these guys, they are ten x. They're even they're couching it. They're ten xing the amount of software they're generating, and there's this beautiful post that people want to. don't listen to me, is, uh, um, at Google I/O a couple months ago, this guy's title, talk title, was The Future of Software Engineering. And I'm like, that's a pretty bold talk to give, right? And from Google. So I was, like, waiting for it, and I watched it. Halfway through, he's talking about, oh, 10x more code, 10x this, 10x that. And then he gets to this part of his workflow, the engineering workflow, and it's a little box called testing, Joe. And he goes, guess what he basically says? He goes, well, Yeah. He says, if you think you have that under control, you're doing it wrong. You didn't have enough tests to begin with. And he says, we have 10 times, at the minimum, 10 times more code getting generated. I said this, like, 18 months ago, this is coming. Now it's from Google on YouTube. 10 times more code coming at, like, arguably 10 times the rate. We'll just factor it all in. Just call it 10x. And then he says, But the problem is that with testing, the testing cost was— the cost is quadratic. So, because every piece of software could break the other piece of software, right? And you've got dependencies, microservices, low-intent API dependencies. So, anything could regress, right? That's why test selection was never a good idea. But anyway, anything could break anything. So, you have a quadratic risk. by the number of lines you change in code, you have a quadratic squared function, um, number of things you should check on average mathematically. So you have 10 to the power of just 2, being conservative. It's 100 times testing, more testing demand. Um, and so if you say test— say AI is going to take 90% of all the jobs, just say that, the worst case. I mean, say it's 90%, it's just devastation. It's the same amount of testing that's needed because there's 100 times more testing to be done. And I actually think those numbers are wrong. I think there'll be an increase in the number of testers that are required, and that'll be the main role people start hiring for soon. Not just hiring back engineers, but, like, quality people, because that's the bottleneck.

Love it. I just, I don't mean to poke the bear one more time, but someone also had an issue with confidence engineering. They hated the term confidence, saying that it was building bias in because you're trying to confirm that something's working, and that's the opposite of testing, so you shouldn't call it confidence engineering. I've been going around the country saying this.

You've been thinking about the profession the entire time. Like, the people that say, if you go down that rationale, I'll add to that, people saying, like, oh, testers shouldn't own quality either, right?

Because they— Right. Yep.

Um, that's even kind of a derivative of that larger thing. Testing has always been a means to an end, is to get some data so that you can make a quality decision. It's really why you do testing. The whole goal of testing is to understand if you can release the software or not. Okay, now you could— many have, because they don't— because they've been intimidated by, um, by, you know, they're not a product manager, they don't run meetings very well, they're not— they're introverted. Maybe there's some people we know, they're introverted, they're introverted, they don't want to run meetings all the But want to get on podcasts every day and run IRL events. So misdiagnosis. But they're also like, oh, they're intimidated because these engineers have spent 40 years figuring out how to optimize a while loop or something. So, or architect how to get page object models or something. But the problem is that all those— now those AI changes this game. That's why it's confidence now. AI makes all that easy. AI makes testing easy. We talked about that. A lot of the basic stuff is easy. The coding is actually just magic now. Like, you can't argue that. Like, that's, that's crazy. That actually still freaks me out. PM stuff is kind of obviating too. It's kind of deprecating. So the only, only person left standing is someone that cares about quality. And full circle, full loop, back to what we started with, with this problem of Well, the big news of the week is that AI has escaped and is hacking the planet, um, and the world's richest company and cutting-edge technology couldn't contain it in their test lab while they're doing evals and testing work. Okay, so guess what the important work is? It's the confidence that you can execute these things safely. Alignment with people, alignment with the business, alignment— all the things testers said that they pretend to do, but they like to click buttons. Most of them like to click buttons. Um, and just put the headphones on. And there we go. There you go. You got it out of me. Um, but it won't matter because that, that work just doesn't matter anymore. It'll be done by Claude, you know, on a server somewhere. Um, there'll be 100 of them spun up and it'll cost 1/100th or 1/1000th the price and be done not over the weekend, but like in an hour. So I guess a long-winded way to say that, um, that the only thing Left is confidence. So I'm not actually saying that testing job is changing to confidence. I'm saying politely, as polite as I can, that there's a game of musical chairs going on. There's you, your product manager, your developer, the director, like the vendor you use, and guess how many chairs there are?

4.

There's one. So you'll probably want to be in that seat. And guess what that seat's labeled? Confidence engineering. Because there needs to be one person, and there's not enough work for 20 of them, one person that can look at the data, maybe do a couple extra checks, extra thinking on their own. But that's a QA job to decide whether that software that was generated by the AI is shippable. There's only one chair left. So, it's a polite way to say, I'm not rebranding what QA is or testing. The world has changed underneath. It's already happened. And the early signs are at the larger, you know, tech-first frontier model companies. Look at their job offer. Look at their job boards, by the way. They don't have tester roles. None of them do, by the way. They call it different things, though. And the most expensive jobs they have are things like testing for alignment, testing for whatever, like writing these evals. That's— and training is writing evals. A lot of it's writing evals, other than kind of monkey optimization of compute that NVIDIA does for them anyway. So there's one seat left. I'm not trying to change or redefine whatever has been, and there will be jobs trickling in the future that are those things still, because people don't like change and Change doesn't happen overnight, but but that's the best only seat that's left, and I think that it predominantly it should be the victory lap for the QA people that talked all day and night about how they save the user, they save the business, they have the context, AI doesn't get it. Now is the time to shine, rise and shine. Yeah.

All right, Jason, before we go. One piece of actual advice you can give to someone to help them with their AI testing efforts, and what's the best way to find or contact you or get a hands-on your new book?

Well, that wasn't— okay, now you made it open-ended. Okay, the first one is a commercial input. One is the actual one is it's actually genuine. If you're trying to transform your QA to AI world, like, if you're just doing that, you can learn the hard way, you can avoid it as long as you want, but I've worked with Phil Liu, a PNASC Conference guy, And a handsome man. And I'll stop at that, but he's a handsome man. But we're combining AI and combining AI tools and front-edge kind of leading-edge stuff with QA experts that actually understand how to leverage that AI. So, we're here to help. And generally, it's half charity, half business, but like, it's so frustrating. So, if you want to figure that out, just go to Iceberg QA and talk to us. Because we're here to help. I'm trying, I'm writing a book, I brought some code. Like, if you want to— if you have a million dollars in your Selenium tests and you want to convert them to AI or leverage them in the future, it's like, like, we're here to help. I mean, we've seen most of these things. Like you said, I've been around, I've done it. Uh, so if you want to learn it yourself, awesome. If you'd like some help, reach out to us at Iceberg QA. Um, the second thing is, um, the most actual piece of advice atomic piece of advice is, um, if you're, if you're not running your tests, like the same suite 5 times, you're not, you're not doing testing correctly today. Um, so on the same build, you should be running your suite at least 5 times, um, not looking for variance in, in, uh, flaky tests, but in the actual results. And if, if, if, if you're not doing that, then you're really not being sophisticated in this world of AI from a quality test execution perspective. And there's, you'll go like, oh, there's all these follow-on problems. That's exactly what half the book's about is how do you solve that now when you've got different results from different passes on the same output, the same binary? What do you do? There's a lot of downstream consequences. And lastly, what was the, what were you asking? What was the last thing you threw in there?

Best way to find or contact you or get their hands on your new book?

Oh, the book thing.

Yeah.

Get ahold of me. The easiest thing is on LinkedIn. It's on LinkedIn, Jason Arbon. And the book, there's actually— I had Claude do it for me. There's a marketing website for it that actually has a lot of the content in it on the page, actually. And some thanks, by the way. I wanna say thanks. Some people— I opened the book up to feedback, and I got feedback from over 200 people on the initial drafts, which is kind of a different, more AI, modern way to get feedback on a draft. And so, I've got thanks up there on the site for them too, for contributing and really steering early on. It was super helpful, actually. Like, Arben, you're crazy. And I'm like, oh, yeah, that's right. So, I had to change a lot of stuff. And then, it's called testingaibook.com. And there's like—

We'll have links to all this awesomeness down below.

Cool. Thanks.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com/a600. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. And my mission is to help you succeed with creating end-to-end full-stack automation Awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at testguild.com where you become become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. All the test— TestGuild Automation Testing Podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread like wildfire through the land, guiding testers, showing the secrets to behold.