How to Move from Prompt Engineering to Harness Engineering in Testing

Hey, what if the future of testing isn't actually writing tests at all, but engineering the system that produces the code? Matt Wynne, who was a longtime Cucumber core team member, is back on the show after more than a decade, and he spent 2 years inside Mechanical Orchard, which is a Silicon Valley startup modernizing legacy mainframe code with LLMs. He shares how he brushed up against the idea of the software factory, a hands-off way of of building where humans never write or read the code, and it changed how he thinks about his own value. I'm sure a lot of you are going through this as well, so you definitely wanna hear and listen how he went from burying his head in the sand about AI to encoding his entire Gherkin coaching practice into 500 lines of Markdown and admits that this was horrifying as it was amazing as well. Now he's teaching testers and developers how to stop prompting and stop building harnesses. And validation loops that let you trust the code you'll actually never read. So if you're drowning in AI-generated pull requests or you're not even sure what your job even looks like in a genetic world, you wanna stay with us all the way to the end, cuz I think if you listen to this episode, it's gonna change or reframe how you think about your role in the future. You don't wanna miss it. Check it out. Hey Matt, welcome back to the show.

Thanks Joe. Yeah, it's good to be back. It's been a long time.

My gosh, I was just looking at it. I think you were on episode 35. I'm almost on episode 600. Wow. Yeah, it was over a decade ago.

Blast from the past.

No doubt.

Yeah, I can't say I remember that much about what we talked about last time. I could guess.

What would you guess?

Was it to do with BDD?

It was indeed, yes. So I'm just curious to know, it's funny how BDD almost seems like it's the user interface of a lot of testing tools I see now using natural language and it just goes out and does the code behind the scenes. Do you find that odd or do you see that as well?

I don't know if I see it. I'm not sure if I'm spending enough time in that space to know what all of the testing tools are doing. But I think the given-when-then structure is a sort of universal language of how tests are organized, right? We used to call it arrange, act, assert before Chris Matz came up with those three keywords. It's just, it's how these little nuggets of behavior, I set something up, I do something to the system, I see what state it was left in afterwards. It's a very natural unit of describing behavior. So I can see why it spread. Absolutely.

Also, I am curious to get your thoughts on this, you know, with the Agentic AI. Now that a lot of the skills in going to solve a problem in automation or in development or testing goes to an agent instead of a developer, is Do you still have the same communication bottlenecks that you found back when BDD was the rage when you were all into it?

The same communication bottlenecks? I mean, I don't know, like, I think the bottleneck— it's really interesting. I'm still puzzling over the bottleneck thing because, you know, I never had one of those t-shirts, but I kind of believed when people had those t-shirts that said, "Typing is not the bottleneck." You know those ones? Yeah. And yet, what these machines are doing is saving us a whole load of typing. We can sort of wish our wishes at the genie, and then the typing comes back at us. We don't have to do the typing anymore. If we can say the right words, we get roughly the typing we wanted. And it is definitely changing things, having that. So maybe typing was the bottleneck in some situations. But I still think that a lot of times, if you zoom out from the typing, the problem was always shared understanding, like shared understanding between the team that are producing the thing to solve the business problem, or maybe even shared understanding between those people who are trying to solve a problem and the world out there that has a problem. Like maybe we haven't even understood the problem properly, right? And we're building something that's useless that nobody needs, or it's missing the target. So I think it depends how far you zoom out. And maybe if, yeah, we've really understood the business problem, we all understand the business problem together and we have a shared understanding of that, then typing is the bottleneck, right? Like now it's just a matter of how fast can we crank out value to meet, to match that problem. And if we're in that lucky position, then the LLMs can really help us. I think the LLMs can also help us with some of the other stuff too, because we are, I think, currently a bit locked into this mindset that the best thing to use LLMs for is to generate code, but they can do lots of stuff to organize and synthesize meaning out of tokens, out of words, including user interviews to help you to better understand what the problem space is.

Yeah, I love that. And it seems like you've been going more into AI, and I, I believe, uh, what caught my attention is you— I think you released a course recently. I saw it on LinkedIn. So maybe talk a little bit about like what you're doing now, and maybe, um, well, we'll dive into that a little bit more.

Yeah, so I've just had two really— a bit over two really fascinating years at this startup in Silicon Valley called Mechanical Orchard. It was founded by Rob Mead, who founded Pivotal Labs, which is the sort of, back in the day, was like the most prestigious, I would say, consultancy that was wholeheartedly and proudly using extreme programming practices in their consulting. So, they would come in and either set up an XP team and deliver a thing for you, or teach your organization how to use XP and have it spread around your organization. So when I landed there, what I found was there were all of these people that, like me, had years of TDD and pair programming and refactoring practice. And we landed there just in this moment when LLMs were starting to become useful. And Rob was just very, very, very encouraging to all of us to just, you know, embrace this technology and try and learn as much as we could about it. So I was in this really lucky position to be like amongst all of these really experienced Agile people, like people that have been doing Agile for like 20 years, and have a boss that was just encouraging me to play with tokens and see what we could do with them as much as possible. This is a really cool learning environment. We're trying to solve a really hard problem, like modernizing legacy code from mainframes, like COBOL code, trying to understand what these, those computers are sometimes like 30, 40 years old, right? These code bases, trying to understand what they do, modernize them. So trying to solve a really hard problem with this new technology, with brilliant people that were really steeped in Agile XP practices, but just an amazing learning environment. So And one of the opportunities I got with that was I got to go hang out for a week with this guy, Justin McCarthy, who was the CTO of StrongDM. And I don't know if you've seen any of the stuff that they published while he was there, but they're basically the people that coined the term the software factory or the dark factory. So this week with Justin just blew my mind, basically. On the scale of ambition that we can have about what we can use LLMs to produce for us, right? Like this way of building software that's totally hands-off. We're really, we're not, like the whole, the rule that they set for themselves to see if it was possible was, what if the humans never write the code, never read the code? What if that was true? What would have to be true for that to be possible? So if you think about it, as a tester, it's terrifying, right? No human's ever going to read the code.

Why would you want to? Yeah.

How do we know whether it works? How do we know whether it's going to be performant? How do we know whether it's going to be secure? How do we know whether it's going to be malleable for the long term? But actually, so many of these things are things that we can check using LLMs, right? We can find out, we can inspect the changes and find out whether they are going to be malleable, whether they are following security best practices. And if they're not, we can heal the code and make changes to it. And if you start to move your work up to this level where what you're doing rather than steering the agent itself, is you're steering the system that the agents work in. So, you're creating these guardrails that the agents have to work within. You're using that non-determinism in a way that makes like the— you're kind of creating a slope so that the behavior, the software that you want, is going to inevitably fall out of that system. So it's a really different mindset because you're having to think about what constraints, what context do I give to the LLM so that they're able to work like that. But the results are phenomenal. Some of the stuff we built that week with Justin, some of the stuff we built afterwards was just amazing. And yeah, I'm really excited about this as a technique. I think it's sort of brought a joy back for me around software development, because it feels like TDD again in so many ways. Because if you want to be able to go hands-off, right, you're going to have to set up this validation loop, and you have to be able to describe to the system, you have to be able to set up your harness to tell it what good looks like. So you have to know, you have to express, what do I mean when I say this thing has been validated, this thing is good? So you have to be able to think about how do I set up that system to validate the output that the random probabilistic machines are going to generate and help it to hone in on the solution that you want.

You don't come off as like a hype guy, you know, like that's drinking the Kool-Aid. So I'm kind of surprised. I mean, it seems like you're all in on this and this is something that I know a lot of developers and testers are terrified by. How did you get there? Like, 'cause I know a lot of people are resisting it or they're like, it's BS. It's just LLMs trying to up their price for when they go, you know.

Yeah, well, I mean, I think I have a lot of reservations about the capitalism at play here. I see these three big players, Anthropic, OpenAI, and Google hoovering up a lot of money that was previously going to humans. Our bosses are paying for tokens from these people rather than paying us to do the work. And if you think about what is happening to our industry happening to many other white-collar industries, you can see that these machines are amplifiers of whatever is happening in our society. They're amplifying The words, the kinds of words that we use come back at us, right? This is why every generated PRD that you read that Claude or Codex has generated reads like some awful Silicon Valley excerpt, right? With all of these buzz phrases that it just puts everywhere. This is why these AI systems are biased towards picking minority ethnic people as being more suspect in police systems. They're just amplifying the existing problems we have. And the same goes for this disparity of wealth problem. They're just amplifying, moving the money towards an ever smaller number of people. So I totally get why people are frustrated about that. But I think that that's like blaming the technology, the large language models, is pointing our ire at the wrong place. We should be pointing our ire at the capitalist system and the way it's organized right now, because the technology is really interesting and can do really interesting things. And we've always built bullshit software, right? For the last several decades, people have been writing software that is useless or even detrimental to humanity, right? And now we can just generate more of it. What a shame. But it's— yeah, I just think we're kind of missing the point by blaming the technology. But the other part of it, I think, is I felt this kind of like around September, October. I spent a long time actually at Mechanical Orchard around all of this buzz. Kind of going, I'm just going to bury my head in the sand and hope this thing goes away because I'm good at this and I don't need these tools. And sometime around about 18 months ago, I started to kind of go, hold on a minute, they're not going away. Some people seem to be getting results out of this. I'm curious. I'd better lean into this and try and figure out what it's all about. And kind of within 6 months of that or so, I found myself starting to write these skills. Say example mapping, right? The thing we probably talked about when I was here 10 years ago. I'm thinking I could probably get the LLM to help me with this. I'll write a skill. Oh, gosh, this thing is really good at writing good gherkin. I had a skill. I used to get paid to go into organizations and coach people on how to write better Gherkin. I've now encapsulated that in 500 lines of Markdown, and you can run it on your feature files, and it'll just be like having me in the room, kind of, not far off, maybe not as good jokes, but it's really pretty good. And that is sort of horrifying, right? It's amazing and horrifying at the same time, because what is left of me? How am I special anymore? If I can encode my knowledge, my skills into these markdown files and the machines can just do it, what am I for? And I think a lot of us who are sort of a bit hypey, like, are excited, let's say, like I am, have been through that. I think actually everybody's been through that. One of my friends described it as like a grieving.

I was just thinking that, the stages of grief, denial, then acceptance. Yeah.

Yeah, the grieving is real, as the LLMs like to say. But yeah, I think that was definitely my experience, was it's like, my identity is tied up in a bunch of stuff that, in being good at a bunch of stuff that these machines can now do. So what is left of me? And I think I've come to a place of understanding that They're like, you know, like I said, they're like amplifiers, they're like robot arms. I can just reach further, I can do more stuff with these things, and still my problem solving, my understanding what the problem domain is, my prioritization of what we should solve in the problem domain, my understanding of software architecture and how we want to organize the code is all still necessary in this space because these things aren't, they're not actually intelligent. They're just transformers of text. They take some text and they turn it into some other tokens and turn it into some other tokens. It's just like, which tokens do we point them at so that we get the tokens that we wanted out? And we have to still be able to understand the whole process to be able to use them effectively.

So I agree. I know some people describe it as the ultimate BS machine where it looks right, but if you don't know the domain, you can get in trouble. I don't know if you've experienced that, because it does produce what looks like incredible results. And then once you start diving in, you're like, this doesn't make any sense. It's not as good as I thought, you know, and then it starts degrading over time. So what you described sounds like next level though. It's almost like a human on the loop where it just it does loops and I don't know, like, what? Well, I don't know what my question is there, but I think like the first—

because I, I was actually talking about this yesterday, that there's kind of like, it's like the Dunning-Kruger thing. There's a, there's a sort of, there's a— initially when you're encountering an LLM, you're like, wow, yeah, this is like magic. And then you kind of go through this disillusionment where you realize that it, it— the magic is not as good as you thought it was, and it's coming up with a lot of bullshit. And then there's this kind of third— I want to say transcendence, but it sounds really pretentious. But there's this kind of third space you get to where you're like, well, of course it's bullshit because it's just an inference machine. That's all it's doing. So now I have to think about, well, how do I use— okay, but given that, how do I still use it to my advantage? So what do I put in? Because the thing is, it's like The best thing you can get an LLM to do, right, is have a really long conversation like this one. Oh, it's not going to be really long, but you know, have a long conversation or like sit down, record a transcript with a customer and then give that to the LLM and say, right, just give me like the 5 most salient points or write me a proposal based on that conversation I just had with the client. And all it's doing is it's It's taking the input, the information that you already have as the input, and compressing it. It's— Lada Kessler describes it as like semantic zoom. So I'm going to just zoom out like you would zoom out in a map. So it's not going to lose any information there because it already had all the information. You're just asking it to summarize it using its giant bucket of all of the words that have ever been written by humankind. To organize the structure of it so that it's going to be legible, it's going to be readable for you. But it's got all of the data there to start off with. It's when you ask it to go outside of that, to go and rifle in its giant bucket for stuff, that it might start making up bullshit. If you start with a tiny seed and then you say, okay, just go and rifle in your giant bucket and try and make something up for me that's an answer for this, Well, there's a much stronger chance now that the answer that you get back is going to contain some bullshit. So, it's really about accepting that that's going to happen and then thinking about either how do I provide it with the right information on the way in? So, that's like my context engineering, where what files am I leaving around the place to guide it and give it signposts, which is actually just what we needed to do for ourselves as humans anyway in these codebases, right? But now we can't rely on folklore. It has to be legible in the codebase. And then when it has produced something, it's probably going to have some bullshit in it. So what's my harness doing to catch that bullshit and find it? Because the thing is, humans make mistakes too. This is kind of one of the other epiphanies I had about this was like, when I would talk with Rob about this stuff, From Rob's level, humans make mistakes too. He couldn't really understand why people were objecting and saying, "Well, these things make mistakes," because from his point of view, humans also make mistakes. And actually, one of the things we teach in the course is this idea that I got from Justin of using the three big brains to review a piece of code or a plan. To review a thing, if you get— because most people are just locked into whichever— they're an Anthropic shop or they're a Codex shop. But if you take your plan or your pull request or whatever and you want to review it, if you get an Anthropic, an Opus or a Sonnet, a a GPT model and a Gemini model to all review that artifact, just like how if you have diverse humans on a team, those 3 models are trained with totally different input data. They've got different training weights. They will come back. They will see different things than the others saw. A lot of them will see the same things, but there'll be some overlap, but there'll also be unique things that they each see. And that can be a really good way of like playing the averages and giving yourself more information to put into that healing loop to fix the bullshit.

Yeah, that's something a lot of testers talk about. Like, you shouldn't use the same LLM to test it that you did to code it. And that's a good point. If you put it through 3 or 4 different LLMs to get a perspective.

I think even just using a new, a fresh session for from the same model is still huge.

Okay, okay, not the model, right, true. Yeah. Right, right.

'Cause it's coming at it with a fresh pair of eyes, if you will.

That's true, that's so true. So, you know, so you talked a little bit about the course, moving from prompt AI to harness engineering. I keep hearing new terms. What is harness engineering?

So this is, I'm not sure if I can say a surname correctly, Brigitte Bockler's, term. I think she's a principal engineer at ThoughtWorks and she's written about this a lot. So the harness is really like the stuff you put around the LLM. It could be just Claude code, but that's kind of not a great harness on its own. So you may well extend Claude code with with skills so that there are going to be prompts that will kick in automatically if you're in a certain situation. You might have hooks that are going to run automatically in certain situations, so deterministic hooks that run. But it's also all of the other infrastructure that you put around the work that the LLM is doing. Like, when do you run your automated tests? When do you fire up an adversarial sub-agent? To read the changes that have just been made and tell you all of the things that it thinks are wrong with it. So your harness is like the system that you're building around those basic calls to the LLM to try and constrain the work it's doing and narrow it in on where you want it to go.

All right. Sounds difficult. How do you get started? Like, so are we moving, like, as you said at the beginning, we're moving away from like what would make it true if it would code and test itself and just almost loop. So are we moving away from prompting then to where we have the systems all set up ahead of time and then we just become the person on the loop rather than in the loop?

Ideally. Ideally. But you've got to start from somewhere, right? So I think about the kind of, all right, so if we zoom back, if we go back, I don't know how long it has to be for you, but like a year, 18 months ago, whatever. Many of us, our experience of using LLMs was copying and pasting this bit of code that doesn't work or whatever into a ChatGPT window in a browser, pressing the help me button, and then copying and pasting the code back out again. It was like Stack Overflow on steroids. That was what we had. Then we had Cursor and we get this slightly annoying tab complete thing where it was flashing up with its guess at what you thought the code would need to be. But if you are beyond that and you are tending to write your code by sitting in one of these terminal apps like Claude Code and you're sending instructions, you can think about that as being in a REPL. In a programming language like Ruby, where you can program like that, right? But it's kind of laborious. And you may well find yourself repeating a lot of the same things over and over, re-explaining things to the machine. And I think it's a fine place to start, but the thing you need to be doing is always asking, like, why do I have to do this? Why do I have to explain this to you? What— and the analogy I used the other day was like, if you— because this is a really common problem you, you hear from people is, well, I prompt the agent and it produces this code, but then I have to spend just as much time repairing that code to be like the code I would have written as it would have taken me to write in the first place. But like, and, and I'm not trying to answer anthropomorphize these things, right? They are machines.

Yes.

But let's imagine that that was a human. You delegated this task to a junior and they'd come back with this code. Would you just take it away and fix it? Or would you— maybe you'd sit down with them and coach them, right? And that would be a learning opportunity for that human. There's no point in sitting and coaching in this Claude session, really, because it's going to forget everything when you start a new session. Unless, we can come back to that. But what if you said to yourself, "Well, I'm disappointed with the work that this human did, but how can I take responsibility for that? What context could I have given them that would've made them more successful? What do I wish they'd have known? Should I have written down some architectural decision records so that they knew that, okay, in this application, we use an MVC pattern," so they wouldn't have, like, written all of their code in the controller. Or did they— yeah, that's a pretty reasonable example.

It's like a sprint review, but with agents. Like a sprint.

So a sprint review, though, would be kind of like, yeah, it would be coming along afterwards and thinking about, yeah, how did it go? How could it have gone better? And I think that's really the thing. It's like when you finish that REPL session, have a little retro with the agent and start to think about what could we do better next time so that I won't have to keep repeating myself? What was missing? And sure, you can have that conversation with the agent. It won't necessarily give you good guidance. You've got to use your skill and knowledge of the codebase that you're working in and what you want. But if you can think about it as, okay, if this went wrong, how can I take responsibility for that? What context could I give it so that next time it will perform better? Gradually, you're going to start getting better results from them. And then the other side of it is, okay, it's produced this wall of text. We're doing this. It's going to be I guess happening before this podcast comes out, but it'll be on the Maven website, this free Lightning lesson on loading human context. Because one of the big things we see is they just spit out a wall of text, right? Either it's like, I've got this plan, blah, there's like 5 pages of plan that you're supposed to read, or it generates this giant pull request, 2,000-line pull request. What's in it takes a lot of work to figure out. But you can, rather than reading it line by line, like you would have had to in the old days, you can use another LLM to inspect that thing, to give you a projection of that wall of text into something that is meaningful to you. So if I'm looking at a large pull request, maybe what I actually want is like a sequence diagram of what are the, how are the objects interacting interacting in this change. What's the impact of this change on the behavior of the system? How's it architected? I can ask the LLM to generate a diagram for me that makes that visual, makes it easier for my brain to understand what's in the change. And again, if we're just talking about a transformation where it has all of the information and what we're doing is reorganizing that information into a form that's useful for me, there's not going to be bullshit in there. It's not going to have to go and reach in its magic bucket.

Right. Because it has the context.

Yeah.

It's not going to hopefully hallucinate. And I think your course had a bullet that interested me. It said something about, "You're drowning in agentic PRs." And I know I've talked to other people that said, "With all this AI-generated code, it's the PRs that are going to be the next bottleneck." But this sounds like this is the solution then to that.

Yeah. And it's scary. And I think we still have to go back to this thing of like, did we definitely need all of this code? But if we are going to have the machines generating the code, we can't be expected to have to read it all. We have to give ourselves tools to allow us to trust that code. So then the question becomes, okay, well, This is like the trick, right, is if you say to yourself, I'm not going to read that code, but I need to ship it to prod. What do I need to know about that code so that I could trust that I could ship it to prod? What are all of the questions I need to ask? And you'd be surprised how many of those checks you could automate with an LLM. You could get an LLM to do those checks. And if you don't trust the first one, run it 3 times, run it 5 times, 5 different sessions, right? Law of, you know, eventually, if there was a problem there, it will find it because each of those sessions is a fresh attempt. And gradually you're going to— so every time you're doing a task, be thinking about, you know, why do I have to do this? Is this not a thing that I could delegate to an LLM to do for me? So the LLM can go and get me the information and then I can make the decisions.

Absolutely. Another thing that caught my attention was, and maybe this is just me, uh, no one else thinks this, uh, something about working in, uh, brownfield code bases, uh, with landmines all around you. I used to work at GE Healthcare. I worked on a system that actually had a 30-year-old mainframe running it behind the scenes. And so when people hear AI, they think, oh, it only needs to be like completely greenfield, not brownfield. Is that true? Or can you really legitimately use AI for even these older applications?

Well, it depends what you're trying to do with the older applications. What we were doing at Mechanical Orchard was taking the entire legacy mainframe estate and gradually moving it into the cloud, into Java. And these tools were amazing at helping us to do that work, not just, I mean, I don't wanna, you know, I don't need to sell that business, but many organizations in that space are just taking COBOL and making JoeBot. They just transpile the COBOL into Java. What we were able to do with the LLMs was put them into a loop where, okay, you've got this set of mysterious code, and here's a metric of how much coverage you have of that code, right? Sometimes we would pull it into GNU COBOL, so you could measure code coverage with the GCC tooling. Other times we just instrument the legacy code, so you could see the coverage. Either way, here's a feedback loop that says, right, this is how much coverage you have. "now generate me test cases." So the, the LLM could just keep fuzzing until it had a set of characterization tests for that code. And now you can take those characterization tests, it's like a mold, and you can cast a Java version that does exactly the same thing as that, that old version did. And all of that is like deterministic checks, right? Because you've generated a set of tests that are deterministic, but the LLM was helping you to get there. And it's this thought about like, how, how can the LLM be useful? How can the transformer be useful and speed up a thing that I could do by hand, but I don't want to have to, or it's going to be too laborious to do it by hand. So that was like the rescuing legacy. I think the other thing about Brownfield is, so I'm consulting with this company right now. I've got a codebase that's like 10 years old and there's a lot of archaeological layers in there. Right. There's a lot of versions of doing things that are old and decrepit and we don't like. And then there are shiny new ways of doing the same thing. And one of the issues that people have been having is, you know, when I generate a new thing, it follows the wrong patterns. It follows the old patterns or it gets mixed up and does a kind of mix of the two. It's not following the best practices in the codebase. But this is because those best practices aren't legible. So what we've been working through as a crew is capturing the implicit folklore architectural decisions that the team have been making over recent years about how they want the code to be so that they're written down in the code base so that the agents can find them, so the agents can generate code that's the shape that they would have generated. The other thing I've been doing with them that's really interesting Do you know anything about Rebecca Wurf's Brock and her work on design heuristics? Have you ever seen any of that stuff?

No, no.

So Rebecca wrote all these wonderful books back in the '90s, 2000s about object design, like responsibility-driven design is her thing. And actually, she inspired Eric Evans' domain-driven design, I think, in a lot of ways.

Okay, yeah. Yes, yes, yes.

Lately, Rebecca's been really interested in this thing of capturing measuring your design heuristics. So, if you are someone who designs software, what are these rules of thumb that you tend to apply, you know, your preferences, your tastes about how you like software to be? And an experiment I've been doing with this client is when the humans review pull requests, we have so much data now in these pull request reviews of what this team's design heuristics are, because they'll take a candidate set of code and they'll litter it with comments. They'll cover it with comments saying, oh, I wish it was, we tend to do it like this here, or I think I would prefer it if it was like that. And I've had an agent go over those pull request reviews to try and distill down some design heuristics that the team has about how they like to organize their code. Because every code has like every, Team has like a house style, right? Try and distill what that house style is. And again, document that in the codebase so that the agents that are generating new code are more likely to generate it in the style that the team would have written anyway to the team's preferences. It's going to have less, you know, churn and comment in when it goes through a pull request review.

Very cool. I want to do some quick lightning questions. All right, so what's the future of testers? I know you have something in your copy that says code reviewer-in-chief. Is that where we're all heading? Like, where— what are we morphing into? Just one generic role of, like, thinking engineer? I don't know. I don't know.

I mean, I definitely, like, I know some people that just say, you know, we're all just builders now. And I saw stuff happening at Mechanical Orchard where we had, it was definitely like blending together. We had, especially what I saw was people on the product side becoming empowered to start making changes to the codebase, which is just amazing. Like people, even if it was just a spike, like a prototype, to be able to make something real and play with it and see it rather than just designing wireframes. Really, really cool. But I definitely saw people who were coming from a product owner role actually start implementing features in the product. So there was definitely a convergence happening there. And let's see. I mean, it's been a while since I worked anywhere where there was like a dedicated tester role. I think for a long time, I've thought of testing as a specialism within the set of T-shaped people that we want to have on a software team anyway. So, some people, their brains are just better wired at thinking about what could go wrong, right? And I don't see why that isn't still useful. In fact, maybe more useful than ever. If we go back to what I was talking about, about this sort of, what I'm trying to call lean software production way of looking at things, right? This hands-off software development way of looking at things where the engineering that we're doing now is we're engineering the system that produces the code rather than engineering the code itself. We really need people that can test that system, right? And can think about where are the leaks in that system? Where's it going to go wrong and how do we mitigate that? So I think that skill, that mindset is still just as valuable as ever. But I think it's important to start to really skill up and understand how these things work. Like, read the Attention Is All You Need paper and understand they aren't magic genies. They are just transformers. And once we start to kind of recognize how the technology works, it becomes easier to test it, right? And work with it. Same as when we're testing any other software, right? Like if we just think of it as a black box, we can't necessarily break it as well as if we look inside.

Love it. And Matt, really quick, job replacement or more jobs, not less?

Well, you already heard me talking about bullshit software. I think as long as there's money around, they'll be paying us to make stuff. I think, yeah, realistically, there may become a crisis where the money's all going to the Silicon Valley shareholders and there just isn't any money to spend. And then they realize there's a problem. But until we get there, I don't know how you say this guy's paradox. I want to say Jevon. Jevons. Jevons paradox, right? You build an extra lane on the freeway and you don't get less congestion, you just get more cars. My suspicion is that's what's going to happen with all of this. We're just going to get— I mean, how many— If you're in the position of being a product owner, you probably know how often you have to say no in that job. Now you just get to say no less, is kind of what I'm seeing. So I don't think we've got a problem there. But I think it might reorganize things. I think the way we organize ourselves in teams when we are moving faster, when things are changing faster, because we are doing more stuff, I think it gets harder and harder to share that context across the human brains, right? So what would have needed a team of 8 to get it done, like that team of 8, if they, let's say they can do twice as much stuff as they could do before, maybe actually what you need is 4 people on that team rather than 8 people trying to keep up with twice as much stuff happening, right? So we have to organize ourselves differently and break that into 2 teams somehow. And I don't know yet how that's going to look, but I think that's an issue that we haven't really thought about yet. I think we're still organizing teams and structuring organizations in quite an old-fashioned way, and we haven't really recognized the impact this is going to have on all that.

Absolutely. Okay, Matt, before we go, two-part question. One piece of actionable advice to help someone with this AI transformation. And the second one is where can people learn more about Prompting AI to Harness Engineering, your course?

So, piece of advice, probably if you haven't got one already, get a modest subscription for yourself and play with it in your spare time. Make things. Make a little thing for your family that organizes your shopping lists or whatever. Just play with it and get a feel for it in a low-risk place, because you might be surprised how good a good code it produces and what it feels like to be steering it at the product level rather than thinking about the code. So, something that you want.

Best way to contact you or find out more about your course?

Yeah, go to mattwyne.net, M-A-T-T-W-Y-N-N-E dot net. We're also launching, and maybe I'll have it done by the time this podcast comes out now because it's a good deadline, leansoftware.ai. That's the kind of brand we're building around all of this Software Factory stuff. So yeah, go to leansoftware.ai.

And we'll have links to all this awesomeness down below. Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com/a596. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the TestGuild Automation Podcast. I'm Joe, my mission is to help you succeed with creating end-to-end full-stack automation awesomeness. As always, test everything and keep the good. Cheers! Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com, where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. Oh, the Test Guild Automation Testing Podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air It spread like wildfire through the land, guiding testers, showing the secrets to behold.