Redefining Test Automation with Dave Piacente

It's been a crazy few months of me running two major online events, Automation Guild 2024 and Robocon. I've been heads down all year and I finally have some time now to focus on other things. One of them I missed from being too busy is that this podcast, the Test Guild Automation Podcast, previously known as Test talks, officially turned 10 years old on February 18th. So I really want to give a big thank you to everyone who's been listening and supporting the show over the years. I I really do appreciate every one of you and I think now this is not only the original podcast dedicated 100% automation testing, but now also is the longest running one and I couldn't do it without you. And today I want to share with you a session we had at this year's Automation Guild. It's from applitools Dave Pecente all about Redefining Test Automation. Dave was actually the second guest ever to be interviewed for this podcast way back in March of 2014. So I thought it'd be fitting to highlight his session from this year's Automation Guild to help celebrate 10 years of automation awesomeness. Before we get to this session, did you know you could still get instant access to all the recordings that took place at the February event? And you can also get in on the monthly training sessions we're going to have up until November. If you join now and for being a loyal listener and to celebrate 10 years of automation awesomeness, use the code celebrate10years and get 33% off your instant Access pass. Grab that ticket and you also get access to our private community as well. Hope to see you in there.

Hello and welcome to my talk on Redefining Test Automation. I'm Dave Piacente. So let's get started.

I think before I get too far, it's worth talking about my story just real briefly for those of you that may not know me. I've been in test automation since around 2009, either attending or speaking at conferences and meetups and I did that until about five years ago. I took a job as a software engineer working at applitools in the research and development team and after a couple years I realized that I wanted to focus more on just being heads down as an engineer and I took a break from being in the public spaces, in the public speaking and still was in test automation, working largely in building helping build products at applitools. So I was still well versed in what's going on in the industry, just not really paying attention to the talks that were going on and, and then more recently I moved into out of R and D and engineering, specifically into this new role where I focus on doing developer advocacy and community focused activities. And so I started more recently paying attention and getting up to speed on what people are talking about again in the industry. I figured, well, it's been some time since I've been doing this. I'll step in and it'll be like jumping ahead light years, right? Five years. It's a long time. And what I realized is that.

We'Re still largely as an industry talking about the same things. And I was a bit surprised by that. And so I thought, well, maybe that could be a good opportunity.

To talk about what I was thinking. And so I figured in that vein of the word opportunity, it's worth mentioning that I think.

There is actually a big opportunity and that's one for us as an industry and as a large community of other practitioners to collectively and collaboratively and clearly define what we think test automation is as an industry. Because where I'm coming from is this, there's a lot of problems that have been solved and I think that there's, there's still people discussing nuances to the solutions or new facets of those solutions. And I think that's great, that's fine. I don't mean to disparage anyone's prior work. What I do think though is there is an opportunity for us to level set across the industry in almost like a standards based way that hey, like as practitioners, like this is the fundamental baseline that we agree people who are wildly successful in test automation do these things and they don't have to be specific to context or vendor. I think there's a way to do it in a vendor agnostic way, in a context, largely context free way. We can actually come up with a way to distill all of that, crystallize it, put it into something. And so that's kind of the, the main thrust or motivation behind this talk in my perspective. And I think that there is some prior work that we can look to to help maybe put some guardrails up to help make the path a little more clear that we can kind of pull from to show us a way forward and kind of help help spell out a bit of what I'm thinking. So some examples, and these are a bit disparate and maybe don't make sense, but they will I think in time. So there's, there's this person, coach John Wooden, I'm going to explain these after I mentioned what they all are. There's also this book called a pattern language. There's something most of us have likely heard of called the Agile Manifesto. And then there's kind of an offshoot which is less known by some, I'm assuming, called the Software Craftsmanship Manifesto. So let's step through this. Coach John Wooden. So there's this link to the New York Times article, but basically there's a lot that's been written and talked about with coach John Wooden. He's a really well known, at least in America, at least for people that care about college sports, basketball coach. And over his time at.

Coaching at UCLA, he had won 10 championships in 12 years coaching this team. And he has a whole lot, whole canyon of content and beliefs around this idea of building like a pyramid of success and how you have to start at the bottom to build the foundation and then you can go up from there. And the most often cited.

Piece of that, that whole concept that he, that's talked about, that's written about even anecdotally, a third party of what he's talked about is the concept of tying your shoes. Really starting from the very, very, very basics of like, okay, he's working with a new player for a team, they're putting on their shoes. What's the first thing they should do? They should tie up their laces. And then he goes, in this article, they actually talk about different quotes of his where it's talking about the different tightness of the laces, saying a little tight. Okay, then maybe you won't get so, so many blisters so you won't miss a couple games and you'll be able to have a little more control and traction on the court, make it even tighter. Well, maybe now you won't sprain an ankle and then you won't be out of the game for a couple weeks. And so just thinking about that in the context of test automation as, as we're getting into this conversation of like, let's articulate what is a baseline. I think starting from kind of fundamentals is really interesting and just want you to put that plant that seed fundamentals.

Now the next example, a pattern language. To borrow from Wikipedia.

Here'S a quick blurb. It says the book creates a new language, what the authors call a pattern language, which is derived from timeless entities called patterns. They describe a problem and then offer a solution. And in doing so, the authors intend to give ordinary people, not only professionals, a way to work with their neighbors to improve a town, a neighborhood, design a house for themselves, work with colleagues to design an office, workshop, public buildings and such as a school and et cetera. And I think this is a fascinating concept where creating a approachable language to describe the building blocks that go into something that's very technical, that typically requires a very specialized person, and then enabling it so you can collaborate with others around you who are not those in that same profession to design a common space or a common outcome. I think that's also interesting as we talk about trying to articulate and define what we want as practitioners. Now let's talk about something we probably all know, Agile Manifesto.

So if we flash back 23 years ago, on a cold February, a Sunday, which kicked off the beginning of a few days, 17 software engineering practitioners met at a ski lodge in the mountains of Utah to find common ground about what mattered to them when it comes to approaching their work. And the byproduct are these values. Agile Manifesto and is largely fueled a huge chunk of our industry, if not nearly all of it, especially those that focus on iterative software development. So since then and so just step through these real briefly here. So the values being individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan.

While there is value in the items on the right, we value the items on the left more. I think it's fascinating, a great way to frame it. I don't know if this is exactly what I have in mind, but I think there's pieces of this that are fascinating and helpful to take with us as we talk about something that I'm thinking about for creating a better definition. Now, if we look at an offshoot of the Agile Manifesto, we have something that came a handful of years later. I think it was around like 2009 maybe.

It'S the manifesto for software craftsmanship. And.

The motivation behind this is to help create a movement around quality and professionalism for software developers. So it's based on four principles which focus on a mixture of software quality and people, both developers and customers. And so it says not only working software, but also well crafted software, not only responding to change, but also steadily adding value. Not only individuals and your actions, but also community of professionals, not only customer collaboration, but also productive partnerships. So similar in similar structure to the Agile Manifesto, there's just a handful and it's a focus on one side versus the other that is in pursuit of the items on the left. We have found the items on the right to be indispensable. So they did the same same format, but flipped it Now I think that both of these are great. It's a great starting point. I think focusing on values, super important. But what I'm also articulating hopefully here and coming to conclusion on hopefully inspiring you all, is to maybe do something like this, but with some additional things. So we're going to. Yes. And this.

And so I think that with that in mind, let's help create a perspective, hopefully this whole, hopefully all of these elements combined. If you put these in your mind.

And treat them as inspiration and a lens to look through, I think it can help inform a path forward for us. And so I like the idea of a manifesto, but I do want something, something more concrete and actionable. So, but let's, let's think about this and how it can be applied to test automation. So really the outcome that I'm looking for, starting with the end in mind, I want to start from the ground up in terms of the things we care about. And so I want to find a common language that we can all agree on and describe the foundational elements that lead to success in test automation. So not just values, but also more general attributes in the practices that we all have for wildly successful teams. And I want to write them down into a living document and so some examples, at least on the value front, to get started, I think we should focus on business values or business value, rather over test coverage. So the idea of just focusing on 100% test coverage or 100% browser and device coverage, when you're, you lose sight of why you're doing it, it's really focusing on delivering business value.

And I think being the conscience over the team over sticking with the status quo just because things have been done a certain way. Like we should always look for ways to try to work smarter, reinvent ourselves, and not just work hard just to get something done because this is the way it's been done or just because somebody says it needs to be done this way, understanding the underlying motivation and trying to be as effective and efficient at the same time as possible, I think is a bit of a superpower that we have. If we can focus on being the conscience of the team.

And then check the things, work for customers instead of works on my machine and really focusing on. It's like you can run tests headlessly, that's great, but customers don't run things headlessly. And there's kind of this, that's just one example, but that conversation where you want to try as close as you can to have parity with the experience that customers have to make sure that you are actually checking things in a way that really gets you as close as possible while balancing efficiency and cost. And then there's no comparison point here, just fast and reliable feedback, full stop. I think that that's really ultimately, at the end of the day, that's what we're trying to do. We're trying to.

Balance the idea of being technical practitioners while also being like business analysts in trying to offer this efficient balance of those two perspectives and creating feedback loops so that we can inform all the relevant stakeholders to make sure that they're aware of when something doesn't work, when something does work, and trying to build high trust, but build it in a way that's like creating a system or set of systems that make, make people stay informed about the health of what's actually being built, both in terms of.

Code in isolation and then, you know, applications in flight, all interacting and working together. So and so in terms of zooming in a bit, some examples to get the conversation started around more general attributes, I think this is, this is where the conversation starts to get a bit more interesting.

And.

Some examples. Team and culture. And by that I mean skills, commitment to quality and testability, pick the right tools for the job.

And.

Robust test framework, simple authoring aligned with team skills that is maintainable and reusable.

And stable and scalable test infrastructure.

Fast execution at scale executed across browsers and devices with stability baked in reporting and analytics. Easily identify flaky tests, see results in real time. Efficient selection and prioritization of tests and test data in isolation. So the ability to mock services systems to generate test data and create deterministic outcomes with minimum dependencies between tests and systems. Now I think we could hopefully, as you hear this list.

Think, oh, are these all the tenants of what I do on my team? Or, or if you think of teams and other practitioners that do things really well, I'm trying to think in terms of fundamental pillars that like prop up the success of a team and in a way that's done again, vendor agnostic, largely context free and in my experience for the teams that I have consulted with, colleagues I've talked with, customers I've interviewed through Apple Tools and the list goes on. I, I think that these are fundamental like table stakes things that everybody.

Ultimately practices and has in their at their disposal, which are not to say that they're easy or necessarily even fungible, but I think that they, they are true. It's like where team and culture are probably the hardest one to get right. It's like getting the right people, the right motivation, right mindset, buy in from leadership, funding and support for either education and focusing on open source or paying for licenses for commercial off the shelf tools or some combination of the two. I think that's one and that takes time to grow and nurture. But I think the teams that have done the most wildly successful things with test automation, that's why I put that one first. The other stuff also still relevant and necessary, some kind of robust framework. It's like you can't do test automation if you don't have a means for making it. So people can author tests simply that actually show up and meet them where they are creating building blocks for tooling that are ideally in the language that they're in or making it. So if it's not something they're familiar with that there's good.

On ramps for them in a way that's also maintainable and reusable. So easy for people to keep working and also solve the problem once and not have to have everyone else solve it as well. And then test infrastructure being stable and scalable, I mean wildly important, the ability to scale horizontally and vertically, horizontally being you want to have as fast as possible so you scale into across as many browser devices as possible. Vertically being across different platforms, operating systems, browsers and devices. And then test flakiness being like the most vile and vein of our existence. But the ability for, as we're seeing with new platforms emerging, the ability to bake in some reliability. So self healing locators being one fantastic example of that. And then reporting and analytics, I mean being able to easily identify and flaky tests and potentially even like quarantine them out to triage them so that they don't introduce doubt and defray trust in your solution. Amazing. Seeing results in real time.

With the ability to efficiently select and prioritize tests.

And understand what's going on at a glance. Also super helpful. And then being able to of course, when something goes wrong, being able to dive in and understand what's going on and then test data in isolation, you know, largely deserves its own because it's such a massive topic. But the ability to.

To do what you need to do to. To simulate a situation, if you catch a bug and you understand that it's like this orchestration between different services or specific data, then you can change the shape as needed and try to really introspect and do what you need or to test different scenarios that a customer would run through by generating test data. And then being able to also test in isolation so that you can actually really hit the other mark of the, the scalable test infrastructure. Because test isolation is really important if you're going to actually get into scalable test execution. And so this is just my list, but I really want to know what's on yours, so I would love it if we could start the conversation at least maybe the simplest low fidelity format, which is to email me. So if you send me an email, the subject redefining test automation to Dave P. That's D A V E P I A C E n t e appletools.com and let me know what's on your list, that would be great.

So.

I think that it's.

Worth talking about clarity. So I think with, when thinking about things in a very specific way around test automation, around what we value and what are the attributes that are important for, that we've seen for teams that are successful, and if we all distill that down into a common definition.

That we largely agree on, I think that that gives us clarity.

Around a foundation of what our practice is. So it becomes a lot easier for us to assess the value and the efficacy of a new technology. And so I want to speak about that briefly in the context of generative AI or anything new that comes potentially after it. And so I think that it's worth asking some questions about this, the new technology.

To try to keep the conversation rounded and realistic.

And so first, the simplest one is always does it solve a real meaningful problem, this new technology, or is it just.

Something that everyone's talking about? There's a lot of hype and really high expectations. But then when you actually get to seeing and using the technology, does reality have a mismatch with the hype? But really putting that aside, because it's very easy to get wound up and excited about the hype or anything new. Even if it's, even if there's no hype, just understanding what are the core tenants and what is it that's important? What are your problems? Does it solve it in a meaningful way?

Does it change the way that you work? Is it new technology that fundamentally changes the paradigm of how you work? Are there certain things on that list that we talked about a couple slides back that, that maybe you can do a lot faster or maybe you don't even have to do anymore? Well then that's super interesting.

Now.

But think about it from the perspective of the side effects and ripple effects from it. Does it create new problems? It's worth considering just because it makes things immensely faster. Maybe it creates a Lot more work product that requires a lot more insight to be able to quickly discern. Maybe it's makes one thing faster, another thing much slower. It's hard to know.

Does it work securely and reliably? If it's a public LLM, you're not going to want to dump your, you know, your secure domain specific business secret stuff into it. And does it work reliably? You know, we've seen hallucinations with ChatGPT for instance. I know these things get better over time. I think over time maybe we'll see it where private LLMs be easier to stand up, they'll be more commoditized, easier to train and be be far more reliable. These things can happen. Again, just questions to ask to understand the context that you're in. Try to keep the conversation grounded. And also when things break, is it clear what steps need to be taken? Do you have the right people? Do you have the resources you need? Is it something that you can reason about? Because if you come to blindly trust the technology once it stops working, what will you do? For instance, quick side quest example, say you're trying to go somewhere you've never gone before in a part of town you've never been to.

And all of a sudden you don't have Internet and you can no longer use Google Maps and you don't have a GPS dedicated in your car. What do you do? I think that we have become so reliant on some technology that we blindly trust it and so to the, to the service of our, of our own needs when it stops working. And so I think we should take that example and that mindset into assessing these new technologies just because even if it works and solves a problem really well, we need to make sure we understand the risks associated with it. And also, just one thing I want to point out, there's this thing called Conway's Law that I think is very relevant and salient for this point of conversation that just because we have new technology and it could fundamentally change.

Let's, let's, let's wave a magic wand. Say it works, say it solves all the problems, makes it so our jobs are fundamentally different and companies adopt it. Now that doesn't solve the people problem. And the thing that's fascinating is this thing called Conway's Law. And according to Wikipedia, as it's written here, describes the link between communication structure of organizations to the systems they design. So specifically organizations that design systems are constrained to produce designs which are copies of the communication structures of these organizations to put it in a different way is to say that the behaviors of the people in organizations, the technology that they come to build and use, will mirror those behavioral conversations and constructs. So just because you magically say, here is this tool that makes it so, we fundamentally had become much like a thousand times more efficient in terms of test automation, and we remove a lot of things that we have to pay attention to. But if it doesn't solve the dynamic between how developers and testers throw things over the wall to each other, there's still going to be this huge friction and these huge people problems laying underneath, laying underneath all the technology. And so technology is not going to magically solve everything. There will still be problems and some of them, a lot of them, will largely be people problems. And so it's just worth putting that into the conversation as well. Just as a side note, now I'd like to open up for questions and love to hear from you. Thanks for listening.

Thanks again, Dave, for your automation awesomeness, for links of everything of value we covered in this episode and on over to test guild.com a487. And while you're there, make sure to register for Automation Guild to get instant access to all the awesomeness that went on at this year's event and get 33% off by using the code. Celebrate 10 years. So that's it for this episode of the Test Guild Automation podcast. As always, test Saratan and keep the good cheers.