Using Your Environment Data for a Better Testing Outcome with Wayne Ariola

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking Wayne all about his concept of an open testing platform. Wayne is a recognized leader on software testing topics such as continuous testing, risk based testing, service virtualization, API testing and open testing platforms. So Wayne has created and marketed products and supported dynamic software development tests and delivery landscapes for many years. He has driven the design of many innovative technologies and received several patents for his innovations and for his inventions. So he's been contributing to the software testing space for many years and in the software industry for more than 20. Really excited to have him on the show. Really cool concept. You don't want to miss a episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Wayne, welcome to the Guild.

Hey Joe, thank you so much. Obviously a big fan of the podcast and I'm really excited that this is my first time.

That's awesome.

I hope to do it again, but I do enjoy the content and it's actually on my iPhone, so thanks for offering it.

Thank you. Awesome. Yeah. Great to have you on the show. I've been wanting to have you on for a while. We chatted previously on a few different things. I'd like to dive into that before we get into it though. Wayne, is there anything I missed in your bio that you want the Guild to know more about?

No, but could you follow me around and just say that all the time for me? That'd be awesome.

No, I guess the one thing I like to differentiate here is I think the software testing space and vendors have a mixed relationship I think over the years and it's something that's pretty interesting. And I think we finally come to a point where I think open source tools are so productive right now and there's a great way to actually assist in acceleration for organizations. And I think, you know, from my perspective, and you know, I've had, you know, multiple patents in the area, you know, some of them are even expressed in open source tools these days. Right. I think that, you know, no matter what organization, I really want to force more collaborative thought around how to achieve what you need to achieve. Because I think we're trying to figure out how to stretch things and then we're catching up, we're stretching it, catching up instead of kind of leapfrogging to the next point. So my whole thing is, I hope we can talk about some topics which assist organizations to kind of leap into that future state rather than kind of trailing along development and evolution of architecture. So that's kind of my thing these days. Like, how can I help you to move forward faster?

Great. So, Wayne, you bring up a good point. You know, I was just having this discussion with someone. I had an online conference and I have sponsors that have vendors. I actually come from enterprise development background, so I'm used to working with vendors and I love vendors, but everyone always gets upset because they always want to do open source. What's the happy medium then between vendors and open source solutions nowadays?

It's a fantastic question. And I think what you need to realize is in the software testing space itself. I think, Joe, I think we've really radically focused on the mousetrap, whether it's building a new mousetrap to building a better mousetrap. But it's really always around when you talk to anybody about automation. It's really around this pretty isolated thing about generating a reusable script or whether it's no code, low code, or coded. It's always really isolated. Now, I think open source has come into the picture and created a really valuable array of tools that are even optimized to particular architectures. So there's no doubt the more technical an organization is, the more focused they are going to be on using those kind of architecturally optimized tools, right? And I'm talking things like Playwright or React or, you know, the more technical frameworks that also have testing components associated with it. I think today that the question associated with how to Partner with Teams, DevTest Teams is changing dramatically. I look at all these tools as nodes, right? Nodes in a process, nodes that feed particular sets of automation. And the question now is, how do I best orchestrate that information? Not creating this kind of divergence between a developer's asset and a tester's asset, but looking at those nodes of information in a much broader picture, right? Not to get too ethereal here, because sometimes this floats away from you and it can. But the idea now is really how do I get this puzzle put back together? So I'm actually building upon the information that's already there in order to produce a better picture for the business to Understand risk. Right. That's what it's all about today. So I think that's where the vendors are coming in and thinking about, okay, what really do we need to do? Do we really need to build a better mousetrap and build another mousetrap and another mousetrap? I just don't think so. I think it's now about creating a platform for more comprehensive effectiveness. In terms of what I'm trying to accomplish.

Yeah, absolutely. And I love how you use the term node because I think your concept that you have is based on like a hub kind of, that assists testers and I think you call it open testing platform. I guess before we dive into that, what is an open testing platform?

So an open testing platform. Thank you for using that word is a collaboration hub. Right. It basically inverts the concept of what we do today around building scripts. It abstracts logic into a digital twin of your critical transaction. I'm going to have to unpack all these terms because I love the word digital twin. People sometimes get it, some people don't. I know if I had to have a digital twin, Joe, it would look a lot better than what I look like now. So a digital twin gives you the ability to create and abstract your logic away from the test script. So today what we're doing is we're building test scripts. And those test scripts are really being buried and very tightly associated with the tool. Right. What you need to do now is abstract that into a collaborative form. And that collaborative form is a representation of the critical transactions associated with your business, which is the digital twin. And the good thing about the digital twin, Joe, is a, it gives people or humans a way to collaborate against something visual. Right. Associated with the transactions. And what I'm talking about is a transaction horizontally across the organization, not necessarily trapped within a vertical of a single application, but really looking at, and I know the word end to end sometimes used in different formats or with different contexts in our industry, but we gotta be looking at the entire experience from a user perspective, whether is stems from business process, whether it stems from a user engagement, and then having that model in a method or a form in which humans can collaborate, number one, is critical. But number two, since we are now in the data age, having that model open and receptive to have machines contribute to it as well. So, meaning that the model not only needs to be influenced by a human who is looking to engage or looking to basically understand the next requirement coming down the pipe, but we also need to mine and curate data from production, from the dev Test infrastructure and inject that into the model to assist us with prioritization, assist us with visibility, reuse, maintainability, to give us this method in which we can actually then push the logic down to the nodes, whatever open source, whatever commercial tool you might want to use to be much more productive. So the idea of the open testing platform is really to create this new method of work, which is called Inform, Act, Automate.

Awesome. So, Wayne, I love this approach because a lot of times people always ask me about tools, what's the tool automation tool? And it seems like we're at a point now where, I mean, there's so many tools use a tool that helps you and use the tool, but people are missing out of all the data all these tools are creating. And it sounds like this information you're talking about is really where the critical information is. The tool is just driving things, but the data is where all the gold is. And it seems like people are really missing out of all this other information all these tools are creating. They're just adding another tool rather than saying, okay, wait a minute, what are we doing with all this data? What are some insights we can look at? So what kind of insights then can people focus in on if they focus more on these type of data that's being produced? I guess so.

Great question. And I think when you look at an open testing platform and back to this idea of Inform, Act, Automate, what we got to realize is our job as testers is actually much, much more complex than anyone else's job within the cycle. A, we have to interpret the logic, and B, we have to be creative around how we're going to exercise how that logic is implemented. Right? And then we need an environment, and then we need data, and then we need to run it. Then we got to go prove whatever we fished out is actually an error or a defect, or we have this whole downstream cycle that creates the most of our work. Right? To go prove back to the organization that what was uncovered is actually a priority to attack. Right. So if you look at the idea of, you know, the. Or the role of testing, it is truly a more insane process than you would typically think when people just say the word test. So when you say what kind of information or what does it do? It promotes a pattern. So the first pattern is Inform. And when I've worked with organizations, the first thing I began to notice in terms of what was delaying the process of testing was essentially late changes. And that can be late changes to the environment, which was one of the main issues. It could be delays in the requests for test data. It could be changes in the requirements or features being pulled at the last minute, feature flags, changes. I can go through a list of 70 of these things, by the way, which are kind of the late stage kind of impacts that stall our process. Right. And when you ask about these conditions in which caused the delay, the one thing that became extraordinarily clear to me was there was somewhere, some point in the system where that change was evident, and that change was evident a long time before the tester actually knew about it. And this is where the idea of curating data for testing really starts to make sense. So if you go into an organization, just as an example, you will see that there's an array of pre production tools, array of dev test infrastructure tools, array of production tools. The level of access to those tools is variable across the organization. I see organizations that have very mature applications, performance monitoring infrastructures, but I also see the fact that testers don't have access to it or they have only access to a dashboard. So the idea of a dashboard is good, but a dashboard still requires me to go to it, click on it, interpret it, and then I have to infer what I need to do to take action. So informed inverts that process. It basically focuses on what the tester is doing today. And let me give you an example. Let's say that the tester is responsible for a very particular. They're assigned to a test, and that test is also tied to a requirement. We could monitor the infrastructure for pull request, right? And automatically notify the tester that a change has been made underneath one of the requirements that they're working on. This information alone is absolutely amazing because you can understand whether we are now in a point where there's going to be a difference in that the code changed and you need to update something. That's the most basic one. But let me continue on. Let's just say that that particular requirement or that particular application has a shift in Persona or a shift in usage. This data is in your application performance monitoring solution. It's sitting there, but the tester is usually not aware of it. You're usually building your Personas, or if you're addressing Personas, you're building it into your test design way before it's time. We can automatically give you that information because you are associated with a particular set of requirements and we can correlate that and produce the data for you in your feed. But we should only do it for the requirements that are in play, that are In Sprint that are hot, that are live, don't get dumped on this massive data of things that you can't act upon. Let's isolate it to what is going to impact your work in the next X days and wherever you are in that Sprint is essentially the framing of it. But curating this data for change is, I think, the first step. Give me the information that's pertinent to what I'm working on now, which is really this inform pattern.

It's almost like real time bubbled up insights. So like you said, I used to create these complex dashboards and people like, you know, it's really pulling teeth to point them to and have them look at it. But this way it's like in Sprint, I'm working on a feature. I guess my team wasn't very agile because we had like eight different Sprint teams and one was building a component that this other team would consume and if they didn't finish it, you know, we wouldn't know after the Sprint. So it sounds like this is kind of like consuming that data in real time so that you're now waiting around to find this data out later after the fact.

It sounds like, yeah, absolutely. Now let me give you another pattern which I love. This particular pattern is, let's say I'm newly assigned to. In your Agile test management system. I'm newly assigned to a, to a requirement or a test. I would automatically get a notification in Slack or Hip Chat or whatever the notification system or email system you're using. I would automatically get a notification of the developer who's working on the code or developers working on the code that implements the results of the last test, what had changed the unit tests associated with it and what were the results of it. And I would actually feed that right into your channel. And the benefit of this is you've basically eliminated the archaeology associated with kind of uncovering where the bones are buried or where the bodies are buried associated with this requirement. Right. You get that information right up. Now this is a very, very simple requirement or, excuse me, a simple pattern. But it's just amazingly valuable to highlight that information. And the reason why I love the idea is that it's so basic, so visceral that it actually makes a lot of sense. We just haven't connected things up. So information can be really highly curated for testers. We've always thought in terms of dashboard, so now we're switching that to make it more proactive. And an open testing platform should have the hook hooks that allow you to correlate These artifacts to give you those patterns. Right. Those information patterns. Now, like I said before, I've uncovered 70 different inform patterns that I think testers would want to know about. You know, this could be config changes in test environments. These can be changes in data patterns or whatnot, impacts to test, data config. Right. Everything that has to or is associated with you being able to execute the test. But we always have this aha moment when we're talking to clients and we get in this.

Can you.

And by the way, you know that they're thinking because their voice goes higher, it goes, can you. And I swear to you, everybody has their own environmental conditions which have forced delays in the process. And so openness in the way we're articulating these patterns is going to be critical.

Absolutely. So if you have a lot of automated tests and they fail because of this insight is not being, not being shared among the teams through collaboration and the tests are failing. Can you automate the automation in a sense that the automation can be alerted in real time on changes to the data and the environment and things like that?

Absolutely. So this is the second piece of the puzzle. So the pattern goes inform. And once you get through the inform piece, what opens up to you is this idea of collapsing the information you're getting to act. So like I said before, when you're in a dashboard view, you basically need to interpret the dashboard, understand the dashboard, and then decide what you're going to go attack. So what we've done with an open testing platform is collapse this. So information then is presented to you with this model or with the digital twin, giving you the ability to take action against the digital twin in a panel. So, for example, the Persona conversation here, we've noticed with Application X that there is a change in the usage pattern of the Persona using it, or there's a change in the endpoint or device that is most prominent in terms of its use, or a change in the browser or changes in one of the vectors associated with Persona. We can feed that into the digital twin to give you the ability to. Now either you have a choice, you could. You get the notification, you create a subset of tests that address that Persona. You could update your test data to better reflect the Persona or you could basically do a combination. Right. So what we're trying to do is collapse this idea of feeding you information with the opportunity to act upon it. Now, once you go through that act motion a number of different times, Joe, I think what you're going to realize is now what some of those things are pretty predictable. We can actually connect live data into automation now. So you never want to really automate. First I would say you gotta learn the pattern and understand the impact of the pattern. And once that pattern now becomes repeatable and it's noticeable, taking the step to automate it is probably your next most logical step. I think one of the things that I've noticed in our industry is that we tend to want to automate and we take massive strides to automate something very complex, but it produces this very brittle automation that we're always fixing. So then that becomes the next failed project. So going through this pattern of looking at the data, creating the connection between the data and acting upon it within a open testing platform, I think is the first two critical steps to the final step where we know we can automate it because we've seen the pattern happen over and over and over again.

Absolutely. I guess the next tricky thing then when you talk about automation is people automatically think automation, functional automation. But I would think that you would have to act on things that can be automated but aren't necessarily tests. So like almost like a remote process automation or something. How do you define automation at this phase then? Is it automation just for testing or what is it?

That's such a great point because as soon as you say automation in our. And by the way, going, coming from your background, you know, I think you're, you're thinking performance testing, you know, I'm thinking functional testing. And by the way, the developers thinking unit test.

Right, right.

And we're all talking about the same word, which is test and test automation. You're absolutely right though. Automation has multiple levels. So there is the, you know, there's the automation. What I would say is understanding and optimizing what needs to happen. There's automation around how and there's automation around when. And all those, by the way, Joe, are separate orchestrated triggers in terms of what we do, the how, which is mostly the test scripts. Right. How am I going to drive that? Interface is what we primarily focus on.

Right.

However, these other two components we've kind of taken on as either adjunct types of projects. But this is where the brittleness comes in, in the end to end process. Now, DevOps and Enterprise. DevOps has forced us to really look at these kind of peripheral subtasks and try to understand whether we can draw them into the process to be much more efficient. You know, ideas around containerization, ideas around leveraging virtualization, ideas around how to aggregate the artifacts in order to actually have more sustainable Automation in aggregate have certainly forced us to relook at this, but when you start looking at the pieces, the idea still comes down to this point where the more we try to aggregate and automate, the more brittle it has become. And this is why the open testing platform has this concept of bringing the pieces together in the form of information, so you can decide what pieces you need to actually to automate in the long run, I guess. In summary, and I've never really said it this way before, Joe, but I think we took a massive leap forward because we had the tools to do so. But in doing so, we really didn't understand the brittleness of the components we were automating from a process perspective. And this is what throws a lot of pause into the process.

So this may seem like circular logic, but you talked about a process. So is open test platform a process or is it a tool? Is it a framework? How could someone say, well, this sounds awesome. How do I actually implement a open testing platform in my organization?

So an open testing platform, basically, by my definition is a concept and it has three steps to it. It has a mechanism to curate data, it has a mechanism to act against, and it has a mechanism to automate the edges when you need to do so. This might be a combination of solutions that you already have, but this pattern is the pattern that's going to allow us as dev testers to keep pace with the ever increasing rate of change. Right? I don't believe that even though that there's a lot of great progress in AI, I don't believe that without this open testing platform pattern, we'll be able to really achieve the business objective and enable AI the way it could be without this pattern. So it's primarily a concept. Now, if there was a person out there or if there was an organization out there that would say, hey, I am a open testing platform vendor, or open testing platform infrastructure, I would say, would have connections into your broader environment which you can manipulate. You could curate those data patterns. So you are actually curating data. So it's impactful to the tester, it's quality focused. And then it would have an automated way for you to actually take action. It's not about a dashboard, it's about taking action. And then it would have kind of ubiquitous tentacles into the open source infrastructure. So if you. The good thing about it is if you wanted to use a project like Selenium, you would be able to do so. If you wanted to say, hey, by the way, I got a shift, we're making architectural Changes and now something like a Cypress is a better fit. You could potentially make that shift without having to look at a load of scripts that you now need to say, how do I make this now valuable in my new environment?

Is there one piece of actual advice you could give to someone to help them with their open testing platform efforts? And what's the best way to find or contact you?

Well, despite the fact, Joe, we just gave away our age to the audience.

This is what I would recommend. Keep a log of what is causing delays. Right? And when things are causing a delay, as simple as it might be, think back to where that data might reside so you would be able to be more informed about that delay. And by the way, you're not allowed to go as far back as developer's head. You need to go to. There is a pull request on a particular piece of code potentially.

Right.

Or there's a feature flag thrown, or there's something in a system that would alert to the fact that you would be aware of a change. And what you're going to find for each one of your delays is that there's some system that you could actually mine that holds this pot of gold. An open testing platform is going to assist you to make sure that that information is then available to you in the future. So that's my only bit of advice. Right?

Awesome. And Wayne, the best way to contact you or find out more about otp.

So we've written a lot about this at Curiosity Software and if you go to CuriositySoftware, ie, you'll see a lot of information on an open testing platform, how to potentially enable it, and a lot of data on the Inform act automate pattern. One of the more interesting pieces of data on that is these inform patterns. I've shared it with a lot of people in our industry and testers in our industry. And this is where the fun comes is you're going to go through that list and go, oh yeah, oh yeah, oh yeah. That happens to me. But once you start understanding how those patterns are then curated, what's going to happen? You're going to have this explosion of ideas. What if I combine X with Y? I would then know when my environment blows up or if the database isn't reset, or if you know the litany of things that actually pause your automation. So it's kind of a fun journey. It might be kind of picking through the skeletons. So I apologize. But what you're going to realize is that a lot of this information that's out there can actually be expressed and used to assist you rather than become barriers to progress.

Thanks again for your automation awesomeness. The links to evident value we covered in this episode. Head on over to testguild.com a346 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.