An Holistic Approach to Automation Testing

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Pam, all about an holistic approach to automation testing. I know a lot of people are always struggling with automation and how to have an end to end solution in place and I think this is going to be a perfect episode for you. For those that don't know, Pam has over 20 years of experience in the software development and process automation industries spanning both federal and commercial sectors. So a lot of experience in all types of different situations. During his career, he recognized one of the main challenges of test automation is the mismatch between technical experience and testing teams and the complex test automation solutions that exist in the market, which I've actually seen myself as well. I've worked at GE Large Enterprises, so this is definitely a struggle I've seen. He has Co founded Subject 7, which is a new tool I've recently heard about in Solution with the intent of developing a comprehensive test automation solution that can encourage and empower businesses and non technical users to develop sophisticated and scalable automation without the need for programming experience, which could be controversial. So we'll dive into that a little bit as well. So he's done a lot of things at Subject 7. He's focused on ensuring success for a lot of different customers and evangelizing the benefits of codeless technology for the test automation industry. You don't want to miss this episode. Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit test skill.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey Bian, welcome to the Guild.

Hi Joe. Thank you for having me.

Great to have you. Before we get into it, say that I missed in your bio that you want the Guild to know more about.

Oh, not really. You covered it pretty well.

Great, great. So I'm excited to have you on the show. Just so folks listening in, I actually saw a demo of this of this particular tool and I started talking with the folks there and realized, hey, this would be a good podcast interview. That's the genesis of this interview. So one of the things I actually saw in that demo was you had a slide that said, I think it was 70% of digital transformations fail. And I'm just curious to know, is that something you've experienced or why do you think that might be so?

Believe that has been taken from one of the research that we have found over the web. But our experience has been that percentage is not that far off. I think the main challenge that I have personally seen over the years is that there is sort of a challenge that projects have in terms of balancing the speed of delivery and the quality of software. So obviously with the rise of CI CD tools, development cycles have tremendously shrunk to few hours, and people are essentially delivering software on a weekly basis, sometimes even shorter cycles. But what has been lagging is mainly automated testing. And historically, projects have tried to scale up their testing through manual efforts and by providing more and more resources to be able to do that. But obviously that becomes a cost barrier to the project. Then obviously there is a limit as to how much you can scale with manual testing. So I think testing and in general automated testing has been main contributor to the failures of digital projects, in my opinion.

So when you see these large enterprises, you think they rely too much on manual testing or they've tried automation testing and they failed and they just reverted back to doing everything manually.

So I think most of the projects that I've seen over the years have tried both. The problem is there is just not enough, you know, automated testers in the market. What I have seen is that to be able to compensate for that lack of, you know, resources in the market, projects will try to provide, you know, 812 weeks, 16 weeks of programming training to testers that they have internally, trying to convert them to automated testers. But it adds complexity because it's yet another code base to maintain. And, and it's not a project that you will ever done with it. You have to maintain it throughout the lifecycle of your project. But more importantly, when you have resource attrition and people leave, finding other resources whom you can train in time to be able to pick up where, you know, the previous people left it off will be, you know, its own challenge as well.

No, that's a good point. I've worked at companies where they had automation testing in place, and they use a certain framework, a certain programming language the key person left, and they, rather than pick up the framework, they develop a new one in a different language and different tool, and it's a repeated cycle. So that's definitely a good point. And another good point you bring up is it's automation is software development. And in my experience, Once again, a lot of developers in my experience haven't even contributed to the automation testing efforts. So you have someone that it's not even a developer trying to do development level automation and failing with it. So it's like a catch 22. So that's definitely something you've seen as well. So it's interesting to hear. So I guess what are some ways to get around this? I know one team I've worked with used behavior driven development and once again that was wasn't successful because they were using just like an automation frame and underneath the covers everything was code anyway. Have you seen those type approaches be.

Successful or so so absolutely you're right. A lot of projects are resorting to bdd. The problem as you well mentioned is that you know, behind the scenes someone has to be coding those, you know, keywords and everything and it sort of falls into similar category of creating your own framework and then what if you know, the person who creates that framework leaves the project? So I guess as an alternative, what I have seen SH in the market is a projects are more and more looking for, you know, no code test automation platform where not only testers or estet people are able to get into the automation, but rather all stakeholders being business, user, subject matter, expert, even clients can use it to potentially do UAT testing, leveraging some of the automated tests that other stakeholders have created earlier in the lifecycle. So more and more projects are trying to to sort of get around it by moving to a no code test automation platform.

All right, so I'll be honest, we're about probably similar age so we've seen a lot of different testing. So when I started off I started with a solution that was an end to end solution and I loved it, I succeeded with it and then we went to open source and then with open source I had to develop a test management bug system. I had to somehow find a way to have the test tooling respond to it or talk back to it and then go into CI cd. It was so tough to actually get all these open source solutions and get them to talk to one another and get the solution I used to have back in the day when it was an end to end solution. But then again the end to end solution didn't always work either. So have things changed since maybe the earlier days when there were these vendor based codeless automation testing solutions and the type of solutions we're seeing now? Because I'm seeing almost like a swing back to like that type of vendor based provided solution where before it was Open source and then before that it was vendor based solutions. Thoughts on that?

So things have changed, are still changing but I think there is, you know, a flood of, of you know, vendors specifically in the test automation space where they are, you know, various vendors are trying to figure this out and trying to come up with solutions. I guess what I have seen in the market, as you well mentioned, you know, the rise of you know, open source that the problem and I guess the main reason that I have seen projects use open source, obviously it's more control over what they develop but more importantly a little bit misnomer that the industry has that they say that in open source is free, why not use a free set of open source products out there as opposed to pay license fees or subscription to vendors. What I think they are sort of not calculating in that mix is that even though open source APIs are free, but building frameworks on top of those open source and then as you well mentioned doing various configuration and integration with other open source products to make the, you know, things like you know, Selenium Appium of the world's work, you have to do a lot of integration, you have to do a lot of coding. Factoring those things into the free open source products will not make it so free anymore. Whereas in addition it adds a lot of overhead to the projects that otherwise they don't necessarily have to deal with.

And it's a good point. And Selenium is awesome. And they actually say that they're an API, they're not a framework. So they don't promote themselves as a framework easy to use. It's more of a developer friendly API that allows you to, to automate a browser. And that's all they do, that's all they claim to do. And like you said, it's very hard to get all this other scaffolding. You need to create like a really true end to end solution. And I had once again another company, they had to build these dashboards with elasticsearch that takes the output of a JSON from the Selenium test. And it's just so hard to get the information you need. So there's pros and cons of both approaches for sure. And so I'm just curious to know, do you see then companies going all in with just one tool solution or do you see where. I've seen a lot of companies where they have pockets of people that are business experts but they're never going to be developers, they don't want to be developers. And so using like an open source tool for them gets them a lot more coverage because they're able to automate a lot of backend things that maybe a normal team wouldn't try to automate to begin with like Salesforce and those type of package based applications. Or do you normally recommend companies go with one solution for everyone in their organization?

So I guess there are pros and cons to each, right? I mean as you all mentioned, there are various projects that are using various tools for their end to end testing. The problem, obviously it gives you the power of each of those tools. Those tools separately are very powerful. But I guess the downside of it is that you will have multiple set of skill sets that now you have introduced in your organization. Now you will have dependencies on various people. Now given the limited number of, I guess automated testers out there in the market and I guess you know a lot of need for these types of skill set. Now you have five, six different tools that you're using. If one of those folks leave, then a portion of your end to end test will run into problems as opposed to if you go with a unified platform that ends up providing your end to end testing capabilities and on top you can reuse them to do other non functional testing. You may lose some of the capabilities of the other products in the long run, but you know everyone in your organization can use it. You no longer have to train folks to be programmers, you no longer have resource dependencies moving forward. But also what you are creating is the ability to be able to reuse more and more of the automated flows that you've created. It's a single repository, so there are pros and cons to each approach. But what I have been seeing in the market, and obviously also the analysts and the market are seeing, is the sort of move, move and interest in projects to move into unified platforms so that they can limit the number of vendors they are dealing with on the testing front and be able to include more and more stakeholders that they have, you know, throughout their life cycle.

All right, so a few things I want to pull out of that before I forget. The first one you mentioned is once again enterprises, a lot of times they need to be certified or they, they work in a regulated environment. So the tools they use need to be like SOC compliant, like all these other things and using vendor sometimes because they already have all those compliances in place, it's easier to get the tool approved because the vendor's already done it rather than going okay developers, what open source libraries are you using? Is this compliant? Doing your Own type of audit. Do you see a lot of companies saving time then not having to worry about those type of issues when they use like a vendor based solution?

No, actually more and more projects are looking for those things. So we at Subject 7, I mean deal with a lot of government clients, with a lot of fintech industries, a lot of projects that have a lot of PI data. So pretty much every single one of those projects look for, you know, various compliances that you mentioned SOC to. They constantly run security scans against the software and the vendors have to address them. At times, you know, as you start approaching high security federal projects, they ask for Fedramp compliancy and all. So you're absolutely right. More and more projects are looking for those types of, you know, certifications.

Absolutely. Another thing you mentioned that was interesting is a lot of times you have business experts that are awesome testers because they know the customer, they know the solution, they know what works, what doesn't work. And my other company, they were trying to convert them to automation testers and so developers and they didn't have those skills. So they got rid of the people and knew the business, knew the application, got these people that weren't necessarily good developers, weren't necessarily good testers. So they got the worst of both worlds. This is just my experience. I'm not, you know, different situations, different people succeed in different. But that was my experience. So I would think using a tool or a codeless solution would help in this sense. Because ideally, you know, you're not so focused, you're not code focused. You know, rather than having to say I, I create code they like, I test software and I just happen to use automation to help me test it. I think it's like a mindset shift almost.

Absolutely, I completely agree.

Awesome. And another thing you mentioned is once again I worked for a large company and so we had code that did one thing. You go to another team, they had the same kind of same exact code that did the same exact thing. You go to another silo, they had the, you know, so you had all those code duplication around these silos. So it seem like even if you had an open source solution that everyone used, using the one code, one code base would be a big win because you're not getting all this reuse and you're getting more reuse rather than recreating the wheel all the time.

Absolutely, absolutely. And it's just mind boggling to see how many projects are creating frameworks from scratch every single time. I remember many years back there were A lot of web frameworks popping up, you know, it was, you know, shale, it was struts, you know, Java server face were just lots and lots of these popping up. But at some point people realized that I think enough is enough. And even with all of those things popping up, you know, I was involved on a homeland security project where they created their own XML based web framework. So they were essentially not using any one of those and created their own. But I guess at some point people realized that, you know, there are enough web frameworks out there. Why do we have to create our own? If our, if we are in a fintech industry, if we are working on a government project or we are in a health sector, we should really just concentrate on what we do best and concentrate on delivering value to our client as opposed to building things internally for our own development. I have a sense that this framework, this test automation framework building has to stop at some point and people will say that there are these zillion vendors out there already who have created these wonderful test automation frameworks. Let's just work with them to you know, sort of morph what we need out of those frameworks and platforms out there as opposed to, you know, keep building our internal test automation platform. So it will stop at some point.

Right. So how do you get up and management to realize that? Because you know, I've seen both sides where up and management looks at a license fee and they're like license fee for a testing tool. We were spending like millions of dollars on this and like, oh let's get rid of that. We'll do all open source and like you said, it's a cost associated with open source. You know, you need to do testing, you need to use tools, but you don't want to pay millions and millions of dollars. I mean is there, how do you convince management and to keep that in front of them? Yes, it's a, it's a cost but you're getting a savings. Is there some sort of calculation or something you recommend customers?

Yes. So what we do is, well first of all I think it's, it's sort of educating the industry and I think industry is gradually getting educated on open source does not necessarily mean less cost. But what we've done is that we've basically developed a series of case studies from our clients. But also we have a simple calculation that you know, the delta of the salary between, you know, an estate who is well versed with automation knows these, you know, test automation open source products pretty well with the salary of, you know, A manual tester, a business user who knows the application that they are test, they're trying to test pretty well, but don't necessarily, are not necessarily the best programmers out there. If we take the salary of the, those, you know, the delta between those two salaries and compare them with, you know, subscription to a test automation platform that out of the box gives you everything, or for the most part everything that you will need is a very easy, you know, calculation that we have been presenting to the decision makers, to people holding dollars in the project. And that seems to be resonating pretty well in the market so far.

That's interesting. I guess also in theory, I know I assume you, like you said, you have case studies and customers that because now you have it' not such a hard in demand skill with the automation piece, you can increase test coverage. So you increase test coverage, you can increase your team because more people can contribute while actually hiring more people because more people in your team who aren't necessarily developers can contribute to the testing effort as well. Is that, do you see that happening?

Absolutely. And I guess what, what we also were running into a few years ago was that the more technical people who were interested in building their own automation frameworks and lear and architecting these, you know, test automation platforms internally we're sort of, you know, hesitant to move to these no code platforms and everything. But if we show those users that, you know, you are an expert in test automation, we are not trying to take away any of the capabilities that you currently have with, you know, programming languages and open source products out there. If there is a test automation platform out there that essentially gives you everything that you need, but at the same time you can write plugins for it, you can write code if you wish to extend the behavior of or the capabilities of that automation product. But at the same time it's enabling other stakeholders that otherwise they do not have the ability to either learn the programming language or know them well enough to be able to contribute to automated testing. The combination of the two is resonating pretty well even with the, you know, the SD folks who are, you know, very technical out there.

Absolutely. Once again, I'm just going back to my experience back in the day we had a tool called WinRunner. You can either record playback or create keywords or you could just straight up code in it and I would just straight up code in it. So it's good to have the option for both teams and I guess it's, you know, I've never seen a real, a true, easy plug play Record and playback. We don't have someone that's technical be able to do something that's maybe slightly different than the use case that the vendor thought about, that can add some code that makes it more specialized for their particular situation. So having both options I think is definitely critical as well. So you know, you mentioned something else earlier about, I think you mentioned non functional testing. This is actually a trend I've been seeing a lot of how testers are getting more involved in not just functional automation, but now they're being asked to do things like security testing and performance testing, accessibility testing. And so once again, that requires more tools and more technologies and more techniques to learn. Have you been seeing this as well as the need to, to get involved in other types of testing and have you seen any workarounds and maybe how you can leverage existing functional tests to do these other types of tests of a test that a lot of people have been asked to do?

Yeah, absolutely. We run into it all the time. The problem is not only you will need various other skill set and various other tools to be able to do non functional testing. The other problem that we have seen and our clients are constantly running into is that, let's take security testing for example. If you want to do security testing, given that there is couple of very highly skilled security people in house to do it, but they do not get to do, you know, security testing of the project, of the product that the team is developing until the very end in the life cycle of that specific Sprint or specific release. So it sort of comes in very late in the game if there are tremendous, I guess, security vulnerabilities in the product that could potentially be a showstopper for the release to go out. What we are prophesying now is that given that there is a unified platform, no technical expertise needed now all stakeholders are able to automate. What if you can reuse these for security testing but not do it at the end of the Sprint? But we do non functional testing via the shift left paradigm as well. What is wrong with developers, testers, business users, as they are building their automated functional tests to be able to, you know, bundle them together to do some load testing, to do it, some, you know, dynamic security testing or you know, accessibility testing early on so that they are found much earlier. So that reduces the feedback loop and everything. You have more time to fix it, less costly to fix it earlier in the Sprint. So a lot of projects, a lot of decision makers are actually getting into the mindset that, you know, a unified testing product where it Allows you to do all types of testing, functional and non functional, to get maximum reuse out of the test assets that you're creating for the project is getting tremendous, tremendous, I guess, interest.

Have you seen successful implementations using this approach? Do you have any examples?

Sure, I mean at Subject 7. So we do have a unified testing platform. Typically when we approach a client we go for, you know, automated functional testing, integrate, integrating them into their DevOps pipeline with various plugins that we have and everything. Once the teams have built enough automated flows in the system, then they come back to us and say, do you guys have any solutions for accessibility testing, security testing, load testing? And when they see how easy it is to take and bundle a series of these flows and repurpose them for load testing, security testing and accessibility testing, it's just a no brainer for them to, you know, pick up or add that specific module of our software so that, you know, the same people were already well versed, created these many automated flows, be able to just use it for, you know, other types of testing. We've seen many of those successes already.

Nice. So you know, different tools, I think, different techniques, different companies, better. So I know if you had a startup, it's just two people, three people, maybe overkill to have like a one, no all encompassing solution. But if you work for a large enterprise, I would think having a solution in place like this would be helpful. So where do you see yourself as the ideal customer? Who do you think would benefit from your solution and who do you think maybe it would not be the right solution for?

So our user base, our clients are very diverse. We have, you know, global consulting, you know, companies who are doing constantly working with Fortune 500, doing their Oracle ERP implementations, doing their workday salesforce implementations using our product. We have, you know, lots of government agencies, whether on the intel side or you know, civilian side who have been using our product for their custom made, you know, government projects. And then we have a lot of E commerce companies large and small who have been using our products successfully. I think it's just, you know, how comprehensive the solution is and how flexible it is to scale as the project scales. And also the fact that it is, you know, subscription base that can grow as the projects grow, have been tremendous success factor and a very instrumental piece of information for our clients large and small to be able to know that, you know, if I'm heading the route of a unified platform. But that's on a roadmap maybe three years from now having the sense of, you know, comfort that this tool that we are picking today, even though for just, just this niche being, you know, aut functional testing, but knowing that it is capable of satisfying our testing needs a year from now, three years from now, 10 years from now, has been, you know, very instrumental in getting them, you know, up and running and you know, feeling sort of comfortable that we are making the right decision because this is sort of addressing the things that we have on our roadmap in the near and long term future.

I love that tip. Just for folks listening, regardless of what solution you're using, you should definitely look at your roadmap. So when you're planning your automation, you're looking at, okay, what's coming down the pike two to three years from now. I had the same issue once again with the enterprise company I was working for about integrations that were coming a few years down the line that we didn't take into account. So great point there about the roadmap and I guess it's good that people can start small and then if they say, oh, now we need a scale, rather than say, okay, well let's go and find a solution where we can scale now against all these different devices, I assume you would then have that type of solution in place for them as well at some point.

Absolutely, absolutely. Yeah. We were using various cloud platforms behind the scenes so we can scale, you know, horizontally, vertically as a project scale. So absolutely.

Okay, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts and what's the best way to find you or contact you? I'll learn more about your awesome solution.

Sure. So the piece of advice that I have is when you are starting to go the route of Agile and Scrum and looking at a DevOps tool and then more importantly telling automation tools, please look at the skill set that you have internally across various stakeholders being business users, subject matter experts and more technical people that you have and see whether there are any tools in the market that can fit the need of them as opposed to trying to revamp the entire organization and just bring in brand new set of people who do not necessarily know the application. They are good programmers, but even though they are good, good at building automation frameworks, but now they don't know your specific application and you will basically lose that part of the subject matter expertise that you have. You can find us at www. Subject-7.com we would be happy to show you guys how our platform works and obviously you will see tremendous number of case studies, quotes from our clients and we hope to be able to address your test testing needs soon. Thank you.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a386 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.