Test Automation Recap for 2019

Five, four, three, two, one. Drop that.

Welcome to the Test Guild Automation Podcast. We're all getting together to learn more about software testing and automation with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today it's just going to be you and me and I like to go over 2019. It's the beginning of the new year. I want to go over predictions I made for automation testing in 2019 and then see where we're going in 2020. So just a recap of the year as well as a look forward to what you can expect as an automation engineer going into 2020 and beyond. And so the first thing that comes to mind for me for 2019, I don't know if you saw this blog post, was I was actually laid on off. My company laid me off in October and so I decided to try to do Test Guild full time. My plan all along was that I wanted to do Test Guild full time, but I wasn't brave enough to do it. So in a sense, being laid off was a blessing in disguise. But you know, I've never worked full time on my own business ever. So kind of nervous. And the reason why I'm telling you this is that the way I hope to generate the majority of my income in 2020 is from my online guild conferences. So I have a conference on automation, conference on performance testing, one on security testing, and one on QA and testing in general. So the first conference I'm actually doing full time in 2020 is Automation Guild. And if you haven't noticed, Registration for the 4th annual online conference dedicated 100% to helping you succeed. Creating Automation AWES. Automation Guild is now open for registration. Head on over to GuildConferences.com automation and register. Today our goal is you learn a tip, tool, technique or best practice that's going to help accelerate your automation efforts in the new year. We bring together some of the biggest names in the automation industry to share their secrets on what works for them and hopefully to help you with your automation. So now that I'm doing this full time, I'm really trying to find ways to really add the most value and things that you need help with. And one of those ways is with online conferences. So make sure to check out the lineup for Automation Guild 2020. I think it's awesome. Basically I survey all previous Automation Guild attendees and ask them what the number one struggle is with automation that they're facing and then try to find speakers and sessions to help address those needs. So hopefully it gives you awesome value in the new year. So check it out. The Test Guild Automation podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit testguild.com saucelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

All right, so Here were my seven predictions for automation testing in 2019. Where I went wrong, where I was spot on, and where I think we're going in 2020. Okay, so prediction number one I made was that maintenance of automated tests will get easier. And so what did I mean by maintenance of automated tests will get easier? And so currently the biggest roadblocks I've seen with automation frameworks is number one, maintenance due to weight conditions. So a weight that you put in didn't work and so it times out or maintenance due to locators. So locator changes or use a locator that's not really robust. And when an element changes, you have to update your scripts in order to update that element so that when the test runs again, it works. And so a lot of times if you're working on a very large project with a bunch of different sprint teams, develop makes a change, doesn't necessarily make the change to the automation framework. You run the automated test, they fail, you don't know why it fails. And after, you know, hours and hours you realize an element changed. So I think a lot of people have addressed this either with certain technique. If you listen to the podcast episode we did with Greg Pascal from Ramsey Solutions, he spoke of a technique that really helped them make their tests more maintainable. And it was basically using using the strategy of using multiple XPath locators. So definitely check that out. That's episode 278 where Greg talks about that. I also interviewed Jeremias from Retest and Retest is a paid solution. However, they have an open source solution called Recheck Web. And Recheck Web uses something called what they call a golden master based test framework that sits on top of Selenium that allows you to easily create and maintain your tests so they become almost unbreakable. And basically one of the biggest struggles team have is if there is not a unique ID that a developer has exposed. Usually what you have to rely on is an XPath or a CSS. And I know some people have been successful in creating a very robust XPath check. But in my experience a lot of times those are still not as stable as you like them to be. What this solution does, it actually creates a unique ID for you by injecting it. And like I said, this is a completely open source solution that came out recently. So here's a quick clip from my interview with Jeremias in episode 277 speaking about how this works.

A lot of tests tend to be brittle in that if you change, I don't know, the XPath, the ID, the text that the test relies on in order to identify the elements it wants to interact with, then the test breaks. However, Gold Master based approach has a benefit that if the test breaks then we still have the Golden Master where that text ID whatever is yet unchanged. So what we do is we have a look, peek into the Golden Master, find the element that we originally wanted to interact with, and then go to the current version of the website and find the element that has the highest match in terms of all the remaining attributes like XPath, name, text and whatnot, and then return this element. So it means that the Golden Master test based approach is much more resilient to change because it can can use all the redundancies that are, you know, inside the website. And we even go one step further, when we create that copy of the website, we introduce into that copy we introduce another attribute that does not exist on the live website. And because it doesn't exist on a live website, it can never be affected by change. So the this means that you now have a virtual constant identifying attribute which we call the retest ID that you can use in order to reference the element that the test has to interact with. And then again using all the various attributes of the past, of the persisted, the Golden Master website and comparing them to the current version of the website, we can find again the highest match and return that and interact with that specific element, making your tests very robust, very resilient to change.

And this is something that came out after I created this blog post. So clearly it's an area that a lot of people and companies have worked on in 2019. I think more work on in 2020. I think not being able to find elements is going to be a thing of the past and I'm happy to see that more and more open source AI based solutions are coming out there that you can then integrate with your Selenium Appium based automation frameworks. So the next prediction kind of lends itself to the previous one and that AI test automation assistance would become a reality. I definitely saw this in multiple vendor based Solutions. And what I mean by AI automated assistance is that because it's collecting all that data for all your different runs from your test, it's able to Azure in your IDE creating a Selenium test to let you know maybe the likelihood of how, how accurate a certain indicator is going to be that you're using. And of course, because it's using machine learning and it's learning over time and it's gathering all the run history, it's able then to take that data and do some sort of correlation to see when something fails, what's the likelihood of why it failed and how to work around it. So one example of this is I did a webinar, I believe in August with the folks at powersoft with their new solution Selenic, that just came out, which actually does this. So let's just listen really quick to how that works.

Here's our first thing. We've got a report here which is not just a simple stack trace. Normally what I would see, if I was wanting to go ahead and look at what my problem was in my Selenium script, I'd be used to pouring through.

A stack trace of in essence Java calls and trying to interpret where this is within my, my test script. What we're going to do now though is we're going to use.

The report.

That comes from Parasolinic right here in Safari. Here I can see that I tried to run three tests, two of them failed and we can see the detailed information for the two that have failed. I can see an overview of the test suite. I can see a high level summary of what happened. I can also go ahead and look at a screenshot. The interesting thing here is this little green dot here or this little green light bulb that's next to this particular issue. So here, this tells me that there's a recommendation, meaning that Selenic saw that there was a problem, analyzed the build or analyzed the test failure. And then it says, you know what, let me see if I can figure out how to fix that. Now, as I said, Selenic has been monitoring my test executions over time. When it sees that it's not able to select on a particular element in the page, what it does is it goes back and analyzes the historic test runs and tries to find a new element locator within the page. Then it assigns that element locator a confidence factor so that I can have an understanding of how well we think that this locator is. Now, that confidence factor is a combination of a weighting which is customizable based upon my preferred locator strategy, as well as a stability ratio, which is done or achieved by analyzing the historic data.

So I just use this PowerSoft solution as an example of how I think other vendors have been doing it and how they will be doing it and even more in 2020. The next prediction I made was Heal thyself Automation. And so basically, at runtime, as you're running your test, if it doesn't find an element, it automatically knows that in relation to another element or using another id, it's able still to continue running your test because it's able to identify at runtime how to interact with that element, even though the explicit ID you may have used wasn't there. So as we saw earlier, that because these tests are running over and over, machine learning can be used to kind of parse all those log files and find correlations between why certain things are failing, why they're not. And so not only can it just bubble up insights with assistance, but it could take it even a step further and heal itself during runtime. And once again, here's a quick example of PowerSoft solution Solanic doing this. But like I said, this is just an example from this particular vendor, but other vendors would use a similar approach.

Okay, so I've got my test. I could go ahead and rerun this. This will work fine. But there's actually a problem with what I just showed you. What I just showed you is that my tests failed and then I went through and I fixed my test and then I now have to rerun them. But if Selenic is so smart and Selenic knows what it needs to do to fix the test, why doesn't Selenic do that for me? That's actually what takes us to the second part of our story. If I'm going to. I just reverted back now, so I've got the old locator that we know does not work. I'm now going to go back into my configuration view and my Selenic view and I'm going to turn on self healing. Now what this is going to do is when Selenix sees the test failure, it's then going to go ahead and perform its analysis, trying to figure out which locators to use and it's going to try them live. It's not going to change the test, it's going to change the behavior of the test at runtime and then use that information to inform me as the test.

What I need to do to update my script. This is the augmenting use of AI, not the replacing use of AI. You don't want to. You have AI just making a whole bunch of random changes to your codes, your test scripts, because it'll just run away with itself. What you want to do is to say, okay, how can it give me some information that's useful, that it can help me improve my workflow? Okay, so let's go ahead and run this with Selenium now. So you'll see the same use cases you can see here. All of my tests have succeeded, but if I look back at my HTML report, you'll see that, yeah, I've got one test that passed, but two were healed. Okay? And they were healed because one of them was healed due to a locator and the other one was healed due to a weight condition. And you can see here in the report that actually both of these now have recommendations. So we go to the first one, my bill pay, and you can see against the same information that you saw previously, this is a locator problem. And it's saying, hey, you know what? Same locator list as before. But now you can see that we actually validated this one. This is the recommendation to use, and we use this to heal the locator at runtime. The second use case is actually the weight condition problem. So this is not a problem with a locator. This is saying, is unable to click on this apply now button. The test was healed by waiting two seconds. So we're recommending that you change your weight conditions on that, on that particular variable or on that locator to give yourself enough time to wait for the particular element. So here, what we're able to do is we're able to guide you not just on the locators, but also the wake conditions, which were those two primary challenges with the tests. And then likewise, when I come back in here and I go, update locator, we'll actually see that we have the same workflow as we had before where we could actually do the recommendations.

All right, so like I said, this tool came out in August of 2019. This is like almost eight months after I made my prediction. So call me Joe Stranamus there. I think I was spot on. And like I said, I think this is definitely direction we're going to see more of in 2020. Along the same lines of AI assisted automation, I also said we see more smart execution of tests, meaning someone checks in code. And based on that, you know that you can run these four sets of tests to Cover that code checking. So wouldn't it be great to know when the developer checks in code, the exact number of tests you need to run and to verify that actual change? I said 2019 is the year when this becomes possible. C Lights was already doing it, or almost doing it. I believe in 2019 they added functionality to make it even more reliable. But other companies started offering solutions like this as well. So once again powersoft came out with the solution Selenic that does this and also appshurify. So once again, let's just go to a quick clip of the PowerSoft Selenic solution that came out in August of 2019 after I made this prediction that actually does this.

How can we reduce the amount of time that it takes to execute the test scenarios themselves? Now with this, this is where we talk about the smart test execution. We're focused on helping organizations optimize their test execution by focusing on the changes inside of the application. This is sometimes referred to as test impact analysis or change based testing. We're able to understand how an individual test affects the underlying code base. At the granularity of each test, we know what parts of the code base that test touches. We can then cross correlate that with the changes that are being made into the code base and then use that to to in essence reduce the set of tests that need to be executed. So you can optimize your test execution and in essence get quicker feedback from your cicd process. You have thousands of tests, and being able to get that feedback very quickly as part of the CI CD process is critical to ensure that you can really optimize an agile DevOps initiative. But if you have to run all 10,000 tests, you're going to have to wait overnight or in some cases over the weekend before you get that feedback. What we can do with this technology is we can actually help optimize the execution of these tests at multiple layers of your testing practice. We actually have technology that embeds into the IDE to assist developers with unit testing working very similar to Solanox creation functionality where we can analyze codes to create unit tests. But then this also has the ability to optimize the execution of those unit tests based upon the code changes the developer is making inside of the ide as well as optimizing the CI process. We also have this same capability for API tests and for Selenium tests, analyzing the changes in the underlying Java binary, so a WAR file, for example, and analyzing what the changes are and analyzing the code coverage against those. By mapping to the actual application itself. It means you remove any ambiguity between understanding.

What parts of the code base a test actually touches because you're looking at the real runtime data.

As I mentioned, I also spoke to James Ferrer Apps Surefy, and their approach is a little bit different. This is James explaining how they use machine learning to help target what tests you need to run against what check in.

We're using machine learning in a way that has already been used by some of the big companies, the Googles and the Facebooks, and in fact, they've actually written papers about it. What we're trying to do is we're trying to say for the set of tests, which is the small subset that you should run based on the changes being made. So we're not building the test for you. What we're doing is we're trying to prioritize your existing test suite, and that's using a classification model. Then we have a couple of other tricks that we do where we try to detect whether a failure has been caused by a real defect or whether it's one of those flaky ones. We have another classification model there. Then finally we have a little bit of a defect prediction model where it's saying, these are the commits that have come in. What's the risk associated with them?

Once again, I would say I was spot on with this prediction. I think more and more companies, once again, are going to do this in 2020. A lot of the predictions for 2020 are pretty much the same as 2019, except it's going to be more advanced. But wait to the end and I'll give you some quick ones I didn't mention in 2019 that I think will be even bigger in 2020. Another prediction I made in 2019 was the return of record and playback automation tools. All right, so I interviewed Dave Hefner November 2018, and he talked about the return of Selenium IDE and how it's better now than it was before. And if you think about it, first off, you're probably cringing when you hear record and playback if you're an older automation engineer like I am. So if you started off in late 90s, early 2000s, a lot of vendors promised the record and playback to companies and it was a nightmare. Nowadays, though, especially like Selenium ID was created from the ground up and that it has built in best practices now that it's able to handle things that the older tools were not able to handle. So after speaking to Dave, I thought, I bet you any money now we're going to see more and more tools coming up with these recorded playback because the technology has gotten better over the years that now you probably can get closer to the original vision of recorded playback, but without all the other pitfalls that people used to encounter previously. So I think it was spot on. If these tools weren't popular in 2018, they were more popular in 2019 for sure. So some folks I spoke to in 2019 that have an IDE type solution that's a recording playback. We spoke to folks from Oxygen and just a shameless plug. Angie Jones actually will be talking about Oxygen and demonstrating how to write test automation using it at the Automation Guild 2020 conference. Definitely check that out. Hand it over to guildconferences.com automation we also talked to the folks at Testproject IO. They have an excellent record and playback solution. I also spoke to the folks at Boozing, which has an IDE solution, also spoke with the folks at Excel, IQ Giljeet, who also will be joining us at Automation Guild 2020 and he talked about why he thinks more and more of these tools are coming out and I believe it's true as well, is that Selenium and appium are just APIs. You need to build all the other stuff around it in order to make it a really full blown framework. And in my experience not everyone has that type of skill. So a lot of these IDE type solutions are coming out that act like a wrapper on top of Selenium and Appium. So they use the best in breed but they add logic on top of it that just makes it easier to create automated tests. Here's a quick example of Gilgit explaining it the reason why they created their solution and the way most vendors explain why they created their solutions to me as well this year to give the.

Power back in tester's hand, right? But it obviously comes with a pretty huge overhead of custom frameworks, automation design, chasing that code maintenance. And unless you write a series of perfect world frameworks, you know the output of Selenium is going to drag you down, is going to drag your ROI down. So you know, again our goal was to kind of bring that power, bring that flexibility, but with the ease of use so that an automation engineer, you know, can, can focus more on their throughput and their efficiency rather than handling the overhead in their, in their times and effort which they spend. So you know, the goal was to kind of take the best world, take the best of both worlds, right?

And just random FYI, Giljeet's Automation podcast was the second most popular podcast in 2019 for the test Guild Automation Show. So definitely check that out. It's Next Generation agile automation, episode 249. All right, so once again, I think I was pretty spot on with my prediction. With more and more Selenium ID like tools coming out once again, I think it's going to be a growing trend. And eventually I think these tools will get smart enough where a lot of the things we're struggling with now will be solved from a technical point of view, but you still need testers to have testing skills in order to figure out if they're doing what they should be doing. Okay, next prediction I made in 2019, let's see how well I did. Was Automation Vendor Merger Madness. So I said there was going to be more and more companies merging together because I always say follow the money. I know it's cliche, but there's a lot of money in the test space right now. So anytime you see that, you know there are going to be companies that acquire other companies to make themselves even bigger. And we definitely saw a bunch of that. So Source Labs in April of 2020 acquired Screener, which is a visual validation test tool. Tricentis acquired Test Project, which we talked about before. It's still going to be a free tool. It's just acquired by Tricentis. Also, Smart Bear actually acquired Cucumber and they acquired Bitbar. So a bunch of acquisitions in 2019. Probably something we'll see even more of in 2020 as well. All right, so the last prediction I made was continuous testing will be the top buzzword in 2019.

Actually, I was wrong. I don't think I heard anyone use the term continuous testing on my podcast. So it was just a term I thought was going to catch on. I thought it was going to be a buzzword, and it really wasn't a buzzword. So just a minor thing there, but I definitely don't think I was on point with that particular prediction. All right, so I may create another episode just for my predictions for 2020.

Just for my predictions for 2020. But basically it's just going to come down to AI. I think it's going to come mainstream. So before when I was in conferences, 2018, 2017, even early 2019, anytime you mentioned AI, people would roll their eyes when it came to testing. But I've seen on Twitter, even some of the biggest.

I don't know how you would call them, but people that maybe cynics are even coming around to say, okay, maybe there's something to this that actually can help me with my testing efforts. Another thing I think is going to be bigger in 2020 is RPA. So RPA has been a term I've been hearing more of 2019, and I think it probably will play a bigger role, especially as we go towards more global data privacy and governance initiatives that we've been seeing around the world. Another area is security testing. For some reason 2019, I heard more about security testing. I've always heard about security testing, but I haven't heard so much from people saying that they actually are implementing security testing. And I believe 2018 we saw the three biggest, I think the second, the third and the sixth or fifth largest security breaches ever happened in 2019. I think 2020 we're definitely see a big uptick in security testing becoming part of a typical tester's or a typical Sprint team's requirement, making sure they're creating secure software, they're testing software for security. And actually I saw this as such a big trend that actually started a brand new podcast dedicated just to security testing called the Test Guild Security Podcast. So definitely check it out. Go to testguild.com podcast and you'll notice now I now have a podcast just to automation, which you're listening to now I have another one on performance testing and site reliability, which I think is going to be a bigger focus area in 2020 and also security. So definitely check out those other podcasts. And Another prediction for 2020, you know, this is just based on conversations with other automation engineers. I was speaking to someone that was saying that on an interview he was going on the end of 2019, he was on like five, 10 interviews. They all asked about experience around API testing, which he didn't have. So I've been talking about API testing since 2014. I think 2020 is going to be the year that if you don't know API testing, you need to know it. So definitely you should add that to your learning plan for 2020. And as I mentioned, I was laid off in October. So my prediction for 2020 is I'm going to be doing Test Guild full time, hopefully if the Guild conferences go well. So if you have anything you want to learn more about any topics, any subjects. This is why I now have time to create more podcasts, more online conferences. If there's a conference you think I should have in 2020 that is underrepresented elsewhere, let me know and I'll try to put it together. And you can always contact me@joetestguild.com with your suggestions for things that I can do to add value for you in the new year. So that's it for my recap of 2019 and just a teaser of what I see happening in 2020. Keep your eyes open for a dedicated blog post around top trends I see for 2020 later on in January. So for links to everything I covered in this episode, head on over to testguild.com a like automation285 and while you're there, make sure to click on the Try it for Free Today link under the Exclusive Sponsor section to learn all about Source Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild. Com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.