234: How Test Automation Will Change in 2019

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. Today we're going to be going over my predictions for trends for test automation in 2019. Just want to wish you all a happy new year. Hopefully you had a great holiday. And if you're looking for ways to accelerate your automation efforts in 2019, one thing I think of right now that you can do is, if you haven't already, register for Automation Guild, the largest online conference in the world dedicated to helping you succeed with creating automation awesomeness. We have some of the top automation experts joining us to share a tip, tool, technique or best practice that you could put into place right away to help either new or an existing test automation project. I actually recently read the Zephyr World test report that they put out every year and one of the things that really shocked me that really stood out was they said that 40% of folks say that the biggest obstacle with test automation is finding testers with the skills needed to succeed with automation. Sending your team to Automation Guild is one of the most cost effective ROI things you could do for your team to have them learn those valuable skills they're going to need in order for your company to be successful with automation testing. There's no travel required, extra time doesn't need to be taken off, you don't need to pay for hotel rooms or food. Really, it's really awesome. Great deal. I try to pack as much value in there as possible. If you haven't already, head on over to automationguild.com and register today. Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testalks.com and click on the sign up now link under the Homepage Sponsor section.

So you're probably wondering what the new automation and testing skills and tools are that you're going to need in 2019. Prior to each automation guild that I put on, I always ask potential attendees to fill out a survey to give me an idea of what things they're struggling with. I also speak with a lot of testers, a lot of thought leaders throughout the year on my Test Talks podcast and also when I go to certain conferences, I get to speak with a lot of big names in our space. And also I love speaking with Test tool vendors because they deal with a lot of customers and all the variety issues that they see across different companies. This information I collect really gives me a clear picture of what the rest of the air probably will look like and what to prepare for. So what I like to do to get a clear picture of what the new trends are going to be, I like to go through those survey results from my gilt conferences. Then I like to look at what new technologies might be coming up that will help folks with those specific struggles that they called out within that particular survey for the coming year. All right, I guess before we actually get into the results, let me just set the stage with a little background about the survey. It's based on the responses of over 295 people. What I found was interesting is as I mentioned earlier in the show, the Zephyr report that came out that said the biggest struggle people reported was that their team doesn't have enough expertise in test automation. Well, the reason why I find that odd is in my particular survey when I ask people to rate their current automation skills, over half, 52% say that they're experts, so they're experts for automation. Then the next half is 30% say they're junior. And then the smallest one is 8.9 as newbie. So one of the first trends I could pull out comparing and contrasting the Zephyr results compared to my results is if they're saying a majority of their companies they speak with are struggling to find testers with automation skills. And the people that listen to test talks that fill up my surveys for guild conferences are saying more than half of them, the majority, that they're experts in test automation. You all are very employable, right? So if you have these skills, first trend is are they going to be more employable than ever in 2019 and beyond, even with AI technologies coming out, which we'll touch on shortly. All right, another question I ask is what is the primary technical stack you need to automate against? Now this is an open ended question, so I get all kinds of answers. But when I put this into a word count frequency app, what it does is it finds common words and it adds up how many times it's been repeated within the survey. The top word that shows up or the word count that shows up the most frequency is Java and then the next most frequent one is dot net, then we have Web, Angular, Mobile, C sharp, and then APIs. So really the only trends I could pull out of this data is I compare this to the survey results for automation Guild Last year. So both Java and. NET were also the top ones last year as well as web. But what jumped into the top five was mobile and API testing. So to me, I mean, we've been hearing about APIs in Mobile forever, but it sounds like just now more and more testers are getting involved in automation for those particular technologies. So that's something to be aware of. And so the other question I always ask, once again is an open ended question. So the response was all over the place was what is the single biggest automation challenge you are facing right now? And the reason why I ask that is I go out and find speakers to talk around those particular topics that folks are struggling with. So that not only are you able to go to a conference, but you're going to a conference that meets the needs that you need for all the things you need to learn for that year to help you with your actual automation efforts. So there are four main buckets I put these into. That was a trend that I saw. The first one was maintenance of test scripts was one of the biggest things people are struggling with. Another one was reporting and analysis of automation test runs. So they were spending a lot of time trying to find out not only how many tests failed, but why they failed. Also a lot of people were struggling with automation frameworks and also a lot of people, I put them in a bucket of what we're calling continuous testing. The need to quickly create automation and to do it throughout your software development lifecycle. So incorporating into DevOps, getting that feedback, all those types of things went into continuous testing. So let's go over one of the biggest trends around addressing the issue of maintenance of test scripts. And let's get this one out of the way early and that's AI and machine learning. So like it or not, it's been one of the hottest topics this year and I think it'll be even More so in 2019. Almost everyone I spoke to on test Talks this year, 2018 brought up something to do around AI or machine learning. And the reason why I think is most AI solutions I've seen, at least up to this point, have been created to address the maintenance of test script issue that we all have seen if you've been doing test automation for any length of time. And I actually spoke to Dan Belcher, who's one of the co founders of Mable, which is one of the first AI based automation test tools that came out and he told me that he started the company before they even wrote one line of code. They surveyed over 100 companies in order to understand what the most frequent test complaint was from these companies. And the number one frequent complaint they heard over and over again was the ability to create and maintain test automation. That was the biggest pain point and was one of the number one struggles. So now how many times have your test scripts failed because someone changed the application under test? They changed the element ID or the change of the name in the application, but they didn't actually go back into the automation script itself made the change. And most of the new functionality in the AI based tools I've seen was designed to help with this very problem. So instead of having to manually update your automated test after every type of new UI change, these tools are smart enough to self correct themselves during runtime. And so I really think this automation test tool, self healing, really is a trend that's going to continue and become more and more sophisticated. As I mentioned, the AI tools I've tried already understand the difference between what might be a very small change to an element and the difference between a new feature or a broken front end. This self healing functionality is possible because after every test run, the tools are collecting tons and tons of data. And this data spans not just output logs, but also HTML data, images, screenshots of the state of the pages at each point that you're executing the test, any errors that may be coming out of Chrome and other browsers, all this data is being gleaned and stored. All this data is feeding into the tool's machine learning algorithms. So in essence, after each run, it creates models based on all the different data sets it's picking up for your application during the test run. And over time, it's able to determine what's normal about the application under test and what's not. So when it finds something that is not typical, it, it can flag insights to let you know, hey, this page is performing slower than it has the previous 10 runs, or this idea has changed. But because I've run so many times, I know this field also has these other identifiers that I've stored. So let me use those other identifiers to identify this field and forget about the ID that it's no longer, that's, that's no longer there. So in that way it's able to self correct itself at runtime without any sort of interaction from you. And you actually could see this in the new Selenium ide. When you record a script with the new Selenium ide, not only does it record the element it thinks is the best one to identify the field, but it actually has about five or six other options. It records for you behind the scenes. So when you run it, it's able to determine if the first element's not found. Let me try these other five to help determine, to help identify the field. So it's already being baked into some of these other tools. Really cool stuff. I think it's really a trend we'll see more and more of in 2019. Now up until now, another trend I've been seeing is where before it used to be vendor tools maybe 10 years ago, then it morphed into all open source with a lot of these AI based technologies. A lot of them are vendor based. So if companies that want to get into AI based automation in the short term, you're probably going to want to have some budget. So that's definitely a, unfortunately a trend that I probably see going on at least in the next couple of years. But I have seen, I saw my first instance of an open source solution by Jason Auburn. Jason is really a thought leader in the AI testing space and he's a serial test entrepreneur. He's the founder of Test AI and also the author of How Google Tests Software. He also, I think recently revealed at the previous Selenium conference an open source AI classifier plugin for Appium to help bring AI based element finding to Appium. Now I don't see this being a huge trend in 2019, but I think over time, just as we saw vendors morph to open source, I think we'll see the same thing with AI based technologies. The vendors have the upper hand as of now, but over time probably open source will then become the dominant solution with more and more contributions to AI based technology being baked into these open source solutions. All right, so another thing that people were struggling with based on my survey was actual finding out not only did a test run, but when it fails, why it failed. And I think machine learning, for reporting, for automation, for looking the test runs and doing analysis is going to be another trend we see more and more of in 2019. You know, within some large enterprises, the amount of time needed to triage failing tests can be all consuming. And this of course leaves little time to innovate. Not to mention the considerable time wasted hunting through a bunch of log files to determine whether failures are due to real issues, system issues, coding issues or real defects. A lot of times, you know, time just runs out, you don't know why it failed, test kicks off again and it passes and then it fails and you have this constant flakiness but you can never really determine why that is. And actually Alan Page has an awesome metric he came up with called Mean Time to Diagnosis. So essentially it's a testing metric that tells you how long it takes you to figure out why a test fails. Machine learning is perfect to help you do this quickly and efficiently. Once again, Jason Arbon, in a test talk to me this year, explained that with AI based solutions, they take a lot of the drudgery out of testing. They actually find things at scale. Now, for example, you as a tester can't go through a million server logs because you're not a machine, but a machine can. A machine can easily show you the things that are very likely to be different between thousands, hundreds of millions of records. So ultimately this AI technology we'll see more and more of is going to do this drudgery for us of serving up insights that are interesting that have a prediction to why it may have occurred. So it ultimately is up to the human judgment to say if the machine is right or wrong. A great example of this is something I started using in 2018. I hope to see this grow in 2019 because I think it's a big time saver. And that is ReportPortal IO. It's a open source tool I've been playing with that I think really is going to take off, that is able to analyze your results using machine learning to bubble up insights for you. And it could automatically bucket things into why something failed, flag things of why they're flaky. Really great for helping you really analyze your results quickly and automatically using AI based technology. So check that out if you haven't already. Reportportportportportportportal IO and once again, after speaking with a bunch of people on test talks, for example, Iran Sher, CEO of C Lights, feels that in 2019 we'll start to see a transition not only to AI and machine learning, but also in managing quality through data analysis, through data analytics, as opposed to relying solely on opinions, feelings and experience. So how cool would it be to have a tool that leverages machine learning to not only analyze the code coverage of your existing test, but including the functional test, but also tell you that here's the code that was checked in. Here are the three tests you need to run to actually cover that change. So rather than just say ah, run all the tests, but not necessarily know strategically what test should pass for that particular check in, wouldn't it be great to know when the developer checked in code to automatically know the exact test you need to do to Verify that change. That's where I think we're heading in 2019. I think we're going to see more of that as well. So as I mentioned earlier, Selenium IDE is back. That is a big trend we'll definitely see more of and that is the return of record and playback. Most of these AI based tools are both codeless and record and playback. So back in the day these types of technologies were really frowned on and really a lot of vendors probably promoted it and you started using it and it never worked how they said it would work. I think record and playback is becoming more and more feasible. I don't think it's perfect. I think you will always need to tweak a test. However, these tools are becoming more and more powerful And I think 2019 is probably one of the first times we'll actually see record and playback as a valid option. And Selenium IDE has been recreated, it's been brought back to life and as I mentioned, it's using some AI based technology behind it to help the playback be more reliable by using AI machine learning to interact with your application and runtime and self correct. So another big trend is recording playback is back. But interestingly enough, even though Selenium IDE is based on Selenium, I think there's going to be a surge in non Selenium automation based tools. Now, way back in 2014, I did a predictive blog post about how browser based automation tool vendors were slowly dying off and being replaced by Selenium. I think we could all agree that in the last four to five years, Selenium has been the de facto browser based automation tool. And as a result, many vendors are now embracing and offering support for it as well. But in 2019, I think we'll start seeing companies that started their automation journey years ago that automation does not equal Selenium. You need to use the right tool for the right job at the right point in your software development pipeline. Now it's possible. If you are using Selenium now, you're realizing it's not the right tool for you. Don't get me wrong, there's nothing wrong with Selenium. It's an awesome tool, but it's just a tool. And not everyone needs to use the same tool the same way. Different teams have different styles and different needs and preferences. I think that's why we've been seeing more and more tools like Test, CAFE, Cypress IO Jess being introduced lately that don't leverage Selenium WebDriver. And I think That's a trend we'll see more and more of in 2019. All right. The new buzzword for 2019, if you haven't heard it enough already, is continuous testing. You're going to hear much more about continuous testing in 2019, and in fact, it has been one of the hottest topics on my test talks interviews. If you're wondering, the best definition of continuous testing I've heard is the ability to instantly assess the risk of a new release or change before it impacts customers. You want to identify unexpected behavior as soon as they're introduced. The sooner you find and prevent a bug, the easier and cheaper it is to fix. So this is really a big development for companies to realize that not only do you need to learn how to do scripting, but but that you need to fundamentally change the way you do software development and the way you do testing to accommodate the fast feedback loops you need. As we move ToWards rapid development, DevOps type practices and continuous testing is a critical piece of this. So I really think continuous testing is the next major wave of testing. Also to go along with this concept of continuous testing, being able to rapidly develop and test code and get them into the hands of the consumer faster and faster. I think 2019 will finally start to see more and more companies also moving towards virtualization. For example, my company moved towards a solution called Skytap which allows us to easily automatically stand up whole environments completely and tear them down at runtime for our automated test. Automating the infrastructure, which can be a big pain point for some teams if you have to maintain multiple environments and manually set them up, this could slow down your development process. So having a way to quickly stand up environments and tear them down is also a huge trend I see in 2019. So see virtualization cloud infrastructure being a big, big push for DevOps and testing in the new year. And the last trend I've been seeing is mergers and acquisitions of tool companies becoming larger and larger. So as we talked about continuous testing, with the rapid growth of continuous testing DevOps, more and more vendors really are jumping at the chance to create an ent end testing solution for the clients. And to accomplish this, they may need a tool that isn't part of their current portfolio, like say, performance testing or a test management tool. And that's why I think we'll see more and more merges happening in 2019. Now we've already seen a bunch of these in 2018, like hip test was recently acquired by Smart Bear Zephyr, I think also was acquired by SmartBear Tricentis acquired Qasymphony and Flight IO. In fact, as I was actually writing an article that I created before I created this podcast, Perfecto Mobile was acquired by Perforce. And the CEO of Perforce stated that one of the reasons why this was done was that the enterprise continues to need more automation to both scale and accelerate their application delivery. And that is why they acquire Perfecto Mobile, which is really the whole point of continuous testing. So follow the money. I know it's cliche, but the main reason I believe 2019 and beyond will be a good time to be a tester is the number of testing quality focused companies being acquired. The market clearly has a demand for developing quality software rapidly and is putting its money behind tools to help that along. And of course you can't do anything with the tool or technique without testers, a fact that should give you and me optimism in the New year and beyond. And notice my prediction in 2019 and beyond is not testers will be replaced by AI. Anyone that thinks that, I think is seriously insane. If you think that someone like Michael Bolton James Bond can be replaced by an algorithm, it's not even a contest. And along these lines, if you're looking for a good book to read, check out Life After Google, the Fall of Big Data and the Rise of Blockchain Economy by George Gilder. He actually calls this type of thinking materialist superstition and has awesome reasons why AI will never replace a human being. And what makes it so credible is George, in early 1990 wrote a book, Life After TV, where he actually predicted the rise of the mobile cell phone. So this guy knows what he's talking about. There's different domains, AI takes care of one domain, testers take care of another, and I don't think they're really in conflict. I think it's really man and machine helping each other, bringing us into the new year and beyond. So let's create more innovation. Don't get left behind, Embrace the change and lead the way to a better automated future. All right, so for links to everything about we covered in this episode, head on over to testtalks.com.

And while you're there, make sure to click on that Sign up for a free trial link under the exclusive Sponsor section to learn all about our awesome sponsor Sauce Labs. So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating automation. Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness.