231: Nodejs Automation Frameworks with Casey Cantwell

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. Today we're going to be test talking with Casey Cantwell, a senior qa architect about WebDriver IO automation. This episode came about after receiving an email from Casey who asked in reading your blog post on Frameworks you need to know, I was surprised to see frameworks based on Node JS didn't get any love in your article. We chose webdriver IO to help build a framework around for our automation efforts.

And we're loving it.

So after reading the email, I asked Casey if he would be open to being interviewed all about his experience and with creating Node JS based automation frameworks. So that's what we'll be test talking all about today.

Also, if you haven't heard already, registration.

For Automation Guild 2019 is open to learn all about the latest in automation tips, tools, techniques and best practices you need to know to help you be the hero of all your automation efforts in 2019.

Make sure to register for Automation Guild at Automation Guild.

You only have one week left till December 7th to get our two for one special. Register now for Automation Guild and get a free ticket, a savings of $197 to Secure Guild, an online conference dedicated 100% to security based testing that I'll be putting on May 20th to the 21st, so make sure to register today. To avoid missing out on this value based deal. Go to automationguild.com and register today.

Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testalks.com and click on the Sign up now link under the Homepage Sponsor section.

Hey Casey, welcome to Test Talks.

Thank you Joe. I appreciate you having me on your show.

It's great to have you on the show today.

You actually reached out to me talking.

About Node JS and why I didn't have much information out on there. I thought it'd be great to get.

You on the show to talk all about Node JS test based automation frameworks.

As well as your experience with implementing automation. But before we get into it, you just tell us a little bit more about yourself.

Sure. My name is Casey Cantwell. I am an automation architect and QA manager for a software company. We have a Development lab here in Las Vegas, Nevada. My previous experience is in QA. I started at a startup called Triga in 2000 where I worked for 17 years as a QA Engineer, Senior QA Engineer, Manager, Solution architect. And in 2017 I left IBM to join a new company as an automation architect to develop and drive a automation strategy for a new cloud platform that we are developing.

Okay, so awesome question I have is how does your QA manager type background help influence how you create an automation framework? Usually people usually just start off as an automation engineer.

Just wanted to know how that kind.

Of colors your perspective on what is an automation framework and what you think are the critical pieces for an automation framework.

That's an interesting question. So as a manager it's helpful. As an automation architect and a manager, it's helpful to kind of set the direction and it gave me opportunity to do some research on the technologies that we were going to use to build an automation framework. So when we, when I started at this company, I was given a period of time to do a POC where I was able to evaluate different technologies and different tools, knowing that ultimately I wanted to build an automation strategy that was going to allow us to release code after we finished a sprint. So our goal is we have a sprint, we develop stories, and when that sprint is finished, we want to be able to promote that code to production with a high level of quality in a very short period of time. So we needed an automation framework that we could build that was going to be.

Flexible, it needed to be cost effective, it needed to be very maintainable, it needed to be.

Have fantastic reporting, and most importantly, we needed an automation framework that we could put into our CI CD pipelines so that when we do go to release on a regular cadence, we'd be able to run our automation and know that we weren't regressing anything with stories that we're currently developing. So in that role as a manager and as an automation architect, I did that PoC.

Against many tools and technologies and ultimately that led us to the direction where we chose webdriver IO, which is the Node JS bindings for Selenium. And it turned out to be a fantastic choice for us. And we've done some really exciting and fantastic things with our automation architecture.

Obviously, the next question is, what drew you to WebDriver IO? Or why did you choose Node JS as the technology you were going to build your framework around?

One of the challenges that we had in developing automation is it was because of the technology stack that our platform is being built on. So our technology stack is for the platform is Node JS in the back end, JavaScript, HTML and CSS in the front end. And specifically we're using a technology called Polymer web components and lit element on the front end, which has.

Shadow doms in that technology. And what's interesting about the technology stack that I'm having to automate against is that Selenium by default does not support shadow dom. You cannot have a selector and be able to say, I want this element by id. If that element is hidden in a shadow dom, that selector is not going to work. So we had a lot of challenges with traditional tools to where those tools did not support Shadow Dom, just like Selenium doesn't support Shadow Dom. But fortunately, with WebDriver IO there is a workaround to penetrate a shadow DOM and to be able to find nested elements. The reason we chose Webdriver IE specifically is because our development stack was primarily JavaScript. Our development team, they're full Stack developers, all developing in JavaScript, so it naturally led us to wouldn't it make the most sense to also have an automation architecture and framework in the same language that our developers are developing in? And that was a really important aspect of why we chose WebDriver IO. Developers develop in JavaScript, the automation team will develop in JavaScript and there's a synergy and they can help us do our jobs when we run into technical issues or need assistance with our framework.

Very cool. So I understand the reason why now that you have an implemented, I'm curious to know if it actually played out the way you thought it would. And the reason why I say that maybe. I know some dysfunctional teams is a lot of times they choose the same exact language as the developers, but then the developers never even touch the framework, and so they may be choosing a technology that they're not even comfortable with. So how's that working for you? Are the developers actually contributing now or is it like a seamless type of integration between your developers and the people doing automation?

It's worked out really well for us. Our Scrum teams are compromised of roughly four to five developers and an automation engineer. And when we're working on those stories, our definition of done is that a story has to be fully tested, has to have no defects, and more importantly, for that story to be completed, the automation framework has to be implemented for that story. Automation scripts for that story also need to be automated. And our API client to support a lot of our development needs also has. So our SCRUM team is kind of a functional team and they have a single goal of putting out a story at the end of that sprint. And because that team works closely together and they're using the same languages to achieve a common goal, it's very common that a developer will be working with an automation engineer to peer program some complex function required in the automation framework to support like a drag and drop of an element. For example, we have some complex controls that might need some specific JavaScript developed and it's really helpful when a develop when a developer can help the automation engineer develop the method to support a certain control when needed.

Cool. So is the automation engineer outside the Sprint team or do they live inside the Sprint team?

So the automation engineer lives inside the Sprint team and they are that team is compromised of it's made up of four to five developers, an automation engineer, there's a designer and a product owner that are responsible for getting out those stories. So it's a Automation is built into our process and we do not complete stories unless the automation is in place to support that story when it's completed.

So if the automation engineer was sick or they leave the company, do your developers fill in that gap or do they leave the story open until you fill that automation role?

Well, the short answer is that we have enough. We have enough. We have more than one Scrum team. So in the event that someone were to leave or someone was sick, other automation engineers on other Scrum teams can help pick up that slack. And we also have a really robust and successful internship program to where we're bringing in computer science folks from UNLV and we are getting them trained on our automation framework and they can develop and write scripts. They can also contribute to the framework directly. So in the event someone was out or sick, we can use some of our interns to help backfill that work where necessary. But developers could get in there and help if necessary as well. But we haven't had a need for that yet.

Now, you did mention webdriver IO is what you ultimately chose. Were there any close contenders that you're debating over?

The short answer is no, because we're using.

To be honest with you, my POCs were not very successful when I was trying to automate against.

Web components using polymer. A lot of the traditional plan record tools couldn't find an element because our elements were hidden within a shadow DOM and that posed a real challenge for us. And then some tools that we were evaluating, they didn't support Safari or they only supported Chrome. One of our criteria was that it needed to be have wide adoption against the things that we needed to support. We needed to support Safari, Firefox, Chrome, Edge and Mobile. Some tools were eliminated because they couldn't support Safari, and a lot of tools were eliminated because they didn't have direct shadow DOM support. Polymer is a Google's web component library. It's essentially a library to allow you to build web components. A web component is like a control, very close to the native browser, similar to React or Angular. Polymer is a similar framework that allows you to build a web component. So you can think of a web component as like a control or a page, or it could be a table that you can write as a single component. And it has its own shadow dom, so it has its own little DOM structure. And it's encapsulated and protected to where you can take that component and have it used in one page and you can pick up that component and use it in another page and it will work and will be styled and will have the same look and feel in that other page. It's a way to develop one component and reuse that in multiple places. So a lot of our applications are web components to where everything's nested in a shadow dom. So elements are essentially hidden from the parent DOM structure. So it's kind of interesting in the way that we have to do our selectors. Generally, if you had a selector, you would say, give me this element by this ID or by this name or by class, or whatever the selector strategy might be. But in an environment where you have web components and you have shadow doms, your selector strategy is way different.

In our automation framework, I have to build a selector which is an array of all the selectors to penetrate a shadow dom. Let me give you an example. We have an application. It has a component that has a shadow dom, and inside that it has a component that has a shadow DOM and that component has something that has a shadow dom. I have something where I have to select five things in order to penetrate each shadow dom. To get to that lower component, I build an array of selectors. The first selector would be app. The next one would be this component with ID of this value. Then we put that array through a loop where it says find this first element. I've got it. I've now penetrated and I can now access that shadow dom. Now take the second element in that array, search for that, find that element. Ok, I've got it. Now I have the ability to penetrate that component's push shadow dom. Then I take the third item in that array, I search for that, I find that element. So the, the location strategy is really, really complicated in our automation framework, but it actually turned out to be pretty simple because of the way we're using object Orient, the way we're using objects, and inheritance, where our automation framework is more of less an API, which we have objects. So at the highest level we have the app, and the app has a method called get this object that will return you that object with its parents element selector path plus what that path was. So each object in our automation framework has.

A selector path based on the object that was calling it. So every component page view form in our automation framework has a selector, and its overall selection is an array of selectors in which you're using to find that element.

Is this outside of WebDriver? Like, just for you to even start using WebDriver IO? This is just JavaScript code you've created to actually interact with the elements and then you hook webdriver IO into that by calling the methods.

That is correct. That's correct. Webdriver I.O. is our test runner and we had to develop the classes to support the ability to get those elements. So we're using page objects. So every page, every component, every view, everything has its own class. Insider Automation framework, at the highest level we can say app, get this object and then the framework will return you that object. But most importantly, that object's selector path will be set up to be its parents selector path plus your current selector path.

We take that array and we send it to a command in webdriver IO to say find this element in the shadow dom, which takes an array of selectors and we'll keep looping through the DOM structure and go through that array one by one, penetrating each shadow DOM until it can find the element that we need. So our selectors, a selector for something that might be nested four or five levels deep. That array would have five selectors and we select one thing at a time, penetrating the shadow DOM each time we go down. So it's a lot of objects and a lot of classes in our framework to support that, but it's a single method that does the query lookup.

Nice. How did you come up with this approach?

The way we came up with this approach was we did a POC using webdriver IO and we knew that we wanted to have scripts that were going to be easy to write, easy to maintain. So our development architect suggested that we come up with a way so that our selector paths were always automatically calculated so that we wouldn't have any static.

Paths in our code. So starting at that highest level, we have an app and its selector is app. And we use the concept of getters and setters to where we can say get me this object. And when we get that object, it sets that object's path up for you. So, so when you get it and that path is its parent's path plus what that object's path, that child path will be. So each object is kind of generic and each object can get its other objects. So our automation framework matches our application kind of one to one. And in the framework we can say this page, get this page, get this form, get this field, set a value, clear a value, click on a value, click on something, get inner HTML, outer HTML, whatever it might be. But it's a lot of objects being passed around. It's more like an API to where in our automation framework you can say, give me this thing. And when it returns to you, it's set up with the selectors set for you. And then you can take a specific action that you want to do in the browser, like setting a value, clearing a value, clicking on something, or asserting something.

Awesome. So like I said, it's almost like a one to one matching with your framework and your application. Does that make it easier for maintenance? So, you know, if someone touches something in their application, it's easy to map it to exactly where that would be in your framework itself.

It matches exactly one to one. So if you were to inspect our HTML and you were to look at the HTML path.

You would see a direct correlation of for every object that you see in there. That's before a shadow DOM that is represented as a class and an object in our automation framework. So it's very easy to know how to if you need something, it's easy to know how to get it. So whenever we develop a new component or a new control, we can add that to our framework directly. But what's really fantastic about our framework in the way that we're using objects is that when a component is used in another place, let's say we have a component and it's used in page one. Page one has a method that will say get that component. If that same component is used in another page, we would just add a method to page 2 that said get that component, and both pages will set its path appropriately. So that way we only have to have one class that represents that component and it might have all of the features that that thing can do. Let's say that component is a control and it has some buttons, it has some fields. That component will have methods to get those specific fields, those buttons, those whatever it might be. And we can reuse and not have to copy and paste code. One component, one class, other classes can call and get that class and get a handle on the methods and fields that it has. So there's a lot of maintainability in that. We're trying to really avoid having to write, copy and pasted code. That's kind of a mantra that I've learned. If you're copying and pasting code, you're probably doing something wrong. And that's kind of a motto that we've lived by. But we've had.

Refactors and some work done in our platform where they decided to move an element where it was buried down two or three shadow doms. We said, you know, we're going to change that and we're going to make that an overlay directly off the app. It was a pretty big restructure of our code and our automation framework was able to adjust to a really big impactful change by only having to change a few lines of code. So instead of those components being available at this lower level object, we just change that get method to the app and it was able to react to that change in development really quickly. So there's a lot to be said. And using objects and object oriented programming to tackle some of these challenges in automation.

Cool. So for that last example, so you have an automation engineer in a Sprint team, but when they make a change like that for the framework, is there like another group outside of the Sprint teams that decide what gets checked in or is there a code review like, how did you decide? Like that approach? Let's do this other approach rather than someone going rogue within the Sprint team.

So we have a fairly robust code review process. So every PR that is checked in by an automation engineer is reviewed by the senior engineers and by myself before that code can be approved. And we also, as the automation engineers ourselves, also have our own small standup. So we, we have our own little Scrum team, which is automation centric and automation focused. And we meet twice, twice a week for 10 or 15 minutes and we kind of review what's going on with our stories, what the automation impact is, so that everyone on the other Scrum teams is kind of aware. So we're very in sync as an automation team in what other folks are doing. But we do have a pretty good review process in place. No one can check in code to the automation framework without it being formally reviewed and approved by others.

So Kasey, you have a very interesting background. Once again, you worked for IBM like a very large enterprise company that probably had a lot of procedures in place. You also have been a QA manager. How are you measuring success with your framework? Like, how do you know you've been successful? Or are there any kind of metrics you use to gauge the health of the framework and how your efforts are going?

So one of the ways that we measure success is we is our ability to execute all of our automated scripts on a nightly basis and to see fantastic reports around our execution. So our automation framework using WebDriver IO also just to talk a little bit about the core technologies, just to kind of set that up, but the core technologies of our automation framework is WebDriver IO. Mocha Chai. We're using Appium for mobile testing and then we're using a Lore framework for the test suite and test case reporting. So we execute all of our tests nightly. We execute them on Chrome, Firefox Edge, we're executing our scripts using Appium on mobile and tablet devices. And then for each one of those categories, it generates and uploads the results to aws. And we have a nice report in the morning that says here are all the automated tests that ran, here's their current status and their state.

We can see the success that we're executing on a nightly basis, which is going to help us in the future when we get to production, having a really solid process in place where we have scripts that are running nightly. The reporting engine is fairly robust. It will tell us when we have things that fail, when things are working.

You mentioned all these different environments you run against at night. How do you manage your test or your test runs to run against all these different kind of configurations. Do you use a Selenium Grid or Sauce Labs or something like that?

So our long term plan is to move to Sauce Labs next year in Q2 when we get closer to production and looking to put out our product. But for now we are using Docker containers. So we stand up a Docker container, we do an NPM install, so we install all the latest technologies that we have in our automation framework and then we execute our suite of tests on a specific browser. So we have a container, it launches, it executes all of our tests in Chrome, it then posts the results of the Chrome run onto a Chrome Allure port. And then we do the same thing for Firefox, we do the same thing for our API tests. Then we have a separate Windows environment that is executing the Appium.

Suites. In addition to Edge, we have all those segments that run every night and then we come in in the morning and we see the results overnight in our development environment.

Very cool. How reliable is that?

It's been really reliable for us. We've had some inconsistencies. When we moved to Docker container, we weren't managing the resources on the container as well as we could have. So we had some inconsistencies. But for the most part, we're running 150 automated tests right now a night and we on average are running at roughly 97 to 98% passing rate. And the issues that we have are attributed to defects that known defects that we have in our platform that haven't been addressed yet.

How do you triage those errors? Do you have a use the dashboard? Is there any other tools you use to say, okay, this is a real issue so you're able to keep track of it next on the next next run.

So the way we manage our scripts and results is we developed a JIRA integration. So to take a step back, we use JIRA to manage all of our release management, to manage our stories and defects. And in JIRA we developed what we call a nightly board. And when our automated scripts run, we were able to extend WebDriver, IO and Mocha to create a custom recorder. When every test runs, it will write the result to a text file.

At the end of the process, we process that test file and our JIRA integration will say this test failed. Check query JIRA and see if a task exists on the nightly board and if it does update that task that the test failed again. But if a task doesn't get does not exist, create a new task representing that script failing and then it will append the error to the task and it will also assign that task to the owner of that script. So the the nightly execution is fully automated. So that we have a board that is loaded up with tasks of what is failing. And then if a test passes the following night, it will check that board to see if an open task exists, saying that it failed previously and will automatically close that task. So everyone on our team, every script is assigned an owner and that owner is responsible for looking at that nightly board and seeing what new tasks were generated for them on a daily basis. And then taking a quick look at that test to determine was that test flaky? Was that test, did it fail because it was environmental or was there a defect or was there a timeout in someplace else that caused it to Fail. There is some triage and there is some maintenance to make sure these scripts are running pretty well. But in general we don't have a lot of, we don't have too many false positives, but we do capture all the metrics and we do track why a script is failing and what fixed it. So it could be that we had a script error, or it could be just an intermittent or a flaky test, or it could be a defect, but we track all those metrics. And it's really nice to have the reporting fully automated so that in the morning you just load the nightly board and it will show you the tests that failed overnight and the reasons why those tests failed.

I really like how your team has overcome some unique challenges to your environment. Is there someone that's leading this or is it just you have a great innovative team and everyone comes up with ideas like this?

It's been kind of a. It's a very collaborative environment. So working directly with developers in the automation team, it's, we, we see some challenges. And, and that's one of the reasons why I love JavaScript so much and why I love being in a JavaScript ecosystem, is that when you have a problem, there are so many ways and so many packages and tools to, to assist you in trying to solve some of those problems. So when we decided that we wanted to develop a JIRA integration because having to read through all the reports manually every night was just taking too much time, we had an automation engineer and his task was, hey, go look at the little reports and figure out which one of these were already reported and which ones weren't. That was taking a lot of time. So we thought, you know, we could probably develop a JIRA integration to automatically do that for us. So because we're using JavaScript, we were able to Google, you know, Jira integration APIs and lo and behold, there's a JavaScript library called Jira Connector. And all we had to do was import the JIRA Connector.

Library into our automation framework and we were very easily able to develop a JIRA integration to create tasks on a board.

So that's one of the reasons why I like our automation framework so much, is that because we're in a JavaScript ecosystem, there's just a lot of, it's really, really flexible and there's a lot of other libraries that you can easily bring into your environment to do some really cool and neat things.

Awesome. So I like to keep this at 30 minutes. I have one last question, but I just want to touch on Test data. So I think you have something in your, in your slides on this as well. So you're running against all these different environments and a lot of issues that come up, I've hear from other teams is how, how to manage the test data. And I think you've come up once again with another unique approach to handle this. Can you just tell us a little.

Bit more about how you handle your.

Test data for your automation test?

Yeah, absolutely. So we're a cloud platform, and in that cloud platform, we have a customer environment where the customer can log in, they can set up a bunch of data, and they can do a lot of work in that environment. So an approach that we decided to take was we decided we wanted to build our own API client that matches our platform one to one, so we can set up any data that's required. So the way our automation framework is developed is it leverages that API client and the concept of a data set. So a data set is a list. It's a set of commands that uses our API client to go set up and environment. So let me take a step back and better explain that. So let me start with the API client. So to help with this problem, we developed an API client, and that API client matches one to one our platform API. So any, or in our platform where you can create data, update data, or delete data, we have in our automation framework an API call to be able to do that same function. So using an MPM package called Request, which is an, which is an HTTP client, we developed our API client using that core technology. So when we go to, when we go to write or run a test case, we call our API client to go create the customer environment, the user, and it will also create all the setup data and the test data required to run a specific test. So at the start of a test, we say.

Use this particular data set and then we will initialize it, and then that data set will call our API and set up all the data required to run that test. So our tests are, we call them more micro scripts. So when we run an automated test, we want to focus on a specific feature or a specific function. So our platform is heavily on scheduling, but there's a lot of setup rules in scheduling and how things might be scheduled, and there's a lot of tests around that. But wouldn't it be really nice if you wanted to focus your test to be on the actual scheduling and not on the setup or the rules? So in our API, we can set up the environment the way we want in an automated way, and we can focus our test to run that specific test.

And then tear down that data at the end of it.

But I encourage folks to really consider leveraging the API in your automation framework, because there's a lot of flexibility in what you can do.

Okay, Casey, before we go, is there one piece of actionable advice you can give someone to help improve their Node JS or automation testing efforts in general? Let us know the best way to find or contact her.

So the one actual advice that I have for folks that are working in an automation framework is to really consider your scripts, your test scripts, and how you're abstracting them. Everyone's goal should be to not have duplicated code or copied code. And you want to be able to abstract tests in a certain way that you can take a lot of that duplicated code out. So one of the ways that we achieved that in our automation framework was we abstracted all of our code into simple keywords like page, form, view, dialog, and overlay. So in a script we can say app autologin, nav scheduling, and the code to execute that one specific thing was abstracted and that code lives somewhere else. So that when you read a test script, it reads very much like English language. And there's not a lot of techie stuff in our scripts. So if something were to change in a form or in the way you navigate to something, we only have to change that code one place and then that automatically gets updated in all of our test scripts. So look for opportunities to abstract your scripts and look for opportunities to remove the copy and paste that you might have in your test scripts and move that to an abstraction somewhere else in your framework so that you have a really easy way to maintain your test scripts. And the easiest way to contact me is through my email address, which is.

Cantwellsmail.Com thank you, Casey, for your Node JS automation awesomeness. For links to everything of value we covered in this episode, head on over to testtalks.com 2.

And while you're there, make sure to click on that sign up for a free trial link under the exclusive sponsor section to learn all about Sauce Labs.

Awesome products and services.

And as I mentioned, at the top of the show, make sure to head on over to automationguild.com and register for the largest online conference in the world dedicated 100% to just helping you succeed.

Recruiting Automation awesomeness.

So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating Automation Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, Amazing blog articles and other automation.

Awesome.

A mission.