Journey to a World-Class Test Automation Framework with Greg Paskal & Max VelDink

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Automation Podcast. We're all getting together to learn more about software testing and automation with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Greg Pascal and Max Feldink all about their journey to a world class test automation framework. So get ready to discover all about major decisions you need to make when creating an automation framework, like vendors versus an open source, tools, how to choose a language, why reporting matters, how to test your framework, and much, much more. You don't want to miss this episode. Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey, Greg and Max, welcome to the Guild.

Hey Joe, it's good to be here.

Yeah, thanks for having us, Joe.

Awesome. It's great to have you on the show. Greg's been on the show. I lost count of how many times Greg's been on the show, so I think most folks know him. But before we get into it, a quick maybe introduction, Greg and then Max, let us know a little bit more about yourself.

Hey, yes, this is Greg. It's really great to do the show today. Joe, with you and have Max. And we also have Nick Temple, our new guy here. Joined the automation team about four months ago. We worked here for this awesome guy named Dave Ramsey at Ramsey Solutions down here in Franklin, Tennessee, not far from Nashville. And we've been kind of officially a team for close to four years now. And we're really excited to come and share some of the things we've been learning about standing up some really cool automation that we think your listeners will really learn from.

Yeah. So the reason I'm called a unicorn is I joined Ramsey all the way back in 2015. So four years ago. Ended up having to move back home for some unforeseen circumstances. And then I rejoined the team in 2017 and that's just a rare thing to do. So I'm rare.

Very cool. So if I remember correctly, Greg, when you got to Ramsey Solutions, I guess you were starting from ground zero on what framework to use, I think you were like me, you came from a vendor based automation background, but I think you end up choosing open source. I'm just curious to know with you there and Max and maybe I don't know if Nick knows the story, how you chose basically WebDriver as the automation infrastructure that you're going to use for your current automation framework.

Yeah, Joe, you're right. I think our background's similar. I came from initially Mercury background to HP and and moved through QTP and uft, all that stuff that HP was selling at the time. That's been my background. When I got to Ramsey, they had been experimenting with Selenium webdriver and they had done a couple proof of concepts. So we started there. It was really my first introduction to using Selenium.

We walk before you run. I had some mentoring early on on just the basic premise of it. Boy, did I find that my background in HP tools helped me realize quickly that regardless of the tool set, there are some fundamentals of automation that applied all sorts of things that are the same across all the tools. And so we just applied those same principles here and we got some great wins. So I was able to take that knowledge and bring it in. That's how we got on Selenium. Then Max and I had a great conversation one day about let's open up the covers of this Selenium toolset and really understand it.

Awesome, Greg, I guess what are some of those principles then that you brought over from a vendor based tool that you notice applies to automation? Because when I talk to people a lot of times they look down on vendor tools and I'm like, it's just a tool. Selenium is just a tool. It's the same principles. If you mess up with your vendor based automation, you're going to have the same issues with Selenium. It's not a magic solution. So any of those principles, I guess those are some of the things we'll talk about in this episode, but at a high level. Anything that comes to mind?

Yeah. Probably the two most important ones I pass on to folks, regardless of tool set, is one, figure out how you're going to do your object location. Where we call them locators, you do them a certain way with the object repository. At least that's how we did it in qtp.

In Selenium, we build locators in our page objects, but we use exactly the same principles which are building strong locators. We can go into that maybe today or in another podcast. And the other one is our synchronization strategy. I'm just a fanatic about it. Nick's heard this a dozen times as we've been one of our new guys here as we brought him on board that these are the two fundamentals you've got to really understand is how do you identify something on a screen? How do you know when it exists, not exists, or how long should I wait for it to exist or wait for it to not exist? Those four synchronization methods are golden if you can really tackle those.

Well, I guess one of the first decisions folks have when they go to Selenium is because it's just an API, you need to actually develop up your whole framework from scratch. So what programming language did you choose? I think Max is a Ruby ninja. So did you choose Ruby because you had resources, already knew Ruby, or how did you go about choosing a language?

Yeah, the choice actually predated me, but it is because we had in house expertise in Ruby. So most of our apps when we made the selection in 2016. Yeah, so the rest of our company, we're predominantly Ruby on Rails and so we decided to go with Ruby just because that was the most in house expertise. We already had the tooling set up for local development.

For our Framework gym where all of our code lives. We have a pipeline set up with web systems already, so it was a lot easier to go with the flow by picking Ruby.

Curious to know Greg, maybe I'm wrong, but did you know Ruby when you chose the language? Because if you didn't then it's pretty brave to choose a language you weren't even familiar with for the framework.

Joe, that's exactly what it was like when I came in and found out we were going to roll with Ruby. And so no, I had no exposure to Ruby before and we just jumped in and decided to pick it up. It was awesome having some folks here that were willing to coach me through it and Max is really great at Ruby and he's teaching Nick and I and we're plugging away at it well.

Now I think so I guess. Max, any tips on I'm sure other people have the same exact issue. They choose a language for the framework and ultimately you have someone on your team that doesn't know that language. Any high level tips on how to get them up to speed on learning a language or learning the framework easily?

The main recommendation I have and we have like a compiled list of Ruby resources in house now, but most of them are geared towards Rails so there's not a ton of pure Ruby tutorials out there. And so just pairing with someone who knows the language really well is very helpful. So we probably Nick and I sat together most days for the first four or five weeks that you were actually working. And that's like the best way to get in, kind of feel out. It helps solidify whoever's teaching their understanding of the language too, when you have to stop and explain literally everything that you're doing and some of the complexities of, well, you know, JavaScript already, but in Ruby, this is how we do it. And these are the five different ways you can do it in Ruby, because there's always five different ways to do something in Ruby. And so just picking out like the basics of the language, but also the team style around the language. So these are the conventions that we follow when we write our Ruby code. This is how all of our UI tests look familiar. So we can change projects pretty easily. So just that pairing process is super helpful.

I guess the next issue is, I'll be honest, I chose Java for the framework for the company I was working in because everyone had Java background. But it turned out no one contributed to the framework, even though they were supposed to. So I regretted that decision because I probably would have chose something like Ruby or Python. So it sounds like, though, since 2015, no issues with this decision.

No, the framework has scaled really well as we continue to extend it out with the new capabilities and automation.

Probably the toughest thing for me working in Ruby is how many different ways you can do the same thing. That's probably the hardest part I've had with the language. There's a lot of shortcuts and alternative ways of getting things done. We tried to standardize on certain approaches and things, and Max has been great about bringing us up to speed if he's learned a new approach and we'll integrate that new approach. It could be working with.

A dozen different types of things, but it's been really helpful.

The process of opening up PRs against our own SDK because it is its own repo, it is up in GitHub with all of our other source code around the company. We actually get other people coming in to inspect our Ruby code. So maybe more senior Ruby developers, more people that are from the Rails background speaking into our code who kind of know about framework development a little bit more. That's been really helpful too.

It sounds like you guys have two of the hottest things in place, already decided on what automation technology to use and also what language. And then we get to what we talked to earlier, the things that affect every automation framework, regardless of what technology or what language. And that is the first one is testability and the Second one is synchronization. So testability. Greg, I think you said something. You had some sort of locator strategy you put into place to help you with this issue.

Yeah, I think when we originally started to figure out how we were going to do locators, I started thinking about the things I learned in QTP using the Object Spy and its Object repository. And I realized even then that finding two to three parameters was kind of the sweet spot for finding great locators. I have kind of this philosophy that locators should be very precise so that if they don't exist, you don't want this test to keep fumbling its way down the road. It's better that it stop running and you figure out you have some maintenance to do. Once we kind of settled on that approach, now we needed a programmatic way to build locators that were testable. And that's when I really started to dig into XPath, which really has a bad rap online. A lot of folks, their exposure to XPath has been from record and playback options, whether in Selenium or other types of tools, where it tries to take its best guess at building locators. But the locators are nuts that they generally come up with when they just kind of try to figure it out on their own. What's nice about using the XPath approach is we can go into Chrome's dev tools and actually test the locators. And I think that's a little bit of our secret sauce, is that all locators are testable and that they live in the page object. So for each page, we have a page object and they live right there. So we know when we're working on the login page that we just simply bring up that Ruby file, the page object file for it, and all the locators that interact with those objects are right there. We can pull them right out and bring them into Chrome dev tools and test them. I'm telling you, that's been a huge benefit to our approach.

So, Greg, can you just give an example for folks that aren't familiar with that approach? How are you able to test it within Google Developer, Google Chrome Developer Tools?

I'm going to let Max answer that.

Yeah, it's actually built in and it's kind of subtle. So if you do. If you're in dev tools, you pop it open, you usually end up on the Elements tab. If you do a command F control F, I think on Windows, inside of that Elements pane, it'll pop up a little Find box And inside there, if you actually paste an XPath, it'll match all of the elements on that page that are queried by that XPath. So you can kind of see, oh, this is matching three elements and I actually only wanted it to match one. So I need to more fine tune this XPath. You can also find and replace just raw text like that. You can do CSS class location that way. So there's a ton of functionality packed into that one little input field in Chrome.

What's nice also is Chrome by right clicking on an object will allow you to inspect it. So it gets you into dev tools. When you do that, you can see the source at that point. So you can look at things like hrefs and div tags and other things that you might need to use to build a strong locator. So it kind of gets you headed in the right direction that way.

And so Greg, if I'm understanding this correctly, you're creating like a compound statement that says locate an element that has multiple attributes.

Yeah, Joe, that's correct. Sometimes we can find great locators like div tags that might have an ID attached to them. This is actually part of what our team is doing. When we meet with our dev teams and we talk about testability, we coach them towards building in testability into the application, which generally we coach them towards ID tags and things like that. Sometimes we have to use a little more compound where we might use two or three attributes in order to identify something. We have our best case and our worst case scenarios of what we like to do. We tend to stay away from anything related to class that's I think, commonly used online. But class always ties generally to look and feel because it's connected oftentimes with css. And so we stay away from anything that we can that's related to the look and feel because that's the most dynamic part of the application and we stick with more functional things that are going to hang around a little bit.

More of why we've standardized on just XPaths when we could use multiple strategies inside of Selenium, because Selenium supports CSS ID name if you want to. With XPath, Greg mentioned the continuum of all the different types of locators you might have to build. You can build all of Those inside of XPath and if your team standardizes on that, you're always speaking the same language. But if you're switching up between using a CSS style locator, or if you have to use an ID or have to use A name. So you have to get your dev team involved to put those in the ui. That can lengthen the process and it's a little bit harder to keep multiple concerns in your head. So by using XPath we can just say this will catch everything. We have our idea of what a phenomenal XPath looks like with only two, two or three predicates. And then we have some hairy, ugly XPaths, especially when we're automating React or a single page app. But we still have the whole gamut covered there. There's never been something we could not select using an XPath.

I could just see people cringing though, because a lot of people would do this wrong. I think they would hard code the XPath within their scripts and if anything changed, they'd have to locate it in all their scripts to see what changed. How do you. How do you get over that with this approach?

Yeah, I think there's a myth that if you use XPath it's going to change every night, and it really doesn't. If you're automating something and it's relatively stable, you're probably going to bind to what the tag is and then some property of the tag. So hopefully there's an ID on there or a data attribute. We started to move ToWards data test IDs here, which are really helpful. But the idea that the button is going to change every single night usually doesn't happen. And if it does happen, that's an allow window of maintenance to go in there and update it to the new style. And also if you are building your XPath correctly when it is changing, that's indicating that the application is actually changing as well. So it might be time to revisit what your test is testing at that point.

Awesome. I think it's just great to clarify. Also, this is not a Ruby specific technique, this is XPath. So any language binding should be able to use the same approach, correct?

Yep. Yeah, it should. It's a type of locator you can pass to the Selenium API and all of the syntax for that is completely agnostic to what language you're sending it from. So it's almost like a little mini SQL if you will, but just for DOM elements.

I love this approach. So this seems like it helps make application more testable. Also makes it so when you have developers that don't add unique IDs, you can kind of create a combination of attributes that can make it unique enough to find it without a lot of overhead. It sounds like. So the Next issue then is synchronization. So how did you guys get over the synchronization of. You know, sometimes people hard code sleeps or they do weird waits where it's not even waiting for the right elements. Be up here in the dom. And so it's like randomly passes or fails.

Yeah, I think for those automation engineers that are listening, I think that this is when you really want to get into the art of what we do. This is a fundamental. You've got to get your arms around, which is if you're using any type of wait or sleep in your synchronization approach, you need to revisit that. That should be your absolute last result. I think out of all of the tests we have in place, we may have one or two sleeps that I can think of that might be anywhere. And we usually only need it if we're debugging something. We definitely don't want it to persist. The way we get around that is we have some methods that we built early on based around these four synchronization strategies. And I've shared them on the show, I think before, but I'm going to go over them again. You should have a synchronization method that you can pass a locator into. Let's say that we want to determine if a button exists. So I pass in a locator and I would say, does this exist right now? So the method's whole purpose is determining at this exact moment in time, does this thing exist? Then it doesn't wait around for that to do it. It just does that. Now we also have another one which is the opposite of that, which is at this moment, does this button not exist. So it's our not exist synchronization. Once we built those, and those were real solid, those were fairly easy to build, became the next two that are a little harder to build. The next one was wait for this button to exist. So it's a way to exist in this method. It's going to look for a locator and it has a certain allowable time for it to keep polling to determine if that locator exists. If it doesn't, it comes back and says, hey, you said I could wait up to 15 seconds to look for this thing. I've continued to pull it at once a second to see if it's there once half a second. That's actually a parameter we pass in.

It has not existed in the time you've given me to do it. Then the wait to not exist is just the opposite of that, so we expect maybe a dialog box to be up on the screen and we will say wait for the dialog box to not exist. So in other words, we're expecting it to go away and it uses a very similar logic. Now, those two last synchronization approaches are a little harder to engineer, but they are so tight right now that our synchronization approach just works fantastic. And we really don't have any flaky test problems due to synchronization at all.

A little bit more towards the framework design. With synchronization, we've made it extremely difficult not to synchronize our UI tests when you're using our framework. So we do all of the synchronization under the hood for you. So if you say I want to click this element, we also have logic in there to make sure the element is indeed present before you try to click it. And we have a lot of built in wait up to 20 seconds. But if you find it early, bail out early and keep going. So our synchronization, unless you need to explicitly use it, is just baked in everywhere. And that's really helped us not have to think about it all the time and think more about what is the actual test and how do I design the test.

Awesome. So are there any Ruby specific libraries you use to help you with this or are you just straight up coding it from scratch without leveraging any existing code or libraries?

We use a little bit of what's built into Selenium's waiting, but. And this could change in selenium 3. I know they've been working on this, but the current version of Selenium, the weight is not great. Like I haven't had great success using the Wait API or anything. So most of our synchronization is like a pass through layer in our framework before we actually get to Selenium. So we query Selenium multiple times, make sure the element's present, but then when you're using one of our page object abstractions, you're not actually like hard coding any of that. That's just happening for you behind the scenes.

What I see is the common causes of test failing is the testability and the synchronization. Another issue I have is when things do fail, you need to be able to triage really quickly. And so a lot of times for some reason, people fail to really invest time in reporting. So have you done anything special with reporting that helps you troubleshoot or give you really good reports to hand out to your team?

Yeah. So we're let me. I think I need to give a little bit of backstory real quick before we get into what we're doing for reporting. So we're using a unit testing like a test runner essentially called RSpec. There's pretty much two in the Ruby community. There's RSpec and there's Minitest. We just opted for RSpec because again, that's what our company used. And RSpec has this concept of a formatter and all. A formatter is fancy name for prints report somewhere. So it's lifecycle hooks. During while your test is running, different things happen that are interesting. While your tests run, we can pull information out of those different lifecycle hooks and then put them somewhere else. So the default email report for RSPEC does not look great. And so we had the additional concern of okay, well all of our test engineers, our manual testers, their consuming most of the information via reports, but we're sending this HTML page to an Outlook inbox and Outlook is not rendering it correctly. It's very much a unit testing runner first. So some of the UI testing concerns that we're wanting to surface don't really come across that way. So it's fairly easy to build your own custom formatter and that's what we did. So we build our own HTML reports. We can completely control all of the HTML, control all of the data when it comes out of the test run and make it so that when they're looking at the report they see the failure and why first and then all of the summary underneath that. And we also provide some reference about, hey, we used these values while we were doing some other testing inside of your or during your test run. So we're using that custom RSpec behavior to get it to the HTML reports. And we're also outputting every single test run anywhere between like 5 or 5, yeah, 500. Reporting like events to this thing called Kibana or it's an elasticsearch instance. So we're actually emitting data and we can go back and query that data historically using a tool called Kibana, which allows us to see from like run to run, how is the test run, how is the duration shifting? Or do we have 12 failures this morning and now they're all cleared up. And Greg's been working on a lot of dashboarding around that so our team can kind of get an overall sense of the health of all of our applications here.

Awesome. You still. My next question, which was I know Greg has talked in the past about the Elk Stack and Kibana with his dashboard and how it integrated. So it sounds like it is integrated there. I'll have a link in the show notes to that article. I think Greg actually wrote for the folks at Elk Stack about Kibana. So good stuff there. Anything you learned along the way while creating this framework? I know it must have taken you time or to get the synchronization right over time to really hone it in, to get it perfect. Like, is there anything you've seen, maybe Ruby specific, that you were doing incorrectly and that by modifying it, it made it easier so that it became more reliable?

It's not really Ruby specific, but I think a good thing to keep in mind, like when your team decides, okay, we want to do open source, we don't want to use an existing framework, we kind of want to roll our own. We didn't just start with a framework, right. We had one project and we were just writing UI tests for it. So along the way there, inside of that one project, we kept identifying pieces of code that we kept reusing over and over again. So if we were making sure we were on the right page, there's a concept there called Validate Locator exists. Does this thing exist currently? Wherever the driver's at, we were just using that over and over again inside of the same project. It's like, okay, well, let's go ahead, let's take the 30 minutes it'll take to create another file, put some of that logic up over there, and let's just require that file back into our test. We did that and then that worked really well. And we kept doing that process inside of that one project until we had a folder that just had a bunch of. In Ruby, they're called modules. Other languages, it's just this idea of a namespace, so a little container of functionality, essentially. So we had quite a few of those. And in Ruby, the way that we package up dependencies is something called gems. And we started looking at that and we said, if this was a gem, this would be really easy to lift and put into another project. So if the first application we automated, all of that code lived locally inside of the repo with that project. If we moved it into a gym, we could just have the gym be required into that same project. All the source code gets loaded the same way, but then this other project can use it. So it's a process of testing it out within one project until it's really tight. Feels really good. The API makes a lot of sense. And then lifting that code into Like a library, essentially, and figuring out how that looks and then putting it into the framework.

Nice. So I guess another question folks usually have is how do you test your testing framework? Sometimes people write code that can test, that can run with people checking code to make sure they didn't break anything, do anything like that. Like, I assume you have more than one person contributing to this framework. How do you test that? Any changes made to your testing framework, it doesn't break it, sure.

Yeah, it's surprisingly hard to do.

It's kind of towards that lifting model of testing out code inside of one project first and then moving it up into the framework. We have a hard fence there that if any code that gets promoted into the framework to be used by all of our other projects, all of our other UI testing projects, it has to have unit tests and it has to have some sort of coverage for that.

It can get kind of tricky. So we use RSpec to run our UI tests and we do a bunch of custom bootstrapping there in order to load our framework when any of our tests start up inside the framework itself. We're also using RSpec as the more traditional, just a unit testing framework. We have unit tests that are written in a certain paradigm, and then we also have UI tests that are written in a slightly different paradigm in different projects. So that's kind of the. We have the freedom when we're testing out code to try out a bunch of different stuff. Don't necessarily need to write unit tests for that, but before it is packaged up and released to everyone else, there has to be some sort of unit testing around there. And most of the time, if you understand what you're extracting into the framework. So if we're changing how we're going to do an email report and know that we need to add a new section, if you can get that down to a pretty small class that's really testable, it's pretty trivial to write a unit test for it. It probably takes another 10 minutes or so to pass in a couple of different variations of input and verify the output. But it saves a ton of time down the road. And that's just unit testing in general. Where it gets a little bit trickier is when you're having to simulate integration tests inside of your framework. So what if we were in a UI test run and this particular bit of functionality needs to send off this data and it needs to look like this? So we do have a lot of custom construction inside of our unit tests in order to act like that? We are running a UI test, but that's just kind of a trial and error, right? You start using the class, realize you need it to be in a certain state, and you start adding some helper tester, test helpers to drive your UI test into that state.

So how formalized is this? I know a lot of people will have code reviews and then once you have code reviews, they actually have a pipeline where developers will check in their code, the unit test will run, there'll be like a certain percentage of code coverage, and only then if that passes like 90%, is it allowed to go into what they call the imp builder.

Do you have a formalized process for your framework like this as well? Is it more like the honor system or the shame system where if you break it, then everyone's going to know you broke it?

It's only the shame system when we're developing, because that's when it breaks all the time. We're really fortunate at Ramsey to have a world class DevOps team that is really focused on getting source code to the place where it needs to be running. So we use a service called GemFury, which if your listeners don't know, it's just a package repository manager. So it can host RubyGems, it can host PHP packages, I think it can host maven dependencies. It's just a place to put the library essentially inside of our source code. We use a docker file which packages up all of our Ruby, installs all the dependencies, runs all the unit tests. And if each one of those tests succeeds, then our infrastructure that our DevOps team has set up for us takes that built RubyGem and just puts it up on GemFury for us. We're following semantic versioning, so if it's the same version, then it doesn't replace whatever is already out there. But if we bump the patch version or the minor version, it's going to be the next available version of the gym. And the UI testing projects are going to just try to go to the next patch version whenever they can. It's worked out really well. I mean, we have a pipeline where we can see all those different tasks and if the unit tests fail, we can go and see why they failed on the build server and figure out why we weren't seeing that locally. And the DevOps team is four or five rows of desks down from us, so we can just go and hang out with them. And I realize most people aren't in that situation, but it's helped out a lot and we haven't had to rely on the honor system because we can rely on the build, pass or failing.

Okay, Greg and Max, before we go, is there one piece of actionable advice you can give someone to help improve their automation framework creation efforts? And what's the best way to find or contact her?

One more tip I guess that we've kind of learned is through the process of creating the framework with just two people, we had the way that we knew how to do it. But as we've been onboarding Nick and seeing kind of the darker parts of the framework, or he starts to use it and he says why do I have to pass this in? Or why does this work kind of this way? It's really eye opening to see how a third individual who has no background who is trying to use your framework because we're building a framework to test faster. So what are the barriers for him when he's trying to use the framework that aren't enabling him to test faster? That's really challenging. Some of the just static APIs that have been in place since 2016. Oh, that maybe is a weird way to do that. And maybe we need to go and reinvestigate why we're expecting the user of the framework to call something like that. So just having someone look at your interface to your framework is super valuable. And since we own all the code, we can change it pretty easily. And so getting the interface to the most valuable spot for everyone, where everyone knows how to use it, where it makes sense, where we're offloading a bunch of the manual labor you used to have to do, that's a really good goal to strive for. And that's something we've been trying to put more and more into our framework.

Joe, I got one last piece too. I think for those that are fortunate to not be a one man show and they've got a couple folks in their automation team, I think one of the pieces I'd like to leave them with as well is begin to identify the different skills and talents each of those individuals bring to the table. I think we're discovering that more and more as a team of three now and we hope to grow here, maybe add a couple more guys down the road, a couple more folks. But we started to recognize that we're all holding down different parts of the kind of the success of this framework going forward.

Some of us are thinking about the customers that are using it or how do we make it more extensible down the road, or how do we maintain this better? That's really added to it. I don't think when we started in 2016 it was just me to begin with. There's no way I would have even been able to identify that. But as we grow, we're starting to notice that more and more and it's been really helpful.

Yeah. So to contact me, probably the easiest way now is if you want to Find me on LinkedIn. My last name is super unique, so if you just look for maxwelldink, I'll pop up there. And then if you want to check out my GitHub page, it's GitHub.com maxveldinc.

And folks can find me pretty easy as well on LinkedIn at Greg Pascal or find my website realworldtestautomation.com and you'll find a lot of blog entries and work that I've done and so it'd be great to connect with you all. Love to do that.

Thank you Greg and Max for your automation awesomeness. Thanks to everything of value we covered in this episode. Head on over to testguild.com alicautomation a278 and while you're there, make sure to click on the Try it for free Today link under the exclusive Site Sponsor section to learn all about Source Labs awesome products and services. And just so you know, I'm about to open up registration For Automation Guild 2020, the only online conference in the world dedicated 100% to helping you succeed with creating Automation Awesomeness. We have a great lineup. If you haven't seen it yet, just head on over to GuildConferences.com forward/automation. It's a must attend conference, all online. No travel required for you to start off the new decade. Right, let's end Flaky test in the new year. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full Stack Automation awesomeness. As always, test everything and keep to good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.