Building a Software Quality Process from the Ground Up with Evan Niedojadlo

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey it's Joe and welcome to another episode of the Test Guild Automation Podcast. And today we'll be talking with Evan all about how to build a software quality process from the ground up. He's done it at a former organization and he's been very successful. So I'm really excited to get his views on how to do this correctly or to share some tips that can help you with your journey as well. If you don't know, Evan is an engineer that currently focuses on reliability, distributed systems and infrastructure and has some security sprinkled on top as well. So he's really a well rounded tester. He also is the co organizer of the Austin Automation Professionals and the Austin Computer Book Club. Really good organizations. If you're in Austin, you definitely need to check that out. And he likes to provide a welcome space to all folks to learn and in automation. So really exciting. I'm on the show, you don't want to miss it, check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out. Hey Evan, welcome back to the Guild.

Hey Joe, it's good to be back as always.

Cool. It's awesome to have you on the show. I failed to mention a YouTube channel. Let's give a quick plug to it. I really like what you're doing there. Just tell us a little bit more about it.

Oh yeah, that sounds good. Yeah. So it's random thoughts tech. So right now I'm kind of focused on site reliability engineering, kind of sharing my thoughts about how, you know, I've built out this process and just kind of my day to day with what I'm doing at work. Just basically just for anyone that's starting out with this, just kind of a resource. It's like SRE for the rest of us outside of Google. So that's kind of what I'm focused on. But eventually I'll turn it into some more automation specific stuff. So whether that's testing or I'll definitely.

Have a link to that in the show notes, it's a must. Subscribe to channel for sure. So that brings us to an interesting point. Evan, you were on our performance. I don't know how many people know I actually have a performance Insight reliability podcast, but I do. We had an interview there there recently and it's curious how you really have a strong background in automation. Automation testing, and yet you moved over to an SRE role. So before we jump into the meat of this topic, just for the folks that may have missed that episode, why move from automation testing to sre? What was the motive behind that?

The main motivation for myself was becoming more of an interest in Linux and really just kind of like the system size of things. So I guess that's a quick answer. I mean, I was like already doing some, a little bit of DevOps ish work at my organization. So it was just this trajectory that I wanted to go down and then I was given the opportunity when a new project was being created. So I was kind of plucked from what I was doing. I was like doing test management at the time. I was blocked into kind of like the individual contributor SRE role. And it was a massive challenge for myself because there's a lot to learn there. But I did it and I tried to share everything that I've learned just for anyone else who's interested in doing it.

Very nice. And in that previous episode we talked about how you actually built the SRE process from the ground up. And I thought we would flip it around now and just aim it towards building a software quality process from the ground up, which I believe you did at your previous company. So I guess what is the first step if someone is struggling with how do I build a quality process from scratch?

Scratch, yeah. So some context. This was a while ago now. So this was like. Wanted to share this because once again, if anyone was found themselves in my situation, I want them to kind of like have a. Have a guideline of as to what to do or actually like what I ended up doing. So the context there is this was like 2015, 2014ish. I was already doing some testing, so I was like a QA analyst, I believe the title was at the time. And I wanted. I guess I ended up moving down with a company to Austin. And with that move I just ended up seeing a lot of organizations doing some amazing things. So whether that was like locally or whether that was on Twitter, whatever it might have been, what I ended up doing was like taking those ideas and then just kind of apply. Well, applying them ultimately. But then initially just getting that buy in, part of that buy in was. Was really convincing leadership. And this was executive leadership. Really like Asking them the question if they wanted to ship more frequently. Because we really had a bottleneck with just the overall quality process where there was just a wait time. There was this period of time where like, we would just like, you know, qa. Qa, QA or testing. And that was kind of an initiative for myself to learn more about it ultimately, like, so it was just like this career initiative to learn more about it, but then at the same time, like the business benefits as well. So I guess that's like the deep, deep context. So a lot of this information I'm kind of like pulling from the past, quite literally.

So nice. But it's kind of similar to what you did with the SRE process. You seem very not afraid to approach executive management and give you a point of view of where you think could help. So is that the same thing? Like, how do you get up? I don't know, the courage of a wallflower and an introvert. So how do you know, how do you approach your executive team to get involved in something, an initiative you really believe in?

Yeah, so the big thing there was this is a small company. That's like the main thing. So like, my background, if you listen to the SRA podcast, as well as something I mentioned, my. My background is I think my entire career has been startups for three various different companies. And arguably, I think over this time period, I think I've made a past seven years of my career off of a technical debt to some degree. So just kind of funny, but that's really the main thing. Like I gu. Size is a big deal, so that's a major, major factor. So like, oftentimes you're at some, like, large organization, you'll probably be communicating with like your manager or like their manager or whatever it might be to voice those opinions. Could be in various ways. You know, you could either set up a meeting, get the courage to set up a meeting, or just kind of like inject some of these ideas within Slack. Like we have, for example, like we had, like we still do, but we have that quality, like a quality Slack channel as well as sre. But like in those channels, I'm oftentimes like posting ideas or just like thoughts and just kind of turning into more of a communicative process. So that helps. I think it's big thing. But yeah, so that's an additional context there.

So you did mention when you were there, they did have automation engineers, testers, quality engineers already. But there was a. There was a lack in, I guess a process in place for automation or testing. Is that correct? Was it just unlike, okay, the software is done, go test it before you were involved?

Yeah, absolutely. That, I mean we were, the shop at the time was just entirely based on exploratory manual testing, whatever. You know, that's a term that's kind of thrown around these days. But we were just quite literally doing testing. And there was, once again, there was just like this, this, the area that I moved to was just kind of like so ahead of what we were doing. And I ended up, I ended up like finding this out through meetups. So like meetup.com, that's kind of how I wanted to create my own meetup eventually with some other folks. But yeah, just understanding the community, understanding what they were doing. Because what we were doing was just this long process where there's so much possibility for us to just innovate on that and you know, not only innovate on it, help the business, but once again help out people's careers. And there's just lot to it. And I think the main thing there was absolutely, it was just that, that barrier, you know, there's the production barrier that we were doing. And in hindsight there is a couple ways to address it. But we, or myself, I ended up, ended up doing it by just kind of selling this idea of automation, you know, just selling the idea of, of, you know, we even had delivery pipelines in place, but there were like no tests. And then there was just a lot going on. There was like a lot of, a lot of like technical debt going on and just some strangeness around like how we tested and how we deployed and a lot of these kinds of things and just really like being the only test. The organization that, you know, we, we did, we had contractors who were testers as well, but like nobody was really pushing that, that automation approach. And I think it was a good time to, to get that buy in as well. So.

Yeah, and so I guess another interesting thing about you is I think a lot of times people say, oh, I'm too young or I, I don't have a PhD in software development or testing or whatever, but it seems like, I think you read in my notes here that you taught yourself to program. So did you teach yourself to program so you can implement test automation? What was the process there?

Yeah, yeah. Context there is. Yeah. I love school early. So I was like 19, got, you know, got into the industry back in the day. So I was doing some fun stuff then, but you know, so 19, 20, 21, so like moving on into My career, the need for like some. I was doing some programming at 19 because I was working at a consultant shop and we were doing some C. So I was like working on existing applications. So C. And hate to say this, VB. Net at the time, this is like 2008. So fast forward getting this quality role that I ended up, this tester role that I ended up getting, there was more of a need for the automation that I saw. So like I did teach myself how to write code and then I think a lot of the community helped me teach myself as well. So I think it was like this community effort of like all these people were doing this and then I ended up working through a lot of the books. So like shout out to Dave Hefner's book, the Selenium Guidebook was extremely helpful for what we were doing. Thankfully, my organization was like, you know, I approached them about covering this code bootcamp, so I wanted to get in depth. I wanted to kind of like put myself in a schedule of really enforcing not only like learning how to code scripting and whatnot, but like to understand, understand like at the time, just the full stack development approach. So at the time I was doing, I did a bootcamp on Ruby on Rails. So the purpose behind that was to understand Ruby in the RubyGem for selenium. So I wanted to go down that route. And it was, it was a lot of, it was great because I ended up learning that and Capybara and RSPEC and working with that, which was a lot of fun. And then the downside was that when I got the buy in for all of this stuff, the team that I was working with was like, hey, we're, we're, you know, this is a contour team, of course, and they're like, hey, we're more familiar with Python. Let's, let's go down that route. So I was like, all right, well, so I did that and then I ended up teaching myself off of that Python and then we ended up utilizing pytest and a whole bunch of. Once again, Dave Hefner's book helped out quite a bit because I was able to use that knowledge that I had previously and then just kind of like kind of run through that book and really understand, you know, what to do with Python and how to, how to approach this, how to build out the whole test suite itself. So.

But yeah, so interesting. I think a lot of times people, what they do is they choose the, the programming language that their team is most familiar with and it could go either way. Either you end up not knowing the programming language and your team does and then your team doesn't contribute and then you have to teach yourself the language and it's just you running it or in a more functional company or more holistic company, you choose the language everyone uses and everyone contributes. So what was your experience? Which way since you, you did change based on what everyone else was using?

Yeah, no, that's, that's exactly it. So it was a small team and that exactly. Like a lot of the stuff that I'm reading about now and like I'm not in depth in like a lot of the quality conversations these days. I'm more like such as more on the op accessory side. And like in hindsight it's like one of those things where like you want the whole team approach and that's what I've been doing a lot at work lately. Just getting like self service in place and just like all of this automation to try and like make everything accessible versus like going through this particular team. But yeah, in hindsight, totally it's one of those scenarios where like I probably should have done C NET at the time, you know, this was before Cypress was a thing and you know a bunch of like. And I have story about that in a bit but like I think the quick answer is absolutely like we had the whole quality team participating which was, which was quite nice because we all ended up contributing and we all did amazing things with this test suite and it's actually still running in production to this day. Whether that's good or bad, I know that's being looked at now but they're being revised in some ways. But we did some amazing things. We took that and built a bunch of things off of it. But like yeah, I totally agree. And in fact my recent suggestions for the quality team was like, you know, go down like utilize like the same language as like the front end. So like you know, we can get like the front end developers like contributing to this as well. So like it's a whole, we just get some fast, you know, some fast contribution in this. I totally agree with you on that whole team approach because we did the whole team approach with the quality perspective. But it really should have been like everyone like it should have been engineering as a whole doing these things.

Absolutely. So I guess the next question is you probably had a bunch of existing tests already. How do you know which test to begin with? When you're starting with that like the team's starting from scratch, you have a bunch of tests already. Okay, what do we begin with that really can kick this off, right to get really buy in because we see early, early gains right off the bat.

Yeah, that was, that was like tough. So I thought about a couple possibilities and the one that made sense to me to kind of push internally was the critical items of the website. So like in our case I, I, I entirely focused on that because we were quite literally like there, we had, we're, I think we had, I had us using like Zephyr and I think there's the team still using Zephyr now, but I was using Zephyr and I think it was like between them and like Testrail or something like that. But we, we tried out both and Zephyr was like nice because it integrated with Jira like heavily using Atlassian. And it was just one of those things where like, and I found it and saw it and we were looking at a bunch of stuff before that too. We were looking at like BDD and whole different, you know, various sides of approaches. But like because we started off with Zephyr we had a bunch of test cases in place. I know that's a word people might laugh at these days as well. But like we had, we had a whole bunch of these and we time boxed our tests and so it kind of evolved and like you know we had this regression suite and then identified the business critical test. Like what made sense in our case like the business at the time it was getting an offer in a vehicle. So I focused all of our efforts initially just on that first like maybe five to ten end to end like happy path tests, checks, whatever you want to call it. So like I wanted to really, really emphasize that and then meeting with once leadership and showing that off and showing it how it worked and, and then that was just doing it locally and then that kind of like led up to the buy in on Sauce labs and that whole thing. But yeah, that, that was how I, how I did it and then that was how I sold it internally was like hey, here's your business critical items that are making the business money. This is making the business, is paying our salaries and making the business go forward. So it helped out a lot to do that and then we continue forward off of that.

Right. I guess another thing I think team struggle with is they do almost waterfall efforts when they're starting out. Let's automate everything, then let's try to get in cicd then let's plan what we do in production. So what was your approach? Do you say let's take one test run it through what I think the pipeline should be and then let's iterate on that based on feedback that we get from each test.

Yeah, I think it was an approach where we were doing this in Sprints, so we were breaking this up per Sprint. So print. And this was a two week sprints and we're still doing two experience, although at a time we're doing three weeks, but that's another story for another day. But what we ended up doing was writing the test in the beginning of the Sprint. So I don't even think shift loss was really a term then. I don't. I know it's one of the, once again one of those wild things to think about, but we wanted to do it in the beginning of the Sprint and we wanted to add it before really before we released. So we were kind of like doing this hybrid approach where we were doing tests in the beginning. So we're trying to run it as much as possible and try to get that feedback and see if there's anything breaking. But ultimately at the end of the day what we ended up doing was kind of.

Applying those tests to our pipeline. So in our case it was TeamCity. So at the time, we're using a bunch of different stuff now, but at the time it was TeamCity and we would do nightly tests and then we would add them to a nightly pipeline and then we would run those each night and see if like we would really look for if the test was solid as well. Like we wanted to ensure like this test was, was passing. Like we wanted to make sure that if we were going to run this in production or staging or whatever, like we, the feedback that we get, we want everyone on the team to be reassured that, you know, it's not just noise. So that was, that was a big thing as well for us. But yeah, ultimately like all of these ended up in TeamCity. It was per. We would define them per service and within pytest we would use markers and stuff like that. So we would kind of like would have this, this large, very large test suite and then divide it up per service. So we were doing like this per service test approach, once again running them nightly, every night. And we would actually integrate it within Slack. So we did like, I guess, I don't know, you don't really hear this term thrown around a lot. But like ChatOps, we would do this ChatOps approach where like we integrated our test suite to Slack and it would tell us what failed and what didn't and then it would produce the link to Sauce Labs, which we ended up ultimately getting buy in for. So it was like this large effort to get all that buy in. But we did end up on Sauce Labs at the time. There really wasn't a whole lot for us to do as far as we could run it locally in like a Mac Mini or something. But it didn't make much sense, like manage all of that and like our own local selenium grid and whatnot. So Sauce Labs at the time was. It was the best approach and we're still using them.

But yeah, nice. So that must get more people involved then with the Slack integration. So your developers then get alerted and then they click on a link and automatically go to the results that were stored in Sauce is what it sounds like.

Yeah, yeah. And it helped out a lot. And that was. I don't, I don't really see a lot of teams doing that. And it scaled fairly well. Like we're actually. If I go in Slack right now, I can actually still see it. So I can still see all that that we built out, which is cool.

Cool.

It might change and whatnot. But yeah, very, very helpful. And we. And also once again we can go in there and identify if we see the same test failing over and over, delete the test, you know, unless. Or refactor it, depending on the kind of effort and the criticality of that test. So.

So is there anything you learned that you had to develop your tests more reliably or how to make your test less flaky based on another thing I see sometimes people develop tests and they never run them in parallel or they don't run it in a cloud provider like Source Labs until after they have a bunch of tests and they have to kind of tweak things. Any lessons learned from that particular area?

Yeah, it was a pain sometimes. Initially it was kind of a pain because we were so used to running tests locally. Just ran really well. So then you add Sauce Labs to the mix or browser stack or, you know, whatever. And then what ended up happening was we found out that there's latency and like that's what kind of got me to learn more a bit about networking. And then we had to add weights in within that test as well. And that was probably the biggest thing. Like we had to understand that some of these tests were failing and you'd kind of review the video within Sauce Labs and then you'd identify why it would fail. And sometimes it was mysterious and sometimes we'd have to have tickets open. Been with Sauce Labs and amazing support team over there, by the way. Yeah, so they helped us a lot on that. But yeah, it was just a big change between getting really, really, really like I guess ahead of ourselves and not understanding like the, the network side of things. Like when we ran this on someone else's cloud, it would be, there'd be problems and we'd have to address those problems by essentially at the time it was just adding weights and we would use parallelization as well. So we would run all of our tests in parallel, which helped just in the speed so the timing on how quickly everything ran.

But you know, when I started my career I was a solo automation engine. I was the king of the realm and I found I was much more successful that way. But then when you start having more teams, more people involved, it gets very, really difficult. So how did you educate your team? I think you told me you actually flew to India or something like that. So maybe a little learning about. I guess an aspect people miss out on is how to educate their teams or training how to get the team up to speed with training.

Oh yeah, for sure, yeah. So the team that I was working with was based in Ahmedabad, so specifically Gujarat. And I flew there, worked with them for a while. Of course I've been working with these all friends over there, lots of friends and just working with them for a while. And we ended up working together and you know, utilizing Git and GitHub which was a big deal for us, just the social coding aspect and then eventually ended up flying to, to Gujarat and yeah, it was, it was a lot of fun because you know, there was just this idea of, you know, seeing everyone in person and meeting everybody. It was really, really great. And I know a lot of us are remote now, so it's like this, you know, so I was doing remote I guess in some sense for a little bit before everything went down but. But yeah, it's just meeting everyone was nice and really the training, the in depth training was just really getting one up to speed on why we're doing this. I think, I think more so is like the why and then it would came down to the specifics of Pytest and which plugins we could use and the plugin architecture and which books we were using and there is lots of that and shout out to Brian Ockin and the Pytest book, which helped us a lot as well.

A great book.

Yeah, yeah, yeah. And I did that twice. I actually flew out there and trained and worked with this team twice now. One was more on the quality side, the other Time was kind of like on some of the security stuff I wanted to do towards the end, which we wanted to do like the. I guess a lot of people call it DevSecOps now, but we were kind of like embedding some OWASP stuff in our pytest suite a long time ago. But it was. Which was a lot of fun. But yeah, yeah, ultimately the, the training thing was. It was I guess a buy in. There's a lot to it. There's just so much. So we could do an entire episode on that.

But yeah, yeah, yeah, no doubt. How do you know if you were being successful? I know a lot of times people come up with really silly metrics. Like 80% of tests will be automated. Did you have any tips or what you used for metrics to make sure you, you were on track with the. How the health of your automation project was?

Ye. I know I have a lot of people don't like this one, but for a little while and this was before I read more into it, but we were tracking like bugs that were caught with automation, which is not a good metric. I mean some people use it and whatnot, but it was just. Yeah, we started off with just some. That was like a big challenge for myself. It was like understanding like that was the community aspect of things. Kind of helped out a lot there because I would ask friends of mine who worked at like some really established organizations and I was like, what are you using for. For these kind of metrics? And what ended up happening was, you know, I guess coverage was. Was a thing and it kind of like, all right, so coverage. I'm trying. Oh, once again pulling all of this from, from like many, many years ago. I was. How many tests did we develop that Sprint, like how. And then like I guess we had a metric on like how many, you know, automated that we had. Once again, probably not. A really, really great metric is kind of a silly metric in hindsight. Once again, bugs was kind of the thing that we followed. Release time was more of like the more matured metric that we were. We were starting to use was like how quickly can we get from like staging to productions? We started tracking that as well. That was actually like a good metric that helped us. So if anyone's listening to that, to this, like that did team quite a bit because ultimately at the end of the day, like the company just. The business just wants the product and depending on the business, but most of them just want the product in production. That's how they're making their money. And then We, I'm trying to think of other ones that we had. I think it was like test failures like we were looking at. We would do metric on, on automated test fails and then we would, would use that as a metric to go back and kind of like do a retro of like why did it fail? And that kind of thing. So that was very, very helpful at the time as well. It's just a few metrics, I mean some that are helpful and some not as much.

So talking about, you know, what was helpful, what wasn't. Now in retrospect, is there anything else you learned what you would have done differently now if for some reason someone said, okay, you're not an SRE anymore, you're going back to that team, you're going to revive it up the way you started it, what would you do differently?

Yeah, that's a good one. Right now I kind of have some communication with all and I work with the teams right now because I pull them into some chaos engineering tests and all that. So it's a lot of fun. So yeah, I mean the one thing right off the bat would be like to get better metrics. I know they're doing their own thing now, which is nice and they're still maintaining some of it like release time and frequency and all that kind of. I think frequency was a big thing, but that's like on the upside as well. So there's kind of like that, that cross pollination thing and then there is the whole team approach thing. That's this. Probably one of the biggest things I would do differently was not to just limit ourselves in just the quality team kind of owning this thing. I would open everything up to all engineering and really get their full buy in on it and just get them involved with the entire process as well. Because it was a bit of a bottleneck in the sense that like we, there's a lot that we wanted to do and there was still a lot that I was thinking about like adding a oas, you know, static analysis and all these kinds of things in our, in our PI test test suites and all that. And then there were performance things. I think getting that whole team approach would, would be you get different opinions on things as well like that. I think that's the biggest thing, right? Like you get varying, differing, differing opinions from, from all these different walks of life. And I think it's an amazing thing because it's, it's just you don't think of everything and that's, that's one of the big things that I found Out. And I was just like, well, we could have done this, this and this. But yeah, whole team approach, better metrics, changing the test suite as well. Like I said, just matching that language to the overall code bas a big deal, but it kind of ties into the whole team approach aspect of things. I think getting more comfortable with deleting repeated failures is a big thing. And I do hear that still once in a while, see some tweets about that. And that's. That's huge. Just that once again, like that once you see your build failing and like, you've realized it's like, oh, it's a Philly key test. It's like, who, like, relies on that? It becomes one of those things where I actually believe one of the. I did go to SASCON, SAS something, some convention in 2015, 2016, and I remember seeing Angie Jones do a talk on, I believe it's called the Build Cred Wolf. And that was like the one thing that, where I was just like, all right, this is. We got to do this. And that was a big thing as well. That helped us. But yeah, that's really. Oh. And nowadays what I would do is look at really just the thing like Mable. The companies like Mable and that are doing more of. More automation to it because it was just this. I think it's a big, big thing right now is really like the AI ops and you hear like a lot of these buzzwords kind of thrown around. But for nowadays, I would try to get that buy in because it would save us a lot of time and we'd be able to scale, I think, more quickly than a lot of our time was spent on refactoring.

Wow, controversial. So you'd actually look to a vendor that's doing kind of like a no code, low code type of AI type of approach. You actually believe in that some companies add value? I think they do. So I'm just curious to know what you think.

And I don't know much about it. So based on what I've seen and what I've read, I think it's something for us to evaluate. I wouldn't go all in on it, but I would say I want to save myself here before people start commenting. They're like, hey, why would you do this? You talked about Surface earlier and doing JavaScript and all that. But yeah, I would totally evaluate it, without a doubt. And that's what we did with all these different. You know, when we're doing Ruby and Python and evaluating it, then I would do the same thing. Like our evaluation process was huge. It was pretty in depth. So I would totally evaluate that and see where and that. And honestly, like the visualization, like stuff like applitools, I think it's a huge thing that we were. At the end when I was involved with that process, I was like, hey, let's check this out. And I did an MVP on it. And I was like, this is amazing. But yeah, I would do that.

So that's a great tip. Evaluation. Don't be afraid. A lot of times we have our own biases we bring to a team and hey, don't be afraid to try it out. If it doesn't work, don't use it. If it does work, do use it. I mean, a lot of times people, people somehow say I only use Selenium or I only use Cypress and it's like, okay, that's cool. But you know, does it help your team? If there's another tool that can help you, another approach, why not explore it? For sure. All right, Evan, so you mentioned Chaos Engineering. So I don't know if a lot of folks know this yet because I haven't announced it, but you're going to be one of the speakers of the 6th Annual Automation Guild happening February 7th to 11th, 2022. Little teaser. What are you going to be talking about?

Yeah, I'm going to be talking about KS Engineering and yeah, cool. And so yeah, and just the whole team approach to it because that's, yeah, I'm pulling like I'm doing it right now. I'm pulling in DevOps and SRE and the quality team. So yeah, I like that whole team approach just with all the different insights.

So yeah, I love it because you have real world experience. So it's not going to be like a theoretical type of sessions actually be real world tips that people can put into place. And Chaos Engineering is hot, hot, hot. So I definitely recommend people to register. Now head on over to automationguild.com okay Evan, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing processes and what's the best way to find or contact you?

Oh yeah, the big helped me with my entire career is just community. Like I embed yourself in community and learn as much as possible. Especially if you're coming from a similar background to myself in the sense that you're like the only person on the team. Which yeah, community is the biggest thing. Like I just learned so much and I still do and you know, that's why I started several of our own like community members and whatnot. And then as far as like reaching me, YouTube, Twitter, I'm all over the Internet, so just and then mostly on Twitter. So if anyone has any questions about anything that's covered or whatever it might be, I'm always, always around.

Great. Yeah, and a big shout out to the Austin Automation professionals. It seems like it is a really active community and it seems like since we are all remote now, I'm sure a lot of these sessions are people can actually access what happened to be in Austin I assume. Thanks again for your automation awesomeness. The links everything value we covered in this episode. Hand on over to test guild.com a374 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and subscription services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.