How Microsoft Uses TestOps with Oren Rubin and Maor Frankel

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we're talking about Oren and more. All about Test Ops and how it can help you scale your automation efforts. Oren, if you don't know, is the founder and CEO of Testim IO. He also has done a lot with the open source community, has been involved in speaking with the automation community for at least 20 years. Maybe I'm overstating it, but he's been around for a bit so he knows a lot of things about automation and more. Is a senior software engineer at Microsoft. I love Microsoft. When I started I used to do visual support from Microsoft Visual Basic support online and it was a third party and I always wished oh wow, Microsoft was the thing in the 90s to work for. So I always love have someone on Microsoft on the show. So it's an honor to have more join us and they both have a deep understanding of how enterprise strength automation automation works at scale. So you really don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey guys. Welcome to the Guild.

Hey.

Hey Joe.

Hey guys, good to have you on. I know I did a short intro, but is there anything I missed in your bio that you want the Guild to know more about? Oren, let's start with you.

I think you're spot on. I've been basically building tools for developers for the last 20 years.

Awesome. How about you? More?

Yeah, I mean I can just say I work for. I'm part of a group called Microsoft Security. We do products for security for enterprises and organizations and I'm part of a team that's called the Front End Infra and we provide different tools for all our developers and that includes the infrastructure for test automation using currently testim IO.

Oh, very cool. So we're going to get into that, but the show is not about Testim I.O. any products. Like I said, Testim I.O. is awesome. But a lot of the tools and techniques we're going to talk about today can be a apply to any situation. So you want to stay tuned all the Way to the end to hear this awesomeness coming your way. So I guess the first thing is, as you know, there's a lot of buzzwords around, and I saw Test Ops, I'm like, okay, what the heck is Test Ops? I've heard of AI ops, business ops, data ops. So why another ops? Oren, let's start with you.

Sure. I think, yeah, I agree that there's so many different types of ops, and specifically, even Test ops, people are describing it in different ways. I think the best way to describe it is about scale. Scale. That means how do I do something? Not when I have 10 tests. How can I have 10,000 tests and continue adding those tests and maintaining them in the same speed? It's all about scale. Make sense?

And speaking of scale. Yeah, absolutely. I'm sure. Microsoft Enterprise company, a lot of automation, a lot of development. So more, did you think of Test Ops when you implemented your solution or where did test Ops come into play here?

Yeah, I think for us at least, I have quite a lot of experience with automation from other companies that I worked on. But when I came to Microsoft, the first thing that we had to think about when we talked about automation was test Ops was scale. Because the amount of people that work on our application and our code base and the amount and the size of our application is huge. So when we thought about what technology we want to use and how we want to approach our solution, first thing we thought about was how do we handle that amount of scale and that amount of developers working on the same thing at the same time.

Very cool. So is there a reason why I hear more and more about scale? Are there any trends that are going on? I know integrating things into CICD has been something so. Oren, any thoughts?

Yeah, I think we see that a lot more. I think that in the past where the majority of the testing was manually tested, you hear more about how can I scale my manual testing. Right. You want to make sure that you cover everything. Now is with the trend is moving to more toward automation. And now you don't have 10 tests, you have thousands of tests. Now you need to make sure how do you scale that?

Absolutely. And more. Is that the reason why you thought about, well, we need to really scale better. We need to look into a process to do this better?

Yeah, I mean, I think, you know, like everything web applications, application general, but specifically web applications have been getting bigger and bigger and also the amount of hands that handle a specific application's increasing. So it makes sense that you need tools to manage that. Writing A testing framework for a team of five, 10 people, where everybody can communicate, that's one thing. But doing it for dozens of developers, sometimes hundreds of developers, which not necessarily can communicate with each other all the time, that's a completely different thing. And you have to think about it differently.

Mos, I think you're also referring to like the shift left that the fact that you want to test earlier.

Yeah.

So it's about scale, not just the amount of tests, but the number of times that you're running them. You're running them earlier and earlier to find the bugs much faster.

Yeah, yeah, we have that approach too. I mean, when automation started, the idea was, you know, just do end to end tests and test everything. And today it's changed a bit. And people trying to catch the bugs as sooner as possible. That also relates to scale because when you have a big organization, you don't want to handle regressions that other people have. You want to catch the bugs that you introduced, and that has to happen even before the code goes in. So you definitely have to start thinking about how you manage that and how you move things sooner in that process.

Yeah, absolutely. So I used to work for a large enterprise company and they went from releasing like once a year to they wanted to do it every six weeks. And so there's more people involved because they wanted developers to do testing as well. We had more tests than were bad tests because everyone's writing tests and more releases, which made it really hard to scale. So are those the typical kinds of challenges you've seen as well, maybe holding teams back that you've spoken with? Warren, I know you speak with a lot of companies probably.

Yes, we see that a lot. I think you said people want to go to the six weeks. I think that's the more slower teams. We see companies that want to be continuous deployment. That means that they can release in a click of a button. The developer changes a few lines of code they want in one click to build everything, test everything, and deploy it.

Yeah, I mean, we release versions to production. Every few minutes there's a new version. We deploy parts. So every few minutes production is changing. And that has to be tested continuously to make sure there's no regressions that can be done manually.

Absolutely. I don't know how long you've been at Microsoft. I know back in the day they're the ones that started this SDOT role where they actually had like a tester embedded with the team. But I've spoken with a lot of engineers like Abel Wang, and it sounded like developers now are responsible for testing. And that was kind of a big shift.

Yeah.

That probably caused issues to scale as well because they're just learning how to test. Is that something you saw when you joined Microsoft or is that a common thing?

Yeah, yeah.

You saw maybe at other companies.

There's still test engineers and manual testers in Microsoft, but definitely Microsoft changed the focus and want to give more of that control to the developers. That doesn't mean there's not test engineers, but you want the developer to be able to test his own changes. That's like what Aaron said. We want to shift left. We want to be able to give that developer the tools to test his own code before it even gets to anybody else. And Microsoft has definitely adopted that approach. And it's a challenge. Not all developers, you know, it's a state of mind to think about automation and testing and not all developers take that very naturally. But it's definitely something that Microsoft is taking in very seriously and it's widespread throughout Microsoft. But it's not easy for sure, right?

No doubt. And Arin, I think when I was researching test ops, costume came up. A few other companies did as well. And like you said, they all have different slants on it. Yours has like four core tenants and I thought maybe we go through each tenant, what you see and how it really works in the wild at Microsoft. So the first one was planning. What are your thoughts on what people need to do to plan or anything that's holding people back from planning to make their test more scalable?

What we want to give is. When I'm talking about, I'm a person inside organization. I want to push test automation to the limit and scale it. So this is where you want to help? How can I plan? Who's doing what? Wait a second. Do have test automation SDATs that helping out? How can we assign. Oh, those are new features that need to be tested. Who does those things? Is there already reusable components that we can use or which one do we need to build? Planning doesn't mean just building new stuff. What are the tests right now that we need to that we have technical data on? Which one are the flaky tests that we need to fix? Make sure that they're not flaky. Which are the tests which are quarantined because of a specific bug that wasn't introduced. So everything here, you need to manage that and say, hey, what am I doing today? What am I doing this week and what is my team going to do? So that's really the thing that you want to manage and make sure that everyone knows what's going on.

Absolutely. When I worked for once again a company, big enterprise and developers started testing, we always had these discussions, well, is this covered by the unit test? Do I need to cover create a fully functional, full blown test for we have a unit test or an integration test or using microservices. So how do you go about planning what to test and who's going to test it and all that type of things?

Yeah, that's a good question. Where should a test is it. Should it be a unit test or an end to end test? Generally our approach is similar to what Oren said is shift left, whatever you can test, the sooner and smaller the better and the further up you go. That is where integrations happen. When you want to test a certain component, that would usually be a unit test. If you wanted test the integration between two components, usually a page or a feature, that would be a UI test without for instance a backend with mock data. And when you want to test the integration between the front end and the back end there you'd probably have a full blown end to end test that tests the whole scenario. So that's kind of the rule of thumb but you know that that's, you know, very naive. Obviously it gets much more complicated than that, but that's the approach that we try and follow.

Absolutely. It does get really complicated. And another thing I saw was we moved from vendor based tools, which I loved because they had like a full suite end to end solution where you write a test, you can map it and we went to open source and then you had to create these weird integrations to try to map what you're testing, what the lifecycle was, what the results were of the test execution. So I think test management comes to be an issue when you're scaling as well. Arin, any thoughts on that?

Yeah, I think we do see that we see especially when we go and tools like Selenium, which is by the way an amazing tool. Everyone keeps asking me about Selenium and think I don't like Selenium. I love Selenium, it's a great tool but it provides the. It's the browser automation. It doesn't provide, as you said, the how do you manage? How do you know the state of your test, the history? Those are the things that Selenium doesn't provide out of the box and not intending to. So those are the things that are kind of missing when you're right now, when you're trying to scale.

So how did you handle management of tests when you started implementing your solution?

Yeah, I'll make Owen a bit happy. And I promote testimao because, like Oren said, when we had the option between writing our own infrastructure and which is, you know, like you said, Selenium and other automation tools generally give you the ability to actually write the test and the API for them, but they don't give you that whole infrastructure around you to manage them and run them. And that's what testimyo helped us. Our application was written very fast and didn't have automation from day one. So we were looking for something that can give that whole suite of tools that we needed around automation. Selenium, grids and test management and user management, all that. And Testim gave us most of that stuff and that's how we kind of had a shortcut and managed to get all that stuff out of the box and. But definitely that's a big part of automation today, is not just writing the tests, it's maintaining that whole thing. And there's so many tools around it that you need to manage everything.

So crazy thought Mark. I don't know why I was thinking this is, as Oren mentioned, Selenium is awesome. No one's dogging Selenium. Selenium is awesome. But there's other tools coming along as well. And I know Microsoft has invested heavily in Playwright. So when you're developing tests, does management tell you what tools or techniques use? Are you kind of have to use Playwright or. Have you used Playwright or anything like that?

Yeah, no. I think maybe a while ago Microsoft might have had that state of mind. That was before my time, but Microsoft today are a completely open company. Use whatever tool you need to get the job done and do it the best way. I use the Mac personally and really there's no constraints on what tools to use. Obviously you have to justify it. You can do whatever you want, but if you have a good justification, use anything. And Playwrights had a lot of momentum when we started our infrastructure. It was just getting started. I know they're doing work around some of the tooling around it too, so it's looking very promising. But again, even with Playwright, you're still going to have to do some work to get that infrastructure around all the management of your tests. There's still that work to be done for sure.

So, Oren, kind of going down this rabbit hole here, Testim, do you only support Selenium, someone who's using Playwright? Do you give them the management type of tools to help them with their test as well.

We've started adding more support, I think, for Playwright and Puppeteer. We even released, for Playwright, we released an open source version of root cause analysis. So one of the aspects that we help with is understanding when there's a fail. What's going on? Is it giving you the screenshots or the logs? The console logs, the network logs, pinpointing, saying, hey, here's the bug. Exactly. So that's one of the tooling that we helped and we even open source that and there's a free version for Playwright.

All right, cool. So let's get back to the topic at hand. I just had to touch on Playwright a little because it is a burning topic that a lot of people like to know about. So the next issue after management planning, management was Control. And once again, my company, when they started, they had me on every single code review for every developer that created a test. And it was. They'd marked me as a roadblock because I wasn't able to get to them. So every Sprint review, Joe's a roadblock. Couldn't get my test done. So how do you handle the control aspect of testing, Oren? And then we'll go to more. Any thoughts on that?

Yeah, I think that's. That's important. You want to keep the high standards. I don't think that control is. It's a word that has a little bit of a negative connotation. I think what people are trying is to actually make it into a good thing. You want to make sure that you stay with high quality, high quality tests and your own code. So I think that giving that intention, like, how can we make it even High standards, code reviews, test reviews, that's a great thing that can help. Making sure that nobody writes directly, for example, to Main or Master, then making sure that someone appear, looks at that. Those kind of things are something that really helps, making sure that you have those nice standards. That's in general, I think we're trying, of course, to provide more value for scanning people's tests and saying, hey, wait a second, you have here, you wrote the same thing that someone else did. And trying to say, hey, you have code duplications. That's not a standout. That's not a good way. So we'll prompt that and say, hey, are you sure you want to add that instead of reusing the same login method that someone else used?

Absolutely. And I actually want to dive into that soon. Probably the next question is going to be on Insights. I think you probably could Help with to have better control before we do though. Amor, how do you handle this at Microsoft? I'm sure people are always curious to know how people test at Microsoft or Google, these big companies. Do you have any tips on how did you have like part of your definition of done, Dial shop, create automation tests for every piece of code you write or like how does it work there?

Yeah, well, we have kind of checklists where you have to go over it when you write a feature and you complete it. We try not to enforce things. I mean the biggest balance, like you said, we don't want to get to a point where we're roadblocked for anything. On one hand we want to have control and we want to make sure that we don't get bad code in or bad tests in and stuff like that. On the other hand, we don't want to block people. People want to let people move. So we have to find that balance between things. So generally we let the teams kind of manage the things themselves. We allow them to, within each team, they can review their own tests, their own code, their own changes. We allow them that freedom to decide what's important for them and whatnot. We do have guidelines, we do have things that they can follow, but we don't enforce it because we don't want to slow them down. That's kind of the way we do it in Microsoft. And we also have, like we said, we have the levels, you know, unit test, write as many as you can. It's pretty quick, pretty easy. We want to make sure that you have a CI test if you have an integration between components and if you have an integration with the backend, you need to have an end to end test of that. We can't always enforce it, but at least when we get bugged and stuff like that, we know where we went, where we're missing coverage. That's our approach.

Very cool. And Arno, I'm sure as people are doing this and they're running the test and they're trying to get a handle on it. You did mention something about finding trends, about almost insights wherever a team is not reusing code. We have this issue as well. We had eight Sprint teams and everyone would write their own login just because they didn't want to touch anyone else's code. So. And it was hard to find sometimes because you're like, they just rename it something different. Like.

Yeah, any help with this?

We all know that. We all know that. Yeah.

This is what happens when you have a big, when you're Talking about scale, when you have. If There are just 10 tests and 10 methods, like usable components, that obviously that's not an issue. But when you're talking about a huge application with thousands of tests, thousands of reusable steps that you can reuse, how do you find those? How do you know not to. How do you know how to change something very safely? I think it's very important for platforms to help out with that, because a human. It's very hard for me to scan a thousand tests, but for a machine to scan a thousand tests and finding out that, hey, those steps exactly. That you use right now, there's an identical one over there. It's called the login. Please use that and try it and see that it works. And if something that machines can help us, and I guess this is where AI comes into play, it's very easy for machines to actually point that out and do that. And we've released something that actually helps even on the fly. Imagine that you're saying, hey, I want to start a test. My test starts with the login. I'm going to the login page. And then imagine something pops up and says, okay, someone already mo already wrote a login step that fits right here. Do you want to use that? So instead of trying to guess or going over thousands of functions out there, if someone suggests, hey, there's only one login that fits this page. Do you want to use that? Yes or no? Very simple.

Absolutely. Now, I guess another issue I'd seen a lot of. If you had a test that used functionality that was. That had a bug in it or was flaky over time, it was hard to know. Like a test pass, pass, failed, pass failed. That type of insight as well. Are you able to find those type of insights as well using a tool like say, testim?

Yeah, it's very easy to. I think there's two things. One is making sure that you are aware of that, hey, show me all my tests that are right now which are flaky. That means a test passed only on the second run. I tried, it passed and then it failed. Those kind of things. That's easy to find. I think AI can help even more in telling you there's a. Wait a second. There's a hundred failures here. I ran a thousand tests, 100 failed. Do I have to go one by one? 100 tests, that's a lot. If AI can tell you, hey, 90 of them actually failed for the same reason. They're exactly the same step in the application that would save you a Big deal. You don't have to go over the 90 tests, just one. And if you. And imagine they will tell you, wait a second, it failed yesterday for the same reason. And here's the bug report that someone did, someone already investigated that. You don't even need to investigate that one issue, one test, because you can see the reports from yesterday. So that's a big help.

Absolutely. Another thing, obviously to scale, you're going to need tools. I always ask people about tooling. So more, what tools did you use to help you? I'm sure when you started this before you were able to scale and then you had issues, then you somehow resolved it to scale better. So what kind of tools do you maybe use to help you skill for all your tests to get. To get it better?

Yeah. One thing that we saw important with scaling, not just test, by the way, is federation fragmentation, is because we have so many different teams and so many different developers, we can't control everything. We can't be one central point where everything is controlled. We have to kind of federate that and allow teams to manage the things themselves. We also want to have some kind of ownership model where we allow these teams and developers themselves and the people who write the test to own that test. So if something happens to one of those tests, you know who to approach. Right? I mean, if you've got thousands of tests, one of them fail, what are you going to do with that? How do you know what most likely you have no idea what that test is about. And another thing that we found is we try as much as possible run specific tests for the granularity, make sure that what we test is exactly what changed. Because the more you test, you write, the more flakiness you're going to have, the more difficult it's going to be to find out who, what change broke it. And the more specific you can be about what you run, the higher resolution you're going to get from your tests. And it's going to be much easier to maintain that and basically find bugs and not have flaky tests that just make noise and make everybody lose faith in the system.

Going on with what Mauro said, by the way, I think it's super important when we're talking about end to end testing. I think there's two meanings when people mean end to end. Some people mean it is I want to have all my components tested. That means front end, back end, all of it at the same time. And I'm testing one specific path, one scenario, but it goes through all the way from the back end to the front end, et cetera. And as opposed to end to end, when someone says the user scenario is end to end, I want to log in, I want to do a checkout, I want to add to cart, and I want to make the payment. That's a user experience, end to end. That experience end to end could be only on the front end. All the network could be mocked. And as opposed to, hey, we're checking only the login or only the checkout. We're only testing that, but we're testing that capability. That's a mini scenario, but with all the components, front end and backend. So having said that, I think just strengthening what Maor said, that if you're testing, you have a test that checks the checkout. Test the checkout. Try not to test every other feature right now and understand that.

Great advice. So I know you mentioned testim a few times in this interview so far. So is there anything specific in testim you use to help you with test ops per se?

Yeah. First of all, like I said, the whole concept of test owner is something that we find useful. That's been a big change for us because before that, when a test failed, everybody was like, all right, you know, who do we need to approach? They generally approach the infra team. And then we had to try and figure out who wrote that test and who owns it and what happens. And here we don't even need to be involved. They can go straight to the person who wrote that test and say, hey, you know, is this. I think the test you written is wrong or is broken, or maybe you can help me figure out how to fix it. So that's been a big improvement. Pull request. Also big help for us because you can. People who are not familiar with how to write tests, they can basically ask somebody, hey, can you look at this test before I put it in and can you review it and tell me if I'm doing something wrong or can I do something better? That's been a big improvement for us. Also, testim have the concept of branches where developers can make their changes separately and not put it in the master of the test. So that way, you know, they don't put in noise into the system without being sure that they're finished with what they were working on. And the biggest thing they've actually added is the whole concept of quarantine and evaluating. Testing, especially evaluating is really good because tests generally take some time to be stable, and that's usually where people lose faith in the system because tests get Added all the time. And those tests are not stable. And that means the system is always not stable. When you have something like evaluating. And what testim have is the concept of having a test run but not fail the suite, you can say, all right, put that test in evaluate and let it run, let it mature for a while. And when I feel ready that it's stable enough, I'll move it to active and that way it can run and actually give value. And also quarantine, where you can just say. That's common in tests, where you can just say, this test has a problem, let's put it on the side and see if somebody can it fix.

I love that feature. Just once again, from my experience, it'd be a lot like we'd have to tag something as flaky. But it was really hard to track and it was a lot of manual work to do that when I used to do it. And it was always hard to tell management, well, these five tests failed because of this. And you'd have to keep track of an Excel spreadsheet or something. So can you just explain this feature, like, how easy is it to use to help, you know, put things in quarantine and things like that?

I'll start with the quarantine. That's so. So every test has a test status. So you can move it at quarantine. If you're saying, for example, you have a bug and you're not going to fix that right now, so you don't want your CI to keep failing. You want to say, hey, I want to release, I'm going to quarantine, that I'm going to quarantine. That that means the test will not even run. And why is that different, by the way, than just taking the test out and saying, hey, I don't want to run this test in a different way. This special tagging helps you with, first of all, not forgetting what happens. I've seen it in so many companies that people say, oh my God, this smallest feature is amazing because if someone took something out, they forgot to bring it back. And you have thousands of tests, who can remember that there's this specific test, I need to put it back. Oh, no, wait a second. Now we have only 999 tests. Oh, we forgot one. So people, this little thing, when you're just talking about company, in most cases, when you have 120 developers, you can forget this small thing. Keeping track and understanding. This test is quarantined. This test is flaky. Understanding the status and giving it meaning and semantics to those labeling, it really helps. So the evaluation, I think that when you write a test, I think we all explained it very, very well. When you write a test when it's not ready, how do you know when it's ready? On one hand you want to connect it to the CI to see it run, but on the other hand, wait a second, if I put it in the CI too early, it's going to break my build could be because it's not ready. How do I know? Do I want to start building a scheduler that's running it every, every night for the next week and then I'll know that my test is ready. So the concept there was, there's a test you lay you instead of saying my test is being drafted right now, I think it's ready, I'm putting it in evaluating. That means it's going to run in the CI but it's not going to of failed build. So I can track it, I can come in two days, can look at it and say, hey, show me all my tests which are evaluating. And I see that it's been running, it's been run like a hundred times and it works all the time. Sure, let's move that and say that it's CI ready and if it doesn't, if it's flaky, then we want to say, hey, let's go back to that test and let's fix that, make sure that it's super CI ready.

I definitely love this feature. 120 developers. How did you resist the urge not to develop a solution yourself? A lot of people like, oh, we only do open source, we're developers, we'll create our own solution. What gave you the thing like, hey, there's a solution out there, it's called Testim and that has all the functionality we actually need to help us.

Like I said, first of all, when I got to the team, we didn't have any automation and we already had quite a large scale product and we needed to get up to speed fast. And Testim was a good option because they allowed us to do that also. There's so many things around tests. Yeah, you can do it yourself. But do you really need to, I mean, do you really need to manage your own test grid? Do you really need to manage your own test runner, screenshot recording? I mean, yeah, you can do that. How much are you going to get by handling that? Because you're going to have to maintain that, you're going to have to support that over time and that's going to take resources and that's going to shift your focus from what you really want to do and that's focused on writing your tests. Obviously using a third party tool does limit you on certain things, but you know, it's a balance where you say, all right, what I get here is worth, you know, giving up some of that control. And I'm not sure I want control over all those aspects. So that's what convinced us, you know, the ramp up speed, which was very fast. I mean, we had hundreds of tests written, you know, in a couple of months and you know, it's very easy for the developers to onboard because it's a UI tool. And also over time we get things that's free from testing that they develop for us or for other people. We don't have to invest that and that allows us to focus on other things.

Okay, Oren, more, before we go, is there one piece of actual advice you can give to someone to help them with their test ops efforts? And what's the best way to find or contact you more? We'll start with you and end with Oren.

Well, first of all, how can, can somebody approach me? I can publish my email, no problem. Anybody who has any questions can, can just send me an email, reach out to me, no problem, or LinkedIn, whatever. And advice for handling test ops. Wow, that's a good question. I think one thing I could say is that there's no perfect solution. You're always going to have to accept certain things. If you want control, then you have to take into account that you're going to slow things down. There's no way you can control everything and still move fast. And if you want to move fast, you're going to have to take into account that things are going to break. But if you move fast, you can fix them fast. So you need to find that balance and what works for you and just accept the fact that it's never going to be perfect. But you know, you need to do what's right for you and your team and your organization. DNA.

Great advice, Noren. Last words.

My suggestion is like land and expand. You want to keep having things. Don't let it be slow, don't let it be flaky. Try start small, having 10 tests. If you're having flaky tests right now, now start to have 10 tests that are running in the CI that you trust and then grow. Have 11 tests. The trust is the most important thing. Always keep the CI super clean and so everyone in the organization would trust their test so that's my advice. Make sure everyone trusts their test.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a353 and while you're there, make sure to click on the Try for free Today link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.